Re: [PATCH] tests: add a special setup where prerequisites fail
- Date: Tue, 14 May 2019 15:39:06 +0200
- From: Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx>
- Subject: Re: [PATCH] tests: add a special setup where prerequisites fail
On Tue, May 14 2019, Johannes Schindelin wrote:
> Hi Ævar,
> On Tue, 14 May 2019, Ævar Arnfjörð Bjarmason wrote:
>> On Tue, May 14 2019, Johannes Schindelin wrote:
>> > What would you think about a mode where random test cases are skipped?
>> > It would have to make sure to provide a way to recreate the problem,
>> > e.g. giving a string that defines exactly which test cases were
>> > skipped.
>> > I am *sure* that tons of test scripts would fail with that, and we
>> > would probably have to special-case the `setup` "test cases", and we
>> > would have to clean up quite a few scripts to *not* execute random
>> > stuff outside of `test_expect_*`...
>> I think it would be neat, but unrelated to and overkill for spotting the
>> practical problem we have now, which is that we *know* we skip some of
>> this now on some platforms/setups due to prereqs.
> I understand, but I am still worried that this is a lot of work for an
> incomplete fix.
> For example, the t7600-merge.sh test script that set off this conversation
> has two prereqs that are unmet on Windows: GPG and EXECKEEPSPID. On Azure
> Pipelines' macOS agents, it is only GPG that is unmet. So switching off
> all prereqs would not help macOS with e.g. a bug where the GPG test cases
> are skipped but the EXECKEEPSPID test case is not.
It won't catch such cases, but will catch cases where a later new test
assumes that whatever the state of the test repo it gets is what's
always going to be there. In practice I think that'll catch most such
The other GIT_TEST_* modes assume similar non-combinatorial explosion
I haven't gone back through the test suite's commit history to try to
dig for other cases, so perhaps this mode is premature etc.