Re: [PATCH] tests: add a special setup where prerequisites fail
- Date: Tue, 14 May 2019 16:04:41 +0200 (CEST)
- From: Johannes Schindelin <Johannes.Schindelin@xxxxxx>
- Subject: Re: [PATCH] tests: add a special setup where prerequisites fail
On Tue, 14 May 2019, Ævar Arnfjörð Bjarmason wrote:
> On Tue, May 14 2019, Johannes Schindelin wrote:
> > On Tue, 14 May 2019, Ævar Arnfjörð Bjarmason wrote:
> >> On Tue, May 14 2019, Johannes Schindelin wrote:
> >> > What would you think about a mode where random test cases are
> >> > skipped? It would have to make sure to provide a way to recreate
> >> > the problem, e.g. giving a string that defines exactly which test
> >> > cases were skipped.
> >> >
> >> > I am *sure* that tons of test scripts would fail with that, and we
> >> > would probably have to special-case the `setup` "test cases", and
> >> > we would have to clean up quite a few scripts to *not* execute
> >> > random stuff outside of `test_expect_*`...
> >> I think it would be neat, but unrelated to and overkill for spotting
> >> the practical problem we have now, which is that we *know* we skip
> >> some of this now on some platforms/setups due to prereqs.
> > I understand, but I am still worried that this is a lot of work for an
> > incomplete fix.
> > For example, the t7600-merge.sh test script that set off this
> > conversation has two prereqs that are unmet on Windows: GPG and
> > EXECKEEPSPID. On Azure Pipelines' macOS agents, it is only GPG that is
> > unmet. So switching off all prereqs would not help macOS with e.g. a
> > bug where the GPG test cases are skipped but the EXECKEEPSPID test
> > case is not.
> It won't catch such cases, but will catch cases where a later new test
> assumes that whatever the state of the test repo it gets is what's
> always going to be there. In practice I think that'll catch most such
That's fair. It will also raise awareness of these issues, which should
also have a quite beneficial effect.
Given that your patch is not even large, I think you're right, it *is* a
lot of bang for the buck.