Re: [PATCH] tests: add an optional test to test git-annex
- Date: Wed, 17 May 2017 19:19:02 +0900
- From: Junio C Hamano <gitster@xxxxxxxxx>
- Subject: Re: [PATCH] tests: add an optional test to test git-annex
Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx> writes:
>> Well, it is one thing to place git-annex under CI to make sure its
>> latest and greatest works together well with our latest and greatest
>> (and it may be something we want to see happen), but driving its
>> tests from our testsuite sounds like a tail wagging the dog, at
>> least to me.
> To me this is just a question of:
> * Is it the case that git-annex tests for a lot of edge cases we don't
> test for: Yes, probably. As evidenced by them spotting this
> regression, and not us.
And I'd encourage them to keep doing so.
> * We can (and should) add a test for the specific breakage we caused
> in 2.13.0, but that's no replacement for other things annex may be
> covering & we may be missing which'll catch future breakages.
> * It's a pretty established practice to test a library (git) along
> with its consumers (e.g. annex) before a major release.
I am not so sure about the division of labor. What you are
advocating would work _ONLY_ if we test with a perfect & bug-free
version of the consumers. If they are also a moving target, then
I do not think it is worth it. After all, we are *not* in the
business of testing these consumers.
Unless I misunderstood you and you were saying that we freeze a
version, or a set of versions, of customer that is/are known to pass
their own tests, and test the combination of that frozen version of
the customer with our daily development. If that is the case, then
I would agree that we are using their test to test us, not them.
But I somehow didn't get that impression, hence my reaction.