Re: [PATCH] tests: add an optional test to test git-annex
- Date: Wed, 17 May 2017 08:47:01 +0200
- From: Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx>
- Subject: Re: [PATCH] tests: add an optional test to test git-annex
On Wed, May 17, 2017 at 4:45 AM, Junio C Hamano <gitster@xxxxxxxxx> wrote:
> Ævar Arnfjörð Bjarmason <avarab@xxxxxxxxx> writes:
>> Add an optional test to test git-annex. It's guarded by a new
>> EXTERNAL_TESTS environment variable. Running this test takes me 10
[Re-arranged your mail because it worked better with my reply]
> I do not mind at all to place the simple reproduction recipe Joey
> posted as a new test in our test suite, though. That kind of test
> that catches changes to externally visible behaviour surely belongs
> to our test suite.
This is not a replacement for having an isolated test for the issue
Joey noted. We should have a separate patch for that, but I did not
have time/interest in writing that up. This change is orthagonal to
> Well, it is one thing to place git-annex under CI to make sure its
> latest and greatest works together well with our latest and greatest
> (and it may be something we want to see happen), but driving its
> tests from our testsuite sounds like a tail wagging the dog, at
> least to me.
To me this is just a question of:
* Is it the case that git-annex tests for a lot of edge cases we don't
test for: Yes, probably. As evidenced by them spotting this
regression, and not us.
* We can (and should) add a test for the specific breakage we caused
in 2.13.0, but that's no replacement for other things annex may be
covering & we may be missing which'll catch future breakages.
* It's a pretty established practice to test a library (git) along
with its consumers (e.g. annex) before a major release.
* This allows us to do that at minimal cost. I think it makes sense to
add this and integration tests for other similar utilities if they're
similarly easy to integrate.