Web lists-archives.com

Re: autopkgtest results influencing migration from unstable to testing




On Thu, May 03, 2018 at 11:52:40AM +0200, Arturo Borrero Gonzalez wrote:
> On 3 May 2018 at 11:21, Colin Watson <cjwatson@xxxxxxxxxx> wrote:

> > (I echo Simon's thanks for doing this, though!)


> Yeah, thanks for this!

> I would say, yeah, please wait a couple of stable releases before
> going full blocker.
> I (and others) may not have the time to polish our autopkgtest tests.

> If we end with less autopkgtests tests because of this, then the
> result of this push would be the opposite of what we intended :-P

I think the status quo is that we have a lot of autopkgtests that are
useless as a CI gate.

In Ubuntu, we have 245 overrides in place to ignore "regressions" in tests
that once passed but no longer do.  Many of these are tests that are simply
flaky and only pass sometimes.  Some are tests that once succeeded but now
fail because something else changed in the overall distribution, and wasn't
/ couldn't be caught by autopkgtest to prevent the regression.  Some are
tests that are treated as "regressions" now because previously they couldn't
be run on a particular architecture, but Ubuntu's infrastructure has
improved to where they now can be (and are seen to fail).

And there has been no release of Ubuntu in which more than 93% of tests
passed, on any architecture. (http://autopkgtest.ubuntu.com/statistics)

I have also been submitting a lot of bug reports about autopkgtests that the
maintainers have allowed to regress in new versions of the Debian package.

  https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=ubuntu-devel@xxxxxxxxxxxxxxxx;tag=autopkgtest

It's fine if the raw number of tests goes down, if the overall quality of
the tests - and therefore the quality of the release - goes up (and the time
wasted hunting buggy tests goes down).

-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek@xxxxxxxxxx                                     vorlon@xxxxxxxxxx

Attachment: signature.asc
Description: PGP signature