Web lists-archives.com

Re: [PATCH 4/9] tests: optionally write results as JUnit-style .xml




Hi Eric,

On Mon, 3 Sep 2018, Eric Sunshine wrote:

> On Mon, Sep 3, 2018 at 5:10 PM Johannes Schindelin via GitGitGadget
> <gitgitgadget@xxxxxxxxx> wrote:
> > This will come in handy when publishing the results of Git's test suite
> > during an automated VSTS CI run.
> >
> > Signed-off-by: Johannes Schindelin <johannes.schindelin@xxxxxx>
> > ---
> > diff --git a/t/test-lib.sh b/t/test-lib.sh
> > @@ -431,11 +434,24 @@ trap 'exit $?' INT
> >  test_failure_ () {
> > +       if test -n "$write_junit_xml"
> > +       then
> > +               junit_insert="<failure message=\"not ok $test_count -"
> > +               junit_insert="$junit_insert $(xml_attr_encode "$1")\">"
> > +               junit_insert="$junit_insert $(xml_attr_encode \
> > +                       "$(printf '%s\n' "$@" | sed 1d)")"
> > +               junit_insert="$junit_insert</failure>"
> 
> This is a genuine failure, so you're creating a <failure> node. Okay.
> 
> > +               write_junit_xml_testcase "$1" "      $junit_insert"
> > +       fi
> > @@ -444,11 +460,19 @@ test_failure_ () {
> >  test_known_broken_ok_ () {
> > +       if test -n "$write_junit_xml"
> > +       then
> > +               write_junit_xml_testcase "$* (breakage fixed)"
> > +       fi
> >         test_fixed=$(($test_fixed+1))
> >         say_color error "ok $test_count - $@ # TODO known breakage vanished"
> >  }
> 
> This was expected to fail but didn't, which means it probably needs
> some sort of attention. test_known_broken_ok_() prints this result in
> the 'error' color, and test_done() re-inforces that by printing a
> message, also in 'error' color:
> 
>     42 known breakage(s) vanished; please update test(s)
> 
> So, should this emit a <failure> node also, perhaps with 'type'
> attribute set to "warning" or something? (<failure type="WARNING"
> message="...">)

My primary aim is to display the test results in the web interface, see
e.g.
https://git.visualstudio.com/git/_build/results?buildId=128&view=ms.vss-test-web.test-result-details

The parser for JUnit XML (and in fact, the JUnit XML schema itself) do not
allow for such a warning. If you add a `<failure>`, then the build fails.

And we do not want the build to fail. Historically, I saw quite a couple
of "vanished" breakages depending on the platform where I ran the tests.

> > @@ -758,9 +793,58 @@ test_at_end_hook_ () {
> > +xml_attr_encode () {
> > +       # We do not translate CR to &#x0d; because BSD sed does not handle
> > +       # \r in the regex. In practice, the output should not even have any
> > +       # carriage returns.
> > +       printf '%s\n' "$@" |
> > +       sed -e 's/&/\&amp;/g' -e "s/'/\&apos;/g" -e 's/"/\&quot;/g' \
> > +               -e 's/</\&lt;/g' -e 's/>/\&gt;/g' \
> > +               -e 's/  /\&#x09;/g' -e 's/$/\&#x0a;/' -e '$s/&#x0a;$//' |
> > +       tr -d '\012\015'
> > +}
> 
> It's possible to insert a literal CR in the 'sed' expression, which
> does match correctly on BSD (and MacOS). For instance:
> 
>     CR=$(printf "\r")
>     sed -e "s/$CR/\&#x0d;/g"

Okay. But since we are talking about displaying some chunk of text, I
would rather just delete the CR here anyway.

Ciao,
Dscho