Web lists-archives.com

PSA: Use newer add_test(NAME <n> COMMAND <c>), check CI, esp. with min ECM >=5.38




Hi,

tl:dr
In your CMake code replace all remaining
    add_test(<n> <c>)
with
    add_test(NAME <n> COMMAND <c>)
to make sure test executables are still found once you bump the minimum ECM 
version required >= 5.38.
And check your product on KDE CI, where not found test executables are now 
properly reported as failure (everything got already up-to-date builds with 
that change, if part of groups "Frameworks", "Applications" or "Extragear").
At least one project missed to see some unit-test regression on released 
versions due to test executables not found and not run, without being 
reported as failure on CI.

Long version:
It was found that on KDE CI some build states had been reported with a blue 
ball (=all good), while actually most of its tests were not run, due to the 
test executable not found. This was due to the logic on CI mapping ctest's 
<notrun> to jenkins <skipped>, with the later not invalidating an "all good" 
state. This has now been changed (D20874), <notrun> with 'Unable to find 
executable' is now reported as <failure>, so will be treated like a failed 
test.
(Locally a "make test" reports not found tests as failed, so we should have 
ourselves run "make test" more often it seems :) ).

Tests executables not or no longer being found can happen when you bumped the 
minimum required ECM version to 5.38 or above and use KDECMakeSettings. 
Because this will trigger the placement of all built executable code into the 
toplevel bin/ directory, not in the normal build dir.
Which breaks the inherit assumption of

   add_executable(testfoo ${SRCS}) # add_executable(<target> ...)
   add_test(foo-test testfoo)      # add_test(<test-name> <command>)

where the command is taken as verbatim command and usually works as the name 
of the executable target is also the name of the executable, which was placed 
in the build dir, which is the default working directory on running the 
tests.

To fix this, use the newer signature of add_test:

   add_test(NAME foo-test COMMAND testfoo)

(so just insert "NAME" & "COMMAND" before the two arguments).

Because "[i]f <command> specifies an executable target (created by 
add_executable()) it will automatically be replaced by the location of the 
executable created at build time." Thus the test binaries being placed in 
bin/ will no longer be an issue.
See https://cmake.org/cmake/help/latest/command/add_test.html
This signature exists already with cmake 2.8.12.

Any other related wrapper calls, like ecm_add_test, are fine and do not need 
any work.

For more background why this is needed at all, see the docs about the efforts 
trying to make tests & apps runnable uninstalled for developer convenience 
(yes, no free lunch):
https://api.kde.org/ecm/kde-module/KDECMakeSettings.html
https://community.kde.org/Guidelines_and_HOWTOs/Making_apps_run_uninstalled

Cheers
Friedrich