QA protip of the day: make sure your test runner fails properly

Source

Just when you thought you were safe...it's time for a blog post!

For the last few days I've been working on fixing Rawhide packages that failed to build as part of the Python 3.6 mass rebuild. In the course of this, I've been enabling test suites for packages where there is one, we can plausibly run it, and we weren't doing so before, because tests are great and running them during package builds is great. (And it's in the guidelines).

I've now come across two projects which have a unittest-based test script which does something like this:

#!/usr/bin/python3

class SomeTests(unittest.TestCase):
    [tests here]

def main():
    suite = unittest.TestLoader().loadTestsFromTestCase(SomeTests)
    unittest.TextTestRunner(verbosity=3).run(suite)

if __name__ == '__main__':
    main()

Now if you just run this script manually all the time and inspect its output, you'll be fine, because it'll tell you whether the tests passed or not. However, if you try and use it in any kind of automated way you're going to have trouble, because this script will always exit 0, even if some or all the tests fail. This, of course, makes it rather useless for running during a package build, because the build will never fail even if all the tests do.

If you're going to write your own test script like this (which...seriously consider if you should just rely on unittest's 'gathering' stuff instead, or use nose(2), or use pytest...), then it's really a good idea to make sure your test script actually fails if any of the tests fail. Thus:

#!/usr/bin/python3

import sys

class SomeTests(unittest.TestCase):
    [tests here]

def main():
    suite = unittest.TestLoader().loadTestsFromTestCase(SomeTests)
    ret = unittest.TextTestRunner(verbosity=3).run(suite)
    if ret.wasSuccessful():
        sys.exit()
    else:
        sys.exit("Test(s) failed!")

if __name__ == '__main__':
    main()

(note: just doing sys.exit() will exit 0; doing sys.exit('any string') prints the string and exits 1).

Packagers, look out for this kind of bear trap when packaging...if the package doesn't use a common test pattern or system but has a custom script like this, check it and make sure it behaves sanely.

Comments

No comments.