Debugging Syntax and Import Errors in OpenStack Unit Tests

Most OpenStack projects use a combination of tox, testr, and subunit to run their unit tests. This works fine as long as you don't have any simple errors in your code, such as a typo in an import or a missing closing paren. In those cases testr or subunit may swallow the error and you won't know exactly where it is (I hear this may be fixed in some version of the tools, but the one I'm currently using still has this limitation). To fix this, we need another way to run unit tests that will expose the actual error to us.

There are a number of different ways to do this, but the one I've been using is:

.tox/py27/bin/python -m testtools.run discover tests

What is this magical incantation? Let me explain:

Python invocation: Using the python from the tox venv means it will find all of the dependencies that were automatically installed by tox, so you don't have to install them all on the host system.
Testtools test runner: Run the tests using the simple testtools test runner. This will report the error that we need.
Testtools discover: Indicates that the test runner should look for tests recursively in the module specified next.
Module containing tests: My example would work for most of the Oslo projects because they put their tests in a top-level module named tests (yes, I know, this isn't a particularly good idea, but in practice it hasn't caused any problems and changing all of the projects to fix it would be an unnecessary headache). For other projects, the module might be something like nova.tests.

And there you have it. Now you should get a useful error message for missing imports or syntax errors. Note that using testtools all the time probably isn't a good idea because it's a much more primitive test runner that doesn't do things like run tests in parallel (at least that I can tell), but in some instances simplicity is what we want.

Run with testr:

${PYTHON:-python} -m subunit.run discover -t ./ ./tests --list 
--- import errors ---
tests.unit.test_lockutilsNon-zero exit code (2) from test listing.
error: testr failed (3)

Run with the method described above:

ERROR: unittest.loader.ModuleImportFailure.unit.test_lockutils
----------------------------------------------------------------------
Traceback (most recent call last):
ImportError: Failed to import test module: unit.test_lockutils
Traceback (most recent call last):
  File "/opt/stack/oslo.concurrency/.tox/py27/lib/python2.7/site-packages/testtools/run.py", line 472, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib64/python2.7/unittest/loader.py", line 230, in _get_module_from_name
    __import__(name)
  File "/opt/stack/oslo.concurrency/tests/unit/test_lockutils.py", line 17, in 
    import multiprocessin
ImportError: No module named multiprocessin