Add subTest support for testtools.TestCase by CyrilRoelandteNovance · Pull Request #630 · testing-cabal/testtools

CyrilRoelandteNovance · 2026-09-08T20:45:42Z

Commit 267c0be added support for
unittest.TestCase.subTest. This meant the following code:

import unittest

class TestSubTests(unittest.TestCase):
    def test_even_numbers(self):
        for i in range(5):
            with self.subTest(i=i):
                self.assertEqual(i % 2, 0)

would list the exact values of "i" for which the test failed:

$ stestr run test_example_subtest 2>&1 | grep ^Captured
Captured traceback (i=1):
Captured traceback (i=3):
Captured traceback (i=1):
Captured traceback (i=3):

But the following code:

import testtools

class TestSubTests(testtools.TestCase):
    def test_even_numbers(self):
        for i in range(5):
            with self.subTest(i=i):
                self.assertEqual(i % 2, 0)

would not produce a similar output:

$ stestr run test_example_subtest_testtools 2>&1 | grep ^Captured
Captured traceback:
Captured traceback:

This commit fixes this so that we get the same output whether we use
unittest.TestCase.subTest or testtools.TestCase.subTest.

Assisted-by: Claude Opus 4.6 (1M context) noreply@anthropic.com
Signed-off-by: Cyril Roelandt cyril@redhat.com

jelmer · 2026-09-08T23:41:23Z

+_subtest_msg_sentinel = object()
+
+
+class _SubTestInfo:


_SubTestInfo is a bare object, but it's passed to result.addSubTest() where a unittest.TestCase is expected. Stdlib's TextTestResult.addSubTest calls getDescription(subtest) ->
subtest.shortDescription(), so running any testtools test that uses subTest under a plain unittest.TextTestRunner aborts the run:

AttributeError: '_SubTestInfo' object has no attribute 'shortDescription'

Reproduced with unittest.TextTestRunner(verbosity=2) on a case with four subtests, two failing.

OK, I ended up making this class inherit from unittest.TestCase so it has shortDescription and every method/attribute one might expect to be able to use.

jelmer · 2026-09-08T23:43:17Z

                        self._run_user(_raise_force_fail_error)
                        failed = True
+                    for subtest, err in getattr(self.case, "_subtest_failures", ()):
+                        self.result.addSubTest(self.case, subtest, err)


If the result object doesn't implement addSubTest, the resulting AttributeError is caught and discarded by the surrounding _run_user machinery, and a test with genuinely failing subtests reports as passing:

events: ['startTest', 'stopTest']
was_successful: True

Silently converting failures into successes is the worst available failure mode here — worth handling the missing-method case explicitly, or guaranteeing addSubTest is present on the decorated result.

OK, I am now calling addFailure() if addSubTest() does not exist. I think we can expect addFailure to be implemented?

jelmer · 2026-09-08T23:43:41Z

+        subtest = _SubTestInfo(self, msg, params)
+        try:
+            yield
+        except Exception:


This catches SkipTest, so self.skipTest(...) inside a subTest is recorded as an error rather than a skip. Against stdlib, for a case that skips one subtest and fails another.

OK I caught SkipTest and reraised the exception.

jelmer · 2026-09-08T23:44:06Z

+        self, msg: object = _subtest_msg_sentinel, **params: Any
+    ) -> Iterator[None]:
+        """Return a context manager for a subTest."""
+        subtest = _SubTestInfo(self, msg, params)


There's no nesting stack, so nested subTests lose the outer parameters.

I did not even think about that use case! I am now keeping a list of parameters.

CyrilRoelandteNovance · 2026-09-09T19:11:36Z

@jelmer Thanks for the review, the new patchset should address all your comments.

jelmer · 2026-09-09T20:25:48Z

+        old_params, self._subtest_params = self._subtest_params, merged_params
+        try:
+            yield
+        except SkipTest:


is raising here correct? I think we should record this subtest as skipped but still run the other ones

Indeed, I should have logged the reason for skipping and moved on to the next subtests. This is fixed in the new patchset.

jelmer · 2026-09-09T20:27:06Z

+                    for subtest, err in getattr(self.case, "_subtest_failures", ()):
+                        try:
+                            self.result.addSubTest(self.case, subtest, err)
+                        except AttributeError:


this also catches AttributeErrors from within addSubTest, not just absence of addSubTest itself

Good catch. I switched to testing whether the method exists before calling it.

Commit 267c0be added support for unittest.TestCase.subTest. This meant the following code: import unittest class TestSubTests(unittest.TestCase): def test_even_numbers(self): for i in range(5): with self.subTest(i=i): self.assertEqual(i % 2, 0) would list the exact values of "i" for which the test failed: $ stestr run test_example_subtest 2>&1 | grep ^Captured Captured traceback (i=1): Captured traceback (i=3): Captured traceback (i=1): Captured traceback (i=3): But the following code: import testtools class TestSubTests(testtools.TestCase): def test_even_numbers(self): for i in range(5): with self.subTest(i=i): self.assertEqual(i % 2, 0) would not produce a similar output: $ stestr run test_example_subtest_testtools 2>&1 | grep ^Captured Captured traceback: Captured traceback: This commit fixes this so that we get the same output whether we use unittest.TestCase.subTest or testtools.TestCase.subTest. Closes: testing-cabal#317 Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Cyril Roelandt <cyril@redhat.com>

Subtest failures were buffered on the test case and replayed at the end of RunTest._run_core, bypassing testtools' own exception machinery. That lost several behaviours relative to a non-subtest failure: - a subclass overriding `skipException` had its skip reported as an error, since `subTest` caught the hardcoded `unittest.SkipTest` - `expectFailure` inside a subtest leaked the private `_ExpectedFailure` as an error instead of recording an expected failure - `MultipleExceptions` was reported verbatim rather than unpacked - details attached inside a failing subtest never reached the result - `AsynchronousDeferredRunTest._run_core` never drained the buffers, so a failing subtest reported as a pass Report each subtest as it completes instead, dispatching on the case's exception handlers and passing the details accumulated inside the block. Since reporting no longer lives in `_run_core`, the Twisted runner is covered too; it only needs to know not to also report success. Also align with unittest where the buffering had diverged. Follow-up to testing-cabal#630.

CyrilRoelandteNovance force-pushed the fix-subtest-testtools-testcase branch 3 times, most recently from 4f38f9d to 0458427 Compare September 8, 2026 23:27

jelmer requested changes Sep 8, 2026

View reviewed changes

CyrilRoelandteNovance force-pushed the fix-subtest-testtools-testcase branch from 0458427 to 18ce8f5 Compare September 9, 2026 19:09

jelmer reviewed Sep 9, 2026

View reviewed changes

CyrilRoelandteNovance force-pushed the fix-subtest-testtools-testcase branch from 18ce8f5 to 77ba2c7 Compare September 14, 2026 15:32

jelmer self-requested a review September 14, 2026 20:14

jelmer approved these changes Sep 14, 2026

View reviewed changes

jelmer enabled auto-merge September 14, 2026 20:14

jelmer force-pushed the fix-subtest-testtools-testcase branch from 77ba2c7 to f662ae4 Compare September 14, 2026 20:15

jelmer merged commit d4a6af3 into testing-cabal:master Sep 14, 2026
9 checks passed

jelmer mentioned this pull request Sep 14, 2026

Report subtests through the exception handlers #633

Merged

Navigation Menu

Add subTest support for testtools.TestCase - #630

Add subTest support for testtools.TestCase#630
jelmer merged 1 commit into
testing-cabal:masterfrom
CyrilRoelandteNovance:fix-subtest-testtools-testcase

CyrilRoelandteNovance commented Sep 8, 2026

Uh oh!

jelmer Sep 8, 2026

Uh oh!

CyrilRoelandteNovance Sep 9, 2026

Uh oh!

jelmer Sep 8, 2026

Uh oh!

CyrilRoelandteNovance Sep 9, 2026

Uh oh!

jelmer Sep 8, 2026

Uh oh!

CyrilRoelandteNovance Sep 9, 2026

Uh oh!

jelmer Sep 8, 2026

Uh oh!

CyrilRoelandteNovance Sep 9, 2026

Uh oh!

CyrilRoelandteNovance commented Sep 9, 2026

Uh oh!

jelmer Sep 9, 2026

Uh oh!

CyrilRoelandteNovance Sep 14, 2026

Uh oh!

jelmer Sep 9, 2026

Uh oh!

CyrilRoelandteNovance Sep 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

CyrilRoelandteNovance commented Sep 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CyrilRoelandteNovance commented Sep 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants