SenTestingKit: How Does it Even?
Have you ever wondered what used to happen when you hit
⌘U to run your unit tests in Xcode 4.6 and prior? Here's a step-by-step sequence of events detailing how tests are loaded and run using SenTestingKit.
As I explain in Probing SenTestingKit, XCTest (née SenTestingKit) has changed surprisingly little since it was first adopted by Apple in 2005. Although what follows is based on the 2005 source code to SenTestingKit, I assume Xcode 7 and XCTest still operate very similarly. I'll test that assumption in future blog posts.
1. Xcode invokes a command-line tool called
otest to run your tests.
In XCTest, this tool has been renamed to
xctest. You can run it yourself from the command line:
otest is a simple program that loads a bundle at a given path.
$ xctest /path/to/bundle
The loaded bundle is assumed to have a class named
+[SenTestProbe runTests:] to kick things off.
SenTestProbe runs from within the loaded bundle. In order to pass along the command-line arguments it was invoked with,
otest also sets various key-value pairs to
NSUserDefaults. For example, running
otest -SenTest All /path/to/bundle, the following is set in
SenTestProbe Determines How it Was Launched
+[SenTestProbe initialize] is called as soon as
+[SenTestProbe runTests:]. The method is a convoluted hack that checks bits of state in
NSUserDefaults to determine how the class came to be initialized.
In the case this post is describing, the method would end up doing nothing, since it would determine that it must have been loaded by
+[SenTestProbe runTests:] directly.
Conversely, if not launched by
SenTestProbe Initializes Observers
The first thing
+[SenTestProbe runTests:] does is initialize a
SenTestObserver is a base class that defines an interface for receiving messages related to the state of a
@interface SenTestObserver : NSObject // ... + (void) testSuiteDidStart:(NSNotification *) aNotification; + (void) testSuiteDidStop:(NSNotification *) aNotification; + (void) testCaseDidStart:(NSNotification *) aNotification; + (void) testCaseDidStop:(NSNotification *) aNotification; + (void) testCaseDidFail:(NSNotification *) aNotification; @end
Notice the design flaws:
- The interface is defined as a concrete subclass of
NSObject, not a protocol. Worse still, the base class defines behavior shared by all of its subclasses. A protocol would have been cleaner.
- The methods take an
NSNotificationas a parameter.
-[NSNotification object]is typed as an
id, but in practice
SenTestRunobjects are sent. A value object parameter would have enforced greater type safety.
- The methods are class methods, not instance methods. This makes it impossible to have the same kind of observer act on notifications more than once.
This underpins one of XCTest's greatest shortcomings: its inflexible logging. We explore this tragedy further below.
The fact that
SenTestObserver defines a way to interface with
SenTestRun notifications, plus the fact that the
+[SenTestObserver initialize] method checks
NSUserDefaults for a key named
@"SenTestObserverClass", may make you assume that, as a consumer of SenTestingKit, you would be able to specify your own, custom test logger.
But the cake is a lie.
+[SenTestObserver initialize] sets the following to
…then immediately sets the class specified in the defaults as the "current test observer." The "current observer" then registers for a series of SenTestingKit-specific
+[SenTestObserver setCurrentObserver:]sets a single class as the observer for the entire test suite. The FIXME source code comment from SenTestingKit describes the situation perfectly: "Not very elegant and only one observer currently possible." (Which isn't exactly true, since one could register for the relevant
Like most problems described in FIXME comments, in practice this remains unfixed even today. Although the version of XCTest packaged with Xcode 6.0.1 included a class named
XCTestObservationCenterthat appeared to be able to register multiple observers, re-entrant unit tests still crash when run with Xcode or
What we're left with is
SenTestLog as the sole listener for
NSNotification notifications regarding the state of a test run.
SenTestLog is a simple listener that writes the results of each test to
stderr–this is the class that would generate the test output seen in the console in Xcode.
SenTestProbe Initializes a Test Suite
After initializing the test observer,
+[SenTestProbe runTests:] proceeds to initialize the test suite itself.
How a test suite is constructed varies based on the test "scope"–the value passed to
otest on the command-line. Here are the options:
otest -SenTest All /path/to/bundle: The default test suite, which includes a test for each test method defined across all subclasses of
otest -SenTest Self /path/to/bundle: A test suite that only contains tests defined in the specified bundle.
otest -SenTest MyTestCaseSubclass/testMyTestMethod /path/to/bundle: A test suite that contains a single test run for the specified test method.
We'll focus on
-SenTest All, since that's what's run when we run tests via Xcode's
+[SenTestSuite defaultTestSuite] creates a test suite named
@"All tests", composed of every test method defined across all
SenTestCase subclasses. Enumerators are used to build the suite:
+[SenTestSuite defaultTestSuite]enumerates each subclass of
SenTestCase, adding the tests in
-[SenTestCase defaultTestSuite]to itself.
-[SenTestCase defaultTestSuite]enumerates each test invocation defined on it, adds it to a new test suite object, and then returns that object.
SenTestingKit defines categories on
NSObject that use the Objective-C runtime's
objc_getClassList to grab all of a class's subclasses, and to create an
NSInvocation for each of its methods.
It also defines categories on
NSInvocation to determine whether an invocation is what SenTestingKit considers a "test method". It must:
- Have a return type of
- Take no arguments
- Its selector must begin with "test"
4. The Test Suite is Run
Once the test suite has been prepared,
-[SenTestSuite run] is called to kick things off. This creates an instance of
SenTestSuiteRun, which is used as a collection of
SenTestCaseRun instances. The
SenTestSuite enumerates its tests and calls
-[SenTestCase run] on each of them, adding the
SenTestCaseRun objects returned to its collection.
-[SenTestCase run] is straightforward:
-[SenTestCase setUp]is called to perform user-specified test setup.
-[SenTestCaseRun start]is called. This posts a
- The test case's invocation is invoked using
-[NSInvocation invoke]. SenTestingKit advertises that it makes a distinction between test failures and exceptions that occur during test runs; hence, the invocation is invoked within a
@catch. If an exception is raised, it's stored as an attribute on the
SenTestCaseRun. Otherwise, the invocation may call
-[SenTestCase failWithException:]to add a failure to a
-[SenTestCaseRun stop]is called. This posts a
-[SenTestCase tearDown]is called to perform user-specified test teardown.
SenTestCaseRunobject is returned. The object maintains a failure count and exception properties.
SenTestingKit aficionados may notice that the interface for reporting failures was different in early versions of SenTestingKit:
-[SenTestCase failWithException:]. The new interface is
The move away from failure reporting based on
NSExceptionwas a great decision. The requirements for reporting are now much clearer. The old interface required assertions to raise exceptions with specific keys in their
SenTestSuiteRun returned by
-[SenTestSuite run] has information on the number of failures and the number of exceptions, but recall those failures were also reported via
NSNotification notifications, and subsequently logged by the
SenTestLog observer. With no additional work to do, the
SenTestProbe exits with a status code equal to the number of failures in the
Well, there you have it–everything that happens in SenTestingKit after hitting
⌘U in Xcode 4.6.
In an upcoming post, I'll explore how this series of events has changed in subsequent versions of XCTest. For example, the version of XCTest packaged with Xcode includes
XCTestObservationCenter, which can (in theory) register more than one test observer.