SenTestingKit: How Does it Even?

Have you ever wondered what used to happen when you hit ⌘U to run your unit tests in Xcode 4.6 and prior? Here's a step-by-step sequence of events detailing how tests are loaded and run using SenTestingKit.

As I explain in Probing SenTestingKit, XCTest (née SenTestingKit) has changed surprisingly little since it was first adopted by Apple in 2005. Although what follows is based on the 2005 source code to SenTestingKit, I assume Xcode 7 and XCTest still operate very similarly. I'll test that assumption in future blog posts.

1. Xcode invokes a command-line tool called otest to run your tests.

In XCTest, this tool has been renamed to xctest. You can run it yourself from the command line:

$ /Applications/Xcode.app/Contents/Developer/usr/bin/xctest

otest is a simple program that loads a bundle at a given path.

$ xctest /path/to/bundle

The loaded bundle is assumed to have a class named SenTestProbe. otest calls +[SenTestProbe runTests:] to kick things off.

SenTestProbe runs from within the loaded bundle. In order to pass along the command-line arguments it was invoked with, otest also sets various key-value pairs to NSUserDefaults. For example, running otest -SenTest All /path/to/bundle, the following is set in NSUserDefaults:

@{
    @"SenTestedUserDefaults": @"/path/to/bundle",
    @"SenTest": @"All" 
}

2. SenTestProbe Determines How it Was Launched

+[SenTestProbe initialize] is called as soon as otest calls +[SenTestProbe runTests:]. The method is a convoluted hack that checks bits of state in NSProcessInfo and NSUserDefaults to determine how the class came to be initialized.

In the case this post is describing, the method would end up doing nothing, since it would determine that it must have been loaded by otest, and otest calls +[SenTestProbe runTests:] directly.

Conversely, if not launched by otest, SenTestProbe calls +[SenTestProbe runTests:] itself.

3. SenTestProbe Initializes Observers

The first thing +[SenTestProbe runTests:] does is initialize a SenTestObserver.

SenTestObserver is a base class that defines an interface for receiving messages related to the state of a SenTestRun:

@interface SenTestObserver : NSObject
// ...
+ (void) testSuiteDidStart:(NSNotification *) aNotification;
+ (void) testSuiteDidStop:(NSNotification *) aNotification;

+ (void) testCaseDidStart:(NSNotification *) aNotification;
+ (void) testCaseDidStop:(NSNotification *) aNotification;

+ (void) testCaseDidFail:(NSNotification *) aNotification;
@end

Notice the design flaws:

  1. The interface is defined as a concrete subclass of NSObject, not a protocol. Worse still, the base class defines behavior shared by all of its subclasses. A protocol would have been cleaner.
  2. The methods take an NSNotification as a parameter. -[NSNotification object] is typed as an id, but in practice SenTestRun objects are sent. A value object parameter would have enforced greater type safety.
  3. The methods are class methods, not instance methods. This makes it impossible to have the same kind of observer act on notifications more than once.

This underpins one of XCTest's greatest shortcomings: its inflexible logging. We explore this tragedy further below.

The fact that SenTestObserver defines a way to interface with SenTestRun notifications, plus the fact that the +[SenTestObserver initialize] method checks NSUserDefaults for a key named @"SenTestObserverClass", may make you assume that, as a consumer of SenTestingKit, you would be able to specify your own, custom test logger.

But the cake is a lie.

In reality, +[SenTestObserver initialize] sets the following to NSUserDefaults…

@{
    @"SenTestObserverClass": @"SenTestLog"
}

…then immediately sets the class specified in the defaults as the "current test observer." The "current observer" then registers for a series of SenTestingKit-specific NSNotification announcements.

+[SenTestObserver setCurrentObserver:] sets a single class as the observer for the entire test suite. The FIXME source code comment from SenTestingKit describes the situation perfectly: "Not very elegant and only one observer currently possible." (Which isn't exactly true, since one could register for the relevant NSNotification strings.)

Like most problems described in FIXME comments, in practice this remains unfixed even today. Although the version of XCTest packaged with Xcode 6.0.1 included a class named XCTestObservationCenter that appeared to be able to register multiple observers, re-entrant unit tests still crash when run with Xcode or xcodebuild.

What we're left with is SenTestLog as the sole listener for NSNotification notifications regarding the state of a test run. SenTestLog is a simple listener that writes the results of each test to stderr–this is the class that would generate the test output seen in the console in Xcode.

3. SenTestProbe Initializes a Test Suite

After initializing the test observer, +[SenTestProbe runTests:] proceeds to initialize the test suite itself.

How a test suite is constructed varies based on the test "scope"–the value passed to otest on the command-line. Here are the options:

  1. otest -SenTest All /path/to/bundle: The default test suite, which includes a test for each test method defined across all subclasses of SenTestCase.
  2. otest -SenTest Self /path/to/bundle: A test suite that only contains tests defined in the specified bundle.
  3. otest -SenTest MyTestCaseSubclass/testMyTestMethod /path/to/bundle: A test suite that contains a single test run for the specified test method.

We'll focus on -SenTest All, since that's what's run when we run tests via Xcode's ⌘U.

+[SenTestSuite defaultTestSuite] creates a test suite named @"All tests", composed of every test method defined across all SenTestCase subclasses. Enumerators are used to build the suite:

  1. +[SenTestSuite defaultTestSuite] enumerates each subclass of SenTestCase, adding the tests in -[SenTestCase defaultTestSuite] to itself.
  2. -[SenTestCase defaultTestSuite] enumerates each test invocation defined on it, adds it to a new test suite object, and then returns that object.

SenTestingKit defines categories on NSObject that use the Objective-C runtime's objc_getClassList to grab all of a class's subclasses, and to create an NSInvocation for each of its methods.

It also defines categories on NSInvocation to determine whether an invocation is what SenTestingKit considers a "test method". It must:

4. The Test Suite is Run

Once the test suite has been prepared, -[SenTestSuite run] is called to kick things off. This creates an instance of SenTestSuiteRun, which is used as a collection of SenTestCaseRun instances. The SenTestSuite enumerates its tests and calls -[SenTestCase run] on each of them, adding the SenTestCaseRun objects returned to its collection.

-[SenTestCase run] is straightforward:

  1. -[SenTestCase setUp] is called to perform user-specified test setup.
  2. -[SenTestCaseRun start] is called. This posts a @"SenTestCaseDidStartNotification" notification.
  3. The test case's invocation is invoked using -[NSInvocation invoke]. SenTestingKit advertises that it makes a distinction between test failures and exceptions that occur during test runs; hence, the invocation is invoked within a @try/@catch. If an exception is raised, it's stored as an attribute on the SenTestCaseRun. Otherwise, the invocation may call -[SenTestCase failWithException:] to add a failure to a SenTestCaseRun.
  4. -[SenTestCaseRun stop] is called. This posts a @"SenTestCaseDidStopNotification" notification.
  5. -[SenTestCase tearDown] is called to perform user-specified test teardown.
  6. A SenTestCaseRun object is returned. The object maintains a failure count and exception properties.

SenTestingKit aficionados may notice that the interface for reporting failures was different in early versions of SenTestingKit: -[SenTestCase failWithException:]. The new interface is -[XCTestCase recordFailureWithDescription:inFile:atLine:expected:].

The move away from failure reporting based on NSException was a great decision. The requirements for reporting are now much clearer. The old interface required assertions to raise exceptions with specific keys in their userInfo, like SenTestFilenameKey and SenTestLineNumberKey.

The SenTestSuiteRun returned by -[SenTestSuite run] has information on the number of failures and the number of exceptions, but recall those failures were also reported via NSNotification notifications, and subsequently logged by the SenTestLog observer. With no additional work to do, the SenTestProbe exits with a status code equal to the number of failures in the SenTestSuiteRun.

Well, there you have it–everything that happens in SenTestingKit after hitting ⌘U in Xcode 4.6.

In an upcoming post, I'll explore how this series of events has changed in subsequent versions of XCTest. For example, the version of XCTest packaged with Xcode includes XCTestObservationCenter, which can (in theory) register more than one test observer.