Probing SenTestingKit

Using class-dump and a short script I wrote, I dumped the SenTesingKit and XCTest Mach-O files included in every version of Xcode since Xcode 3.

Even years after SenTestingKit was included in Xcode, its API barely changed. For years the API was nearly identical to the last open-source release by Sen:te.

In fact, based on the class-dump headers, the implementation didn't change very much either. The first major change doesn't occur until the introduction of XCTest in Xcode 5.

It stands to reason, then, that a good understanding of how SenTestingKit used to work will help us demystify how XCTest works now.

I'll write a summary of the entire system once I've had a chance to look at the command-line interface, otest. In the meantime, below are my debugging notes for SenTestingKit v41.

What follows is pretty much stream of consciousness, so don't expect much.

Diving In: Finding a Starting Point

SenTest itself is like an abstract base class; it implements methods like -testCaseCount, but these are empty and must be overridden by subclasses.

SenTestCase is one such subclass. It returns a -testCaseCount of 1–huh? Shouldn't it return the number of test methods?

On second thought, probably not. I happen to know that each test method is tested using a separate instance of the SenTestCase it belongs to. (This makes sense when you think about how -setUp and -tearDown are used to prepare and destroy instance variables defined on the test class instance.) Each instance of a test case will test only one thing: the invocation representing the test method. Hence, -[SenTestCase testCount] returns 1.

Speaking of invocations, I also happen to know SenTestCase objects have a designated initializer: -initWithInvocation:, which in still in use today. What part of SenTestingKit is responsible for initializing SenTestCase objects with invocations?

Well, there's +[SenTestSuite testSuiteForTestCaseClass:], which calls +[SenTestCase testCaseWithInvocation:], passing it an invocation returned by our old friend +[SenTestCase testInvocations] (Quick and all the other testing frameworks hinge upon this method). But that method is only called by… SenTestCase? In +[SenTestCase defaultTestSuite]. Hmm.

I guess I'm still wondering about the entry point to all of this. What kicks this whole thing off?

An aside: I just noticed SenTestingKit defines a category on NSInvocation: NSInvocation (SenTesting). Among its methods:

  • -[NSInvocation testMethodPrefix], which references a constant value of–you guessed it–@"test".
  • -[NSInvocation hasTestCaseSignature], which returns true if -[NSInvocation isReturningVoid] and -[NSInvocation hasTestPrefix].
  • -[NSInvocation compare:], which determines the order in which SenTestingKit runs the methods. Sure enough, it's alphabetical! Thus the origin of the classic Objective-C testing hack: if you want some test setup to be run before anything else in your test suite, put it in a test case called AAATests in a test method named testAAA.

A fun aside, but hey–I wonder if I can explore where -[NSInvcation compare:] is being used. That'll take me to where SenTestingKit is sorting test methods, anyway.

It's in SenTestCase.m. The default implementation of +[SenTestCase testInvocations] is to grab all the instance method invocations (provided by a method named -senAllInstanceInvocations–where is this defined?), filter by those with -[NSInvocation hasTestCaseSignature], and sort using -[NSInvocation compare:].

Hmm, so investigating NSInvocation (SenTesting) didn't turn up much.

How about we check who calls -[SenTestCase performTest:]? Looks like there's a callsite in SenTest.m–seems promising. Specifically, it's -[SenTest run]. OK, so who calls -run?

Searching for the string run] turned up a few results in the Documentation folder. Speaking of which, maybe we should actually read that…

The Documentation

Ugh, it's HTML 4 or something. The source code is a pain to read, maybe I can open it in a browser.

IntroSenTestingKit.html is a primer on how to write test cases using SenTestCase subclasses. But here's the kicker: it explains you can run tests yourself, by writing [SenTest testWithSelector:@selector(testFoo)]!

Ah, but it goes on: if you stick with SenTestCase subclasses, the framework takes care of running them. (But how?)

The intro page also breaks down the classes a bit:

And finally, it explains: the default SenTestSuite is composed of every test case found in the runtime environment: "all methods with no parameters, returning no value, and prefixed with 'test' in all subclasses of SenTestCase".

Looks like a lot of the other docs are automatically generated via source code comments. Let's give them a once over.

Back to the Source

I guess we'll continue by:

  1. Looking at SenTestProbe and where it's used. The header contains some terms I recognize, like Self–which I know as an option to the otest or xctest command-line programs.
  2. The documentation stated that SenTestingKit will automatically run tests, by using +[SenTestCase defaultTestSuite]. So what part of the system triggers this? Or was this a manual process back then?

Ah! I was confused by SenTestProbe and its tiny header file, but it turns out it's all about the implementation, which uses runtime magic to start itself and kick off the tests. Looks like it's a pretty complicated set of events:

  1. +[SenTestProbe initialize] is called, since the probe is included by SenTestingKit, which is included by the consumers' tests. A check is made for +[SenTestProbe isTesting].
  2. +[SenTestProbe isTesting] checks the current +[SenTestProbe testScope].
  3. +[SenTestProbe testScope] returns the value for a SenTestScopeKey set in user defaults.
  4. +[SenTestProbe isTesting] is determined by +[SenTestProbe testScope]. It returns false if no value has been set for that key or if it's equal to SenTestScopeNone, which is the string @"None". Looks like this is how otest and xctest communicate to SenTestingKit proper.
  5. So anyway, if +[SenTestProbe isTesting], then… hmm. Some more conditionals. First of all, there's a check to see whether the probe is running within an application (so I guess in production?). If so, it's disturbing to see that the probe calls +[SenTestProbe runTests:] with a delay of 0.0 in order to, as a code comment explains, "wait for the application to complete its own loading". If the probe isn't running within an application, it checks whether its been launched by a process named "otest", and if so it calls +[SenTestProbe runTests:] immediately.
  6. Right then, +[SenTestProbe runTests:]. This strange method takes a parameter of type id, named ignored. Baffling! Anywho, it does a few things.
  7. First of all, it calls +[SenTestObserver class] to trigger +[SenTestObserver initialize], which contains user default checks of its own to initialize the appropriate observer instance. (Looks like I was hasty to say that class was "just as it should be".)
  8. Then +[SenTestProbe runTests:] determines what the +[SenTestProbe specifiedTestSuite] is, based on the test scope–either +[SenTestSuite defaultTestSuite] for "All", +[SenTestProbe testedBundlePath] (whatever that is) for "Self", or "None" (which I don't think is an option in this case, since a check was made earlier that the scope was not "None").
  9. Finally, +[SenTestProbe runTests:] runs the tests in the test suite determined in the last step.

So really, SenTestProbe has all the power. How unfortunate that it's implemented as a magical, runtime-powered, global object! All of its methods are class methods; it is a singleton, plain and simple.

Next Steps

SenTestingKit isn't just about the API consumers use to write their tests; it also contains otest, the command-line tool they use to run their tests. How does that work? I'll write this up in another post, along with a reader-friendly overview and some diagrams.