class-dump and a short script I wrote, I dumped the SenTesingKit and XCTest Mach-O files included in every version of Xcode since Xcode 3.
Note: If you find this post on XCTest and SenTestingKit internals useful, consider helping me write more of these posts, by supporting me on Patreon.
Even years after SenTestingKit was included in Xcode, its API barely changed. For years the API was nearly identical to the last open-source release by Sen:te.
In fact, based on the
class-dump headers, the implementation didn't change very much either. The first major change doesn't occur until the introduction of XCTest in Xcode 5.
It stands to reason, then, that a good understanding of how SenTestingKit used to work will help us demystify how XCTest works now.
I'll write a summary of the entire system once I've had a chance to look at the command-line interface,
otest. In the meantime, below are my debugging notes for SenTestingKit v41.
What follows is pretty much stream of consciousness, so don't expect much.
Diving In: Finding a Starting Point
SenTest itself is like an abstract base class; it implements methods like
-testCaseCount, but these are empty and must be overridden by subclasses.
SenTestCase is one such subclass. It returns a
1–huh? Shouldn't it return the number of test methods?
On second thought, probably not. I happen to know that each test method is tested using a separate instance of the
SenTestCase it belongs to. (This makes sense when you think about how
-tearDown are used to prepare and destroy instance variables defined on the test class instance.) Each instance of a test case will test only one thing: the invocation representing the test method. Hence,
-[SenTestCase testCount] returns
Speaking of invocations, I also happen to know
SenTestCase objects have a designated initializer:
-initWithInvocation:, which in still in use today. What part of SenTestingKit is responsible for initializing
SenTestCase objects with invocations?
+[SenTestSuite testSuiteForTestCaseClass:], which calls
+[SenTestCase testCaseWithInvocation:], passing it an invocation returned by our old friend
+[SenTestCase testInvocations] (Quick and all the other testing frameworks hinge upon this method). But that method is only called by…
+[SenTestCase defaultTestSuite]. Hmm.
I guess I'm still wondering about the entry point to all of this. What kicks this whole thing off?
An aside: I just noticed SenTestingKit defines a category on
NSInvocation (SenTesting). Among its methods:
-[NSInvocation testMethodPrefix], which references a constant value of–you guessed it–
-[NSInvocation hasTestCaseSignature], which returns true if
-[NSInvocation compare:], which determines the order in which SenTestingKit runs the methods. Sure enough, it's alphabetical! Thus the origin of the classic Objective-C testing hack: if you want some test setup to be run before anything else in your test suite, put it in a test case called
AAATestsin a test method named
A fun aside, but hey–I wonder if I can explore where
-[NSInvcation compare:] is being used. That'll take me to where SenTestingKit is sorting test methods, anyway.
SenTestCase.m. The default implementation of
+[SenTestCase testInvocations] is to grab all the instance method invocations (provided by a method named
-senAllInstanceInvocations–where is this defined?), filter by those with
-[NSInvocation hasTestCaseSignature], and sort using
Hmm, so investigating
NSInvocation (SenTesting) didn't turn up much.
How about we check who calls
-[SenTestCase performTest:]? Looks like there's a callsite in
SenTest.m–seems promising. Specifically, it's
-[SenTest run]. OK, so who calls
Searching for the string
run] turned up a few results in the
Documentation folder. Speaking of which, maybe we should actually read that…
Ugh, it's HTML 4 or something. The source code is a pain to read, maybe I can open it in a browser.
IntroSenTestingKit.html is a primer on how to write test cases using
SenTestCase subclasses. But here's the kicker: it explains you can run tests yourself, by writing
Ah, but it goes on: if you stick with
SenTestCase subclasses, the framework takes care of running them. (But how?)
The intro page also breaks down the classes a bit:
SenTestCaseimplements one or more test methods.
SenTestSuiteis composed of one or more test cases and has a name.
SenTestRunis the result of running a test. It's the base class for
SenTestObserverlistens for results from
And finally, it explains: the default
SenTestSuite is composed of every test case found in the runtime environment: "all methods with no parameters, returning no value, and prefixed with 'test' in all subclasses of
Looks like a lot of the other docs are automatically generated via source code comments. Let's give them a once over.
- Oh! Looks like
-[NSObject senAllInstanceInvocations]is defined as a category method on
NSObjectwithin this framework. Good to know.
SenTestObserveris pretty much exactly what it should be, aside from the fact that it's an abstract base class, not a protocol.
SenTestProbeis probably really interesting. We should investigate that further.
- Ha! There's something called
SenTestDistributedNotifier. At first I thought, "Oh cool! Does it use an announcer, or maybe XPC?"–nope! It posts
NSNotificationnotifications. NOPE NOPE NOPE. Although it does so via
NSDistributedNotificationCenter–is that any different from
- Further research shows this is like a cross-app notification center. Interesting. I think I'd heard of this a few years ago, but I've never used it. Anyway, I guess it's not as bad as I thought.
Back to the Source
I guess we'll continue by:
- Looking at
SenTestProbeand where it's used. The header contains some terms I recognize, like
Self–which I know as an option to the
- The documentation stated that SenTestingKit will automatically run tests, by using
+[SenTestCase defaultTestSuite]. So what part of the system triggers this? Or was this a manual process back then?
Ah! I was confused by
SenTestProbe and its tiny header file, but it turns out it's all about the implementation, which uses runtime magic to start itself and kick off the tests. Looks like it's a pretty complicated set of events:
+[SenTestProbe initialize]is called, since the probe is included by SenTestingKit, which is included by the consumers' tests. A check is made for
+[SenTestProbe isTesting]checks the current
+[SenTestProbe testScope]returns the value for a
SenTestScopeKeyset in user defaults.
+[SenTestProbe isTesting]is determined by
+[SenTestProbe testScope]. It returns false if no value has been set for that key or if it's equal to
SenTestScopeNone, which is the string
@"None". Looks like this is how
xctestcommunicate to SenTestingKit proper.
- So anyway, if
+[SenTestProbe isTesting], then… hmm. Some more conditionals. First of all, there's a check to see whether the probe is running within an application (so I guess in production?). If so, it's disturbing to see that the probe calls
+[SenTestProbe runTests:]with a delay of
0.0in order to, as a code comment explains, "wait for the application to complete its own loading". If the probe isn't running within an application, it checks whether its been launched by a process named
"otest", and if so it calls
- Right then,
+[SenTestProbe runTests:]. This strange method takes a parameter of type
ignored. Baffling! Anywho, it does a few things.
- First of all, it calls
+[SenTestObserver class]to trigger
+[SenTestObserver initialize], which contains user default checks of its own to initialize the appropriate observer instance. (Looks like I was hasty to say that class was "just as it should be".)
+[SenTestProbe runTests:]determines what the
+[SenTestProbe specifiedTestSuite]is, based on the test scope–either
+[SenTestSuite defaultTestSuite]for "All",
+[SenTestProbe testedBundlePath](whatever that is) for "Self", or "None" (which I don't think is an option in this case, since a check was made earlier that the scope was not "None").
+[SenTestProbe runTests:]runs the tests in the test suite determined in the last step.
SenTestProbe has all the power. How unfortunate that it's implemented as a magical, runtime-powered, global object! All of its methods are class methods; it is a singleton, plain and simple.
SenTestingKit isn't just about the API consumers use to write their tests; it also contains
otest, the command-line tool they use to run their tests. How does that work? I'll write this up in another post, along with a reader-friendly overview and some diagrams.