XCTest: The Good Parts

Since my last post about testing, I've been involved with a discussion on Twitter with Apple's Joar Wingfors:

I was especially excited at the opportunity to provide feedback on XCTest:

First, some background on where my critiques are coming from:

XCTest: The Good Parts

First of all: there are a lot of things XCTest does very well. I want to commend the Xcode and XCTest teams on the following points.

Good Part #1: WWDC 2013 - A New Dawn

WWDC 2013 was amazing. I mean, what can I say? Prior to that point, I wasn't sure anyone at Apple cared about unit tests at all. Then, in a single day, Apple engineers introduced:

  1. xcodebuild test: Finally, unit tests on the command line!
  2. Xcode unit test integration: The test navigator, combined with inline test status indicators, really blew me away. It demonstrated the amazing integration that could be achieved with XCTest and Xcode.
  3. WWDC talks that referenced unit testing: At one point, a presenter commented that since he cares about quality, he makes sure to write unit tests. I couldn't believe my ears. I thought it signified a real change in how the iOS developer community would think of testing their code.
  4. New Xcode 5 projects no longer provided an option to include unit tests: New Xcode projects would include unit tests by default, and there was no way to opt out. This encourages much better behavior.

Good Part #2: Extensibility

There are two main styles of unit tests, from what I've seen:

XCTest is an "xUnit" framework. But, thankfully, it is extensible. Developers can override +[XCTestCase testInvocations] to provide an array of test invocations. So while we are forced to write XCTestCase subclasses, we can define test methods at runtime.

I am extremely grateful that the creators of SenTestingKit and XCTest chose to include this amount of extensibility. This has allowed for a great deal of innovation. It's the primary mechanism that makes frameworks like Cedar, Kiwi, Specta, and Quick (all xSpec frameworks) possible.

What's more, Xcode integration is nearly seamless, even when overriding +[XCTestCase testInvocations]. Test results for those invocations appear in the test navigator, and I can re-run individual invocations either via the navigator or the icon in the gutter next to my test source code. Phenomenal.

Good Part #3: XCTAssert is a Function in Swift

The STAssert and XCTAssert family of macros had strange, implicit dependencies in Objective-C: they required that a variable named self was defined, and that variable was a pointer to a subclass of SenTestCase or XCTestCase.

I don't know how it's possible, but in Swift, the XCTAssert family of assertions are functions, and they don't take a reference to an XCTestCase subclass as an argument.

Why do I care? Well, this also helps with extensibility. Let's say I wanted to define a custom assertion, like the following:

func assertContains<T: Equatable>(
  xs: [T], x: T, file: String = __FILE__, line: UInt = __LINE__) {
    XCTAssertTrue(
      contains(xs, x), 
      "\(xs) should contain \(x)",
      file,
      line
    )
}

Prior to Swift, this function would have also had to take a parameter called self, of type XCTestCase.

XCTest: Areas for Improvement

There are a lot of things that XCTest does really well. Below are some suggestions for what it could do even better, in my order of my personal preference.

Potential Improvement #1: Customizing XCTestCase Invocations in Swift

In "Good Part #2", I explained that one of the best aspects of XCTest is that it allows developers to customize it's behavior, by overriding +[XCTestCase testInvocations]. But NSInvocation isn't available in Swift, so it's not possible to use that API from Swift. My radar for this issue has remained open since June 6th, 2014.

Were I able to use this API from Swift, I'd rewrite Quick almost entirely in Swift. I say "almost" because to convert it completely I'd need a solution for my next suggestion.

Potential Improvement #2: Add XCTAssertThrows to Swift

Since XCTAssertThrows is unavailable in Swift, there's no way to write unit tests for code that may raise an NSException. In other words, there's no way to write a test, in Swift, for the following Objective-C code:

void sayHello(NSString *name) {
  NSParameterAssert(name != nil);
  NSLog(@"Hello, %@", name);
}

I think one of the principal values of unit tests is that they force the developer to think of edge cases. Without being able to test exceptions, there's no way to test my assumptions of what happens in some edge cases.

My radar for this issue was closed as a duplicate of radar #16453199, which is still open.

I've worked around this issue by writing a function in Swift that takes a closure as an argument, and then calls a function written in Objective-C. The Objective-C function runs the block in a try-catch:

// Swift

func assertThrows(closure: () -> ()) {
  XCTAssertTrue(throws(closure))
}

// Objective-C

typedef void (^VoidBlock)(void);
BOOL throws(VoidBlock block) {
  @try {
    block();
  }
  @catch {
    return YES;
  }
  return NO;
}

Potential Improvement #3: Testing Swift assert and precondition

The Swift standard library doesn't provide any concept of an "exception". In order to indicate user error, Swift conventionally uses assert or precondition. However, there is no way to test that production code–code I'll be shipping to users of my app–triggers an assert or precondition. All three of the following functions are untestable:

func decrement(x: UInt) -> UInt {
  return x - 1  // Crashes if x == 0
}

func decrementWithAssert(x: Int) -> Int {
  assert(x > 0)
  return x - 1
}

func decrementWithPrecondition(x: Int) -> Int {
  precondition(x > 0)
  return x - 1
}

I think it's absolutely crazy that the only way to test whether my Swift code triggers an assert is by waiting for crash reports from my users, so I filed a radar.

Potential Improvement #4: Decoupling XCTest and Xcode

I mentioned in "Good Part #2" that XCTest is "extensible". This is true: because I can override +[XCTestCase testInvocations], I can configure that behavior.

But there is so much more XCTest behavior that is totally opaque to me:

The problem, as I see it, is that XCTest conflates three responsibilities:

  1. Running a suite of tests (whether that be all of the tests in a suite, or just one of them).
  2. Providing a way for developers to write tests, via an xUnit framework.
  3. Displaying the results of a test suite from within Xcode.

Ideally, I'd like to be able to do #1 myself, and opt into #3 if I so choose.

Code speaks louder than words–I'd be thrilled if XCTest exposed something like the following API for running unit tests:

// MARK: Test Instances

/// A location represents a particular line
/// in a file containing source code.
struct XCTLocation {
  let file: String
  let line: UInt
}

/// A single test, run as part of a test suite.
struct XCTestInstance {
  /// The name of this test.
  let name: String
  /// Assertions should be made from within this closure,
  /// notifying the reporters of any failures.
  let closure: (reporters: [XCTestReporter]) -> ()
  /// If `true`, this test will not be run as part
  /// of the test suite.
  let shouldSkip: Bool
  /// The location of the test (optional).
  let location: XCTLocation?
}

// MARK: Reporting

/// A notification payload for when a test suite
/// has begun running.
/// Defined as a struct, in case we'd like to change
/// the notification payload in future versions.
struct XCTestSuiteBegin {
  let totalTestCount: UInt
}

/// A notification payload for when a test instance
/// is about to be run.
/// Defined as a struct, in case we'd like to change
/// the notification payload in future versions.
struct XCTestInstanceBegin {
  let instance: XCTestInstance
}

/// A notification payload for when a test instance
/// has succeeded.
/// Defined as a struct, in case we'd like to change
/// the notification payload in future versions.
struct XCTestInstanceSuccess {
  let instance: XCTestInstance
}

/// A notification payload for when a test instance has failed.
/// Defined as a struct, in case we'd like to change
/// the notification payload in future versions.
struct XCTestInstanceFailure {
  let instance: XCTestInstance
  let reason: String
}

/// An instance of a test that was skipped.
/// Defined as a struct, in case we'd like to change
/// the notification payload in future versions.
struct XCTestInstanceSkipped {
  let instance: XCTestInstance
}

/// An instance of a completed test suite.
/// Defined as a struct, in case we'd like to change
/// the notification payload in future versions.
struct XCTestSuiteComplete {
  let run: XCTestRun
}

/// A reporter is responsible for presenting failures to the
/// user. A private instance of this protocol, `XCTXcodeReporter`,
/// would be responsible for displaying test results in Xcode.
protocol XCTestReporter {
  func reportSuiteBegin(begin: XCTestSuiteBegin)
  func reportInstanceBegin(begin: XCTestInstanceBegin)
  func reportInstanceSuccess(success: XCTestInstanceSuccess)
  func reportInstanceFailure(failure: XCTestInstanceFailure)
  func reportInstanceSkipped(skipped: XCTestInstanceSkipped)
  func reportSuiteComplete(complete: XCTestSuiteComplete)
}

/// A set of options to configure the behavior of a test run.
struct XCTestRunConfiguration {
  /// The number of threads that should be used to
  /// execute the tests. If zero, the runner determines
  /// the optimal number of cores.
  let numberOfThreads: UInt
  /// A closure that is executed every time a test
  /// instance finishes running. Return `true` to
  /// continue running the test suite, `false` to exit early.
  let shouldContinue(success: XCTestInstanceSuccess?,
                     failure: XCTestInstanceFailure?,
                     skipped: XCTestInstanceSkipped?) -> Bool
}

/// The default configuration for a test suite run.
/// This mirrors the current behavior of XCTest in Xcode 6.3β2.
let XCTestRunDefaultConfiguration = XCTestRunDefaultConfiguration(
  numberOfThreads: 1,
  shouldContinue: { _, _, _ in return true }
)

extension XCTestCase {
  /// Collects all XCTestCase subclasses' test methods, wraps them
  /// in an instance of XCTestInstance, and returns them
  /// in alphabetical order.
  /// This mirrors the current behavior of XCTest in Xcode 6.3β2.
  class func testInstances() -> [XCTestInstance]
}

/// Runs an ordered collection of test instances.
/// :param: tests An ordered collection of test instances.
///               Each test instance is run, and the results are
///               reported to the reporter.
/// :param: reporter The repoter that is provided to each
///                  By default, the reporter is an instance of
///                  XCTXcodeReporter, which knows how to
///                  display those results in Xcode. However,
///                  a different reporter may be supplied by the
///                  user. For example, the user may choose
///                  to supply a reporter that only prints
///                  test failures.
/// :param: configuration A set of options dictating
///                       how a test suite run should behave.
func runTests(
  tests: [XCTestInstance] =
    XCTestCase.testInstances(),
  reporters: [XCTestReporter] =
    [XCTXcodeReporter.sharedReporter()],
  configuration: XCTestRunConfiguration =
    XCTestRunDefaultConfiguration
)

In a new Xcode project, a test bundle should be a simple target that has the following "main function":

func main() -> Int {
  runTests()
  return 0
}

The defaults for the runTests mirrors XCTest behavior in Xcode 6.3β2. But if so desired, the function could be given a different set of reporters, or a different set of tests. Furthermore, because Xcode integration is "opt-in", there's nothing preventing me from running the tests from wherever I like–including from within another test.

I'm not saying that the API above is the only one I'd tolerate–I'd prefer any API that decouples the running of the tests from the reporting of the test results in Xcode.

Potential Improvement #5: Better Support for 1,000+ Test Cases/Methods

If your project has more than a few thousand unit tests, the test navigator icon in Xcode is like a booby-trap: accidentally tap it, and Xcode will spend a solid minute frozen. I assume it's loading all the test methods and displaying them, synchronously, on the main thread. Xcode is completely unresponsive during this time, which absolutely kills my productivity–and that's just displaying the tests.

Even worse is accidentally running the unit tests–with over a few thousand unit tests, Xcode freezes for a minute or so, then crashes. I assume it's running out of memory as it attempts to collect all the test invocations.

The only way to run test suites with 1,000+ tests via Xcode is in small batches. In order to do so, I edit the scheme containing the test target, disabling everything but the subset of tests I need immediate feedback on. But opening Edit Scheme > Test > Info pane also freezes Xcode, as it once again synchronously loads every single test case and method. It is just way too slow to work with.

Potential Improvement #6: Display Test Invocations in Test Navigator Prior to Running Them

One of the best parts of WWDC 2013 was hearing that you could run an individual test from within the Xcode test navigator. But that's only possible with test methods defined on XCTestCase. Test invocations returned via +[XCTestCase testInvocations] only show up after those tests have been run once. It'd be great if Xcode could discover these invocations before running the tests.