A Humble Proposal for swift-corelibs-xctest

EDIT (March 8, 2016): I've since reconsidered some of the APIs I proposed in this post from last December. My number one priority right now is achieving feature parity with Apple's XCTest, but once that's accomplished I still think swift-corelibs-xctest development can help shape the future of testing on Apple's platform.

  1. Have XCTMain take a list of test closure/run option tuples. XCTestCase is merely an API compatibility layer for people used to Apple's XCTest.
  2. XCTMain also takes a configuration object as a parameter. Based on the configuration, it runs the test closures in parallel, in separate processes, and more. Alternatively, it can be configured to run the tests sequentially, as Apple's XCTest does. Sub-processes for tests would make it easy to implement performance testing, Swift assert tests, parameterized tests, add a timeout on each test, and more.
  3. The configuration object passed to XCTMain can be given a set of reporters. Like XCTObservation.h, the reporters are sent messages regarding test events, like "test started" or "test failed".
  4. Apple's XCTest uses XPC to translate XCTest events into GUI updates in Xcode, via an internal protocol caled XCTestManager_IDEInterface. This could theoretically be implemented using an swift-corelibs-xctest reporter. File radars to try and convince the Xcode team to use swift-corelibs-xctest for unit tests, thereby bringing parallel test execution and other test features to Apple XCTest.
  5. Claim Internet Bucks™.

Here's an example of the API I have in mind:

public protocol TestStartedNotification {
    let timestamp: UInt
    // ...and more parameters related to "test started" notification.
}
public protocol TestFailedNotification {
    let timestamp: UInt
    // ...and more parameters related to "test failed" notification.
}
// ...and more notifications.

public protocol Reporter {
    func testStarted(notification: TestStartedNotification)
    func testFailed(notification: TestFailedNotification)
    // ...and more notification methods.
}

public protocol MainOptions {
    let reporters: [Reporter]
    let runEachTestInSeparateProcess: Bool
    // ...and more XCTMain options.
}

public protocol TestOptions {
    let timeoutInSeconds: UInt
    // ...and more individual test options.
}

public func XCTMain(tests: [(() -> (), TestOptions)], 
    options: MainOptions = MainOptions.defaultOptions())

The design would be inspired primarily by JUnit 4. It's a really great API.

Please tweet me with any thoughts. I'd like to flesh out this idea and discuss it more on the mailing lists.