by Brian Gesiak

Swift compiler and swift-corelibs-xctest committer. Creator of Quick, the Swift (and Objective-C) testing framework. Former Japanese literature translator.

Page 2

A Humble Proposal for swift-corelibs-xctest

EDIT (March 8, 2016): I’ve since reconsidered some of the APIs I proposed in this post from last December. My number one priority right now is achieving feature parity with Apple’s XCTest, but once that’s accomplished I still think swift-corelibs-xctest development can help shape the future of testing on Apple’s platform.

  1. Have XCTMain take a list of test closure/run option tuples. XCTestCase is merely an API compatibility layer for people used to Apple’s XCTest.
  2. XCTMain also takes a configuration object as a parameter. Based on the configuration, it runs the test closures in parallel, in separate processes, and more. Alternatively, it can be configured to run the tests sequentially, as Apple’s XCTest does. Sub-processes for tests would make it easy to implement performance testing, Swift assert tests, parameterized tests, add a timeout on each test, and more.
  3. The configuration object

Continue reading →


Xcode 7 includes a project setting named “Apple LLVM 7.0 - Warnings - All languages > Incorrect Uses of Nullable Values”, or CLANG_WARN_NULLABLE_TO_NONNULL_CONVERSION.

You can pass it as a compiler flag using -Wnullable-to-nonnull-conversion.

 What does it do?

To demonstrate, consider the following command-line app:

#import <Foundation/Foundation.h>

void welcomeToClowntown(NSString *_Nonnull greeting) {
  NSLog(@"%@, welcome to clowntown!", greeting);

int main(int argc, const char * argv[]) {
  // Warning: Null passed to a callee that requires 
  //          a non-null argument
  return 0;

Notice that we’re passing a nil literal to our function welcomeToClowntown(), even though it takes a parameter of type NSString *_Nonnull. This always emits a warning, regardless of whether CLANG_WARN_NULLABLE_TO_NONNULL_CONVERSION is enabled.

But what about

Continue reading →

SenTestingKit: How Does it Even?

Have you ever wondered what used to happen when you hit ⌘U to run your unit tests in Xcode 4.6 and prior? Here’s a step-by-step sequence of events detailing how tests are loaded and run using SenTestingKit.

As I explain in Probing SenTestingKit, XCTest (née SenTestingKit) has changed surprisingly little since it was first adopted by Apple in 2005. Although what follows is based on the 2005 source code to SenTestingKit, I assume Xcode 7 and XCTest still operate very similarly. I’ll test that assumption in future blog posts.

 1. Xcode invokes a command-line tool called otest to run your tests.

In XCTest, this tool has been renamed to xctest. You can run it yourself from the command line:

$ /Applications/Xcode.app/Contents/Developer/usr/bin/xctest

otest is a simple program that loads a bundle at a given path.

$ xctest /path/to/bundle

The loaded bundle is assumed to have a class

Continue reading →

Probing SenTestingKit

Using class-dump and a short script I wrote, I dumped the SenTesingKit and XCTest Mach-O files included in every version of Xcode since Xcode 3.

Even years after SenTestingKit was included in Xcode, its API barely changed. For years the API was nearly identical to the last open-source release by Sen:te.

In fact, based on the class-dump headers, the implementation didn’t change very much either. The first major change doesn’t occur until the introduction of XCTest in Xcode 5.

It stands to reason, then, that a good understanding of how SenTestingKit used to work will help us demystify how XCTest works now.

I’ll write a summary of the entire system once I’ve had a chance to look at the command-line interface, otest. In the meantime, below are my debugging notes for SenTestingKit v41.

What follows is pretty much stream of consciousness, so don’t expect much.

 Diving In: Finding a

Continue reading →

Diffing APIs with class-dump

In order to stay abreast on the latest changes to XCTest, I run class-dump on it whenever I download a new version of Xcode.

You can download the latest version and try it out yourself. From the command line:

$ class-dump -H /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks/XCTest.framework

This creates a list of headers found in the framework:

// ...and so on.

This allows me to see what’s changed between two versions of XCTest on a per-file basis. It’s fun to see how XCTest changed between Xcode 7.0 and Xcode 7.1b2, for example.

Because the changes are split up into files, I can clearly see that there’s a brand new interface named XCUIRecorder_tvOS.

Try it out on a framework you’re interested in!

Continue reading →

A Swift Summit Retrospective: Better Together

In a panel discussion that closed Swift Summit, Thoughtbot’s Jack Nutting took the mike to say that “conferences like this are important. They give us a chance to discuss ideas and collaborate. And the Swift team at Apple has shown that they’re listening. They’ve responded to feedback and fixed many issues that we’ve raised as a community.” [1]

I couldn’t agree more.

Over the course of two days, Ayaka Nonaka spoke about writing command-line scripts in Swift, but remarked that while one could link frameworks to the script using xcrun swift -F, this couldn’t be done with projects generated with CocoaPods. That same night, Boris Bügling–a.k.a. @NeoNacho–open-sourced Rome, which is a tool that allows you to do just that. Jorge Izquierdo, a student from Madrid, used the new knowledge and tools to build a CLI to his Swift HTTP server, called Taylor.

The collaboration didn’t stop there, of

Continue reading →

XCTest: The Good Parts

Since my last post about testing, I’ve been involved with a discussion on Twitter with Apple’s Joar Wingfors:

I was especially excited at the opportunity to provide feedback on XCTest:

First, some background on where my critiques are coming from:

  • I’ve been writing unit tests in iOS since 2011, beginning with SenTestingKit.
  • Also around 2011, I began using Kiwi, an open-source testing framework that hooked into +[XCTestCase testInvocations] in order to provide a different way for developers to write unit tests.
  • Also around this time I helped set up continuous integration for the iOS

Continue reading →

Quick 1.0: Better Tests for All

A few days ago, Quick reached 2,000 stars on GitHub. This is a great opportunity to explain my thoughts on Quick, and what role I think it should play in the iOS and OS X ecosystem.

 iOS/OS X Testing: It’s Not About Frameworks or Features

The amount of interest surrounding Quick has been breathtaking. And while I’m excited that so many developers are choosing to use Quick to test their apps’ behavior, I’m even more excited that they’re choosing to write unit tests in the first place.

The quality of a test suite doesn’t have a lot to do with the testing framework being used. A test suite could be composed of clear, behavioral tests written in XCTest, or it could be made up of brittle, confusing tests written in Quick.

Don’t get me wrong–Quick provides tools that can be used to write killer tests:

  • Incremental test setup and teardown using beforeEach and afterEach
  • Suite-level setup

Continue reading →

Swift & C: What I Learned Building Swift Bindings to libgit2

I’ve spent the last few weekends working on Gift, Swift bindings to libgit2. It was a really fun introduction to using C from Swift, which isn’t always easy. Here are some of the things I learned.

 Interacting with C Pointers from Swift

It’s a good thing many people have already written on this subject. I found the following links extremely helpful:

  • Interacting with C Pointers, from Apple’s Swift blog
  • Using Legacy C APIs with Swift, by @jparishy
  • Interacting with C Pointers in Swift, by @develtima

 C Pointers in libgit2

Most functions in libgit2 return an error code, and take a pointer as an argument. If the error code is 0, we know the function succeeded in populating the pointer with a value:

 * Parse a SHA, like "f9e7a", into an object ID.
 * @param out A pointer to a struct, of type `git_oid`, 
*             that the result is written to.
 * @param str input The SHA

Continue reading →

Thoughts on “Out of the Tar Pit”

Dave Lee recommended I read Out of the Tar Pit, a paper on software complexity. Here are some thoughts I jotted down while reading:

  • Page 2: Dijkstra has got to be the most quotable computer scientist. On software complexity: “…we have to keep it crisp…”. My new motto.
  • Page 5: The critique of testing, that the “huge number of different possible inputs usually rules out the possibility of testing them all, hence the unavoidable concern with testing will always be–have you performed the right tests?”, echoes my thoughts since encountering QuickCheck, Fox, and property-based testing. I have not, in recent years, been practicing strictly “test-driven” development–I sometimes write my tests after the implementation. Colleagues have claimed that disciplined test-driven development “eliminates assumptions one makes in writing tests”, but I simply do not think that is true. To be clear

Continue reading →