An Introduction to the Swift Compiler Driver

November 4, 2017

This article contains some content especially for patrons. Although it reads coherently as-is, to read the full article, please consider supporting me on Patreon, or click here if you are already a patron. $10/month gives you access to all content I will ever write on this website.

The first article of this series on the Swift compiler explained that the compiler is just a command-line program, named swift or swiftc. This article explains what that command-line program does.

Specifically, this article explains the Swift "driver". This is the first bit of code that is executed when users invoke swift or swiftc on the command line.

Like any other C, C++, or Objective-C command-line program, swift and swiftc begin by executing an int main() function:

111  int main(int argc_, const char **argv_) {
...
207  }

The main() function calls out to code in a Swift compiler library, libswiftDriver, in order to break up a single invocation of swift or swiftc into smaller chunks of work. These smaller chunks are called "jobs".

Swift driver jobs by example: compiling Swift source code into an executable

For example, let's say I wanted to compile the following Swift source file, hello.swift, into an executable:

print("Hello!")

To produce an executable named hello, I'll invoke swiftc on the command line:

swiftc hello.swift -o hello

This produces an executable named hello, by first compiling the Swift code into a .o object file, and then linking that file into an executable named hello. I can request to see the exact series of jobs that led to the final hello executable by using the -driver-print-jobs option:

swiftc -driver-print-jobs hello.swift -o hello

This outputs two jobs:

1. The first is an invocation of swift -frontend that compiles the hello.swift source code file into an object file named hello-6ab6fd.o.
2. The second is an invocation of the linker, ld, which links the hello-6ab6fd.o object file with the Swift and Objective-C runtimes in order to produce the executable I requested: hello.

Here's the full output:

/Users/bgesiak/Source/apple/build/Ninja-ReleaseAssert/swift-macosx-x86_64/bin/swift \
    -frontend -c -primary-file hello.swift \
    -target x86_64-apple-darwin16.7.0 \
    -enable-objc-interop -color-diagnostics -module-name hello \
    -o /var/folders/ry/2ryfdsb56b30092626qprw6d3rb3ss/T/hello-6ab6fd.o

/usr/bin/ld \
    /var/folders/ry/2ryfdsb56b30092626qprw6d3rb3ss/T/hello-6ab6fd.o \
    -lobjc -lSystem -arch x86_64 -L /Users/bgesiak/Source/apple/build/Ninja-ReleaseAssert/swift-macosx-x86_64/lib/swift/macosx \
    -rpath /Users/bgesiak/Source/apple/build/Ninja-ReleaseAssert/swift-macosx-x86_64/lib/swift/macosx \
    -macosx_version_min 10.12.0 -no_objc_category_merging \
    -o hello

It would be tedious if users of the Swift compiler had to invoke two commands, swift -frontend and ld, every time they wanted to generate an executable from a Swift source code file. The Swift driver exists to make this more convenient: it translates a single invocation of swift or swiftc into a series of swift -frontend and ld jobs that do what the user wants.

My example above may have been too simple to demonstrate just how useful the Swift driver is – it only took two jobs to do what I wanted. However, compiling multiple Swift source files requires a separate swift -frontend job for each one. Producing a .swiftmodule from those source files requires invocations of swift -frontend -merge-modules, as well as other jobs. The Swift driver coordinates all of this work for me, translating a simple invocation such as swiftc -emit-module into what is sometimes dozens of jobs.

If that weren't enough, the Swift driver is also responsible for determining when jobs do not need to be run, because their output is already available and up-to-date. This is known as "incremental building". If you're interested in helping reduce the amount of time it takes for your Swift source code project to compile, you'll want to understand how the driver accomplishes this. It's a topic I plan on writing about at length in future articles.

When I think of the Swift compiler, I think of a program that parses the Swift source code in a file, outputs warnings or errors if that source code isn't syntactically correct, and then translates that source code into binary 0's and 1's that my operating system knows how to execute as a program. However, it's important to make a distinction: it's invocations of swift -frontend that are responsible for all that good stuff.

Invocations of swift and swiftc, without the -frontend argument, invoke the driver, not the frontend. The Swift driver doesn't type-check Swift source code, it generates jobs that invoke swift -frontend. Those invocations of swift -frontend, in turn, do all the cool compiler stuff I think of when I think of compilers, such as type-checking.

A bird's-eye view of the code in the Swift driver

The basic idea of the Swift driver is that it splits up invocations of swift and swiftc into other, more granular invocations of swift -frontend, ld, and other programs. These child invocations are called "jobs". So, how does it accomplish this? What does the source code in the Swift driver look like, at a high level?

The driver source code exists in two primary components: the driver executable C++ code, and the C++ libswiftDriver library code. The driver executable calls into the libswiftDriver library code to perform most of its tasks. libswiftDriver's interfaces are declared in its .h header files, in swift/include/swift/Driver. The implementation of those interfaces is in .cpp implementation files in swift/lib/Driver. The driver executable is implemented in .cpp files in swift/tools/driver, primarily swift/tools/driver/driver.cpp.

The above diagram isn't a complete graph of library dependencies for the swift and swiftc executables. Both the driver executable and libswiftDriver make use of other apple/swift and LLVM libraries, such as libswiftOption (for parsing command-line arguments like -frontend) and libLLVMSupport (which provides utility functions for reading and creating files, among other things). I'll cover these in more detail later.

CMake files, specifically swift/tools/driver/CMakeLists.txt and swift/lib/Driver/CMakeLists.txt, define how the swift and swiftc executables and how the libswiftDriver library are built.

How swift, swiftc, and libswiftDriver are built (and just what is the difference between swift and swiftc, anyway?)

I explained the Swift build system and how the swift executable is built in previous articles. To summarize those articles:

• The apple/swift build system is defined primarily in CMake. The top-level swift/CMakeLists.txt file is executed by the cmake executable, and that file in turn calls the CMake built-in function add_subdirectory() in order to execute the nested CMakeLists.txt files in other directories.
• To understand how a particular set of source files are built, it makes sense to first look at the CMakeLists.txt file that exists within the same directory as the source files.
• The CMakeLists.txt file at the same level as the Swift driver executable source file, swift/tools/driver/driver.cpp, is swift/tools/driver/CMakeLists.txt. That CMakeLists.txt file calls the add_swift_host_tool() function, a custom CMake function defined in the apple/swift project, in order to define how the swift executable should be built.

The swift/tools/driver/CMakeLists.txt file doesn't just define how the swift executable is built. It also defines a symlink, swiftc, that points to swift:

 1  add_swift_host_tool(swift
 2    api_notes.cpp
 3    driver.cpp
 4    autolink_extract_main.cpp
 5    modulewrap_main.cpp
 6    swift_format_main.cpp
 7    LINK_LIBRARIES
 8      swiftDriver
 9      swiftFrontendTool
10    LLVM_COMPONENT_DEPENDS
11      DebugInfoCodeView
12    SWIFT_COMPONENT compiler
13  )
..
34  add_swift_tool_symlink(swiftc swift compiler)
35  add_swift_tool_symlink(swift-autolink-extract swift autolink-driver)
36  add_swift_tool_symlink(swift-format swift editor-integration)

The name of the function add_swift_tool_symlink() is fairly straightforward, and so I can guess that it creates a symlink named swiftc that points to the executable swift. To confirm this guess, I can use the techniques I described in Reading and Understanding the CMake in apple/swift. A git grep for function(add_swift_tool_symlink reveals the function's definition in swift/cmake/modules/AddSwift.cmake. The function calls through to two LLVM CMake functions. Two more git grep invocations, this time in the LLVM repository, can point me to those functions' definitions, in llvm/cmake/modules/AddLLVM.cmake.

As the CMake confirms, the swiftc executable is really just a symlink to the swift executable. But why create a symlink?

Swift "driver modes", or: "A swift by any other name"

You may have noticed, in the CMake snippet above, that not only does the apple/swift build system produce a swiftc symlink, it also creates symlinks named swift-autolink-extract and swift-format. Each one of these represents a different "driver mode" or, as it's referenced in the libswiftDriver source code, a swift::driver::Driver::DriverKind:

120    enum class DriverKind {
121      Interactive,     // swift
122      Batch,           // swiftc
123      AutolinkExtract, // swift-autolink-extract
124      SwiftFormat      // swift-format
125    };

Each of these driver modes:

• Accepts a different set of arguments. For example, swiftc accepts the argument -dump-ast, but swift -dump-ast is invalid.
• Performs a different set of tasks. For example, swift-format tidies up the whitespace and indentation in Swift source code files. It does not perform any of the compilation that swiftc does.

The "driver mode" is determined by libswiftDriver, which contains code that checks the name of the executable that is currently being run:

 79  void Driver::parseDriverKind(ArrayRef<const char *> Args) {
...
 94    Optional<DriverKind> Kind =
 95    llvm::StringSwitch<Optional<DriverKind>>(DriverName)
 96    .Case("swift", DriverKind::Interactive)
 97    .Case("swiftc", DriverKind::Batch)
 98    .Case("swift-autolink-extract", DriverKind::AutolinkExtract)
 99    .Case("swift-format", DriverKind::SwiftFormat)
100    .Default(None);
...
106  }

That is, when I invoke swift on the command line, that causes libswiftDriver to run in DriverKind::Interactive. swiftc results in DriverKind::Batch. In other parts of libswiftDriver, the current DriverKind is referenced when deciding which command-line options to accept as valid, or which actions to perform.

So while swiftc executes the exact same driver code as swift, the fact that it's named swiftc causes libswiftDriver to behave differently.

This is actually a fun quirk in how Swift works: you can't invoke swift through a symlink the driver code isn't aware of. For example, I tried creating a symlink to my swift executable, like so:

# Change directories to where my 'swift' executable lives:
cd ~/local/Source/apple/build/Ninja-ReleaseAssert/swift-macosx-x86_64/bin/

# Create an alias: 'my-swift-alias' points to 'swift':
ln -s swift my-swift-alias

# Attempt to print the help text for 'swift', by invoking
# my alias:
my-swift-alias -help

Sure enough, this results in an error. The Swift driver doesn't recognize the my-swift-alias driver mode:

<unknown>:0: error: invalid value 'my-swift-alias' in '--driver-mode='

As the error message hints at, I can override the Swift driver behavior that infers the driver mode from the name of the executable by using the --driver-mode= command-line option:

# Print out help text as if I were invoking
# the 'swift' executable:
my-swift-alias --driver-mode=swift -help

Another look at the driver executable and libswiftDriver source code, and next steps

To recap:

• The CMake code for the driver executable not only defines an executable named swift, it also defines symlinks pointing to that executable, such as swiftc and swift-format.
• The driver determines the mode in which it's being run in, in part based on the name of the executable or symlink its being run with, and then it parses its command-line options based on that driver mode.
• Based on its mode and the arguments passed to it, the driver executable calls out to libswiftDriver to build a series of "jobs". These jobs are invocations of swift -frontend, ld, or other executables. It's swift -frontend that performs the actual type-checking and compilation of Swift source code.

Parsing command-line arguments and splitting up a driver invocation into child jobs involves a lot of different classes, most of which are defined in libswiftDriver. To name just a few, there's swift::Driver, swift::Compilation, swift::driver::ToolChain, swift::Action, swift::JobAction, swift::Job, and more. I'll go over the interactions between these classes in more detail in the next article in this series, and then I'll start writing about incremental compilation and other advanced features in libswiftDriver.

In the meantime, try exploring this treemap of the source code in the Swift driver. Each rectangle is sized proportional to the number of source lines of code it contains. Click or tap on a rectangle to view it in GitHub.

The majority of the source code we'll cover next time is in three files:

1. swift/tools/driver/driver.cpp
2. swift/lib/Driver/Driver.cpp
3. swift/lib/Driver/ToolChains.cpp