The Swift Compiler's Build System

This article contains some content especially for patrons. Although it reads coherently as-is, to read the full article, please consider supporting me on Patreon, or click here if you are already a patron. $10/month gives you access to all content I will ever write on this website.

My last post explained that invoking swift/utils/build-script compiles the C++ source code in the apple/swift project and produces a swift compiler executable.

swift/utils/build-script is a Python script that invokes another script, swift/utils/build-script-impl. That, in turn, invokes cmake, in order to configure and then compile the source code in apple/swift.

I'll refer to this process as the Swift build system, and I'll explain it in detail below.

swift/utils/build-script invokes swift/utils/build-script-impl, which invokes cmake, which configures and builds apple/swift.

There are many benefits to having a deep understanding of the build system:

I'll begin with the build system's core: CMake.

Using CMake: An example

The cmake executable reads script files that describe how a software project builds. For example, here's how I'd use it to build a simple C++ program, composed of a single source file, /tmp/src/hello.cpp:

/tmp/class/hello.cpp

#include <iostream>

int main() {
  std::cout << "Hello!" << std::endl;
  return 0;
}

I can describe how to build this program using the CMake language, in a file named src/CMakeLists.txt:

/tmp/src/CMakeLists.txt

# CMake version 3.2 or greater must
# be used to configure this project.
cmake_minimum_required(VERSION 3.2)

# The name of the project is 'Hello'.
project(Hello)

# When built, create an executable
# named 'hello', by compiling the
# source file 'hello.cpp'.
add_executable(hello
               hello.cpp)

Running cmake on the command line generates build files that are used to actually build the project:

# Read the CMakeLists.txt in the '/tmp/src' directory,
# and generate build files at '/tmp/build'.
cmake -H/tmp/src -B/tmp/build

I can then compile the hello executable by executing the build files in /tmp/build. By default, CMake generates a Makefile, which I can execute like so:

make -C /tmp/build

Running the above command compiles and links the executable /tmp/build/hello. Running that program outputs the expected text, "Hello!".

It's important to note that CMake itself doesn't compile and link the hello executable. CMake generates build files, and those build files compile and link hello. I can instruct CMake to generate different kinds of build files, including an Xcode project:

cmake -H/tmp/src -B/tmp/build-with-xcode -G Xcode

The above command generates /tmp/build-with-xcode/Hello.xcodeproj. I can open Hello.xcodeproj in Xcode and build hello by clicking the "Run" button:

Building the CMake project via the Xcode GUI.

I can also build hello via the command line, by invoking xcodebuild:

xcodebuild -project /tmp/build-with-xcode/Hello.xcodeproj

Notice how, when I didn't specify a build file generator using the cmake -G argument, cmake generated a Makefile, that I could build by invoking make. The Xcode project, on the other hand, was built by invoking xcodebuild. Remembering which invocation to use is tiresome, so cmake provides a convenient way to build using whichever files were generated:

# Build using the Makefile in the
# /tmp/build directory.
cmake --build /tmp/build

# Build using the .xcodeproj in the
# /tmp/build-with-xcode directory.
cmake --build /tmp/build-with-xcode

CMake has many more features than just the three lines from my example above. I'll cover additional concepts below, as we encounter them in apple/swift's CMake code. To learn about CMake itself, try LLVM's CMake primer, or the latest official CMake documentation.

Why does apple/swift use CMake?

If you're an iOS developer, you've likely used Xcode project files to specify how your iOS application is built. So why couldn't the apple/swift project just use Xcode, too? The simple hello example above demonstrates several reasons why:

$10+ patron-only content

Three reasons why CMake is used in the apple/swift project.

I am a patronBecome a patron

Building apple/swift with CMake

The apple/swift project is essentially the same as my "Hello" project above. Yes, it is a larger project with more CMake code, but it can be configured and built in the same way "Hello" can. Doing so is a useful exercise in understanding the apple/swift build infrastructure.

First, I'll clone the source code from the apple/swift project, as well as the three projects it depends upon:

$10+ patron-only content

A short explanantion of each of the three projects apple/swift depends upon.

I am a patronBecome a patron

I'll clone these projects into a directory named ~/local/Source/apple/standalone:

git clone https://github.com/apple/swift-cmark.git \
    ~/local/Source/apple/standalone/swift-cmark

git clone https://github.com/apple/swift-llvm.git \
    ~/local/Source/apple/standalone/swift-llvm

# A bug in Swift's CMake requires this repository
# to be named 'clang', not 'swift-clang'.
# https://bugs.swift.org/browse/SR-5778
git clone https://github.com/apple/swift-clang.git \
    ~/local/Source/apple/standalone/clang

git clone https://github.com/apple/swift.git \
    ~/local/Source/apple/standalone/swift

$10+ patron-only content

A demonstration of configuring and building apple/swift, as well as its dependencies, as a "standalone" CMake project.

I am a patronBecome a patron

Once the build finishes, the built swift compiler executable is located at ~/local/Source/apple/standalone/swift-build/bin/swift. It works exactly as well as the executable I built using swift/utils/build-script in my last post.

What do swift/utils/build-script and swift/utils/build-script-impl do?

$10+ patron-only content

An explanation of the purpose and behavior of swift/utils/build-script and build-script-impl. This section also contains a brief discussion of the tradeoffs of having users invoke them instead of cmake.

I am a patronBecome a patron

This is a frequent stumbling block for new contributors to apple/swift.

Many experienced contributors recommend newcomers invoke ninja -C ~/local/Source/apple/build/swift-macosx-x86_64 directly in order to perform faster incremental builds of apple/swift. But a new contributor who doesn't know what swift/utils/build-script does will have a lot of questions:

  • "What is ninja?"
  • "What is the swift-macosx-x86_64 directory?"
  • "I used swift/utils/build-script --xcode in order to generate an Xcode project; attempting to invoke ninja results in an error. Why?"

Contributors familiar with the interaction between swift/utils/build-script and CMake know that swift/utils/build-script --xcode eventually calls cmake -G Xcode when configuring the apple/swift project, and so it stands to reason that invoking ninja on the configured project would not work. A slight improvement would be to recommend newcomers use cmake --build instead.

One last trick: "In-tree" builds of apple/swift

Configuring and building apple/swift using direct invocations of cmake gave me a new appreciation of the work done by swift/utils/build-script and swift/utils/build-script-impl. But it also made me notice that configuring and building Clang did not require a cmake invocation of its own.

The recommended way to build the Clang project is slightly different than the way I built apple/swift with CMake. Whereas building apple/swift required me to build LLVM first, Clang is built as part of the LLVM project. LLVM's CMake automatically detects when Clang is present at llvm/tools/clang. If it is, then LLVM's CMake includes Clang in the build.

It turns out that apple/swift can be built in this way as well:

$10+ patron-only content

A demonstration of an "in-tree" build of apple/swift.

I am a patronBecome a patron

Compared to the "standalone" build I first tried, in-tree builds of apple/swift don't require me to remember to recompile apple/swift-llvm. If I make any changes to the apple/swift-llvm source code, they will be recompiled the next time I invoke cmake --build.

However, in-tree builds are not the official documented way to build apple/swift; if I encounter an error related to the build system, it's on me to file a bug and maybe even fix it myself.

For newcomers to apple/swift, I would recommend either using swift/utils/build-script, or the standalone setup.

Summary

$10+ patron-only content

A summary of the topics explored in this post.

I am a patronBecome a patron

Next Sunday morning, I'll post a "deep dive" into apple/swift's CMake code: how it uses a "recursive make" pattern, examples of how I read the code to see what it does, and some summaries of how things like the compiler itself, the standard library, and the test suite are built.

Like this post, portions of next Sunday's post will only be available to patrons. You can sign up to support me on Patreon here!