Getting Started with Swift Compiler Development
Over the next few months, I'll be writing a guide to each section of the Swift compiler source code: swift/lib/Driver, swift/lib/Frontend, swift/lib/Parse, and so on. First, here's how I set up a workspace for building and making patches to the Swift compiler.
Being able to modify, experiment with, and attach a debugger to my own copy of the Swift compiler has been crucial to my learning process. If you want to get started contributing, try following along with the setup below.
Step 0: Minimum disk space and hardware
The apple/swift README recommends at least 20GB of free disk space for build products. In addition, a powerful CPU will reduce the amount of time it takes to build the apple/swift project. I use a MacBook Pro.
No matter the CPU, I always spend the majority of my time reading the code. Chances are your machine at home is perfectly capable of building apple/swift. If not, use a $20/month Digital Ocean "droplet" running Ubuntu 16.04. It can compile apple/swift in a few hours.
Step 1: Clone apple/swift
The instructions in the apple/swift README have everything anyone would ever want to know. Based on that, I went on the command line and entered the following three commands:
# 1. Create the directory where apple/swift # and its accompanying repositories will # live. mkdir ~/local/Source/apple # 2. Clone apple/swift within the directory # created in step (1). git clone \ https://github.com/apple/swift.git \ ~/local/Source/apple/swift # 3. Run a script inside the apple/swift # repository that automatically downloads # the other repositories you'll need. ~/local/Source/apple/swift/utils/update-checkout --clone
Step 2: Build and invoke my own copy of the Swift compiler
In order to build a copy of the Swift compiler that I can attach a debugger to, I run the following command:
~/local/Source/apple/swift/utils/build-script \ --release \ --debug-swift
If the command fails, I double- and triple-check that I have all the prerequisites listed in the apple/swift README installed. The most common cause of failure for me, on macOS, is because I don't have the latest beta version of Xcode installed.
The utils/build-script executable builds the libraries and executables in the apple/swift project. Passing it the combination of the --release and --debug-swift options makes it build the Swift compiler with debugging symbols. Debugging symbols take up a lot of extra hard drive space, and the build takes a lot longer with them included, but having them allows me to attach the lldb debugger to the Swift compiler to learn what it's doing. That's invaluable to me.
The build script will produce a ton of build products, including a Swift compiler executable, swiftc. The location of this executable depends on the options passed to utils/build-script. Having used the options above, I can invoke my swiftc executable at the following path:
~/local/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swiftc --version
The command above should output version information such as the following:
Swift version 4.1-dev (LLVM 260a9efcdb, Clang 6af0830132, Swift 539894568f) Target: x86_64-apple-darwin16.7.0
And of course, I can use my built swiftc to compile and run Swift source code. For example, I created a new file named hello.swift and typed the following:
hello.swift
let greeting: String? = "Hello, Swift!" print(greeting)
I can compile and run this file using my compiler, like so:
# 1. Compiles the Swift source code to produce an # executable named 'hello'. # (This also outputs some warnings -- more on # these in the next section.) ~/local/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swiftc \ hello.swift # 2. Executes the program. This prints # 'Optional("Hello, Swift!")' to the console. ./hello
Step 3: Modify the Swift compiler
Compiling our program with swiftc hello.swift above produced an executable, but the compiler also output some warnings:
hello.swift:2:9: warning: string interpolation produces a debug description for an optional value; did you mean to make this explicit? print("\(greeting)") ^~~~~~~~~~ hello.swift:2:10: note: use 'String(describing:)' to silence this warning print("\(greeting)") ~^~~~~~~~~ String(describing: ) hello.swift:2:10: note: provide a default value to avoid this warning print("\(greeting)") ~^~~~~~~~~ ?? <#default value#>
Just for fun, instead of suggesting the programmer provide a default value with ?? <#default value#>, I'll modify the compiler so that it suggests using a force unwrap ! instead.
To do so, I'll open swift/include/swift/AST/DiagnosticsSema.def and swift/lib/Sema/MiscDiagnostics.cpp with a text editor and make the following changes:
~/local/Source/apple/swift/include/swift/AST/DiagnosticsSema.def
2732 NOTE(default_optional_to_any,none, 2733 "provide a default value to avoid this warning", ())++++ NOTE(force_optional_yolo,none,++++ "force unwrap the value, because why not? YOLO!!", ())2734 NOTE(force_optional_to_any,none, 2735 "force-unwrap the value to avoid this warning", ())
~/local/Source/apple/swift/lib/Sema/MiscDiagnostics.cpp
3381 // Suggest inserting a default value.3382 TC.diagnose(segment->getLoc(), diag::default_optional_to_any)++++ TC.diagnose(segment->getLoc(), diag::force_optional_yolo)3383 .highlight(segment->getSourceRange())3384 .fixItInsert(segment->getEndLoc(), " ?? <#default value#>");++++ .fixItInsertAfter(segment->getEndLoc(), "!");
Now, if I rebuild apple/swift and compile the source code again, I'll see the new warnings I added:
# 1. Rebuild apple/swift, by invoking build-script again. ~/local/Source/apple/swift/utils/build-script \ --release \ --debug-swift # 2. Use the freshly build 'swiftc' to compile the source code # in hello.swift again. ~/local/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swiftc \ hello.swift
This should produce a new set of warnings:
hello.swift:2:9: warning: string interpolation produces a debug description for an optional value; did you mean to make this explicit? print("\(greeting)") ^~~~~~~~~~ hello.swift:2:10: note: use 'String(describing:)' to silence this warning print("\(greeting)") ~^~~~~~~~~ String(describing: ) hello.swift:2:10: note: force unwrap the value, because why not? YOLO!! print("\(greeting)") ~^~~~~~~~~ !
If you've been following along with me, then congratulations! You've just made your first change to the Swift compiler. A change like this one could be sent to apple/swift as a pull request. If accepted, it would be shipped to each and every Swift user in the next Apple release.
You could also distribute your modified Swift compiler as an Xcode toolchain. I'll write more about that, and about the apple/swift build system in general, in my next post.
If you found this post useful, and stay tuned for my next post, on the apple/swift build system: a deep dive into its CMake, shellscript, and Python guts.
Bonus step 4: Debug the Swift compiler
I build apple/swift with debugging symbols so that I can attach the lldb debugger to it. As an example, here's how I'd debug the command I executed above: first, I'll use the -### option in order to have the Swift compiler to print out its sub-commands:
~/local/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swiftc \ hello.swift \ -###
This prints the following two sub-commands:
/Users/modocache/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swift \ -frontend \ -c \ -primary-file /Users/modocache/local/Source/tmp/hello.swift \ -target x86_64-apple-darwin16.7.0 \ -enable-objc-interop \ -color-diagnostics \ -module-name hello \ -o /var/folders/ry/2ryfdsb56b30092626qprw6d3rb3ss/T/hello-f994f3.o /usr/bin/ld ... -o hello
- The first sub-command is an invocation of the Swift compiler with arguments to produce an object file.
- The second sub-command is an invocation of the system linker. This invocation takes the object file produced by the first sub-command, then generates an executable from that file.
I'm interested in debugging the Swift compiler, so I attach lldb to the first sub-command, by copy-pasting the command from the -### output above, but prefixing it with lldb --, like so:
lldb -- /Users/modocache/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swift \ -frontend \ -c \ -primary-file /Users/modocache/local/Source/tmp/hello.swift \ -target x86_64-apple-darwin16.7.0 \ -enable-objc-interop \ -color-diagnostics \ -module-name hello \ -o /var/folders/ry/2ryfdsb56b30092626qprw6d3rb3ss/T/hello-f994f3.o
On macOS environments with Xcode installed,
lldbshould be available on the command line as/usr/bin/lldb. If it's not, double-check that you have the latest version of Xcode installed. If you're using Ubuntu Linux, you should be able to install lldb from here.
The above command attaches lldb to the process and displays a prompt where I can enter commands:
(lldb)
I commonly want to see how the Swift compiler determines whether to print a warning. For example, here's how I would do that for the string optional warning I modified above: first, I set the breakpoint on that file and line number:
(lldb) b MiscDiagnostics.cpp:3377 Breakpoint 1: where = swift`diagnoseUnintendedOptionalBehavior(swift::TypeChecker&, swift::Expr const*, swift::DeclContext const*)::UnintendedOptionalBehaviorWalker::walkToExprPre(swift::Expr*) + 2931 at MiscDiagnostics.cpp:3378, address = 0x0000000101fce333
Then, I run the Swift compiler. It runs until it hits the breakpoint:
(lldb) run Process 57027 launched: '/Users/bgesiak/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swift' (x86_64) hello.swift:2:9: warning: string interpolation produces a debug description for an optional value; did you mean to make this explicit? print("\(greeting)") ^~~~~~~~~~ hello.swift:2:10: note: use 'String(describing:)' to silence this warning print("\(greeting)") ~^~~~~~~~~ String(describing: ) Process 57027 stopped * thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1 frame #0: 0x0000000101fce333 swift`diagnoseUnintendedOptionalBehavior(this=0x00007fff5fbf40b0, E=0x00000001178d4a18)::UnintendedOptionalBehaviorWalker::walkToExprPre(swift::Expr*) at MiscDiagnostics.cpp:3378 3375 .fixItInsert(segment->getEndLoc(), ")"); 3376 3377 // Suggest inserting a default value. -> 3378 TC.diagnose(segment->getLoc(), diag::force_optional_yolo) 3379 .highlight(segment->getSourceRange()) 3380 .fixItInsertAfter(segment->getEndLoc(), "!"); 3381 } Target 0: (swift) stopped. (lldb)
The debugger is now stopped just before the point where the Swift compiler would output the warning I modified. To see how it got here, I can use lldb's bt command, which stands for "backtrace":
(lldb) bt thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1 * frame #0: 0x0000000101fce333 swift`diagnoseUnintendedOptionalBehavior(this=0x00007fff5fbf40b0, E=0x00000001178d4a18)::UnintendedOptionalBehaviorWalker::walkToExprPre(swift::Expr*) at MiscDiagnostics.cpp:3378 frame #1: 0x00000001027ee90a swift`(anonymous namespace)::Traversal::doIt(this=0x00007fff5fbf4078, E=0x00000001178d4a18) at ASTWalker.cpp:1032 frame #2: 0x00000001027f3ec0 swift`(anonymous namespace)::Traversal::visitImplicitConversionExpr(this=0x00007fff5fbf4078, E=0x0000000117201890) at ASTWalker.cpp:552 frame #3: 0x00000001027f17b8 swift`swift::ASTVisitor<(anonymous namespace)::Traversal, swift::Expr*, swift::Stmt*, bool, swift::Pattern*, bool, void>::visitErasureExpr(this=0x00007fff5fbf4078, E=0x0000000117201890) at ExprNodes.def:139 frame #4: 0x00000001027ef788 swift`swift::ASTVisitor<(anonymous namespace)::Traversal, swift::Expr*, swift::Stmt*, bool, swift::Pattern*, bool, void>::visit(this=0x00007fff5fbf4078, E=0x0000000117201890) at ExprNodes.def:139 frame #5: 0x00000001027ef036 swift`(anonymous namespace)::Traversal::visit(this=0x00007fff5fbf4078, E=0x0000000117201890) at ASTWalker.cpp:98 frame #6: 0x00000001027ee940 swift`(anonymous namespace)::Traversal::doIt(this=0x00007fff5fbf4078, E=0x0000000117201890) at ASTWalker.cpp:1037 frame #7: 0x00000001027f2d00 swift`(anonymous namespace)::Traversal::visitIdentityExpr(this=0x00007fff5fbf4078, E=0x00000001172018c0) at ASTWalker.cpp:461 frame #8: 0x00000001027f06e8 swift`swift::ASTVisitor<(anonymous namespace)::Traversal, swift::Expr*, swift::Stmt*, bool, swift::Pattern*, bool, void>::visitParenExpr(this=0x00007fff5fbf4078, E=0x00000001172018c0) at ExprNodes.def:90 frame #9: 0x00000001027ef37c swift`swift::ASTVisitor<(anonymous namespace)::Traversal, swift::Expr*, swift::Stmt*, bool, swift::Pattern*, bool, void>::visit(this=0x00007fff5fbf4078, E=0x00000001172018c0) at ExprNodes.def:90 frame #10: 0x00000001027ef036 swift`(anonymous namespace)::Traversal::visit(this=0x00007fff5fbf4078, E=0x00000001172018c0) at ASTWalker.cpp:98 frame #11: 0x00000001027ee940 swift`(anonymous namespace)::Traversal::doIt(this=0x00007fff5fbf4078, E=0x00000001172018c0) at ASTWalker.cpp:1037 frame #12: 0x00000001027f13fd swift`(anonymous namespace)::Traversal::visitTupleShuffleExpr(this=0x00007fff5fbf4078, E=0x0000000117201908) at ASTWalker.cpp:592 frame #13: 0x00000001027ef6c4 swift`swift::ASTVisitor<(anonymous namespace)::Traversal, swift::Expr*, swift::Stmt*, bool, swift::Pattern*, bool, void>::visit(this=0x00007fff5fbf4078, E=0x0000000117201908) at ExprNodes.def:132 frame #14: 0x00000001027ef036 swift`(anonymous namespace)::Traversal::visit(this=0x00007fff5fbf4078, E=0x0000000117201908) at ASTWalker.cpp:98 frame #15: 0x00000001027ee940 swift`(anonymous namespace)::Traversal::doIt(this=0x00007fff5fbf4078, E=0x0000000117201908) at ASTWalker.cpp:1037 frame #16: 0x00000001027f3ab1 swift`(anonymous namespace)::Traversal::visitApplyExpr(this=0x00007fff5fbf4078, E=0x00000001178d4aa0) at ASTWalker.cpp:713 frame #17: 0x00000001027f12a8 swift`swift::ASTVisitor<(anonymous namespace)::Traversal, swift::Expr*, swift::Stmt*, bool, swift::Pattern*, bool, void>::visitCallExpr(this=0x00007fff5fbf4078, E=0x00000001178d4aa0) at ExprNodes.def:121 frame #18: 0x00000001027ef600 swift`swift::ASTVisitor<(anonymous namespace)::Traversal, swift::Expr*, swift::Stmt*, bool, swift::Pattern*, bool, void>::visit(this=0x00007fff5fbf4078, E=0x00000001178d4aa0) at ExprNodes.def:121 frame #19: 0x00000001027ef036 swift`(anonymous namespace)::Traversal::visit(this=0x00007fff5fbf4078, E=0x00000001178d4aa0) at ASTWalker.cpp:98 frame #20: 0x00000001027ee940 swift`(anonymous namespace)::Traversal::doIt(this=0x00007fff5fbf4078, E=0x00000001178d4aa0) at ASTWalker.cpp:1037 frame #21: 0x00000001027ee8a0 swift`swift::Expr::walk(this=0x00000001178d4aa0, walker=0x00007fff5fbf40b0) at ASTWalker.cpp:1664 frame #22: 0x0000000101fa8db0 swift`diagnoseUnintendedOptionalBehavior(TC=0x00007fff5fbf6730, E=0x00000001178d4aa0, DC=0x00000001178d48d0) at MiscDiagnostics.cpp:3391 frame #23: 0x0000000101fa8a3b swift`swift::performSyntacticExprDiagnostics(TC=0x00007fff5fbf6730, E=0x00000001178d4aa0, DC=0x00000001178d48d0, isExprStmt=true) at MiscDiagnostics.cpp:3406 frame #24: 0x0000000102039c85 swift`swift::TypeChecker::typeCheckExpression(this=0x00007fff5fbf6730, expr=0x00007fff5fbf6320, dc=0x00000001178d48d0, convertType=TypeLoc @ 0x00007fff5fbf4390, convertTypePurpose=CTP_Unused, options=(Storage = 33), listener=0x0000000000000000, baseCS=0x0000000000000000) at TypeCheckConstraints.cpp:1818 frame #25: 0x00000001021d4855 swift`(anonymous namespace)::StmtChecker::visitBraceStmt(this=0x00007fff5fbf6460, BS=0x00000001178d4ad0) at TypeCheckStmt.cpp:1234 frame #26: 0x00000001021d438c swift`swift::ASTVisitor<(anonymous namespace)::StmtChecker, void, swift::Stmt*, void, void, void, void>::visit(this=0x00007fff5fbf6460, S=0x00000001178d4ad0) at StmtNodes.def:43 frame #27: 0x00000001021d250a swift`bool (anonymous namespace)::StmtChecker::typeCheckStmt<swift::BraceStmt>(this=0x00007fff5fbf6460, S=0x00007fff5fbf6448) at TypeCheckStmt.cpp:372 frame #28: 0x00000001021d2448 swift`swift::TypeChecker::typeCheckTopLevelCodeDecl(this=0x00007fff5fbf6730, TLCD=0x00000001178d48a0) at TypeCheckStmt.cpp:1620 frame #29: 0x000000010221d299 swift`swift::performTypeChecking(SF=0x0000000117003c00, TLC=0x00007fff5fbf7180, Options=(Storage = 0), StartElem=2, WarnLongFunctionBodies=0, WarnLongExpressionTypeChecking=0, ExpressionTimeoutThreshold=0) at TypeChecker.cpp:720 frame #30: 0x00000001019329b5 swift`swift::CompilerInstance::performSema(this=0x0000000114818600) at Frontend.cpp:525 frame #31: 0x00000001000e20de swift`performCompile(Instance=0x0000000114818600, Invocation=0x00007fff5fbfd0a0, Args=ArrayRef<const char *> @ 0x00007fff5fbf9398, ReturnValue=0x00007fff5fbfbec4, observer=0x0000000000000000, Stats=0x0000000000000000) at FrontendTool.cpp:571 frame #32: 0x00000001000de812 swift`swift::performFrontend(Args=ArrayRef<const char *> @ 0x00007fff5fbfc1d0, Argv0="/Users/bgesiak/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swift", MainAddr=0x000000010000a260, observer=0x0000000000000000) at FrontendTool.cpp:1381 frame #33: 0x000000010000b573 swift`main(argc_=13, argv_=0x00007fff5fbff870) at driver.cpp:160 frame #34: 0x00007fffcd3e8235 libdyld.dylib`start + 1
Wow, that's a deep backtrace! Skimming through it from the bottom up, I can see that the Swift compiler starts in main, calls performFrontend, then performCompile, and so on, until it begins type checking, and eventually it invokes diagnoseUnintendedOptionalBehavior, which is the function that outputs the warning in question.
The wall of text from the backtrace is intimidating at first, but no worries: in upcoming posts, I'll explain what each portion of the Swift compiler does. In the end, the Swift compiler is just a command-line program; there's nothing magical or scary about it.
Tweet