modocache.io

How the Swift Compiler Emits Diagnostics, Part 2: Swift's Wrappers of LLVM Abstractions

2018-06-01T18:00:00+00:00

In the last two articles in this series on Swift compiler development, I wrote about the abstractions provided by LLVM to emit diagnostics. How the Swift Compiler Emits Diagnostics, Part 1 explained how the llvm::SourceMgr class prints diagnostic messages along with lines of source code in an llvm::MemoryBuffer. Then I wrote about the internals of that llvm::MemoryBuffer class in How Swift and Clang Use LLVM to Read Files into Memory.

In this article, I explain how these abstractions come together in the Swift compiler. Specifically, I'll cover:

How the Swift frontend reads source files in as llvm::MemoryBuffer instances.
How the Swift frontend registers those buffers with a swift::SourceManager, a libswiftBasic wrapper around llvm::SourceMgr.
How the Swift lexer records swift::SourceLoc locations as attributes on the tokens it lexes. (swift::SourceLoc is defined in libswiftBasic, as a wrapper around llvm::SMLoc.)
How the Swift parser instructs swift::DiagnosticEngine and swift::SourceManager to print a diagnostic in the event it has encountered a parse error.

It's these mechanisms above that result in the printing of the parser error I wrote about in the last article:

uhoh.swift

1  // uhoh.swift
2  
3  print(("Yikes.")

swiftc uhoh.swift
uhoh.swift:3:17: error: expected ')' in expression list
print(("Yikes.")
                ^
uhoh.swift:3:6: note: to match this opening '('
print(("Yikes.")
     ^

Finally, this article will write about how the Swift compiler defines its diagnostics strings – for example, the "expected ')' in expression list" string above – by using macros. This is similar to the way token kinds are defined, as described in my article Getting Started with the Swift Frontend: Lexing & Parsing.

Step 1: Creating an `llvm::MemoryBuffer` for each input source file

My article An Introduction to the Swift Compiler Driver explained how an invocation of swift is normally split up, by libswiftDriver, into a series of swift -frontend and ld linker invocations. And in Getting Started with the Swift Frontend: Lexing & Parsing, I explained how an invocation of swift -frontend results in libswiftFrontendTool and libswiftFrontend instantiating a swift::CompilerInvocation, parsing command-line arguments via the swift::CompilerInvocation::parseArgs member function, and then using those parsed arguments to instantiate a swift::CompilerInstance. To recap, here's that code again:

swift/lib/FrontendTool/FrontendTool.cpp

1304  int swift::performFrontend(ArrayRef<const char *> Args,
1305                             const char *Argv0, void *MainAddr,
1306                             FrontendObserver *observer) {
....  
1348    std::unique_ptr<CompilerInstance> Instance =
1349      llvm::make_unique<CompilerInstance>();
....  
1371    CompilerInvocation Invocation;
....  
1379    // Parse arguments.
1380    if (Invocation.parseArgs(Args, Instance->getDiags(), workingDirectory)) {
1381      return finishDiagProcessing(1);
1382    }
....  
1464    if (Instance->setup(Invocation)) {
1465      return finishDiagProcessing(1);
1466    }
....  
1542  }

In Getting Started with the Swift Frontend: Lexing & Parsing, I pointed out that the swift::CompilerInstance::setup function instantiated a swift::ASTContext object (a crucially important object that represents the syntax tree of the Swift program being compiled). But it's also in this function that source files are read into llvm::MemoryBuffer objects. The setup function calls through to swift::CompilerInstance::setUpInputs, which calls setUpForInput, and so on, until eventually getInputBufferAndModuleDocBufferIfPresent instantiates an llvm::MemoryBuffer via the llvm::MemoryBuffer::getFileOrSTDIN function (which I explained in detail in the last article):

swift/lib/Frontend/Frontend.cpp

135  bool CompilerInstance::setup(const CompilerInvocation &Invok) {
136    Invocation = Invok;
...  
164    return setUpInputs();
165  }
...  
238  bool CompilerInstance::setUpInputs() {
...  
243    for (const InputFile &input :
244         Invocation.getFrontendOptions().InputsAndOutputs.getAllInputs())
245      if (setUpForInput(input))
246        return true;
...  
260  }
261  
262  bool CompilerInstance::setUpForInput(const InputFile &input) {
263    bool failed = false;
264    Optional<unsigned> bufferID = getRecordedBufferID(input, failed);
265    if (failed)
266      return true;
...  
280    return false;
281  }
282  
283  Optional<unsigned> CompilerInstance::getRecordedBufferID(const InputFile &input,
284                                                           bool &failed) {
...  
291    std::pair<std::unique_ptr<llvm::MemoryBuffer>,
292              std::unique_ptr<llvm::MemoryBuffer>>
293        buffers = getInputBufferAndModuleDocBufferIfPresent(input);
...  
308    // Transfer ownership of the MemoryBuffer to the SourceMgr.
309    unsigned bufferID = SourceMgr.addNewSourceBuffer(std::move(buffers.first));
...  
313  }
314  
315  std::pair<std::unique_ptr<llvm::MemoryBuffer>,
316            std::unique_ptr<llvm::MemoryBuffer>>
317  CompilerInstance::getInputBufferAndModuleDocBufferIfPresent(
318      const InputFile &input) {
...  
326    using FileOrError = llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>>;
327    FileOrError inputFileOrErr = llvm::MemoryBuffer::getFileOrSTDIN(input.file());
328    if (!inputFileOrErr) {
329      Diagnostics.diagnose(SourceLoc(), diag::error_open_input_file, input.file(),
330                           inputFileOrErr.getError().message());
331      return std::make_pair(nullptr, nullptr);
332    }
...  
342  }

Notice above that, after calling getInputBufferAndModuleDocBufferIfPresent to instantiate an llvm::MemoryBuffer, the swift::CompilerInstance::getRecordedBufferID function adds the buffer to the source manager.

Step 2: Registering the `llvm::MemoryBuffer` with the `swift::SourceManager`

The Swift codebase defines a library named libswiftBasic. It's the Swift codebase's equivalent of libLLVMSupport: a grab-bag of helper classes and functions without much in common.

Included in this helper library are the Swift classes swift::SourceManager and swift::SourceLoc, which wrap llvm::SourceMgr and llvm::SMLoc. They even define a lot of the same member functions, and in their implementations simply forward these along to the internal LLVM class – for example, swift::SourceManager::addNewSourceBuffer simply calls through to llvm::SourceMgr::AddNewSourceBuffer:

swift/include/swift/Basic/SourceManager.h

 22  namespace swift {
 23  
 24  /// \brief This class manages and owns source buffers.
 25  class SourceManager {
 26    llvm::SourceMgr LLVMSourceMgr;
 ..  
103    /// Adds a memory buffer to the SourceManager, taking ownership of it.
104    unsigned addNewSourceBuffer(std::unique_ptr<llvm::MemoryBuffer> Buffer);
...  
234  };
235  
236  } // end namespace swift

swift/lib/Basic/SourceLoc.cpp

42  unsigned
43  SourceManager::addNewSourceBuffer(std::unique_ptr<llvm::MemoryBuffer> Buffer) {
..  
46    auto ID = LLVMSourceMgr.AddNewSourceBuffer(std::move(Buffer), llvm::SMLoc());
..  
49  }

libswiftBasic wraps these LLVM abstractions in order to expose only some of their functions to the rest of the Swift codebase, or to add convenience methods. For example swift::SourceLoc is documented as being defined mainly to deter other parts of the Swift codebase from calling llvm::SMLoc::getFromPointer:
swift/include/swift/Basic/SourceLoc.h
28  /// SourceLoc in swift is just an SMLoc.  We define it as a different type
29  /// (instead of as a typedef) just to remove the "getFromPointer" methods and
30  /// enforce purity in the Swift codebase.
31  class SourceLoc {
..  
80  };

So in step one above, after instantiating an llvm::MemoryBuffer in the swift::CompilerInstance::getInputBufferAndModuleDocBufferIfPresent function, the swift::CompilerInstance::getRecordedBufferID function calls swift::SourceManager::addNewSourceBuffer, a trivial wrapper around llvm::SourceMgr::AddNewSourceBuffer.

The swift::SourceManager is stored as a member variable on swift::CompilerInstance, called swift::CompilerInstance::SourceMgr – which is confusing, since that's the same spelling as the llvm::SourceMgr class. But one is an member variable, the other is a class.

swift/lib/Frontend/Frontend.cpp

283  Optional<unsigned> CompilerInstance::getRecordedBufferID(const InputFile &input,
284                                                           bool &failed) {
...  
291    std::pair<std::unique_ptr<llvm::MemoryBuffer>,
292              std::unique_ptr<llvm::MemoryBuffer>>
293        buffers = getInputBufferAndModuleDocBufferIfPresent(input);
...  
308    // Transfer ownership of the MemoryBuffer to the SourceMgr.
309    unsigned bufferID = SourceMgr.addNewSourceBuffer(std::move(buffers.first));
...  
313  }

I explained in How the Swift Compiler Emits Diagnostics, Part 1: LLVM Abstractions how llvm::SourceMgr prints diagnostics at locations in an llvm::MemoryBuffer. The Swift lexer is responsible for recording those locations, by instantiating swift::SourceLoc objects (wrappers for llvm::SMLoc) as it lexes tokens.

Step 3: Recording the locations of lexed tokens

In Getting Started with the Swift Frontend: Lexing & Parsing I wrote about how libswiftFrontend eventually calls swift::parseIntoSourceFile, which instantiates a swift::Parser. The swift::Parser initializer instantiates a new swift::Lexer, passing in the swift::SourceManager:

swift/lib/Parse/Parser.cpp

329  Parser::Parser(unsigned BufferID, SourceFile &SF, SILParserTUStateBase *SIL,
330                 PersistentParserState *PersistentState)
331      : Parser(
332            std::unique_ptr<Lexer>(new Lexer(
333                SF.getASTContext().LangOpts, SF.getASTContext().SourceMgr,
334                BufferID, &SF.getASTContext().Diags,
335                /*InSILMode=*/SIL != nullptr,
336                SF.getASTContext().LangOpts.AttachCommentsToDecls
337                    ? CommentRetentionMode::AttachToNextToken
338                    : CommentRetentionMode::None,
339                SF.shouldKeepSyntaxInfo()
340                    ? TriviaRetentionMode::WithTrivia
341                    : TriviaRetentionMode::WithoutTrivia)),
342            SF, SIL, PersistentState) {}

Besides the swift::SourceManager, the swift::Lexer also takes a swift::DiagnosticEngine as a parameter to its initializer. This class is used to actually print the diagnostics – I'll cover it more in step 4 of this article.

In that article I also wrote about how the swift::Lexer iterates over each character in its input memory buffer in order to form tokens. For example, if it finds the character 'p' in its buffer, it'll attempt to lex an identifier. To do so, it advances its pointer into the buffer as long as it finds characters that form valid identifiers ('0' through '9', 'a' through 'Z', underscores, etc.). Once it can advance no further, it calls swift::Lexer::formToken in order to reset the current swift::Token with the correct source location.

Here's the code that lexes an identifier and forms a token based on a pointer to its text:

swift/lib/Parse/Lexer.cpp

591  /// lexIdentifier - Match [a-zA-Z_][a-zA-Z_$0-9]*
592  void Lexer::lexIdentifier() {
593    const char *TokStart = CurPtr-1;
594    CurPtr = TokStart;
595    bool didStart = advanceIfValidStartOfIdentifier(CurPtr, BufferEnd);
...  
599    // Lex [a-zA-Z_$0-9[[:XID_Continue:]]]*
600    while (advanceIfValidContinuationOfIdentifier(CurPtr, BufferEnd));
601  
602    tok Kind = kindOfIdentifier(StringRef(TokStart, CurPtr-TokStart), InSILMode);
603    return formToken(Kind, TokStart);
604  }

The swift::Token::getLoc member function can then be used to instantiate an llvm::SMLoc and swift::SourceLoc based on the buffer pointer:

swift/include/swift/Parse/Token.h

 34  class Token {
...  
221    /// getLoc - Return a source location identifier for the specified
222    /// offset in the current file.
223    SourceLoc getLoc() const {
224      return SourceLoc(llvm::SMLoc::getFromPointer(Text.begin()));
225    }
...  
280  };

So that explains how the lexer constructs tokens and their locations, how the frontend instantiates memory buffers for each of the input files, and how those memory buffers are owned and managed by a swift::SourceManager. Finally, the parser calls through to llvm::SourceMgr::PrintMessage in order to actually print the diagnostic – just as my sample program did in How the Swift Compiler Emits Diagnostics, Part 1.

But the Swift compiler uses several layers of indirection to do so: swift::DiagnosticEngine, swift::Diagnostic, swift::InFlightDiagnostic, and more. The next and final step in this article writes about those classes in detail.

Step 4: Printing the diagnostic

When the swift::Parser parses invalid Swift source code and determines it must print a diagnostic, it doesn't call llvm::SourceMgr::PrintMessage directly. As an example, let's take a closer look at the note diagnostic from the beginning of this article:

uhoh.swift:3:6: note: to match this opening '('
print(("Yikes.")
     ^

This diagnostic is printed because the parser encounters the opening '(' token, and its Parser::parseMatchingToken function kicks off a loop that tries to parse an expression list and a closing ')' token. If a closing ')' token isn't found, it calls Parser::diagnose, passing in two arguments: the location of the '(' token, and the diagnostic to print:

swift/lib/Parse/Parser.cpp

 858  /// parseMatchingToken - Parse the specified expected token and return its
 859  /// location on success.  On failure, emit the specified error diagnostic, and a
 860  /// note at the specified note location.
 861  bool Parser::parseMatchingToken(tok K, SourceLoc &TokLoc, Diag<> ErrorDiag,
 862                                  SourceLoc OtherLoc) {
 863    Diag<> OtherNote;
 864    switch (K) {
 865    case tok::r_paren:  OtherNote = diag::opening_paren;    break;
 866    case tok::r_square: OtherNote = diag::opening_bracket;  break;
 867    case tok::r_brace:  OtherNote = diag::opening_brace;    break;
 868    default:            llvm_unreachable("unknown matching token!"); break;
 869    }
 870    if (parseToken(K, TokLoc, ErrorDiag)) {
 871      diagnose(OtherLoc, OtherNote);
 872  
 873      TokLoc = PreviousLoc;
 874      return true;
 875    }
 876  
 877    return false;
 878  }

Notice that the note diagnostic text isn't represented with a string, but instead with a type swift::Diag<>, set to the value diag::opening_paren. This value is defined using macros:

swift/include/swift/AST/DiagnosticsParse.def

34  #ifndef NOTE
35  #  define NOTE(ID,Options,Text,Signature) \
36    DIAG(NOTE,ID,Options,Text,Signature)
37  #endif
..  
47  NOTE(opening_paren,none,
48       "to match this opening '('", ())

Another file defines the DIAG macro and then includes this file, in order to define a new global variable, named swift::diag::opening_paren:

swift/include/swift/AST/DiagnosticsParse.def

23  namespace swift {
24    namespace diag {
25    // Declare common diagnostics objects with their appropriate types.
26  #define DIAG(KIND,ID,Options,Text,Signature) \
27    extern detail::DiagWithArguments<void Signature>::type ID;
28  #include "DiagnosticsParse.def"
29    }
30  }

The extern variable opening_paren is an instance of swift::detail::DiagWithArguments<void ()>::type, which is a type alias for swift::Diag<void ()>. The template parameter void () indicates that the diagnostic takes no arguments.

There are other swift::diag kinds that do take arguments, such as operator_static_in_protocol, which takes a StringRef for the operator name:
swift/include/swift/AST/DiagnosticsParse.def
345  ERROR(operator_static_in_protocol,none,
346        "operator '%0' declared in protocol must be 'static'",
347        (StringRef))
348  
In a future article, I'll cover how these arguments are used. For now, I'll focus on opening_paren, which takes no arguments.

As shown above, Parser::parseMatchingToken calls Parser::diagnose with the location of the opening parenthesis token and the diag::opening_paren. This function instantiates a swift::Diagnostic based on the swift::Diag<>, and calls through to swift::DiagnosticEngine::diagnose (recall above that the swift::Parser was instantiated with an instance of swift::DiagnosticEngine, a member of swift::ASTContext):

swift/include/swift/Parse/Parser.h

545    InFlightDiagnostic diagnose(SourceLoc Loc, Diagnostic Diag) {
...  
549      return Diags.diagnose(Loc, Diag);
550    }
...  
556    template<typename ...DiagArgTypes, typename ...ArgTypes>
557    InFlightDiagnostic diagnose(SourceLoc Loc, Diag<DiagArgTypes...> DiagID,
558                                ArgTypes &&...Args) {
559      return diagnose(Loc, Diagnostic(DiagID, std::forward<ArgTypes>(Args)...));
560    }

The Swift compiler defines several abstractions around diagnostics, and it can be hard to keep track of them all. So far I've covered swift::Diag<> and swift::detail::DiagnosticWithArguments<>::type, which represent a single unique diagnostic. The code snippet above introduces swift::Diagnostic, swift::DiagnosticEngine, swift::DiagnosticConsumer, swift::DiagnosticInfo, and swift::InFlightDiagnostic. I'll introduce them each before stepping through the code:

swift::Diagnostic: A diagnostic and all of the arguments it requires. For example, this is used to represent diag::opening_paren and its zero arguments. It's also used to represent diag::operator_static_in_protocol and store its one StringRef argument.
swift::DiagnosticEngine: This is an "announcer" class. It stores a list of swift::DiagnosticConsumer "listeners", which it delegates to in order to print diagnostics text.
swift::DiagnosticInfo: Information about the diagnostic that is passed to swift::DiagnosticConsumer isntances.
swift::InFlightDiagnostic: This class is, confusingly, called a "diagnostic", but it is actually used to represent state in the swift::DiagnosticEngine. The swift::DiagnosticEngine::diagnose function creates an "active" diagnostic, and returns an instance of InFlightDiagnostic. When this instance goes out of scope and its destructor ~InFlightDiagnostic is called, it calls swift::DiagnosticEngine::flushActiveDiagnostic, which in turn notifies each swift::DiagnosticConsumer that it should emit the current active diagnostic.

That's a lot of abstraction, but I found that after I followed the code, things became a little clearer. Recall that the swift::Parser::diagnose function above instantiated a swift::Diagnostic for diag::opening_paren, and then called swift::DiagnosticEngine::diagnose:

swift/include/swift/AST/DiagnosticEngine.h

646      /// \brief Emit an already-constructed diagnostic at the given location.
647      ///
648      /// \param Loc The location to which the diagnostic refers in the source
649      /// code.
650      ///
651      /// \param D The diagnostic.
652      ///
653      /// \returns An in-flight diagnostic, to which additional information can
654      /// be attached.
655      InFlightDiagnostic diagnose(SourceLoc Loc, const Diagnostic &D) {
656        assert(!ActiveDiagnostic && "Already have an active diagnostic");
657        ActiveDiagnostic = D;
658        ActiveDiagnostic->setLoc(Loc);
659        return InFlightDiagnostic(*this);
660      }

Here the swift::DiagnosticEngine sets its ActiveDiagnostic to the swift::Diagnostic that it was given. It then instantiates a swift::InFlightDiagnostic and returns it.

Again, the swift::InFlightDiagnostic being returned here isn't actually a "diagnostic." It doesn't hold any information about what text should be printed out or at which location. Its only purpose is to be returned to the caller – the Parser::parseMatchingToken function – in case that caller wants to attach additional fix-its to the swift::DiagnosticEngine::ActiveDiagnostic.

In this case, the Parser::parseMatchingToken function doesn't store the InFlightDiagnostic at all. So it immediately goes out of scope, and its destructor is called:

swift/lib/Parse/Parser.cpp

861  bool Parser::parseMatchingToken(tok K, SourceLoc &TokLoc, Diag<> ErrorDiag,
862                                  SourceLoc OtherLoc) {
...  
870    if (parseToken(K, TokLoc, ErrorDiag)) {
871      diagnose(OtherLoc, OtherNote);
...  
878  }

swift/include/swift/AST/DiagnosticEngine.h

393      ~InFlightDiagnostic() {
394        if (IsActive)
395          flush();
396      }

The InFlightDiagnostic::flush member function calls through to DiagnosticEngine::flushActiveDiagnostic, which loops over each diagnostic consumer to call DiagnosticConsumer::handleDiagnostic, passing them a reference to the swift::SourceManager, along with a DiagnosticInfo instance:

swift/lib/AST/DiagnosticEngine.cpp

241  void InFlightDiagnostic::flush() {
...  
247      Engine->flushActiveDiagnostic();
248  }
...  
682  void DiagnosticEngine::flushActiveDiagnostic() {
...  
685      emitDiagnostic(*ActiveDiagnostic);
...  
690  }
...  
699  void DiagnosticEngine::emitDiagnostic(const Diagnostic &diagnostic) {
...  
705    SourceLoc loc = diagnostic.getLoc();
...  
818    // Pass the diagnostic off to the consumer.
819    DiagnosticInfo Info;
820    Info.ID = diagnostic.getID();
821    Info.Ranges = diagnostic.getRanges();
822    Info.FixIts = diagnostic.getFixIts();
823    for (auto &Consumer : Consumers) {
824      Consumer->handleDiagnostic(SourceMgr, loc, toDiagnosticKind(behavior),
825                                 diagnosticStringFor(Info.ID),
826                                 diagnostic.getArgs(), Info);
827    }
828  }

The announcer and listener pattern employed by DiagnosticEngine and its DiagnosticConumer instances allow the Swift compiler to handle diagnostics in different ways. For example, when I invoke swiftc -serialize-diagnostics, libswiftFrontendTool registers a swift::SerializedDiagnosticConsumer instance, which writes diagnostics data to a file.

By default, libswiftFrontendTool registers a swift::PrintingDiagnosticConsumer, in the swift::performFrontend function:

swift/lib/FrontendTool/FrontendTool.cpp

1607  int swift::performFrontend(ArrayRef<const char *> Args,
1608                             const char *Argv0, void *MainAddr,
1609                             FrontendObserver *observer) {
....  
1612    PrintingDiagnosticConsumer PDC;
....  
1648    std::unique_ptr<CompilerInstance> Instance =
1649      llvm::make_unique<CompilerInstance>();
1650    Instance->addDiagnosticConsumer(&PDC);
....  
1824  }

When DiagnosticEngine::emitDiagnostic invokes the PrintingDiagnosticConsumer::handleDiagnostic function, the llvm::SourceMgr functions described in How the Swift Compiler Emits Diagnostics, Part 1 are called. llvm::SourceMgr::GetMessage is used to get the text to print, and llvm::SourceMgr::PrimtMessage is used to output that text to the console:

swift/lib/FrontendTool/FrontendTool.cpp

 66  void PrintingDiagnosticConsumer::handleDiagnostic(
 67      SourceManager &SM, SourceLoc Loc, DiagnosticKind Kind,
 68      StringRef FormatString, ArrayRef<DiagnosticArgument> FormatArgs,
 69      const DiagnosticInfo &Info) {
 70    // Determine what kind of diagnostic we're emitting.
 71    llvm::SourceMgr::DiagKind SMKind;
 72    switch (Kind) {
 73      case DiagnosticKind::Error:
 74        SMKind = llvm::SourceMgr::DK_Error;
 75        break;
 76      case DiagnosticKind::Warning: 
 77        SMKind = llvm::SourceMgr::DK_Warning; 
 78        break;
 79  
 80      case DiagnosticKind::Note:
 81        SMKind = llvm::SourceMgr::DK_Note;
 82        break;
 83  
 84      case DiagnosticKind::Remark:
 85        SMKind = llvm::SourceMgr::DK_Remark;
 86        break;
 87    }
 ..  
106    const llvm::SourceMgr &rawSM = SM.getLLVMSourceMgr();
107    
108    // Actually substitute the diagnostic arguments into the diagnostic text.
109    llvm::SmallString<256> Text;
110    {
111      llvm::raw_svector_ostream Out(Text);
112      DiagnosticEngine::formatDiagnosticText(Out, FormatString, FormatArgs);
113    }
114    
115    auto Msg = SM.GetMessage(Loc, SMKind, Text, Ranges, FixIts);
116    rawSM.PrintMessage(out, Msg);
117  }

Summarizing the last three articles

There's a lot of different machinery involved in printing diagnostics, spanning both the Swift and LLVM codebases:

The Swift frontend uses libLLVMOption to parse command-line arguments. The frontend instantiates an llvm::MemoryBuffer for each command-line argument that is determined to be an input file path. An llvm::MemoryBuffer either reads the entire file into memory, or it uses the operating system call mmap to read chunks of it into memory as needed.
The Swift frontend registers each instantiated llvm::MemoryBuffer with a swift::SourceManager. This is a libswiftBasic wrapper around llvm::SourceMgr. libLLVMSupport implements llvm::SourceMgr, which defines logic to print a line of source code from an llvm::MemoryBuffer, along with carets ^ or underlines ~~~~~ at specific locations or ranges, plus some arbitrary text.
The Swift lexer records swift::SourceLoc locations as attributes on the tokens it lexes. swift::SourceLoc is defined in libswiftBasic, as a wrapper around llvm::SMLoc.
The Swift parser is instantiated with a swift::DiagnosticEngine, which in turn holds a reference to the swift::SourceManager that owns the llvm::MemoryBuffer objects. When the Swift parser encounters a series of tokens that don't fit Swift's grammar rules, it instructs its swift::DiagnosticEngine to print a diagnostic.
The swift::DiagnosticEngine uses an announcer & listener pattern to call handleDiagnostic on each of its swift::DiagnosticConsumer instances, passing them a reference to the swift::SourceManager and some information about the diagnostic.
The default diagnostic consumer is swift::PrintingDiagnosticConsumer, which calls llvm::SourceMgr::PrintMessage in order to print the diagnostic text to the console.

How Swift and Clang Use LLVM to Read Files into Memory

2018-05-07T18:00:00+00:00

The prior article in this series explained how the Swift and Clang compilers used llvm::SourceMgr to emit diagnostics for source locations in memory buffers, represented by the class llvm::MemoryBuffer. This article focuses on llvm::MemoryBuffer, the primary abstraction for reading files and streams into memory. Since it's used by Swift, Clang, and LLVM tools like llvm-tblgen, I found it valuable to understand how it works.

Reading a file into memory using C++

The documentation for libLLVMSupport's llvm::MemoryBuffer class says it "provides simple read-only access to a block of memory, and provides simple methods for reading files and standard input into a memory buffer." To better understand how it does that, I tried writing a simple C++ program, called read.cpp, that reads a file – itself, in this case – into memory. For simplicity's sake my program is only meant to operate on Unix systems.

My read.cpp program reads a file into memory by using various system calls. These are requests made to the operating system for things like "open a file and give me its file descriptor," or "read 8 bytes from the file with this file descriptor." Julia Evans has a wonderful comic that explains them further:

My read.cpp program uses four system calls:

open(2) to get a file descriptor for the file.
fstat, which returns information about a file descriptor. Specifically, read.cpp allocates memory based on the file's size.
read(2), which reads a given number of bytes from a file into a pre-allocated block of memory.
close(2) to close a file descriptor once I'm done using it.

Once the read.cpp program allocates memory and reads its own source file into that memory, it increments the char * pointer into the memory and prints out the first line of the file:

read.cpp

 1  #include <cerrno>
 2  #include <iostream>
 3  #include <system_error>
 4  
 5  #include <fcntl.h>
 6  #include <sys/stat.h>
 7  #include <unistd.h>
 8  
 9  int main() {
10    // I'll open this file itself and read it into memory.
11    auto FileName = __FILE__;
12  
13    // The system call open(2) gets a file descriptor
14    // representing the open file.
15    int OpenFlags = O_RDONLY;
16    int FD = open(FileName, OpenFlags);
17  
18    // open(2) returns a -1 if the file could not be opened.
19    // In this case, print an error and return.
20    if (FD < 0) {
21      std::error_code Err(errno, std::generic_category());
22      std::cerr << "[ERROR] Could not open file \""
23                << FileName << "\": " << Err.message()
24                << std::endl;
25      return 1;
26    }
27  
28    // Syscall fstat populates the struct stat pointer
29    // with information about the given file descriptor,
30    // including the file's size in bytes.
31    struct stat Stat;
32    if (fstat(FD, &Stat) < 0) {
33      std::error_code Err(errno, std::generic_category());
34      std::cerr << "[ERROR] Could not acquire information "
35                << "on file descriptor \"" << FD
36                << "\": " << Err.message() << std::endl;
37      return 1;
38    }
39  
40    off_t FileSize = Stat.st_size;
41    std::cout << "[NOTE] File size: " << FileSize << " bytes"
42              << std::endl;
43  
44    // Allocate memory in size equal to the number of bytes
45    // in the file.
46    char *Memory = static_cast<char *>(operator new(
47        FileSize + 1, std::nothrow));
48    Memory[FileSize] = 0;
49  
50    // Use syscall read(2) to read in bytes from the given
51    // file descriptor, into the prepared buffer, 16 bytes
52    // at a time.
53    const ssize_t ChunkSize = 16;
54    ssize_t Offset = 0;
55    ssize_t ReadBytes = 0;
56    do {
57      ReadBytes = read(FD, Memory + Offset, ChunkSize);
58      if (ReadBytes < 0) {
59        std::error_code Err(errno, std::generic_category());
60        std::cerr << "[ERROR] Could not read from file "
61                     "descriptor \""
62                  << FD << "\": " << Err.message()
63                  << std::endl;
64        delete Memory;
65        return 1;
66      }
67      Offset += ReadBytes;
68    } while (ReadBytes != 0);
69  
70    // I've now read the file into memory. To demonstrate:
71    std::cout << "[NOTE] Here's the first line "
72              << "of the file: \"";
73    char *Ptr = Memory;
74    while (*Ptr != '\n' && *Ptr != '\0') {
75      std::cout << *Ptr;
76      ++Ptr;
77    }
78    std::cout << "\"" << std::endl;
79  
80    // Once I'm done with the file, I need to delete the
81    // memory I allocated, otherwise this is a memory leak.
82    delete Memory;
83  
84    // Finally, I need to close the open file descriptor,
85    // using the system call close(2).
86    if (close(FD) < 0) {
87      std::error_code Err(errno, std::generic_category());
88      std::cerr << "[ERROR] Could not close file "
89                << "descriptor \"" << FD << "\":"
90                << Err.message() << std::endl;
91      return 1;
92    }
93  
94    return 0;
95  }

I can compile and run this program like so:

clang++ read.cpp -o my-read-example
./my-read-example
[NOTE] File size: 2820 bytes
[NOTE] Here's the first line of the file: "#include <cerrno>"

This is a good initial implementation of reading a file into memory in C++. In fact, this is very similar to what the llvm::MemoryBuffer::getFile function does. However, there's room for improvement.

Reading a large file into memory using `mmap(2)`

Recall that we allocated memory on the heap using operator new, and then used the syscall read(2) to populate that memory with the contents of our file:

read.cpp

46    char *Memory = static_cast<char *>(operator new(
47        FileSize + 1, std::nothrow));
48    Memory[FileSize] = 0;
..  
56    do {
57      ReadBytes = read(FD, Memory + Offset, ChunkSize);
..  
67      Offset += ReadBytes;
68    } while (ReadBytes != 0);

This allocation would be problematic if we had a huge file to read into memory. A file with a size of 1 gigabyte would result in 1 gigabyte of memory being allocated – that's a lot of RAM!

Thankfully, the syscall mmap(2) allows us to read in bits of the file at a time. Once again, Julia Evans explains it best with another great comic:

I can modify the read.cpp program to use mmap(2) when reading from large files:

read.cpp

  5  #include <fcntl.h>
  +  #include <sys/mman.h>
  7  #include <sys/stat.h>
  8  #include <unistd.h>
  9  
 10  int main() {
 ..  
 ++    // For "large" files over 1024 bytes in size, I'll use
 ++    // syscall mmap(2).
 ++    char *Memory = nullptr;
 ++    bool UseMMap = (FileSize > 1024);
 ++    if (UseMMap) {
 ++      std::cout << "[NOTE] Using mmap" << std::endl;
 ++      int ProtectedOptions = PROT_READ;
 ++      int Flags = MAP_SHARED;
 ++      Memory = static_cast<char *>(mmap(nullptr, FileSize,
 ++                                        ProtectedOptions,
 ++                                        Flags, FD, 0));
 ++      if (Memory == MAP_FAILED) {
 ++        std::error_code Err(errno, std::generic_category());
 ++        std::cerr
 ++            << "[ERROR] Could not mmap file descriptor \""
 ++            << FD << "\": " << Err.message() << std::endl;
 ++      }
 ++    } else {
 ..      // ...use operator new as before.
 89    }
 90  
 91    // I've now read the file into memory.
 ++    // Note that this works exactly as before, we
 ++    // don't have to worry about whether it's an mmap:
 94    std::cout << "[NOTE] Here's the first line "
 95              << "of the file: \"";
 96    char *Ptr = Memory;
 97    while (*Ptr != '\n' && *Ptr != '\0') {
 98      std::cout << *Ptr;
 99      ++Ptr;
100    }
101    std::cout << "\"" << std::endl;
102  
+++    if (UseMMap) {
+++      // Once I'm done with the mmap'ed region, I need to
+++      // release it.
+++      munmap(Memory, FileSize);
+++    } else {
108      // Once I'm done with the file, I need to delete the
109      // memory I allocated, otherwise this is a memory leak.
110      delete Memory;
+++    }
...  
123    return 0;
124  }

Compiling and running this program produces the exact same results as before, with the important distinction that this program can open even very large files, without allocating a ton of memory.

To experiment, you could try adding millions of lines of comments to the bottom of read.cpp. Flip the (FileSize > 1024); conditional to < in order to use operator new, and you'll allocate hundreds of megabytes of memory up front. Then flip it back, to use mmap(2), and you'll allocate almost no memory.

For the most part, llvm::MemoryBuffer works exactly the same way as the read.cpp program above. It has a few extra bells and whistles, too: it works on both Unix and Windows, it uses a more complex hueristic to decide whether to use mmap(2) or not, and it uses some interesting syscalls and options on platforms that support them. I'll explain these as I write about it in detail below.

The LLVM implementation of `read.cpp`: `llvm::MemoryBuffer::getFileOrSTDIN`

Swift and Clang both use the llvm::MemoryBuffer::getFileOrSTDIN static member function to open input file arguments passed to them on the command-line. For example, below is the code in libswiftFrontend converts the string filenames it was passed on the command-line into llvm::MemoryBuffer objects. The filename is a std::string stored as swift::InputFile::file.

swift/lib/Frontend/Frontend.cpp

315  std::pair<std::unique_ptr<llvm::MemoryBuffer>,
316            std::unique_ptr<llvm::MemoryBuffer>>
317  CompilerInstance::getInputBufferAndModuleDocBufferIfPresent(
318      const InputFile &input) {
...  
326    using FileOrError = llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>>;
327    FileOrError inputFileOrErr = llvm::MemoryBuffer::getFileOrSTDIN(input.file());
328    if (!inputFileOrErr) {
329      Diagnostics.diagnose(SourceLoc(), diag::error_open_input_file, input.file(),
330                           inputFileOrErr.getError().message());
331      return std::make_pair(nullptr, nullptr);
332    }
...  
342  }

As I wrote in the previous article, these llvm::MemoryBuffer will then be passed over to the llvm::SourceMgr, which takes ownership of them. The swift::Parser will then interact with llvm::SourceMgr (or more precisely, a wrapper called swift::SourceManager) in order to emit diagnostics at particular locations in the buffer.

The llvm::MemoryBuffer::getFileOrSTDIN function returns either a std::unique_ptr to an llvm::MemoryBuffer for the given file, or an error. This is represented by the llvm::ErrorOr type. (I'll write more about llvm::ErrorOr in the future, but in the meantime you can watch this 5-minute lightning talk from LLVM Developers Meeting 2016 to learn more about them.)

The getFileOrSTDIN function just checks for a file name of "-" and then delegates its logic to either llvm::MemoryBuffer::getSTDIN or getFile. It may optionally be given an int64_t FileSize argument, but if not the default value of -1 signals the function to find out on its own – just as my example read.cpp program above did, by using the fstat system call.

llvm/include/llvm/Support/MemoryBuffer.h

125    /// Open the specified file as a MemoryBuffer, or open stdin if the Filename
126    /// is "-".
127    static ErrorOr<std::unique_ptr<MemoryBuffer>>
128    getFileOrSTDIN(const Twine &Filename, int64_t FileSize = -1,
129                   bool RequiresNullTerminator = true);

llvm/lib/Support/MemoryBuffer.cpp

 143  ErrorOr<std::unique_ptr<MemoryBuffer>>
 144  MemoryBuffer::getFileOrSTDIN(const Twine &Filename, int64_t FileSize,
 145                               bool RequiresNullTerminator) {
 146    SmallString<256> NameBuf;
 147    StringRef NameRef = Filename.toStringRef(NameBuf);
 148  
 149    if (NameRef == "-")
 150      return getSTDIN();
 151    return getFile(Filename, FileSize, RequiresNullTerminator);
 152  }

I'll focus on the getFile case for now, which delegates in turn to a function called getFileAux. The getFileAux static function implements some of the logic I implemented in the read.cpp example above: it opens the file in order to obtain a file descriptor, it reads that file, and then it calls close(2) in order to close the file descriptor:

llvm/include/llvm/Support/MemoryBuffer.h

73    /// Open the specified file as a MemoryBuffer, returning a new MemoryBuffer
74    /// if successful, otherwise returning null. If FileSize is specified, this
75    /// means that the client knows that the file exists and that it has the
76    /// specified size.
77    ///
78    /// \param IsVolatile Set to true to indicate that the contents of the file
79    /// can change outside the user's control, e.g. when libclang tries to parse
80    /// while the user is editing/updating the file or if the file is on an NFS.
81    static ErrorOr<std::unique_ptr<MemoryBuffer>>
82    getFile(const Twine &Filename, int64_t FileSize = -1,
83            bool RequiresNullTerminator = true, bool IsVolatile = false);

llvm/lib/Support/MemoryBuffer.cpp

229  ErrorOr<std::unique_ptr<MemoryBuffer>>
230  MemoryBuffer::getFile(const Twine &Filename, int64_t FileSize,
231                        bool RequiresNullTerminator, bool IsVolatile) {
232    return getFileAux<MemoryBuffer>(Filename, FileSize, FileSize, 0,
233                                    RequiresNullTerminator, IsVolatile);
234  }
...  
242  template <typename MB>
243  static ErrorOr<std::unique_ptr<MB>>
244  getFileAux(const Twine &Filename, int64_t FileSize, uint64_t MapSize,
245             uint64_t Offset, bool RequiresNullTerminator, bool IsVolatile) {
246    int FD;
247    std::error_code EC = sys::fs::openFileForRead(Filename, FD);
248  
249    if (EC)
250      return EC;
251  
252    auto Ret = getOpenFileImpl<MB>(FD, Filename, FileSize, MapSize, Offset,
253                                   RequiresNullTerminator, IsVolatile);
254    close(FD);
255    return Ret;
256  }

Unlike read.cpp, the getFileAux function does not call the open(2) system call directly in order to obtain an open file descriptor for given filename. Instead, it uses the llvm::sys::fs::openFileForRead function. This LLVM helper function, unlike open(2), works on both Windows and Unix platforms.

Per-platform implementations of system calls in LLVM

The llvm::sys::fs::openFileForRead function has a single delcaration, in the header file FileSystem.h:

llvm/include/llvm/Support/FileSystem.h

...  /// @brief Opens the file with the given name in a read-only mode, returning
...  /// its open file descriptor.
...  ///
...  /// @param Name The name of the file to open.
...  /// @param ResultFD The location to store the descriptor for the opened file.
...  /// @param RealPath If nonnull, extra work is done to determine the real path
...  ///                 of the opened file, and that path is stored in this
...  ///                 location.
...  /// @returns errc::success if \a Name has been opened, otherwise a
...  ///          platform-specific error_code.
822  std::error_code openFileForRead(const Twine &Name, int &ResultFD,
823                                  SmallVectorImpl<char> *RealPath = nullptr);

But the LLVM codebase defines two separate implementations of this function: one that's used on Windows platforms, and another that's used on Unix. It accomplishes this using CMake.

I've found that a working knowledge of CMake is a gift that really keeps on giving when it comes to compiler development. If you haven't already, you can read about it more in my articles The Swift Compiler's Build System and Reading and Understanding the CMake in apple/swift.

LLVM's root CMakeLists.txt file appends two directories to its modules path, and then includes one file from each of those directories: llvm/cmake/config-ix.cmake and llvm/cmake/modules/HandleLLVMOptions.cmake. Finally, it configures a header file named config.h.cmake:

llvm/CMakeLists.txt

184  set(CMAKE_MODULE_PATH
185    ${CMAKE_MODULE_PATH}
186    "${CMAKE_CURRENT_SOURCE_DIR}/cmake
187    "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules"
188    )
...  
588  include(config-ix)
...  
602  include(HandleLLVMOptions)
...  
737  configure_file(
738    ${LLVM_MAIN_INCLUDE_DIR}/llvm/Config/config.h.cmake
739    ${LLVM_INCLUDE_DIR}/llvm/Config/config.h)

The config-ix.cmake file uses the built-in CMake function check_symbol_exists in order to determine which system calls are available in the target build environment. For example, it checks whether pread is available and, if it is, has CMake define a variable named HAVE_PREAD:

llvm/cmake/config-ix.cmake

205  check_symbol_exists(pread unistd.h HAVE_PREAD)

Then, in HandleLLVMOptions.cmake, it uses the built-in CMake platform variables, WIN32 and UNIX, to set the CMake variables LLVM_ON_WIN32 and LLVM_ON_UNIX to True or False:

llvm/cmake/modules/HandleLLVMOptions.cmake

108  if(WIN32)
...  
114      set(LLVM_ON_WIN32 1)
115      set(LLVM_ON_UNIX 0)
...  
117  else(WIN32)
118    if(UNIX)
119      set(LLVM_ON_WIN32 0)
120      set(LLVM_ON_UNIX 1)
...  
129  endif(WIN32)

At this point, CMake variables like HAVE_PREAD and LLVM_ON_UNIX would only be visible from within CMake. To make their values visible in C++, the config.h.cmake file is configured via a call to the CMake built-in function configure_file, as shown in a code snippet above. The config.h.cmake file is full of #cmakedefine directives, which configure_file transforms into #define statements for consumption in C++. For example, config.h.cmake contains these #cmakedefine statements…

llvm/include/llvm/Config/config.h.cmake

142  /* Define to 1 if you have the `pread' function. */
143  #cmakedefine HAVE_PREAD ${HAVE_PREAD}
...  
311  /* Define if this is Unixish platform */
312  #cmakedefine LLVM_ON_UNIX ${LLVM_ON_UNIX}
313  
314  /* Define if this is Win32ish platform */
315  #cmakedefine LLVM_ON_WIN32 ${LLVM_ON_WIN32}

…which on a Unix-like platform, such as macOS, are transformed into these statements, placed in a file in the build directory named include/llvm/Config/config.h:

build/include/llvm/Config/config.h

142  /* Define to 1 if you have the `pread' function. */
143  #define HAVE_PREAD 1
...  
311  /* Define if this is Win32ish platform */
312  #define LLVM_ON_UNIX 1

And in llvm/lib/Support/Path.cpp, instead of finding an implementation of the llvm::sys::fs::openFileForRead function, instead there's a condiitonal include based on these definitions:

llvm/lib/Support/Path.cpp

1072  // Include the truly platform-specific parts.
1073  #if defined(LLVM_ON_UNIX)
1074  #include "Unix/Path.inc"
1075  #endif
1076  #if defined(LLVM_ON_WIN32)
1077  #include "Windows/Path.inc"
1078  #endif

It's in the included llvm/lib/Support/Unix/Path.inc file that I can find the actual implementation of llvm::sys::fs::openFileForRead that's used on Unix platforms.

Opening a file on Unix

As in the read.cpp example at the beginning of this article, the Unix implementation of the llvm::sys::fs::openFileForRead function uses the system call open(2) in order to open a file and get its file descriptor:

llvm/lib/Support/Unix/Path.inc

719  std::error_code openFileForRead(const Twine &Name, int &ResultFD,
720                                  SmallVectorImpl<char> *RealPath) {
721    SmallString<128> Storage;
722    StringRef P = Name.toNullTerminatedStringRef(Storage);
723    int OpenFlags = O_RDONLY;
724  #ifdef O_CLOEXEC
725    OpenFlags |= O_CLOEXEC;
726  #endif
727    if ((ResultFD = sys::RetryAfterSignal(-1, open, P.begin(), OpenFlags)) < 0)
728      return std::error_code(errno, std::generic_category());
729  #ifndef O_CLOEXEC
730    int r = fcntl(ResultFD, F_SETFD, FD_CLOEXEC);
731    (void)r;
732    assert(r == 0 && "fcntl(F_SETFD, FD_CLOEXEC) failed");
733  #endif
...  
758    return std::error_code();
759  }

The implementation above is long-winded because of two pieces of Unix trivia.

First off, instead of calling open(2) directly, it calls llvm::sys::RetryAfterSignal, which invokes open(2) in a while loop. This loop retries the open(2) call if it fails with an EINTR error code:

llvm/include/llvm/Support/Errno.h

33  template <typename FailT, typename Fun, typename... Args>
34  inline auto RetryAfterSignal(const FailT &Fail, const Fun &F,
35                               const Args &... As) -> decltype(F(As...)) {
36    decltype(F(As...)) Res;
37    do
38      Res = F(As...);
39    while (Res == Fail && errno == EINTR);
40    return Res;
41  }

I'm not a C++ expert. In case you aren't either, allow me to offer an explanation for the templates being used in the code above.

The RetryAfterSignal function has three template parameters:

const FailT &Fail, representing a value returned when the function call fails.

const Fun &F, representing the callable function.

A template parameter pack const Args &... As, representing the arguments passed to function F.

RetryAfterSignal uses the trailing return type syntax, of the form auto function -> return_type. Its return type is specified as decltype(F(As...)). In other words, the return type is the type returned by the expression F(As...).

To map this all to the concrete example we were looking at in llvm::sys::fs::openFileForRead, recall that function had the expression sys::RetryAfterSignal(-1, open, P.begin(), OpenFlags). Here -1 is the failure value const FailT &Fail, open is the function value const Fun &F, and (P.begin(), OpenFlags) are the template parameter pack arguments passed into the open function. The return type is the type returned by open(P.begin(), OpenFlags), which is int.

The llvm::sys::RetryAfterSignal function ignores the EINTR and retries because "blocking" Unix functions like open(2) and read(2) return EINTR whenever they are interrupted by a Unix signal. Interruptions like this can occur for all sorts of reasons, some of which you can read more about here. In these cases, LLVM will simply try again.

The other quirk in the llvm::sys::fs::openFileForRead implementation is the check for O_CLOEXEC, an open(2) flag that only exists on Linux 2.6.23 and above. This option has the OS automatically close the file descriptor if the process forks. If it's not available, the implementation uses the syscall fcntl in order to set a similar flag.

Reading the file into an `llvm::WritableMemoryBuffer`

The llvm::sys::fs::openFileForRead function opens a file and returns its file descriptor. Then control is returned back to the getFileAux function, which passes the open descriptor into the getOpenFileImpl static function:

llvm/lib/Support/MemoryBuffer.cpp

242  template <typename MB>
243  static ErrorOr<std::unique_ptr<MB>>
244  getFileAux(const Twine &Filename, int64_t FileSize, uint64_t MapSize,
245             uint64_t Offset, bool RequiresNullTerminator, bool IsVolatile) {
246    int FD;
247    std::error_code EC = sys::fs::openFileForRead(Filename, FD);
248  
249    if (EC)
250      return EC;
251  
252    auto Ret = getOpenFileImpl<MB>(FD, Filename, FileSize, MapSize, Offset,
253                                   RequiresNullTerminator, IsVolatile);
254    close(FD);
255    return Ret;
256  }

The getOpenFileImpl implements the same logic the read.cpp example at the beginning of this article did. If the file's size was not provided, it finds out how large the file is by calling llvm::sys::fs::status, which on Unix calls fstat. It then makes a decision as to whether to use mmap(2) or to allocate memory up front using operator new. If it allocates memory, then it uses the system call read(2) (or pread, if HAVE_PREAD is true) in order to read the bytes of the file into memory:

llvm/lib/Support/MemoryBuffer.cpp

416  template <typename MB>
417  static ErrorOr<std::unique_ptr<MB>>
418  getOpenFileImpl(int FD, const Twine &Filename, uint64_t FileSize,
419                  uint64_t MapSize, int64_t Offset, bool RequiresNullTerminator,
420                  bool IsVolatile) {
421    static int PageSize = sys::Process::getPageSize();
422  
423    // Default is to map the full file.
424    if (MapSize == uint64_t(-1)) {
425      // If we don't know the file size, use fstat to find out.  fstat on an open
426      // file descriptor is cheaper than stat on a random path.
427      if (FileSize == uint64_t(-1)) {
428        sys::fs::file_status Status;
429        std::error_code EC = sys::fs::status(FD, Status);
430        if (EC)
431          return EC;
...  
441        FileSize = Status.getSize();
442      }
443      MapSize = FileSize;
444    }
445  
446    if (shouldUseMmap(FD, FileSize, MapSize, Offset, RequiresNullTerminator,
447                      PageSize, IsVolatile)) {
448      std::error_code EC;
449      std::unique_ptr<MB> Result(
450          new (NamedBufferAlloc(Filename)) MemoryBufferMMapFile<MB>(
451              RequiresNullTerminator, FD, MapSize, Offset, EC));
452      if (!EC)
453        return std::move(Result);
454    }
455  
456    auto Buf = WritableMemoryBuffer::getNewUninitMemBuffer(MapSize, Filename);
457    if (!Buf) {
458      // Failed to create a buffer. The only way it can fail is if
459      // new(std::nothrow) returns 0.
460      return make_error_code(errc::not_enough_memory);
461    }
462  
463    char *BufPtr = Buf.get()->getBufferStart();
464  
465    size_t BytesLeft = MapSize;
466  #ifndef HAVE_PREAD
467    if (lseek(FD, Offset, SEEK_SET) == -1)
468      return std::error_code(errno, std::generic_category());
469  #endif
470  
471    while (BytesLeft) {
472  #ifdef HAVE_PREAD
473      ssize_t NumRead = sys::RetryAfterSignal(-1, ::pread, FD, BufPtr, BytesLeft,
474                                              MapSize - BytesLeft + Offset);
475  #else
476      ssize_t NumRead = sys::RetryAfterSignal(-1, ::read, FD, BufPtr, BytesLeft);
477  #endif
478      if (NumRead == -1) {
479        // Error while reading.
480        return std::error_code(errno, std::generic_category());
481      }
482      if (NumRead == 0) {
483        memset(BufPtr, 0, BytesLeft); // zero-initialize rest of the buffer.
484        break;
485      }
486      BytesLeft -= NumRead;
487      BufPtr += NumRead;
488    }
489  
490    return std::move(Buf);
491  }

The functions llvm::sys::Process::getPageSize and llvm::sys::fs::status above use the same CMake tricks as llvm::sys::fs::openFileForRead did in order to include a platform-specific implementation: getPageSize is implemented in llvm/lib/Support/Unix/Process.inc and Windows/Process.inc, and status is implemented in Unix/Path.inc and Windows/Path.inc. On Unix they use system calls getpagesize and fstat in order to get the information they need from the operating system.

The code above instantiates either an llvm::MemoryBufferMMapFile or an llvm::WritableMemoryBuffer based on whether the helper function shouldUseMMap returns true or false. As it was in the read.cpp example at the beginning of this article, one criteria for that decision is the size of the file – for example, if it's smaller than a page on the system, or smaller than 16 kilobytes, then mmap(2) is not used:

llvm/lib/Support/MemoryBuffer.cpp

308  static bool shouldUseMmap(int FD,
309                            size_t FileSize,
310                            size_t MapSize,
311                            off_t Offset,
312                            bool RequiresNullTerminator,
313                            int PageSize,
314                            bool IsVolatile) {
...  
321    // We don't use mmap for small files because this can severely fragment our
322    // address space.
323    if (MapSize < 4 * 4096 || MapSize < (unsigned)PageSize)
324      return false;
...  
360    return true;
361  }

Assuming mmap(2) is not used, then the getOpenFileImpl function calls the static function llvm::WritableMemoryBuffer::getNewUninitMemBuffer. This function allocates the buffer memory just as the read.cpp example did, by using operator new. Unlike the read.cpp example program, however, this function not only allocates memory for a buffer to store the file's contents, it also allocates space for an instance of the llvm::MemoryBuffer class, and for the name of the file:

llvm/lib/Support/MemoryBuffer.cpp

273  std::unique_ptr<WritableMemoryBuffer>
274  WritableMemoryBuffer::getNewUninitMemBuffer(size_t Size, const Twine &BufferName) {
275    using MemBuffer = MemoryBufferMem<WritableMemoryBuffer>;
276    // Allocate space for the MemoryBuffer, the data and the name. It is important
277    // that MemoryBuffer and data are aligned so PointerIntPair works with them.
...  
280    SmallString<256> NameBuf;
281    StringRef NameRef = BufferName.toStringRef(NameBuf);
282    size_t AlignedStringLen = alignTo(sizeof(MemBuffer) + NameRef.size() + 1, 16);
283    size_t RealLen = AlignedStringLen + Size + 1;
284    char *Mem = static_cast<char*>(operator new(RealLen, std::nothrow));
285    if (!Mem)
286      return nullptr;
287  
288    // The name is stored after the class itself.
289    CopyStringRef(Mem + sizeof(MemBuffer), NameRef);
290  
291    // The buffer begins after the name and must be aligned.
292    char *Buf = Mem + AlignedStringLen;
293    Buf[Size] = 0; // Null terminate buffer.
294  
295    auto *Ret = new (Mem) MemBuffer(StringRef(Buf, Size), true);
296    return std::unique_ptr<WritableMemoryBuffer>(Ret);
297  }

Based on the code above, I can see that the memory that's being allocated here is laid out in three distinct segments:

The first segment of memory allocated is sized such that an instance of llvm::MemoryBufferMem<llvm::WritableMemoryBuffer> could fit within it. Note that the size is calculated using sizeof(MemBuffer), and then the memory buffer is instantiated by calling new (Mem) MemBuffer(...). As I mentioned in my article on Getting Started with the Swift Frontend: Lexing & Parsing, this is a "placement" new operator call. It doesn't allocate any memory, and instead calls the MemBuffer constructor, and then places the constructed instance in the chunk of memory Mem. (You can read more about "placement new" here.)
The second segment of memory stores the name of the buffer. It's sized using the call to NameRef.size() above, and then the name is copied by calling the static helper function CopyStringRef.
Finally comes the rest of the buffer, which is the same size as the file being read into it.

The memory buffer allocated and returned by the llvm::WritableMemoryBuffer::getNewUninitMemBuffer function is an llvm::MemoryBufferMem<llvm::WritableMemoryBuffer>. MemoryBufferMem<T> is defined as a subclass of T. In this case, T is an llvm::WritableMemoryBuffer, which in turn derives from llvm::MemoryBuffer. The constructor of MemoryBufferMem calls through to llvm::MemoryBuffer::init:

llvm/lib/Support/MemoryBuffer.cpp

 83  /// MemoryBufferMem - Named MemoryBuffer pointing to a block of memory.
 84  template<typename MB>
 85  class MemoryBufferMem : public MB {
 86  public:
 87    MemoryBufferMem(StringRef InputData, bool RequiresNullTerminator) {
 88      MemoryBuffer::init(InputData.begin(), InputData.end(),
 89                         RequiresNullTerminator);
 90    }
 91  
 92    /// Disable sized deallocation for MemoryBufferMem, because it has
 93    /// tail-allocated data.
 94    void operator delete(void *p) { ::operator delete(p); }
...  
104  };

And the llvm::MemoryBuffer::init function simply sets private members pointing to the beginning and end of the buffer:

llvm/include/llvm/Support/MemoryBuffer.h

  42  class MemoryBuffer {
  43    const char *BufferStart; // Start of the buffer.
  44    const char *BufferEnd;   // End of the buffer.
  ..
 154  };

llvm/lib/Support/MemoryBuffer.cpp

 44  /// init - Initialize this MemoryBuffer as a reference to externally allocated
 45  /// memory, memory that we know is already null terminated.
 46  void MemoryBuffer::init(const char *BufStart, const char *BufEnd,
 47                          bool RequiresNullTerminator) {
 48    assert((!RequiresNullTerminator || BufEnd[0] == 0) &&
 49           "Buffer is not null terminated!");
 50    BufferStart = BufStart;
 51    BufferEnd = BufEnd;
 52  }

In summary, on a Unix system:

The llvm::MemoryBuffer::getFileOrSTDIN static function checks whether its been given a filename of "-" and, if it has, calls llvm::MemoryBuffer::getSTDIN. Otherwise, it calls llvm::MemoryBuffer::getFile.
llvm::MemoryBuffer::getFile calls through to getFileAux.
getFileAux gets an open file descriptor by calling llvm::sys::fs::openFileForRead, then getOpenFileImpl to instantiate a new llvm::MemoryBuffer and read in the contents of the file, and finally`close(2) in order to close the file descriptor.
getOpenFileImpl checks the file size and determines whether to use mmap(2). If mmap(2) is not used, then getOpenFileImpl allocates memory for an llvm::MemoryBuffer, its name, and its contents. It then reads in the contents of the file using read(2) or pread, depending on what's available on the operating system.

Mapping the file into an `llvm::MemoryBufferMMapFile`

Recall that getOpenFileImpl instantiates an llvm::MemoryBufferMMapFile if shouldUseMMap returns true:

llvm/lib/Support/MemoryBuffer.cpp

416  template <typename MB>
417  static ErrorOr<std::unique_ptr<MB>>
418  getOpenFileImpl(int FD, const Twine &Filename, uint64_t FileSize,
419                  uint64_t MapSize, int64_t Offset, bool RequiresNullTerminator,
420                  bool IsVolatile) {
...  
446    if (shouldUseMmap(FD, FileSize, MapSize, Offset, RequiresNullTerminator,
447                      PageSize, IsVolatile)) {
448      std::error_code EC;
449      std::unique_ptr<MB> Result(
450          new (NamedBufferAlloc(Filename)) MemoryBufferMMapFile<MB>(
451              RequiresNullTerminator, FD, MapSize, Offset, EC));
452      if (!EC)
453        return std::move(Result);
454    }
455  
456    auto Buf = WritableMemoryBuffer::getNewUninitMemBuffer(MapSize, Filename);
...  
490    return std::move(Buf);
491  }

The llvm::MemoryBufferMMapFile class makes use of the llvm::sys::fs::mapped_file_region class, a wrapper around the mmap(2) and munmap system calls:

llvm/lib/Support/MemoryBuffer.cpp

166  /// \brief Memory maps a file descriptor using sys::fs::mapped_file_region.
167  ///
168  /// This handles converting the offset into a legal offset on the platform.
169  template<typename MB>
170  class MemoryBufferMMapFile : public MB {
171    sys::fs::mapped_file_region MFR;
...  
185  public:
186    MemoryBufferMMapFile(bool RequiresNullTerminator, int FD, uint64_t Len,
187                         uint64_t Offset, std::error_code &EC)
188        : MFR(FD, MB::Mapmode, getLegalMapSize(Len, Offset),
189              getLegalMapOffset(Offset), EC) {
190      if (!EC) {
191        const char *Start = getStart(Len, Offset);
192        MemoryBuffer::init(Start, Start + Len, RequiresNullTerminator);
193      }
194    }
...  
208  };

The mapped_file_region constructor calls mapped_file_region::init, which calls mmap(2). Its destructor calls munmap:

llvm/lib/Support/Unix/Path.inc

597  std::error_code mapped_file_region::init(int FD, uint64_t Offset,
598                                           mapmode Mode) {
...  
623    Mapping = ::mmap(nullptr, Size, prot, flags, FD, Offset);
624    if (Mapping == MAP_FAILED)
625      return std::error_code(errno, std::generic_category());
626    return std::error_code();
627  }
628  
629  mapped_file_region::mapped_file_region(int fd, mapmode mode, size_t length,
630                                         uint64_t offset, std::error_code &ec)
631      : Size(length), Mapping(), FD(fd), Mode(mode) {
...  
634    ec = init(fd, offset, mode);
635    if (ec)
636      Mapping = nullptr;
637  }
638  
639  mapped_file_region::~mapped_file_region() {
640    if (Mapping)
641      ::munmap(Mapping, Size);
642  }

What I learned

Looking into llvm::MemoryBuffer and how LLVM reads source files into memory taught me a lot:

At build time LLVM's CMake code determines which platform it's being built for. Based on this, it includes Unix- or Windows-specific implementations, such as llvm/lib/Support/Unix/Path.inc or Windows/Path.inc.
Also at build time LLVM CMake determines which system calls are available on the target platform. For example, if pread is available, then getOpenFileImpl will use pread to read the file into an llvm::WritableMemoryBuffer, instead of`read(2).
I can use mmap(2) to access the contents of a very large file without allocating a large amount of memory. LLVM's shouldUseMMap function references the file size, among other characteristics, to determine whether to use pre-allocated memory with llvm::WritableMemoryBuffer, or mmap(2) with llvm::MemoryBufferMMapFile.
llvm::MemoryBuffer maintains a buffer for the contents of a source file as a "trailing object" – a block of memory that is allocated when the class is constructed, but is not a member of the class itself. LLVM uses this trailing object pattern extensively. (It even defines an llvm::TrailingObjects class template, which I plan on writing more about in the future.)

How the Swift Compiler Emits Diagnostics, Part 1: LLVM Abstractions

2018-04-16T18:00:00+00:00

The prior article in this series explained how the Swift compiler lexes and parses source code, by looking at this syntactically valid Swift program:

hello.swift

1  // hello.swift
2
3  print("Hello, world!")

In this and the next few articles, I'd like to take a look at this ill-formed Swift program (note the extra opening parenthesis):

uhoh.swift

1  // uhoh.swift
2
3  print(("Yikes.")

Compiling this program results in two diagnostics: one error diagnostic, and one note diagnostic:

swiftc uhoh.swift
uhoh.swift:3:17: error: expected ')' in expression list
print(("Yikes.")
                ^
uhoh.swift:3:6: note: to match this opening '('
print(("Yikes.")
     ^

As I explained in the previous article, the Swift parser instructs its lexer to lex tokens in the source program. It inspects each token, and determines what tokens need to be lexed next in order for the program to be valid. This is what I imagine the parser is "thinking" when it parses the program above:

"I instruct the lexer to lex the first token in the source code. The lexer returns to me that token, which is print. I know that this is an identifier. I need to keep parsing to determine whether this identifier is being used as a variable, as part of a function call, or in some other way. To find out, I'll instruct the lexer to lex another token."
"The lexer has lexed the token (, an opening parenthesis. I am probably parsing a function call, and as such I expect to find a list of zero or more expressions – the function arguments – followed by a closing parenthesis ). I'll instruct the lexer to lex another token."
"The lexer has lexed the token (, another opening parenthesis. OK, so maybe there's a tuple argument being passed in as a function argument, such as print((true, 1)) or something." (Keep in mind that the parser doesn't know anything about whether the print function provided by the Swift standard library can take a tuple as an argument – that's something for the Swift type-checker, in libswiftSema, to determine.) "So now I need to parse another list of zero or more expressions to fit inside this tuple or whatever, and then find two closing parentheses: one for the print(, and one for the print((. I'll instruct the lexer to lex another token."
"The lexer has lexed the string literal token "Yikes.". OK, I guess this is the first expression inside the tuple. Let's keep going: I instruct the lexer to lex another token."
"The lexer has lexed the token ), a closing parenthesis. Aha, so it looks like I've parsed a tuple: ("Yikes."). OK, so that's the first expression in a list of expressions being passed as arguments to print(. I need to keep parsing expressions in that list of arguments, and then find a closing parenthesis ) to end the list. I instruct the lexer to lex another token."
"The lexer has lexed the token… eof?? End of file? What the heck? Where's the closing parenthesis for print(? OK, this source code is messed up. I should let the user know by emitting a diagnostic."

I explained the mechanics of steps 1 through 5 above in the previous article. In this article, I'll skim over those steps and focus instead on the diagnostic output itself:

swiftc uhoh.swift
uhoh.swift:3:17: error: expected ')' in expression list
print(("Yikes.")
                ^
uhoh.swift:3:6: note: to match this opening '('
print(("Yikes.")
     ^

How does the parser know where to place the ^ carets in the output above? How is the text actually printed? Where is that logic defined?

It turns out the answer to these questions span both the Swift and LLVM codebases.

Representing source text and locations, and emitting diagnostics, using libLLVMSupport

You may recall from my article on Option Parsing in the Swift Compiler that the apple/swift codebase makes use of LLVM libraries. Specifically, that article described how the actual logic of parsing command-line strings was handled in libLLVMOption.

Likewise, the actual logic involved in storing source program text in memory, representing locations in that source text, and printing diagnostic messages, is handled in the LLVM library libLLVMSupport.

Here's a small sample program that demonstrates how libLLVMSupport can be used to emit a diagnostic:

libLLVMSupport-Example.cpp

 1  #include "llvm/Support/MemoryBuffer.h"
 2  #include "llvm/Support/SourceMgr.h"
 3  
 4  int main() {
 5    // This string will represent our source program.
 6    llvm::StringRef Input =
 7      "func foo() {\n"
 8      "  print(\"Hello!\")\n"
 9      "}";
10  
11    // The llvm::MemoryBuffer class is used to store
12    // large strings, along with metadata such as a
13    // buffer or file name. Here we instantiate a
14    // llvm::MemoryBuffer to store the contents of
15    // our source program.
16    std::unique_ptr<llvm::MemoryBuffer> InputBuffer =
17      llvm::MemoryBuffer::getMemBuffer(Input);
18  
19    // The llvm::SourceMgr class is used to emit
20    // diagnostics for one or more llvm::MemoryBuffer
21    // instances. Here we instantiate a new
22    // llvm::SourceMgr and transfer ownership of our
23    // input buffer over to it.
24    llvm::SourceMgr SourceManager;
25    SourceManager.AddNewSourceBuffer(
26        std::move(InputBuffer),
27        /*IncludeLoc*/ llvm::SMLoc());
28  
29    // Here we grab a pointer into the buffer.
30    // Incrementing and decrementing this pointer
31    // allows us to traverse the source program.
32    const llvm::MemoryBuffer *SourceBuffer =
33      SourceManager.getMemoryBuffer(1);
34    const char *CurrentCharacter =
35      SourceBuffer->getBufferStart();
36  
37    // The llvm::SMLoc class is used to represent a
38    // location in an llvm::MemoryBuffer that is managed
39    // by llvm::SourceMgr. We instantiate an llvm::SMLoc
40    // here, for the starting location.
41    llvm::SMLoc BufferStartLocation =
42      llvm::SMLoc::getFromPointer(CurrentCharacter);
43  
44    // The llvm::SourceMgr::PrintMessage function allows
45    // us to print a caret ^ at a specific llvm::SMLoc
46    // location.
47    SourceManager.PrintMessage(
48        BufferStartLocation,
49        llvm::SourceMgr::DiagKind::DK_Remark,
50        "This is the very beginning of the "
51        "source buffer.");
..
96    return 0;
97  }

Running the above program results in the following output:

remark: This is the very beginning of the source buffer.
func foo() {
^

By incrementing the buffer pointer, I can print diagnostics at other locations as well. In the expanded example below, I use llvm::SMRange to print a warning that points to the string "Hello!" in the source program:

libLLVMSupport-Example.cpp

 1  #include "llvm/Support/MemoryBuffer.h"
 2  #include "llvm/Support/SourceMgr.h"
 3  
 4  int main() {
..  
53    // Let's increment our buffer pointer until we find
54    // the first quotation mark character: the first "
55    // in the source text line 'print("Hello!")'. Then
56    // let's record that in an llvm::SMLoc location.
57    while (*CurrentCharacter != '"')
58      ++CurrentCharacter;
59    llvm::SMLoc StartLocation =
60      llvm::SMLoc::getFromPointer(CurrentCharacter);
61  
62    // Next, let's get the llvm::SMLoc location
63    // representing the end of the string "Hello!",
64    // by finding the first character past the last
65    // quotation mark.
66    while (*CurrentCharacter != ')')
67      ++CurrentCharacter;
68    llvm::SMLoc EndLocation =
69      llvm::SMLoc::getFromPointer(CurrentCharacter);
70  
71    // The llvm::SMRange class represents a range: a
72    // beginning and an end llvm::SMLoc location.
73    llvm::SMRange Range = llvm::SMRange(StartLocation,
74                                        EndLocation);
75  
76    // We can print a warning that points to this
77    // llvm::SMRange range.
78    SourceManager.PrintMessage(
79        StartLocation,
80        llvm::SourceMgr::DiagKind::DK_Warning,
81        "This is the range of source text in which "
82        "a string literal appears.",
83        Range);
..
96    return 0;
97  }

Now running the program results in two diagnostics being emitted:

remark: This is the very beginning of the source buffer.
func foo() {
^
warning: This is the range of source text in which a string literal appears.
  print("Hello!")
        ^~~~~~~~

And I can use llvm::SMFixIt in order to display a suggestion for a string to replace "Hello!":

libLLVMSupport-Example.cpp

 1  #include "llvm/Support/MemoryBuffer.h"
 2  #include "llvm/Support/SourceMgr.h"
 3  
 4  int main() {
..
78    SourceManager.PrintMessage(
79        StartLocation,
80        llvm::SourceMgr::DiagKind::DK_Warning,
81        "This is the range of source text in which "
82        "a string literal appears.",
83        Range);
84  
85    // The llvm::SMFixIt class allows us to print
86    // a replacement suggestion underneath the
87    // caret ^ output.
88    SourceManager.PrintMessage(
89        StartLocation,
90        llvm::SourceMgr::DiagKind::DK_Note,
91        "This is a fix-it that suggests an "
92        "alternative string.",
93        llvm::None,
94        llvm::SMFixIt(Range, "\"Good-bye!\""));
95
96    return 0;
97  }

The text associated with the llvm::SMFixIt is displayed below the underline:

remark: This is the very beginning of the source buffer.
func foo() {
^
warning: This is the range of source text in which a string literal appears.
  print("Hello!")
        ^~~~~~~~
note: This is a fix-it that suggests an alternative string.
  print("Hello!")
        ^~~~~~~~
        "Good-bye!"

llvm::SourceMgr also has some rudimentary support for emitting diagnostics for files included by other files. For example, I can mark our source buffer as having been included by another buffer, like so:

libLLVMSupport-Example.cpp

  1  #include "llvm/Support/MemoryBuffer.h"
  2  #include "llvm/Support/SourceMgr.h"
  3  
  4  int main() {
  +   llvm::StringRef IncludeInput = "import foo";
  +   llvm::SourceMgr SourceManager;
  +    SourceManager.AddNewSourceBuffer(
  +        std::move(
  +          llvm::MemoryBuffer::getMemBuffer(IncludeInput)),
 ++        /*IncludeLoc*/ llvm::SMLoc());
 ++    llvm::SMLoc IncludeLocation =
 ++      llvm::SMLoc::getFromPointer(
 ++          SourceManager.getMemoryBuffer(1)->getBufferStart());
 14  
 15    // This string will represent our source program.
 16    llvm::StringRef Input =
 17      "func foo() {\n"
 18      "  print(\"Hello!\")\n"
 19      "}";
 20  
 21    // The llvm::MemoryBuffer class is used to store
 22    // large strings, along with metadata such as a
 23    // buffer or file name. Here we instantiate a
 24    // llvm::MemoryBuffer to store the contents of
 25    // our source program.
 26    std::unique_ptr<llvm::MemoryBuffer> InputBuffer =
 27      llvm::MemoryBuffer::getMemBuffer(Input);
 28  
 29    // The llvm::SourceMgr class is used to emit
 30    // diagnostics for one or more llvm::MemoryBuffer
 31    // instances. Here we instantiate a new
 32    // llvm::SourceMgr and transfer ownership of our
 33    // input buffer over to it.
 --   llvm::SourceMgr SourceManager;
 34    SourceManager.AddNewSourceBuffer(
 35        std::move(InputBuffer),
 --       /*IncludeLoc*/ llvm::SMLoc());
 ++       /*IncludeLoc*/ IncludeLocation);
 37  
 38    // Here we grab a pointer into the buffer.
 39    // Incrementing and decrementing this pointer
 40    // allows us to traverse the source program.
 41    const llvm::MemoryBuffer *SourceBuffer =
 --     SourceManager.getMemoryBuffer(1);
 ++     SourceManager.getMemoryBuffer(2);
 43    const char *CurrentCharacter =
 44      SourceBuffer->getBufferStart();
...
106  }

Now the diagnostics emitted for each llvm::SMLoc location also include a line indicating where those locations were "included" from:

Included from :1:
remark: This is the very beginning of the source buffer.
func foo() {
^
Included from :1:
warning: This is the range of source text in which a string literal appears.
  print("Hello!")
        ^~~~~~~~
Included from :1:
note: This is a fix-it that suggests an alternative string.
  print("Hello!")
        ^~~~~~~~
        "Good-bye!"

Since our source buffers are not named, llvm::SourceMgr just outputs the line on which the "include" occurs – in our case, the first line in the buffer, 1 – and so the output is "Included from :1:".

Assigning identifers to the llvm::MemoryBuffer improves the output:

libLLVMSupport-Example.cpp

  4  int main() {
  5    llvm::StringRef IncludeInput = "import foo";
  6    llvm::SourceMgr SourceManager;
  7    SourceManager.AddNewSourceBuffer(
  8       std::move(
  -         llvm::MemoryBuffer::getMemBuffer(IncludeInput)),
  +         llvm::MemoryBuffer::getMemBuffer(IncludeInput,,
 ++                                          "include.swift")),
 11        /*IncludeLoc*/ llvm::SMLoc());
 ..
 27     std::unique_ptr<llvm::MemoryBuffer> InputBuffer =
 --      llvm::MemoryBuffer::getMemBuffer(Input);
 ++      llvm::MemoryBuffer::getMemBuffer(Input, "foo.swift");
...
107  }

Now the output includes the file names:

Included from include.swift:1:
foo.swift:1:1: remark: This is the very beginning of the source buffer.
func foo() {
^
Included from include.swift:1:
foo.swift:2:9: warning: This is the range of source text in which a string literal appears.
  print("Hello!")
        ^~~~~~~~
Included from include.swift:1:
foo.swift:2:9: note: This is a fix-it that suggests an alternative string.
  print("Hello!")
        ^~~~~~~~
        "Good-bye!"

The diagnostics are pretty much identical to Swift's (and Clang's, the C/C++/Objective-C compiler), because the apple/swift codebase uses the same llvm::MemoryBuffer I used above in order to store source text in memory, and it uses the same llvm::SourceMgr and llvm::SMLoc to emit its diagnostics.

How exactly do `llvm::SourceMgr` and `llvm::SMLoc` work?

llvm::SourceMgr owns a vector of memory buffers – the ones added via calls to llvm::SourceMgr::AddNewSourceBuffer. The SourceMgr.h header file also includes:

A declaration of the llvm::SourceMgr::DiagKind enum I used above to indicate the diagnostic severity.
The definition of a SrcBuffer struct used to wrap both the llvm::MemoryBuffer and its llvm::SMLoc include location.
The implementation of the llvm::SourceMgr::AddNewSourceBuffer member function. Note how simple it is: it just creates a SrcBuffer to wrap the llvm::MemoryBuffer and its include location, and pushes that SrcBuffer onto its vector of buffers.
Implementations of some incredibly simple functions: llvm::SourceMgr::isValidBufferID, for example, simply checks whether the given buffer index is out of bounds of the llvm::SourceMgr::Buffers vector. llvm::SourceMgr::getMainFileID, in particular, makes me chuckle: it just returns 1! (When I first read the apple/swift source code and saw references to a "main buffer ID", I thought it was something more significant than just the first source buffer added to the llvm::SourceMgr.)

llvm::SourceMgr also stores a vector of "include directories," but I'm not quite sure why yet. As far as I can tell, it's only used by one or two internal LLVM tools, like TableGen (remember llvm-tblgen? I wrote about it in this article).

llvm/include/llvm/Support/SourceMgr.h

 40  /// This owns the files read by a parser, handles include stacks,
 41  /// and handles diagnostic wrangling.
 42  class SourceMgr {
 43  public:
 44    enum DiagKind {
 45      DK_Error,
 46      DK_Warning,
 47      DK_Remark,
 48      DK_Note,
 49    };
 ..  
 56  private:
 57    struct SrcBuffer {
 58      /// The memory buffer for the file.
 59      std::unique_ptr<MemoryBuffer> Buffer;
 ..
 85      /// This is the location of the parent include, or null if at the top level.
 86      SMLoc IncludeLoc;
 87  
 88      SrcBuffer() = default;
 ..
 93    };
 94  
 95    /// This is all of the buffers that we are reading from.
 96    std::vector<SrcBuffer> Buffers;
 97  
 98    // This is the list of directories we should search for include files in.
 99    std::vector<std::string> IncludeDirectories;
...  
104    bool isValidBufferID(unsigned i) const { return i && i <= Buffers.size(); }
105  
106  public:
107    SourceMgr() = default;
...
131    const MemoryBuffer *getMemoryBuffer(unsigned i) const {
132      assert(isValidBufferID(i));
133      return Buffers[i - 1].Buffer.get();
134    }
135  
136    unsigned getNumBuffers() const {
137      return Buffers.size();
138    }
139  
140    unsigned getMainFileID() const {
141      assert(getNumBuffers());
142      return 1;
143    }
...
150    /// Add a new source buffer to this source manager. This takes ownership of
151    /// the memory buffer.
152    unsigned AddNewSourceBuffer(std::unique_ptr<MemoryBuffer> F,
153                                SMLoc IncludeLoc) {
154      SrcBuffer NB;
155      NB.Buffer = std::move(F);
156      NB.IncludeLoc = IncludeLoc;
157      Buffers.push_back(std::move(NB));
158      return Buffers.size();
159    }
...
225  };

The llvm::SourceMgr header file also declares the llvm::SourceMgr::PrintMessage member function, which I used above to print diagnostics. This is a convenience function that creates an llvm::SMDiagnostic instance, and then calls the llvm::SMDiagnostic::print function.

llvm/include/llvm/Support/SourceMgr.h

 42  class SourceMgr {
...
186    /// Emit a message about the specified location with the specified string.
187    ///
188    /// \param ShowColors Display colored messages if output is a terminal and
189    /// the default error handler is used.
190    void PrintMessage(raw_ostream &OS, SMLoc Loc, DiagKind Kind,
191                      const Twine &Msg,
192                      ArrayRef<SMRange> Ranges = None,
193                      ArrayRef<SMFixIt> FixIts = None,
194                      bool ShowColors = true) const;
195  
196    /// Emits a diagnostic to llvm::errs().
197    void PrintMessage(SMLoc Loc, DiagKind Kind, const Twine &Msg,
198                      ArrayRef<SMRange> Ranges = None,
199                      ArrayRef<SMFixIt> FixIts = None,
200                      bool ShowColors = true) const;
201  
202    /// Emits a manually-constructed diagnostic to the given output stream.
203    ///
204    /// \param ShowColors Display colored messages if output is a terminal and
205    /// the default error handler is used.
206    void PrintMessage(raw_ostream &OS, const SMDiagnostic &Diagnostic,
207                      bool ShowColors = true) const;
...
225  };

llvm/lib/Support/SourceMgr.cpp

231  void SourceMgr::PrintMessage(raw_ostream &OS, const SMDiagnostic &Diagnostic,
232                               bool ShowColors) const {
...  
239    if (Diagnostic.getLoc().isValid()) {
240      unsigned CurBuf = FindBufferContainingLoc(Diagnostic.getLoc());
241      assert(CurBuf && "Invalid or unspecified location!");
242      PrintIncludeStack(getBufferInfo(CurBuf).IncludeLoc, OS);
243    }
244  
245    Diagnostic.print(nullptr, OS, ShowColors);
246  }
247  
248  void SourceMgr::PrintMessage(raw_ostream &OS, SMLoc Loc,
249                               SourceMgr::DiagKind Kind,
250                               const Twine &Msg, ArrayRef<SMRange> Ranges,
251                               ArrayRef<SMFixIt> FixIts, bool ShowColors) const {
252    PrintMessage(OS, GetMessage(Loc, Kind, Msg, Ranges, FixIts), ShowColors);
253  }
254  
255  void SourceMgr::PrintMessage(SMLoc Loc, SourceMgr::DiagKind Kind,
256                               const Twine &Msg, ArrayRef<SMRange> Ranges,
257                               ArrayRef<SMFixIt> FixIts, bool ShowColors) const {
258    PrintMessage(errs(), Loc, Kind, Msg, Ranges, FixIts, ShowColors);
259  }

The llvm::SMDiagnostic class is also declared in the llvm::SourceMgr header file. It simply stores the data necessary when printing a diagnostic, and it declares the llvm::SMDiagnostic::print member function. This header also declares llvm::SMFixIt, which is also a simple bag of data:

llvm/include/llvm/Support/SourceMgr.h

227  /// Represents a single fixit, a replacement of one range of text with another.
228  class SMFixIt {
229    SMRange Range;
230  
231    std::string Text;
...
256  };
257  
258  /// Instances of this class encapsulate one diagnostic report, allowing
259  /// printing to a raw_ostream as a caret diagnostic.
260  class SMDiagnostic {
261    const SourceMgr *SM = nullptr;
262    SMLoc Loc;
263    std::string Filename;
264    int LineNo = 0;
265    int ColumnNo = 0;
266    SourceMgr::DiagKind Kind = SourceMgr::DK_Error;
267    std::string Message, LineContents;
268    std::vector<std::pair<unsigned, unsigned>> Ranges;
269    SmallVector<SMFixIt, 4> FixIts;
270  
271  public:
...  
278    // Diagnostic with a location.
279    SMDiagnostic(const SourceMgr &sm, SMLoc L, StringRef FN,
280                 int Line, int Col, SourceMgr::DiagKind Kind,
281                 StringRef Msg, StringRef LineStr,
282                 ArrayRef<std::pair<unsigned,unsigned>> Ranges,
283                 ArrayRef<SMFixIt> FixIts = None);
...  
303    void print(const char *ProgName, raw_ostream &S, bool ShowColors = true,
304               bool ShowKindLabel = true) const;
305  };

Note that llvm::SMDiagnostic is initialized with not just the std::string message that will be printed, but also the std::string source line to print as well. The llvm::SourceMgr::GetMessage member function, which is called when llvm::SourceMgr::PrintMessage is used to emit a diagnostic, grabs the source line, called LineStr below, by searching for newline characters in the source buffer.

llvm/lib/Support/SourceMgr.cpp

169  SMDiagnostic SourceMgr::GetMessage(SMLoc Loc, SourceMgr::DiagKind Kind,
170                                     const Twine &Msg,
171                                     ArrayRef<SMRange> Ranges,
172                                     ArrayRef<SMFixIt> FixIts) const {
...
187      // Scan backward to find the start of the line.
188      const char *LineStart = Loc.getPointer();
189      const char *BufStart = CurMB->getBufferStart();
190      while (LineStart != BufStart && LineStart[-1] != '\n' &&
191             LineStart[-1] != '\r')
192        --LineStart;
193  
194      // Get the end of the line.
195      const char *LineEnd = Loc.getPointer();
196      const char *BufEnd = CurMB->getBufferEnd();
197      while (LineEnd != BufEnd && LineEnd[0] != '\n' && LineEnd[0] != '\r')
198        ++LineEnd;
199      LineStr = std::string(LineStart, LineEnd);
...    
226    return SMDiagnostic(*this, Loc, BufferID, LineAndCol.first,
227                        LineAndCol.second-1, Kind, Msg.str(),
228                        LineStr, ColRanges, FixIts);
229  }

The implementation of the llvm::SMDiagnostic::print function is what actually prints textual output (by default to llvm::errs, which is just std::cerr, also known as STDERR). It's a fairly simple function, composed mostly of if and print statements:

llvm/lib/Support/SourceMgr.cpp

368  void SMDiagnostic::print(const char *ProgName, raw_ostream &S, bool ShowColors,
369                           bool ShowKindLabel) const {
...  
376    if (ProgName && ProgName[0])
377      S << ProgName << ": ";
378  
379    if (!Filename.empty()) {
380      if (Filename == "-")
381        S << "<stdin>";
382      else
383        S << Filename;
384  
385      if (LineNo != -1) {
386        S << ':' << LineNo;
387        if (ColumnNo != -1)
388          S << ':' << (ColumnNo+1);
389      }
390      S << ": ";
391    }
392  
393    if (ShowKindLabel) {
394      switch (Kind) {
395      case SourceMgr::DK_Error:
...
398        S << "error: ";
399        break;
400      case SourceMgr::DK_Warning:
...
403        S << "warning: ";
404        break;
405      case SourceMgr::DK_Note:
...
408        S << "note: ";
409        break;
410      case SourceMgr::DK_Remark:
...
413        S << "remark: ";
414        break;
415      }
...
421    }
422
423    S << Message << '\n';
...
442    // Build the line with the caret and ranges.
443    std::string CaretLine(NumColumns+1, ' ');
444    
445    // Expand any ranges.
446    for (unsigned r = 0, e = Ranges.size(); r != e; ++r) {
447      std::pair<unsigned, unsigned> R = Ranges[r];
448      std::fill(&CaretLine[R.first],
449                &CaretLine[std::min((size_t)R.second, CaretLine.size())],
450                '~');
451    }
452
453    // Add any fix-its.
...
455    std::string FixItInsertionLine;
456    buildFixItLine(CaretLine, FixItInsertionLine, FixIts,
457                   makeArrayRef(Loc.getPointer() - ColumnNo,
458                                LineContents.size()));
459
460    // Finally, plop on the caret.
461    if (unsigned(ColumnNo) <= NumColumns)
462      CaretLine[ColumnNo] = '^';
463    else 
464      CaretLine[NumColumns] = '^';
465
466    // ... and remove trailing whitespace so the output doesn't wrap for it.  We
467    // know that the line isn't completely empty because it has the caret in it at
468    // least.
469    CaretLine.erase(CaretLine.find_last_not_of(' ')+1);
470    
471    printSourceLine(S, LineContents);
...
519    S << '\n';
520  }

What I wrote about above represents pretty much everything libLLVMSupport provides in the way of emitting diagnostics. I enjoy reading code like this in libLLVMSupport because it demystifies things that I would otherwise take for granted. The functionality in llvm::SourceMgr and llvm::SMDiagnostic really boils down to storing strings in memory and printing to the console, but for some reason as a compiler user I had always assumed there was something more magical going on behind the scenes. Maybe you felt similarly after reading this article.

I did, however, leave out some details, such as the internals of llvm::MemoryBuffer, and some of the complex caching logic that speeds up functions like llvm::SourceMgr::getLineAndColumn. The former is especially interesting and I hope to write about it more someday – take a look yourself if you have the chance, by reading through llvm/include/llvm/Support/MemoryBuffer.h and llvm/lib/Support/MemoryBuffer.cpp.

In the next article, I'll explain more about how the apple/swift codebase wraps llvm::SourceMgr with the swift::SourceManager class, how swift::CompilerInstance instantiates a llvm::MemoryBuffer for each source file, and about other Swift-specific abstractions, such as swift::DiagnosticEngine.

Getting Started with the Swift Frontend: Lexing & Parsing

2018-02-05T05:00:00+00:00

A previous article in this series explained two primary ways of invoking the swift compiler executable: swift and swift -frontend.

When invoking swift -frontend, the swift executable enters its main entry point and, once it sees the -frontend option, it begins to do everything you and I think of when we think of compilers: it attempts to lex the source file it's given, parse that file into a syntax tree, type-check it, produce an object file, and so on.
When invoking just swift, without the -frontend option, the swift executable splits itself up into child invocations of swift -frontend. The logic that Swift uses to split itself up is in the libswiftDriver library.

Reading and Understanding the Swift Driver Source Code explains the libswiftDriver code that is executed in the second case. This article focuses on the first case: the "compiler-y" parts of the swift compiler executable.

In a nutshell, I aim to answer the question. "what happens when I compile this simple Swift program, hello.swift?"

hello.swift

1  // hello.swift
2
3  print("Hello, world!")

Parsing a Swift source file

First, I'll recap some details covered in previous articles. For example, I've explained that I can compile the hello.swift program on the command line, by invoking swiftc hello.swift. Even this simple invocation of swiftc, because it does not include the -frontend option, is split up into child jobs by the code in libswiftDriver. I can see these child jobs by invoking swiftc hello.swift -driver-print-jobs, which outputs something like the following:

swift -frontend \
    -c hello.swift \
    -o /tmp/hello.o
ld /tmp/hello.o \
    -lSystem -arch x86_64 -macosx_version_min 10.13.0 \
    -L /Users/bgesiak/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/lib/swift/macosx \
    -rpath /Users/bgesiak/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/lib/swift/macosx \
    -o hello

The first job invokes swift -frontend in order to produce an object file named hello.o, and the second job invokes the linker ld in order to link that object file into an executable named hello.

The first invocation appears to be very short, but make no mistake: it executes a lot of code, from a diverse set of libraries. These libraries include libswiftFrontend, libswiftParse, libswiftAST, libswiftSema, libswiftSIL, and more.

Covering each of these libraries in a single article would be exhausting. Instead, this article focuses on the first few phases of swift -frontend -c hello.swift. It'll cover libswiftFrontendTool, libswiftFrontend, and libswiftParse. These three libraries are used, in conjunction with libswiftAST, to build a tree structure that represents the untyped syntax tree of the Swift source file.

I can display the untyped syntax tree by invoking swiftc hello.swift -dump-parse, which outputs the following:

(source_file
  (top_level_code_decl
    (brace_stmt
      (call_expr type='<null>' arg_labels=_:
        (unresolved_decl_ref_expr type='<null>' name=print function_ref=unapplied)
        (paren_expr type='<null>'
          (string_literal_expr type='<null>' encoding=utf8 value="Hello, world!" builtin_initializer=**NULL** initializer=**NULL**))))))

Note the difference between the untyped tree above and the typed syntax tree that swiftc -dump-ast produces:

(source_file
  (top_level_code_decl
    (brace_stmt
      (call_expr type='()' location=hello.swift:3:1 range=[hello.swift:3:1 - line:3:22] nothrow arg_labels=_:
        (declref_expr type='(Any..., String, String) -> ()' location=hello.swift:3:1 range=[hello.swift:3:1 - line:3:1] decl=Swift.(file).print(_:separator:terminator:) function_ref=single)
        (tuple_shuffle_expr implicit type='(Any..., separator: String, terminator: String)' location=hello.swift:3:7 range=[hello.swift:3:6 - line:3:22] scalar_to_tuple elements=[-2, -1, -1] variadic_sources=[0] default_args_owner=Swift.(file).print(_:separator:terminator:)
          (paren_expr type='Any' location=hello.swift:3:7 range=[hello.swift:3:6 - line:3:22]
            (erasure_expr implicit type='Any' location=hello.swift:3:7 range=[hello.swift:3:7 - line:3:7]
              (string_literal_expr type='String' location=hello.swift:3:7 range=[hello.swift:3:7 - line:3:7] encoding=utf8 value="Hello, world!" builtin_initializer=Swift.(file).String.init(_builtinStringLiteral:utf8CodeUnitCount:isASCII:) initializer=**NULL**))))))))

Specifically, the untyped tree features nodes such as unresolved_decl_ref_expr, and many nodes also have a type='<null>' value. These are later filled in with type information as part of the type-checker, which is implemented in libswiftSema. I'll cover libswiftSema in a future article.

The first six stages of the Swift frontend: lexing & parsing

Just how does the swift executable read the text in the hello.swift file to construct an untyped syntax tree? How does it determine that print("Hello, world!") is a call_expr that wraps a paren_expr that wraps a string_literal_expr?

At a high level, the frontend completes parsing in about 6 stages:

The main function in swift/tools/driver/driver.cpp sees that the first argument it's been passed is -frontend, and so it invokes the libswiftFrontendTool library's performFrontend function.
performFrontend parses command-line arguments to determine the FrontendOptions::RequestedAction, by using the CompilerInvocation::parseArgs member function. It then instantiates CompilerInstance and ASTContext objects based on those arguments. Finally, it calls the libswiftFrontendTool function performCompile.
performCompile uses the requested FrontendOptions::RequestedAction to determine whether to call CompilerInstance::performParseOnly or CompilerInstance::performSema. Our invocation of swift -frontend -c results in a requested action of ActionType::EmitObject, and so CompilerInstance::performSema is called.
The CompilerInstance::performSema member function calls through to several other member functions. It opens a bitstream cursor into the standard library module Swift.swiftmodule (it'll use this cursor later, in order to determine the type of the parsed print(...) expression). It adds a SourceFile node to the root of the AST. It also calls the parseIntoSourceFile function, which instantiates a Parser and calls the Parser::parseTopLevel member function. This member function begins the process of lexing and parsing the text in the hello.swift source file.
The Parser works in tandem with a Lexer, which it creates and stores internally as part of the Parser initializer. When the Parser::parseTopLevel member is called, that kicks off an endless loop that:
1. Lexes a "token". A token is a series of characters in the source file that form a cohesive unit. For example, the first token lexed in hello.swift is "print". That's because the lexer first sees a 'p', determines that it must be the start of an identifier (i.e.: an alphabetical character, followed by any number of alphanumeric characters or underscores), and so continues to read in characters until it sees a '(', at which point it stops.
2. The Parser looks at the token that the Lexer has lexed, and determines what to "parse." For example, when the Parser sees that the Lexer has lexed an identifier token of print, it asks the Lexer to lex the next token. Because that next token is a '(', it determines that it must be parsing a call expression. It asks the Lexer to lex the appropriate number of tokens in order to close the print(...) expression. Once it has, the Parser instantiates a new AST node: CallExpr.
The Parser and Lexer continue their endless loop, instantiating new AST nodes, such as CallExpr or StringLiteralExpr, and adding them to the ASTContext. Eventually they reach the end of the source file. Normally at this point libswiftFrontend would continue by kicking off the type-checker for hello.swift. However, that's beyond the scope of this article, so we'll stop here.

The rest of this article steps through the code behind these six steps, explaining them in more detail.

Stage 1: Parsing the `-frontend` argument

As explained in my libswiftDriver article, swift is just a C++ executable. Any invocation of a C++ executable begins in its main function. The swift executable's main function is defined in swift/tools/driver/driver.cpp.

Recall that one of the first things the Swift compiler's main function does is check for the first argument it's given. If that argument is -frontend, it calls the performFrontend function:

swift/tools/driver/driver.cpp

111  int main(int argc_, const char **argv_) {
...
158    StringRef FirstArg(argv[1]);
159    if (FirstArg == "-frontend") {
160      return performFrontend(llvm::makeArrayRef(argv.data()+2,
161                                                argv.data()+argv.size()),
162                             argv[0], (void *)(intptr_t)getExecutablePath);
163    }

Note that -frontend must be the first argument; invoking swift -c hello.swift -frontend is not considered a frontend invocation, and the performFrontend function will not be called. The machinery described in my previous article, Option Parsing in the Swift Compiler, hasn't been initialized yet, and so libswiftOption and libLLVMOption are not used here to check for swift::options::ID::OPT_frontend. Instead, this is a naive string comparison.

Stage 2: Instantiating a `CompilerInstance` based on command-line arguments

The performFrontend function is part of the libswiftFrontendTool library. libswiftFrontendTool is a tiny Swift compiler library whose single purpose is to drive the frontend compilation process.

The build code for libswiftFrontendTool – which determines the files that are included in the library, how it is built, and which libraries it depends on – is all defined in CMake. Check out the swift/lib/FrontendTool/CMakeLists.txt file to learn more.

There's nothing particularly exciting about the libswiftFrontendTool CMake, so I won't cover it here. A previous article, Reading and Understanding the CMake in apple/swift, has tips on how to read the CMake yourself.

The performFrontend function instantiates a CompilerInstance and a CompilerInvocation, and then performs argument parsing:

swift/lib/FrontendTool/FrontendTool.cpp

1304  int swift::performFrontend(ArrayRef<const char *> Args,
1305                             const char *Argv0, void *MainAddr,
1306                             FrontendObserver *observer) {
....  
1348    std::unique_ptr<CompilerInstance> Instance =
1349      llvm::make_unique<CompilerInstance>();
....
1371    CompilerInvocation Invocation;
....
1379    // Parse arguments.
1380    if (Invocation.parseArgs(Args, Instance->getDiags(), workingDirectory)) {
1381      return finishDiagProcessing(1);
1382    }
....  
1542  }

As I mentioned in previous articles, I'll elide source code that isn't relevant to understanding the topic at hand: how the Swift compiler parses Swift source code. Besdies what's shown above, the performFrontend function also calls functions like llvm::InitializeAllTargets. If it didn't, the Swift compiler would not be able to generate object files. But this article isn't about generating object files, so I won't include that code in these snippets.

The CompilerInstance class is arguably one of the most important classes in the Swift codebase. It's defined as part of the libswiftFrontend library. It's significant because it holds unique, owning references of several important singletons. For example, it holds a unique pointer to the ASTContext, which is a singleton that stores all AST nodes that are created by the compiler.

swift/include/swift/Frontend/Frontend.h

304  /// A class which manages the state and execution of the compiler.
305  /// This owns the primary compiler singletons, such as the ASTContext,
306  /// as well as various build products such as the SILModule.
307  ///
308  /// Before a CompilerInstance can be used, it must be configured by
309  /// calling \a setup.  If successful, this will create an ASTContext
310  /// and set up the basic compiler invariants.  Calling \a setup multiple
311  /// times on a single CompilerInstance is not permitted.
312  class CompilerInstance {
313    CompilerInvocation Invocation;
314    SourceManager SourceMgr;
315    DiagnosticEngine Diagnostics{SourceMgr};
316    std::unique_ptr<ASTContext> Context;
317    std::unique_ptr<SILModule> TheSILModule;
...
578  };

The performFrontend function also instantiates a CompilerInvocation object. Not to be confused with CompilerInstance, the CompilerInvocation class is practically a glorified bag of options. It's responsible for parsing command-line arguments to swift -frontend, and storing their values in various "options" classes, like FrontendOptions or LangOptions.

swift/include/swift/Frontend/Frontend.h

 53  /// The abstract configuration of the compiler, including:
 54  ///   - options for all stages of translation,
 55  ///   - information about the build environment,
 56  ///   - information about the job being performed, and
 57  ///   - lists of inputs.
 58  ///
 59  /// A CompilerInvocation can be built from a frontend command line
 60  /// using parseArgs.  It can then be used to build a CompilerInstance,
 61  /// which manages the actual compiler execution.
 62  class CompilerInvocation {
 63    LangOptions LangOpts;
 64    FrontendOptions FrontendOpts;
 65    ClangImporterOptions ClangImporterOpts;
 66    SearchPathOptions SearchPathOpts;
 67    DiagnosticOptions DiagnosticOpts;
 68    MigratorOptions MigratorOpts;
 69    SILOptions SILOpts;
 70    IRGenOptions IRGenOpts;
 ..  
 80  public:
 ..
 83    /// Initializes the compiler invocation for the list of arguments.
 ..
 92    /// \returns true if there was an error, false on success.
 93    bool parseArgs(ArrayRef<const char *> Args, DiagnosticEngine &Diags,
 94                   StringRef workingDirectory = {});
 ..
302  };

A CompilerInvocation object, then, is basically just a collection of parameters that are used to initialize the all-important CompilerInstance. The command-line arguments to swift -frontend are translated into the appropriate settings on a CompilerInvocation object, via its CompilerInvocation::parseArgs member function.

2.1: Parsing frontend command-line arguments

If you've read Option Parsing in the Swift Compiler, the body of the CompilerInvocation::parseArgs member function should look very familiar to you. It parses arguments using the exact same libswiftOption and libLLVMOption abstractions as the Swift driver does: createSwiftOptTable and llvm::opt::OptTable::ParseArgs.

swift/lib/Frontend/CompilerInvocation.cpp

 962  bool CompilerInvocation::parseArgs(ArrayRef<const char *> Args,
 963                                     DiagnosticEngine &Diags,
 964                                     StringRef workingDirectory) {
 ...
 973    std::unique_ptr<llvm::opt::OptTable> Table = createSwiftOptTable();
 974    llvm::opt::InputArgList ParsedArgs =
 975        Table->ParseArgs(Args, MissingIndex, MissingCount, FrontendOption);
 ...  
 990    if (ParseFrontendArgs(FrontendOpts, ParsedArgs, Diags)) {
 991      return true;
 992    }
 993  
 994    if (ParseLangArgs(LangOpts, ParsedArgs, Diags, FrontendOpts)) {
 995      return true;
 996    }
 ...
1030    return false;
1031  }
1032

After converting the command-line strings into an llvm::opt::InputArgList via the llvm::opt::OptTable::ParseArgs member function, the CompilerInvocation::parseArgs member function calls functions like ParseFrontendArgs, ParseLangArgs, and so on, in order to set values on members such as CompilerInvocation::LangOpts.

For example, ParseLangArgs is responsible for translating the swift::options::ID::OPT_swift_version stored on the llvm::opt::InputArgList, and using that to set CompilerInvocation::LangOpts::EffectiveLanguageVersion. If the version passed in is invalid, it emits an error diagnostic:

swift/lib/Frontend/CompilerInvocation.cpp

125  static bool ParseLangArgs(LangOptions &Opts, ArgList &Args,
126                            DiagnosticEngine &Diags,
127                            const FrontendOptions &FrontendOpts) {
...
136    if (auto A = Args.getLastArg(OPT_swift_version)) {
137      auto vers = version::Version::parseVersionString(
138        A->getValue(), SourceLoc(), &Diags);
139      bool isValid = false;
140      if (vers.hasValue()) {
141        if (auto effectiveVers = vers.getValue().getEffectiveLanguageVersion()) {
142          Opts.EffectiveLanguageVersion = effectiveVers.getValue();
143          isValid = true;
144        }
145      }
146      if (!isValid)
147        diagnoseSwiftVersion(vers, A, Args, Diags);
148    }
...  
355  }

To test this out, we can try passing swift -frontend an invalid language version, such as swift -frontend -c hello.swift -swift-version foo. This outputs:
<unknown>:0: error: version component contains non-numeric characters
<unknown>:0: error: invalid value 'foo' in '-swift-version foo'
<unknown>:0: note: valid arguments to '-swift-version' are '3', '4', '5'
If you're curious what other combinations of Swift language versions are valid, you can take a look at the libswiftBasic functions that the ParseLangArgs function uses above: Version::parseVersionString and Version::getEffectiveLanguageVersion.

One of the most important parts of this argument parsing is done in the ParseFrontendArgs function. This calls through to the ArgsToFrontendConverter::determineRequestedAction member function, in order to set the CompilerInvocation object's FrontendOptions::RequestedAction, based on whether the frontend was invoked with -emit-object, -emit-sil, or some other option. This "requested action" will determine what logic the frontend executes:

swift/lib/Frontend/ArgsToFrontendOptionsConverter.cpp

269  FrontendOptions::ActionType
270  ArgsToFrontendOptionsConverter::determineRequestedAction() const {
271    using namespace options;
272    const Arg *A = Args.getLastArg(OPT_modes_Group);
...
283    Option Opt = A->getOption();
284    if (Opt.matches(OPT_emit_object))
285      return FrontendOptions::ActionType::EmitObject;
286    if (Opt.matches(OPT_emit_assembly))
287      return FrontendOptions::ActionType::EmitAssembly;
288    if (Opt.matches(OPT_emit_ir))
289      return FrontendOptions::ActionType::EmitIR;
...
308    if (Opt.matches(OPT_dump_parse))
309      return FrontendOptions::ActionType::DumpParse;
310    if (Opt.matches(OPT_dump_ast))
311      return FrontendOptions::ActionType::DumpAST;
...
330    llvm_unreachable("Unhandled mode option");
331  }

My invocation of swift -frontend -c hello.swift does not appear to include the argument OPT_emit_object. However, a quick peek at Options.td (covered in depth in the article on Option Parsing in the Swift Compiler) reveals that -c is an alias for -emit-object:

swift/include/swift/Option/Options.td

571  def c : Flag<["-"], "c">, Alias<emit_object>,
572    Flags<[FrontendOption, NoInteractiveOption]>, ModeOpt;

2.2: Instantiating the `ASTContext` via the `CompilerInstance::setup` member function

After the arguments have been parsed, the performFrontend function continues by finishing the initialization of the CompilerInstance. To do so, it calls the member function CompilerInstance::setup:

swift/lib/FrontendTool/FrontendTool.cpp

1304  int swift::performFrontend(ArrayRef<const char *> Args,
1305                             const char *Argv0, void *MainAddr,
1306                             FrontendObserver *observer) {
....  
1348    std::unique_ptr<CompilerInstance> Instance =
1349      llvm::make_unique<CompilerInstance>();
....
1371    CompilerInvocation Invocation;
....
1379    // Parse arguments.
1380    if (Invocation.parseArgs(Args, Instance->getDiags(), workingDirectory)) {
1381      return finishDiagProcessing(1);
1382    }
....  
1464    if (Instance->setup(Invocation)) {
1465      return finishDiagProcessing(1);
1466    }
....
1542  }

CompilerInstance::setup allocates a new ASTContext. ASTContext is responsible for creating, allocating memory for, and owning the nodes of the syntax tree.

swift/lib/Frontend/Frontend.cpp

 76  bool CompilerInstance::setup(const CompilerInvocation &Invok) {
 77    Invocation = Invok;
 ..  
 93    Context.reset(new ASTContext(Invocation.getLangOptions(),
 94                                 Invocation.getSearchPathOptions(), SourceMgr,
 95                                 Diagnostics));
 96  
 ..
106  }

There's no overstating the importance of the ASTContext class; if you do any work on the Swift compiler, you'll see this class used everwhere.

swift/include/swift/AST/ASTContext.h

178  /// ASTContext - This object creates and owns the AST objects.
179  /// However, this class does more than just maintain context within an AST.
180  /// It is the closest thing to thread-local or compile-local storage in this
181  /// code base. Why? SourceKit uses this code with multiple threads per Unix
182  /// process. Each thread processes a different source file. Each thread has its
183  /// own instance of ASTContext, and that instance persists for the duration of
184  /// the thread, throughout all phases of the compilation. (The name "ASTContext"
185  /// is a bit of a misnomer here.) Why not use thread-local storage? This code
186  /// may use DispatchQueues and pthread-style TLS won't work with code that uses
187  /// DispatchQueues. Summary: if you think you need a global or static variable,
188  /// you probably need to put it here instead.
189  
190  class ASTContext {
...
924  };

I'll write more on ASTContext below. For now, let's return to performFrontend. Having called CompilerInstance::setup, it calls performCompile.

swift/lib/FrontendTool/FrontendTool.cpp

1304  int swift::performFrontend(ArrayRef<const char *> Args,
1305                             const char *Argv0, void *MainAddr,
1306                             FrontendObserver *observer) {
....  
1348    std::unique_ptr<CompilerInstance> Instance =
1349      llvm::make_unique<CompilerInstance>();
....
1371    CompilerInvocation Invocation;
....
1379    // Parse arguments.
1380    if (Invocation.parseArgs(Args, Instance->getDiags(), workingDirectory)) {
1381      return finishDiagProcessing(1);
1382    }
....  
1464    if (Instance->setup(Invocation)) {
1465      return finishDiagProcessing(1);
1466    }
....
1508    int ReturnValue = 0;
1509    bool HadError =
1510      performCompile(*Instance, Invocation, Args, ReturnValue, observer,
1511                     StatsReporter.get());
....
1542  }

Stage 3: Kicking off libswiftParse (and libswiftSema)

The performCompile function (also defined as part of the libswiftFrontendTool library) looks at the FrontendOptions::RequestedAction (set in stage 2.1 above) and calls either CompilerInstance::performParseOnly or CompilerInstance::performSema.

swift/lib/FrontendTool/FrontendTool.cpp

528  static bool performCompile(CompilerInstance &Instance,
529                             CompilerInvocation &Invocation,
530                             ArrayRef<const char *> Args,
531                             int &ReturnValue,
532                             FrontendObserver *observer,
533                             UnifiedStatsReporter *Stats) {
534    FrontendOptions opts = Invocation.getFrontendOptions();
535    FrontendOptions::ActionType Action = opts.RequestedAction;
...  
608    if (Action == FrontendOptions::ActionType::Parse ||
609        Action == FrontendOptions::ActionType::DumpParse ||
610        Action == FrontendOptions::ActionType::EmitSyntax ||
611        Action == FrontendOptions::ActionType::DumpInterfaceHash ||
612        Action == FrontendOptions::ActionType::EmitImportedModules)
613      Instance.performParseOnly();
614    else
615      Instance.performSema();
...
857  }

As explained in stage 2.1 above, when I invoke swift -frontend -c hello.swift, it results in a request for ActionType::EmitObject, so the CompilerInstance::performSema member function is called.

Stage 4: Opening a `Swift.swiftmodule` bitstream cursor and kicking off the parsing loop

The CompilerInstance::performSema member function loads the Swift standard library and then calls through to CompilerInstance::parseAndCheckTypes:

swift/lib/Frontend/Frontend.cpp

369  void CompilerInstance::performSema() {
...
382      if (!loadStdlib())
383        return;
...  
398    if (MainBufferID != NO_SUCH_BUFFER)
399      addMainFileToModule(implicitImports);
400
401    parseAndCheckTypes(implicitImports);
402  }

The CompilerInstance::loadStdlib member function opens a cursor into the Swift standard library module file. Swift module files are Swift ASTs, serialized into a binary format called an LLVM bitstream. Later, when libswiftSema type-checks the hello.swift file, it will lookup the print function using the cursor created here.

This article only covers parsing, not type-checking. I'll write more about the CompilerInstance::loadStdlib member function, as well as about Swift modules in general, in a future article, which I'll publish before writing about the Swift type-checker.

Next, CompilerInstance::performSema calls CompilerInstance::addFileToMainModule, which calls through to CompilerInstance::createSourceFileForMainModule.

swift/lib/Frontend/Frontend.cpp

744  SourceFile *CompilerInstance::createSourceFileForMainModule(
745      SourceFileKind fileKind, SourceFile::ImplicitModuleImportKind importKind,
746      Optional<unsigned> bufferID) {
...
749    SourceFile *inputFile = new (*Context)
750        SourceFile(*mainModule, fileKind, bufferID, importKind, keepSyntaxInfo);
751    MainModule->addFile(*inputFile);
...  
757    return inputFile;
758  }

This creates the SourceFile node at the root of the AST that is printed out when invoking swiftc -dump-parse hello.swift. The SourceFile class derives from the DeclContext class, which is used in the AST code as a container for arbitrary declarations.

Back in CompilerInstance::performSema, a call to CompilerInstance::parseAndCheckTypes is made, which then calls through to CompilerInstance::parseAndTypeCheckMainFile. This calls the parseIntoSourceFile function multiple times – as many times as that function continues to find Swift code at the top level of the file to parse, until it hits the end of the file.

swift/lib/Frontend/Frontend.cpp

659  void CompilerInstance::parseAndTypeCheckMainFile(
660      PersistentParserState &PersistentState,
661      DelayedParsingCallbacks *DelayedParseCB,
662      OptionSet<TypeCheckingFlags> TypeCheckOptions) {
...
677    bool Done;
678    do {
...
683      parseIntoSourceFile(MainFile, MainFile.getBufferID().getValue(), &Done,
684                          TheSILModule ? &SILContext : nullptr, &PersistentState,
685                          DelayedParseCB);
...
695    } while (!Done);
...
713  }

The parseIntoSourceFile function is defined as part of a tiny compiler library named libswiftParseSIL (the function was recently split out into its own library in order to break circular library dependencies in the compiler). It instantiates a Parser and calls Parser::parseTopLevel. Note that it sets the Done pointer based on whether it's found the eof "token":

swift/lib/ParseSIL/ParseSIL.cpp

101  bool swift::parseIntoSourceFile(SourceFile &SF,
102                                  unsigned BufferID,
103                                  bool *Done,
104                                  SILParserState *SIL,
105                                  PersistentParserState *PersistentState,
106                                  DelayedParsingCallbacks *DelayedParseCB) {
...
108    Parser P(BufferID, SF, SIL ? SIL->Impl.get() : nullptr, PersistentState);
...  
116    bool FoundSideEffects = P.parseTopLevel();
117    *Done = P.Tok.is(tok::eof);
118  
119    return FoundSideEffects;
120  }

4.1: More on the `Lexer` and tokens

The Parser initializer that's used in the parseIntoSourceFile function creates a new Lexer object:

swift/lib/Parse/Parser.cpp

329  Parser::Parser(unsigned BufferID, SourceFile &SF, SILParserTUStateBase *SIL,
330                 PersistentParserState *PersistentState)
331      : Parser(
332            std::unique_ptr<Lexer>(new Lexer(
333                SF.getASTContext().LangOpts, SF.getASTContext().SourceMgr,
334                BufferID, &SF.getASTContext().Diags,
335                /*InSILMode=*/SIL != nullptr,
336                SF.getASTContext().LangOpts.AttachCommentsToDecls
337                    ? CommentRetentionMode::AttachToNextToken
338                    : CommentRetentionMode::None,
339                SF.shouldKeepSyntaxInfo()
340                    ? TriviaRetentionMode::WithTrivia
341                    : TriviaRetentionMode::WithoutTrivia)),
342            SF, SIL, PersistentState) {}

A Lexer is responsible for reading in the individual characters in a source file and forming logical chunks, called tokens.

The Clang compiler, which compiles C, C++, and Objective-C source code, can print the tokens it lexes, using the -dump-tokens frontend option. For example, consider the following simple C program hello.c:

int main() {
  return 0;
}

I can invoke the Clang frontend, clang -cc1, to dump the tokens in this file (Clang has a driver and frontend system that is nearly identical to Swift's, except that Clang takes the argument -cc1 instead of -frontend). clang -cc1 -dump-tokens hello.c outputs the following:

int 'int'             Loc=<hello.c:1:1>  [StartOfLine]
identifier 'main'     Loc=<hello.c:1:5>  [LeadingSpace]
l_paren '('           Loc=<hello.c:1:9>
r_paren ')'           Loc=<hello.c:1:10>
l_brace '{'           Loc=<hello.c:1:12> [LeadingSpace]
return 'return'       Loc=<hello.c:2:3>  [StartOfLine] [LeadingSpace]
numeric_constant '0'  Loc=<hello.c:2:10> [LeadingSpace]
semi ';'              Loc=<hello.c:2:11>
r_brace '}'           Loc=<hello.c:3:1>  [StartOfLine]
eof ''                Loc=<hello.c:3:2>

The Swift compiler executable does not have an option to print the tokens in a file (although if any readers are interested in contributing, this would be a great feature to add!), but if it did, the tokens in hello.swift would be output like this:

identifier 'print'                Loc=<hello.swift:3:1>  [StartOfLine]
l_paren '('                       Loc=<hello.swift:3:6>
string_literal '"Hello, world!"'  Loc=<hello.swift:3:7>
r_paren ')'                       Loc=<hello.swift:3:22>
eof ''                            Loc=<hello.swift:3:23>

Note that the comment at the top of the file, // hello.swift, and the empty line below that comment, are not represented as tokens. The Swift compiler can be invoked such that it creates tokens for comments and whitespace, but normally they are discarded entirely by the compiler.

The first column of the output above displays the token "kind". Kinds of Swift tokens include identifier, l_paren, and eof. You can find a list of all the different Swift token kinds in swift/include/swift/Syntax/TokenKinds.def. An enum of all the different token kinds is defined in swift/include/swift/Syntax/TokenKinds.h, using a trick readers of my option parsing article should be familiar with: it defines the TOKEN macro, and then includes the TokenKinds.def file, which contais a call to TOKEN for each token kind.

swift/include/swift/Syntax/TokenKinds.h

21  enum class tok {
22  #define TOKEN(X) X,
23  #include "swift/Syntax/TokenKinds.def"
24  
25    NUM_TOKENS
26  };

This creates an enum case for, for example, the if keyword. In that case, the enum case is named tok::kw_if:

swift/include/swift/Syntax/TokenKinds.def

 39  /// KEYWORD(kw)
 40  ///   Expands by default for every Swift keyword and every SIL keyword, such as
 41  ///   'if', 'else', 'sil_global', etc. If you only want to use Swift keywords
 42  ///   see SWIFT_KEYWORD.
 43  #ifndef KEYWORD
 44  #define KEYWORD(kw) TOKEN(kw_ ## kw)
 45  #endif
 46  
 47  /// SWIFT_KEYWORD(kw)
 48  ///   Expands for every Swift keyword.
 49  #ifndef SWIFT_KEYWORD
 50  #define SWIFT_KEYWORD(kw) KEYWORD(kw)
 51  #endif
 ..  
 59  /// STMT_KEYWORD(kw)
 60  ///   Expands for every Swift keyword used in statement grammar.
 61  #ifndef STMT_KEYWORD
 62  #define STMT_KEYWORD(kw) SWIFT_KEYWORD(kw)
 63  #endif
...  
169  STMT_KEYWORD(if)

The Token class stores information about a token: its kind and its text. It also defines member functions such as Token::is, so that other parts of the compiler can quickly check "is this token an if keyword?", by invoking Tok.is(tok::kw_if). Or, to check simply that the token is a keyword, the Token::isKeyword member function is implemented using a macro and an include:

swift/include/swift/Parse/Token.h

 33  class Token {
 34    /// Kind - The actual flavor of token this is.
 35    ///
 36    tok Kind;
 51  
 52    /// Text - The actual string covered by the token in the source buffer.
 53    StringRef Text;
 ..  
 61  public:
 ..    
 76    /// is/isNot - Predicates to check if this token is a specific kind, as in
 77    /// "if (Tok.is(tok::l_brace)) {...}".
 78    bool is(tok K) const { return Kind == K; }
...  
214    /// True if the token is any keyword.
215    bool isKeyword() const {
216      switch (Kind) {
217  #define KEYWORD(X) case tok::kw_##X: return true;
218  #include "swift/Syntax/TokenKinds.def"
219      default: return false;
220      }
221    }
...
302  };

4.2: Priming the `Lexer` to form a token for the identifier `"print"`

Normally the Parser prompts the Lexer to lex the next token in a file. The Lexer also, when it's initialized, lexes the first token in the file by calling the Lexer::primeLexer member function.

swift/include/swift/Parse/Lexer.h

 64  class Lexer {
...
169    Lexer(const LangOptions &Options,
170          const SourceManager &SourceMgr, unsigned BufferID,
171          DiagnosticEngine *Diags, bool InSILMode,
172          CommentRetentionMode RetainComments = CommentRetentionMode::None,
173          TriviaRetentionMode TriviaRetention = TriviaRetentionMode::WithoutTrivia)
174        : Lexer(Options, SourceMgr, Diags, BufferID, InSILMode, RetainComments,
175                TriviaRetention) {
176      primeLexer();
177    }
...
523  };

Lexer::primeLexer calls through to Lexer::lexImpl. This member function implements the core of the lexing functionality in the Swift compiler; it is the method that, based on the first character in a series, determines whether to lex an identifier, a number literal, an operator, or some other token.

The Lexer keeps a pointer to the character it's currently lexing using a member Lexer::CurPtr. Lexer::lexImpl moves the pointer to the next character and then enters a switch statement based on the character's value. Here are the switch statement cases that are relevant to my source file hello.swift:

swift/lib/Parse/Lexer.cpp

2041  void Lexer::lexImpl() {
....
2068  Restart:
....
2074    switch ((signed char)*CurPtr++) {
....
2150    case ' ':
2151    case '\t':
2152    case '\f':
2153    case '\v':
2154      goto Restart;  // Skip whitespace.
....  
2193    case '(': return formToken(tok::l_paren, TokStart);
2194    case '}': return formToken(tok::r_brace, TokStart);
2195    case ']': return formToken(tok::r_square, TokStart);
2196    case ')':
2197      return formToken(tok::r_paren, TokStart);
....  
2209    case '/':
2210      if (CurPtr[0] == '/') {  // "//"
2211        skipSlashSlashComment(/*EatNewline=*/true);
....
2215        goto Restart;
2216      }
....
2264    case 'A': case 'B': case 'C': case 'D': case 'E': case 'F': case 'G':
2265    case 'H': case 'I': case 'J': case 'K': case 'L': case 'M': case 'N':
2266    case 'O': case 'P': case 'Q': case 'R': case 'S': case 'T': case 'U':
2267    case 'V': case 'W': case 'X': case 'Y': case 'Z':
2268    case 'a': case 'b': case 'c': case 'd': case 'e': case 'f': case 'g':
2269    case 'h': case 'i': case 'j': case 'k': case 'l': case 'm': case 'n':
2270    case 'o': case 'p': case 'q': case 'r': case 's': case 't': case 'u':
2271    case 'v': case 'w': case 'x': case 'y': case 'z':
2272    case '_':
2273      return lexIdentifier();
....
2282    case '"':
2283    case '\'':
2284      return lexStringLiteral();
....
2289  }

When the Lexer is first initialized in the frontend when parsing hello.swift, the Lexer::primeLexer member function is called, which calls Lexer::lexImpl. This skips the // comments and whitespace at the beginning of hello.swift, until it reaches the 'p' character in "print", and so it enters the case in which the Lexer::lexIdentifier member function is called. This function consumes characters as long as they are valid identifier characters, and then creates a Token with a token kind of either tok::identifier, or a keyword token kind like tok::kw_if.

swift/lib/Parse/Lexer.cpp

591  /// lexIdentifier - Match [a-zA-Z_][a-zA-Z_$0-9]*
592  void Lexer::lexIdentifier() {
593    const char *TokStart = CurPtr-1;
594    CurPtr = TokStart;
595    bool didStart = advanceIfValidStartOfIdentifier(CurPtr, BufferEnd);
...  
599    // Lex [a-zA-Z_$0-9[[:XID_Continue:]]]*
600    while (advanceIfValidContinuationOfIdentifier(CurPtr, BufferEnd));
601  
602    tok Kind = kindOfIdentifier(StringRef(TokStart, CurPtr-TokStart), InSILMode);
603    return formToken(Kind, TokStart);
604  }

In the case of hello.swift, Lexer::lexIdentifier would advance, from 'p', and end at "print", stopping when it saw the '(' character immediately after "print". It would then call Lexer::kindOfIdentifier, to determine whether "print" was a keyword. The Lexer::kindOfIdentifier member function uses, sure enough, the macro and include trick:

swift/lib/Parse/Lexer.cpp

570  tok Lexer::kindOfIdentifier(StringRef Str, bool InSILMode) {
571    tok Kind = llvm::StringSwitch<tok>(Str)
572  #define KEYWORD(kw) \
573      .Case(#kw, tok::kw_##kw)
574  #include "swift/Syntax/TokenKinds.def"
575      .Default(tok::identifier);
...
588    return Kind;
589  }

So, as soon as the Lexer is initialized, the Lexer::primeLexer member function results in the Lexer::NextToken member being set to a token with kind tok::identifier, and with the text "print".

The next time the Parser requests a token from its Lexer, the Lexer will immediately return to it this "print" token and, if it is not at the end of the file, it'll prepare the next Lexer::NextToken in the same way, by calling Lexer::lexImpl again.

Stage 5: The lex & parse loop

To recap: the libswiftFrontendTool function performFrontend parsed the arguments passed into our swift -frontend invocation and determined it would call the CompilerInstance::performSema member function. That eventually called the parseIntoSourceFile function, which instantiated a Parser and its internal Lexer (which resulted in Lexer being "primed" with the "print" token). Finally, parseIntoSourceFile calls Parser::parseTopLevel, which kicks off the lexing and parsing loop that results in the full syntax tree being constructed for our file.

swift/lib/Parse/ParseDecl.cpp

189  bool Parser::parseTopLevel() {
...
194      consumeTokenWithoutFeedingReceiver();
...  
234      parseBraceItems(Items,
235                      allowTopLevelCode() ? BraceItemListKind::TopLevelCode
236                                          : BraceItemListKind::TopLevelLibrary);
...
281  }

The Parser::parseTopLevel member function first requests the next token from the Lexer, by calling consumeTokenWithoutFeedingReceiver. This sets member Parser::Tok to the token for "print", and has the Lexer ready the next token, which is an tok::l_paren '('.

It then calls Parser::parseBraceItems. "Brace items" here refers to a sequence of expressions or statements that are contained with a pair of braces { ... }. Our hello.swift file does not contain braces, but when parsing top-level code in a file the Swift parser pretends that there are a set of braces wrapping that top level code.

Parser::parseBraceItems enters a while loop that continues to parse the file until it reaches the file's end. Within that loop, it creates AST nodes for expressions or statements that it finds, by calling Parser::parseExprOrStmt. If it finds an expression or statement, it wraps it in a BraceStmt node:

swift/lib/Parse/ParseStmt.cpp

226  ParserStatus Parser::parseBraceItems(SmallVectorImpl<ASTNode> &Entries,
227                                       BraceItemListKind Kind,
228                                       BraceItemListKind ConditionalBlockKind) {
...
250    while ((IsTopLevel || Tok.isNot(tok::r_brace)) &&
...
254           Tok.isNot(tok::eof) &&
...
263            !isTerminatorForBraceItemListKind(Kind, Entries))) {
...
345        // If this is a statement or expression at the top level of the module,
346        // Parse it as a child of a TopLevelCodeDecl.
347        auto *TLCD = new (Context) TopLevelCodeDecl(CurDeclContext);
...
357        ParserStatus Status = parseExprOrStmt(Result);
...
373        if (!Result.isNull()) {
374          // NOTE: this is a 'virtual' brace statement which does not have
375          //       explicit '{' or '}', so the start and end locations should be
376          //       the same as those of the result node
377          auto Brace = BraceStmt::create(Context, Result.getStartLoc(),
378                                         Result, Result.getEndLoc());
379          TLCD->setBody(Brace);
...
381        }
...
424    }
...
427  }

With the creation of a TopLevelCodeDecl and a BraceStmt to wrap the result of the Parser::parseExprOrStmt member function call, the AST for hello.swift would look like this:

(source_file
  (top_level_code_decl
    (brace_stmt
      ...)))

The actual expression print("Hello, world!") will be parsed as part of the call to Parser::parseExprOrStmt.

5.1: A quick note on custom allocators in C++

Before getting into the Parser::parseExprOrStmt member function, take a closer look at the instantiation of TopLevelCodeDecl, which uses an interesting C++ feature:

swift/lib/Parse/ParseStmt.cpp

226  ParserStatus Parser::parseBraceItems(SmallVectorImpl<ASTNode> &Entries,
227                                       BraceItemListKind Kind,
228                                       BraceItemListKind ConditionalBlockKind) {
...
347        auto *TLCD = new (Context) TopLevelCodeDecl(CurDeclContext);
...
427  }

I didn't have any experience writing C++ before I began working on Swift and Clang, so this call to new (Context) TopLevelCodeDecl(...) confused me. I was familiar with expressions such as new Foo(), which allocate memory for an instance of Foo. But the expression new (Context) TopLevelCodeDecl(...) seemed to have an extra element: what is (Context) here?

It turns out that C++ allows you to provide overrides of the new operator for specific classes, and those overrides can take additional parameters. The new operator's first argument must be a size_t that indicates how many bytes should be allocated, but beyond that you can define an arbitrary list of parameters. Here, Context is an argument being passed to new.

Swift's Decl class not only defines a custom new operator that takes an ASTContext argument, it also deletes the default new operator implementation:

swift/include/swift/AST/Decl.h

235  /// Decl - Base class for all declarations in Swift.
236  class alignas(1 << DeclAlignInBits) Decl {
237  protected:
865  
866    // Make vanilla new/delete illegal for Decls.
867    void *operator new(size_t Bytes) = delete;
...  
870    // Only allow allocation of Decls using the allocator in ASTContext
871    // or by doing a placement new.
872    void *operator new(size_t Bytes, const ASTContext &C,
873                       unsigned Alignment = alignof(Decl));
...
878  };

This means that you cannot allocate memory for a Decl by calling new Decl() – you must call new (Context) Decl(). Doing so calls the ASTContext::Allocate member function:

swift/lib/AST/Decl.cpp

97  // Only allow allocation of Decls using the allocator in ASTContext.
98  void *Decl::operator new(size_t Bytes, const ASTContext &C,
99                           unsigned Alignment) {
100    return C.Allocate(Bytes, Alignment);
101  }

You may also have noticed this syntax earlier in this article, when CompilerInstance::performSema called through to a function that created the SourceFile root node in the AST. The SourceFile class inherits from DeclContext, which declares its own overload of the new operator:

swift/include/swift/AST/DeclContext.h

186  class alignas(1 << DeclContextAlignInBits) DeclContext {
...
554    // Only allow allocation of DeclContext using the allocator in ASTContext.
555    void *operator new(size_t Bytes, ASTContext &C,
556                       unsigned Alignment = alignof(DeclContext));
...
560  };

This also calls through to the ASTContext::Allocate function:

swift/lib/AST/DeclContext.cpp

38  // Only allow allocation of DeclContext using the allocator in ASTContext.
39  void *DeclContext::operator new(size_t Bytes, ASTContext &C,
40                                  unsigned Alignment) {
41    return C.Allocate(Bytes, Alignment);
42  }

The ASTContext::Allocate member function is too interesting to explain in detail here. I'll write about it in a future article.

5.2: Parsing the `print(...)` expression

To parse the expression or statement at the top level of my hello.swift file, the Parser must first determine whether it's an expression or a statement. It makes use of a helper function Parser::isStartOfStmt to do so. For the most part, this just checks for keywords that clearly demarcate a statement:

swift/lib/Parse/ParseStmt.cpp

37  bool Parser::isStartOfStmt() {
38    switch (Tok.getKind()) {
39    default: return false;
40    case tok::kw_return:
41    case tok::kw_throw:
42    case tok::kw_defer:
43    case tok::kw_if:
44    case tok::kw_guard:
45    case tok::kw_while:
46    case tok::kw_do:
47    case tok::kw_repeat:
48    case tok::kw_for:
49    case tok::kw_break:
50    case tok::kw_continue:
51    case tok::kw_fallthrough:
52    case tok::kw_switch:
53    case tok::kw_case:
54    case tok::kw_default:
55    case tok::pound_if:
56    case tok::pound_sourceLocation:
57      return true;
..
85    }
86  }

Using this function, Parser::parseExprOrStmt determines that print(...) is an expression, and so it calls Parser::parseExpr:

swift/lib/Parse/ParseStmt.cpp

 88  ParserStatus Parser::parseExprOrStmt(ASTNode &Result) {
 ..    
 96    if (isStartOfStmt()) {
 97      ParserResult<Stmt> Res = parseStmt();
 98      if (Res.isNonNull())
 99        Result = Res.get();
100      return Res;
101    }
...  
117    ParserResult<Expr> ResultExpr = parseExpr(diag::expected_expr);
118    if (ResultExpr.isNonNull()) {
119      Result = ResultExpr.get();
120    } else if (!ResultExpr.hasCodeCompletion()) {
...
127    }
...  
133    return ResultExpr;
134  }
135

Parsing an expression in Swift is complicated, because expressions can be composed of sequence of expressions of arbitrary length. For example, a && (b || (c && d)) is an expression itself, but (c && d) and (b || (c && d) are also expressions on their own.

I'll skip over some of the complexity that this introduces in order to showcase just the path that the code takes when parsing our single element expression sequence print("Hello, world!"). Suffice it to say that Parser::parseExpr eventually calls Parser::parseExprPostfixWithoutSuffix, a function that enters a large switch statement on the current token kind: for example, if the current token is a string literal, it calls parseExprStringLiteral.

swift/lib/Parse/ParseExpr.cpp

1463  ParserResult<Expr>
1464  Parser::parseExprPostfixWithoutSuffix(Diag<> ID, bool isExprBasic) {
....
1466    ParserResult<Expr> Result;
1467    switch (Tok.getKind()) {
....
1498    case tok::string_literal:  // "foo"
1499      Result = parseExprStringLiteral();
1500      break;
....        
1562    case tok::identifier:  // foo
....  
1598      LLVM_FALLTHROUGH;
1599    case tok::kw_self:     // self
1600    case tok::kw_Self:     // Self
1601      Result = makeParserResult(parseExprIdentifier());
1602  
1603      // If there is an expr-call-suffix, parse it and form a call.
1604      if (Tok.isFollowingLParen()) {
1605        Result = parseExprCallSuffix(Result, isExprBasic);
....
1607        break;
1608      }
....
1610      break;
.... 
1883    }
....  
1885    return Result;
1886  }

In the case of print(...), the Parser::parseExprPostfixWithoutSuffix function enters the tok::identifier case. In this case, it falls through to the same case as for self or Self, and so it parses the identifier with Parser::parseExprIdentifier. Then, if the next token is a left parentheses '(', it parses a call expression by calling Parser::parseExprCallSuffix.

The Parser::parseExprIdentifier member function attempts to lookup the identifer name in the current scope and, if it can't find a definition, it creates an UnqualifiedDeclRefExpr node in the AST:

swift/lib/Parse/ParseExpr.cpp

2218  Expr *Parser::parseExprIdentifier() {
....
2249    ValueDecl *D = nullptr;
....
2251      D = lookupInScope(name);
....    
2278    Expr *E;
2279    if (D == nullptr) {
....
2286      E = new (Context) UnresolvedDeclRefExpr(name, refKind, loc);
....    }
2307    return E;
2308  }

An UnqualifiedDeclRefExpr is an expression that references an "unqualified" identifier. The identifier might not have been defined yet, or it may never be defined and so this code will fail during type-checking, or the identifier could have been defined in a different module.

In this case, print is defined in a different module: it's defined in Swift.swiftmodule, the Swift standard library module. As a result, any references to print will be treated as an UnqualifiedDeclRefExpr, at least until the type-checker runs and the print identifier is read from the Swift module.

As explained above Parser::parseExprPostfixWithoutSuffix parses the print identifier, then checks for an opening parenthesis '('. If one exists, it calls Parser::parseExprCallSuffix.

Just as with parsing expression sequences, parsing arguments to a call expression is complicated. Swift functions can take an arbitrary number of expressions as arguments, and each of those arguments can itself be a sequence of expressions. I won't go into the details here; the basic idea is that Parser::parseExprList parses a list of expressions that are passed into a call to CallExpr::create:

swift/lib/Parse/ParseExpr.cpp

3223  ParserResult<Expr>
3224  Parser::parseExprCallSuffix(ParserResult<Expr> fn, bool isExprBasic) {  
3227
3256    ParserStatus status = parseExprList(tok::l_paren, tok::r_paren,
3257                                        /*isPostfix=*/true, isExprBasic,
3258                                        lParenLoc, args, argLabels,
3259                                        argLabelLocs,
3260                                        rParenLoc,
3261                                        trailingClosure,
3262                                        SyntaxKind::FunctionCallArgumentList);
3263  
3265    auto Result = makeParserResult(status | fn, 
3266                                   CallExpr::create(Context, fn.get(), lParenLoc,
3267                                                    args, argLabels, argLabelLocs,
3268                                                    rParenLoc, trailingClosure,
3269                                                    /*implicit=*/false));
3278    return Result;
3279  }

In the case of hello.swift, Parser::parseExprList only finds a single expression. It eventually calls through to the string_literal case in the Parser::parseExprPostfixWithoutSuffix function posted above, which results in a call to Parser::parseExprStringLiteral and, in turn, the createStringLiteralExprFromSegment function:

swift/lib/Parse/ParseExpr.cpp

1888  static StringLiteralExpr *
1889  createStringLiteralExprFromSegment(ASTContext &Ctx,
1890                                     const Lexer *L,
1891                                     Lexer::StringSegment &Segment,
1892                                     SourceLoc TokenLoc) {
....
1902    return new (Ctx) StringLiteralExpr(EncodedStr, TokenLoc);
1903  }

The Parser::parseExprList function allocates a ParenExpr. Whatever expresions it parses, such as the StringLiteralExpr above, are nested within the ParenExpr.

All told, the call to Parser::parseExprOrStmt results in the following nodes being added to the AST:

(call_expr type='<null>' arg_labels=_:
  (unresolved_decl_ref_expr type='<null>' name=print function_ref=unapplied)
  (paren_expr type='<null>'
  (string_literal_expr type='<null>' encoding=utf8 value="Hello, world!" builtin_initializer=**NULL** initializer=**NULL**)))

As shown above, the CallExpr result is then nested within the BraceStmt that is created within the Parser::parseBraceItems member function. So, in sum, the AST looks like this:

(source_file
  (top_level_code_decl
    (brace_stmt
      (call_expr type='<null>' arg_labels=_:
        (unresolved_decl_ref_expr type='<null>' name=print function_ref=unapplied)
        (paren_expr type='<null>'
          (string_literal_expr type='<null>' encoding=utf8 value="Hello, world!" builtin_initializer=**NULL** initializer=**NULL**))))))

Stage 6: Reaching the end of the file

When Parser::parseExprList is called, the Parser tasks the Lexer with consuming each token within the parentheses that come after print. The parsing and lexing stops at ')', at which point the Lexer::NextToken is set to tok::eof – the end of the file.

Recall that all this parsing was occuring within a while loop in Parser::parseBraceItems. One of the termination conditions for that while loop was encountering an EOF marker. Now that the end of the file has been reached, Parser::parseBraceItems returns control back to Parser::parseTopLevel, which returns control back to parseIntoSourceFile, which returns back to CompilerInstance::parseAndTypeCheckMainFile. The stage is now set for type-checking, which is done by calling the performTypeChecking function:

swift/lib/Frontend/Frontend.cpp

659  void CompilerInstance::parseAndTypeCheckMainFile(
660      PersistentParserState &PersistentState,
661      DelayedParsingCallbacks *DelayedParseCB,
662      OptionSet<TypeCheckingFlags> TypeCheckOptions) {
...
677    bool Done;
678    do {
...
683      parseIntoSourceFile(MainFile, MainFile.getBufferID().getValue(), &Done,
684                          TheSILModule ? &SILContext : nullptr, &PersistentState,
685                          DelayedParseCB);
...
688        performTypeChecking(MainFile, PersistentState.getTopLevelContext(),
689                            TypeCheckOptions, CurTUElem,
690                            options.WarnLongFunctionBodies,
691                            options.WarnLongExpressionTypeChecking,
692                            options.SolverExpressionTimeThreshold);
...
695    } while (!Done);
...
713  }

I'll cover type-checking in a future article.

Recap: lexing and parsing in the Swift compiler

This article covered a lot of ground:

It explained that Swift's main function determines whether to call performFrontend by checking for the presence of the -frontend argument.
If the -frontend argument is provided, performFrontend parses the other arguments passed into swift -frontend. It determines the requested action based on those arguments, and instantiates a CompilerInstance and ASTContext to carry out that action. If compilation is requested, it calls performCompile.
performCompile checks the requested action and decides whether to call CompilerInstance::performSema (parsing and type-checking), or CompilerInstance::performParseOnly (just parsing). Our invocation of swift -frontend -c results in a requested action of ActionType::EmitObject, and so CompilerInstance::performSema is called.
A Parser is initialized along with its Lexer. A Lexer reads characters and transforms strings of them into Token objects. A Parser drives the Lexer, based on the tokens its seeing and its knowledge of Swift syntax. The parsing process begins when Parser::parseTopLevel is called. UnresolvedDeclRefExpr. It sees an opening parenthesis immediately following the identifier, so it wraps the expression in a CallExpr. The arguments to the call expression are parsed: a single StringLiteralExpr, wrapped in a ParenExpr.
The end of the file is reached, and so control goes back to CompilerInstance::performSema. Next stop: the type-checker!

To add new pieces of Swift syntax, or to modify existing Swift syntax, it's helpful to understand the libswiftParse source code. For example:

The pull request that implemented mulitline string literals in Swift modified how the Lexer formed string literal tokens.
The pull request that implemented #error and #warning in Swift added tok::pound_warning and tok::pound_error to the list of token kinds in TokenKinds.def. It then added the Parser::parseDeclPoundDiagnostic function. This function, unlike many of the Parser functions we covered above, does not create new AST nodes. Instead, it simply emits a warning or error when it sees tok::pound_warning or tok::pound_error. Pretty clever, huh?

To learn more about how parsing works in the Swift compiler, try writing small programs and walking through the code in the compiler to see how they're parsed. A good way to do this is by attaching a debugger – read the instructions from the first article in this series, Getting Started with Swift Compiler Development, to learn how to do so. There's still a lot left to learn that I didn't cover in this article. Here's just two examples of parser mechanics that this article didn't cover:

The parser is capable of "backtracking". That is, it can begin parsing tokens assuming a certain kind of statement or expression and, if it determines it is not actually parsing an expression of that kind, it can "undo" and reset back to where it started.
The parser can conditionally parse portions of the program based on directives such as #if and #else.

Option Parsing in the Swift Compiler

2017-12-16T05:00:00+00:00

The last article in this series explained how libswiftDriver split up swift executable invocations into smaller sub-jobs. Parsing command-line arguments is a big part of that work, but I didn't go into much detail about it. This article will now explain command-line argument parsing in depth.

Specifically, this article describes how:

Within the Swift build system, LLVM TableGen is used to transform the options specified in swift/include/swift/Option/Options.td. The transformed output is written to a file named Options.inc.
A libswiftOption header includes the Options.inc file in order to define an enum named swift::options::ID. By defining a macro before including Options.inc file, it's able to define an enum case for each option defined in the original Options.td file: swift::options::ID::OPT_driver_print_jobs, swift::options::ID::OPT_driver_print_actions, and so on.
In its implementation, libswiftOption defines a macro, then includes the Options.inc file a second time, this time in order to initialize an llvm::opt::OptTable. The OptTable class is defined in libLLVMOption and provides argument parsing utilities.
In its swift::Driver::buildCompilation method, libswiftDriver calls the llvm::opt::OptTable::ParseArgs method. This takes the array of strings passed into the swift compiler executable's main function as an argument, and it returns an llvm::opt::InputArgList. This class defines the methods used throughout the Swift compiler codebase. The InputArgList::hasArg method, which checks for the presence of an argument, is perhaps the most common.

Many contributions to the Swift compiler involve modifying or adding command-line options. Understanding how these options are parsed has helped me make such contributions.

An introduction to LLVM TableGen

TableGen is a utility program for LLVM developers. An llvm-tblgen executable is built as part of LLVM, and can be found in the LLVM build directory of a Swift build tree, in /path/to/build/llvm-macosx-x86_64/bin/llvm-tblgen.

TableGen is documented in detail here, but for my purposes it's sufficient to understand it as a tool that transforms the syntax in Swift's option files – swift/include/swift/Option/Options.td, swift/include/swift/Option/FrontendOptions.td, and swift/tools/SourceKit/tools/sourcekitd-test/Options.td – into a syntax that looks like C macro invocations.

For example, the -driver-print-jobs compiler option, mentioned in several past articles, is defined in Options.td like so:

swift/include/swift/Option/Options.td

18  include "llvm/Option/OptParser.td"
..
82  def driver_print_jobs : Flag<["-"], "driver-print-jobs">, InternalDebugOpt,
83    HelpText<"Dump list of jobs to execute">;

I can use llvm-tblgen to transform the file in which it's defined:

/path/to/build/llvm-macosx-x86_64/bin/llvm-tblgen \
    -I ~/local/Source/apple/llvm/include \
    -I ~/local/Source/apple/swift/include/swift/Option \
    ~/local/Source/apple/swift/include/swift/Option/Options.td \
    -gen-opt-parser-defs

This takes all of the options in Options.td, and outputs them as calls to a C macro named OPTION. For example, here's the output that corresponds to the -driver-print-jobs definition shown above:

OPTION(
  prefix_1,
  "driver-print-jobs",
  driver_print_jobs,
  Flag,
  internal_debug_Group,
  INVALID,
  nullptr,
  HelpHidden | DoesNotAffectIncrementalBuild,
  0,
  "Dump list of jobs to execute",
  nullptr,
  nullptr)

The LLVM TableGen executable llvm-tblgen can output many different formats. I instructed it to output calls to the OPTION macro, by using the -gen-opt-parser-defs argument. Other arguments include -gen-ctags, which generates definitions and source locations for the popular source indexing utility ctags, and -print-records, which prints TableGen's internal representation of each entry.

With that prerequisite explanation of LLVM TableGen out of the way, we're ready to look at the four stages of how arguments are parsed in the Swift compiler.

Stage 1: Swift's CMake instructs TableGen to transform `Options.td` into `Options.inc`

If you haven't already, try reading The Swift Compiler's Build System and Reading and Understanding the CMake in apple/swift before continuing – you'll need to know the basics of CMake in order to enjoy this section.

A git grep for "driver-print-jobs" reveals that this option is defined in swift/include/swift/Option/Options.td. In fact, nearly every option supported by the Swift compiler is defined in this file. And in that same directory, a CMakeLists.txt file defines how these options are transformed by TableGen:

swift/include/swift/Option/CMakeLists.txt

1  set(LLVM_TARGET_DEFINITIONS Options.td)
2  swift_tablegen(Options.inc -gen-opt-parser-defs)
3  swift_add_public_tablegen_target(SwiftOptions)

Note that, unlike CMake functions that take their input files as arguments, the swift_tablegen CMake macro requires its input files be specified using a global variable named LLVM_TARGET_DEFINITIONS. The function also allows arbitrary options to be passed to the llvm-tblgen executable. In this case, that's -gen-opt-parser-defs – the option that transforms TableGen definitions into OPTION macro calls.

swift_tablgen and swift_add_public_tablegen_target are CMake macros defined in swift/cmake/modules/AddSwiftTableGen.cmake. These two macros call through to LLVM CMake functions tablegen and add_public_tablegen_target:

The LLVM CMake function tablegen instructs CMake on how to produce the specified output file (Options.inc, in this case).
The LLVM CMake function add_public_tablegen_target creates a public CMake target that depends on the production of the Options.inc file.

Here's the tablegen LLVM CMake function. It uses the built-in CMake function add_custom_command to define how llvm-tblgen should be run:

llvm/cmake/modules/TableGen.cmake

 11  function(tablegen project ofn)
 ..
 43    if (IS_ABSOLUTE ${LLVM_TARGET_DEFINITIONS})
 44      set(LLVM_TARGET_DEFINITIONS_ABSOLUTE ${LLVM_TARGET_DEFINITIONS})
 45    else()
 46      set(LLVM_TARGET_DEFINITIONS_ABSOLUTE
 47        ${CMAKE_CURRENT_SOURCE_DIR}/${LLVM_TARGET_DEFINITIONS})
 48    endif()
 ..
 65    add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${ofn}.tmp
 ..
 67      COMMAND ${${project}_TABLEGEN_EXE} ${ARGN} -I ${CMAKE_CURRENT_SOURCE_DIR}
 68      ${LLVM_TABLEGEN_FLAGS}
 69      ${LLVM_TARGET_DEFINITIONS_ABSOLUTE}
 ..
 77      COMMENT "Building ${ofn}..."
 78      )
 79    add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${ofn}
 80      # Only update the real output file if there are any differences.
 81      # This prevents recompilation of all the files depending on it if there
 82      # aren't any.
 83      COMMAND ${CMAKE_COMMAND} -E copy_if_different
 84          ${CMAKE_CURRENT_BINARY_DIR}/${ofn}.tmp
 85          ${CMAKE_CURRENT_BINARY_DIR}/${ofn}
 86      DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${ofn}.tmp
 87      COMMENT "Updating ${ofn}..."
 88      )
 ..
 93  
 94    set(TABLEGEN_OUTPUT ${TABLEGEN_OUTPUT} ${CMAKE_CURRENT_BINARY_DIR}/${ofn} PARENT_SCOPE)
 ..
 97  endfunction()

And here's the add_public_tablegen_target LLVM CMake function:

llvm/cmake/modules/TableGen.cmake

100  function(add_public_tablegen_target target)
...
104    add_custom_target(${target}
105      DEPENDS ${TABLEGEN_OUTPUT})
...
111  endfunction()

The add_public_tablegen_target calls the built-in CMake function add_custom_target in order to define a CMake target. In this case, the name of that target is SwiftOptions. I can manually generate the TableGen for SwiftOptions by invoking cmake --build on the command line, specifying SwiftOptions as the target I'd like to build:

cmake --build \
    /path/to/build/swift-macosx-x86_64 \
    --target SwiftOptions

The above command does essentially the same thing as the manual llvm-tblgen -gen-opt-parser-defs invocation shown earlier in this article.

Instead of having users manually build the SwiftOptions target, the CMake targets for several Swift compiler libraries, such as libswiftOption and libswiftDriver, specify a dependency upon SwiftOptions. For example, here's libswiftOption:

swift/lib/Option/CMakeLists.txt

1  add_swift_library(swiftOption STATIC
.
4    DEPENDS SwiftOptions

As a result of this declared dependency, building libswiftOption results in SwiftOptions being built first, which means llvm-tblgen is run on swift/include/swift/Option/Options.td in order to produce the file /path/to/build/swift-macosx-x86_64/include/swift/Option/Options.inc. And again, because the swift/include/swift/Option/CMakeLists.txt file specifies that llvm-tblgen be invoked with the -gen-opt-parser-defs argument, the Options.inc file is populated with one call to an OPTION macro for each option defined in the original Options.td file.

Stage 2: The `libswiftOption` headers declare an enum containing a case for each of the options

In stage one, llvm-tblgen transformed the Options.td file into a file named Options.inc. For each option defined in the original Options.td, the new Options.inc file contains an invocation of a macro named OPTION:

OPTION(
  prefix_1,
  "driver-print-jobs",
  driver_print_jobs,
  Flag,
  internal_debug_Group,
  INVALID,
  nullptr,
  HelpHidden | DoesNotAffectIncrementalBuild,
  0,
  "Dump list of jobs to execute",
  nullptr,
  nullptr)

This is useful because I can define the OPTION macro to do whatever I want, then #include "Options.inc" in order to have the macro invoked once for each option.

This is exactly what the libswiftOption header Options.h does in order to define the swift::options::ID enum. That enum contains a case for each option defined in Options.td. Without TableGen, that would mean manually listing each of these options out in the enum, like this:

namespace swift {
namespace options {

enum ID {
  OPT_driver_print_jobs,
  OPT_driver_print_actions,
  OPT_driver_skip_execution,
  OPT_driver_use_frontend_path,
  // ...182 more options.
};

Every time a Swift compiler developer wanted to add or remove a Swift option, they'd have to manually add or remove it from that enum. That's not only tedious, it's also error-prone.

Instead, the swift/include/Option/Options.h header defines the OPTION macro such that it concatenates the tokens OPT_<option-identifier> for each definition in Options.inc:

swift/include/swift/Option/Options.h

24  namespace swift {
25  namespace options {
..  
39    enum ID {
40      OPT_INVALID = 0,
41  #define OPTION(PREFIX, NAME, ID, KIND, GROUP, ALIAS, ALIASARGS, FLAGS, PARAM,  \
42                 HELPTEXT, METAVAR, VALUES)                                      \
43    OPT_##ID,
44  #include "swift/Option/Options.inc"
45      LastOption
46  #undef OPTION
47    };

This results in an enum with several hundred cases, one for each option defined in Options.inc. Note that the macro definition of OPTION above only uses the third argument passed into it, which it calls ID. For -driver-print-jobs, that third parameter is driver_print_jobs:

OPTION(
  prefix_1,
  "driver-print-jobs",
  driver_print_jobs,
  Flag,
  internal_debug_Group,
  INVALID,
  nullptr,
  HelpHidden | DoesNotAffectIncrementalBuild,
  0,
  "Dump list of jobs to execute",
  nullptr,
  nullptr)

Stage 3: `libswiftOption` instantiates an `llvm::opt::OptTable`, with an `llvm::opt::OptTable::Info` element for each of the options

LLVM provides a library, named libLLVMOption, that encapsulates common operations related to command-line argument parsing. For instance, libLLVMOption defines the llvm::opt::OptTable and llvm::opt::InputArgList classes:

The OptTable class is instantiated with a list of options to parse, and provides a method named ParseArgs. This method takes the list of strings passed into an executable, compares them to the options the OptTable is supposed to parse, and returns an InputArgList (as well as a list of arguments that could not be parsed).
InputArgList defines a method named hasArg. This method can be used to check whether an option was specified.

libswiftOption is responsible for instantiating an llvm::opt::OptTable that's used to parse arguments passed to the swift executable. The llvm::opt::OptTable initializer takes an array of options it's supposed to parse. These options are represented using the llvm::opt::OptTable::Info struct:

llvm/include/llvm/Option/OptTable.h

39  class OptTable {
..
42    struct Info {
..
45      const char *const *Prefixes;
46      const char *Name;
47      const char *HelpText;
48      const char *MetaVar;
49      unsigned ID;
50      unsigned char Kind;
51      unsigned char Param;
52      unsigned short Flags;
53      unsigned short GroupID;
54      unsigned short AliasID;
55      const char *AliasArgs;
56      const char *Values;
57    };
..
83  protected:
84    OptTable(ArrayRef<Info> OptionInfos, bool IgnoreCase = false);

In order to create this array of options, libswiftOption once again defines the OPTION macro and includes the Options.inc file. This time, it defines the OPTION macro such that it list-initializes an OptTable::Info struct for each call, and it uses those to statically define an array named InfoTable[]:

swift/lib/Option/Options.cpp

13  #include "swift/Option/Options.h"
..
16  #include "llvm/Option/OptTable.h"
17  #include "llvm/Option/Option.h"
18  
19  using namespace swift::options;
20  using namespace llvm::opt;
..
26  static const OptTable::Info InfoTable[] = {
27  #define OPTION(PREFIX, NAME, ID, KIND, GROUP, ALIAS, ALIASARGS, FLAGS, PARAM,  \
28                 HELPTEXT, METAVAR, VALUES)                                      \
29    {PREFIX, NAME,  HELPTEXT,    METAVAR,     OPT_##ID,  Option::KIND##Class,    \
30     PARAM,  FLAGS, OPT_##GROUP, OPT_##ALIAS, ALIASARGS, VALUES},
31  #include "swift/Option/Options.inc"
32  #undef OPTION
33  };

After pre-processing, the OPTION macro calls are expanded such that the InfoTable[] initializer above looks like this:

static const OptTable::Info InfoTable[] = {
  {{"-", nullptr}, "driver-print-jobs", "Dump list of jobs to execute", nullptr, OPT_driver_print_jobs, Option::FlagClass, 0, HelpHidden | DoesNotAffectIncrementalBuild, OPT_internal_debug_Group, OPT_INVALID, nullptr, nullptr},
  {{"-", nullptr}, "driver-print-actions", "Dump list of actions to perform", nullptr, OPT_driver_print_actions, Option::FlagClass, 0, HelpHidden | DoesNotAffectIncrementalBuild, OPT_internal_debug_Group, OPT_INVALID, nullptr, nullptr},
  // ...184 more options.
};

Note that the enum values from stage two are used in these static option definitions: OPT_driver_print_jobs, OPT_driver_print_actions, and so on.

To allow other parts of the Swift compiler to have access to this table of options, libswiftOption creates an llvm::opt::OptTable subclass named SwiftOptTable, and initializes it with the InfoTable[] array from above:

swift/lib/Option/Options.cpp

37  class SwiftOptTable : public OptTable {
38  public:
39    SwiftOptTable() : OptTable(InfoTable) {}
40  };

It also defines a function that allows other libraries to grab a reference to the option table:

swift/lib/Option/Options.cpp

44  std::unique_ptr<OptTable> swift::createSwiftOptTable() {
45    return std::unique_ptr<OptTable>(new SwiftOptTable());
46  }

Stage 4: `libswiftDriver` calls the `llvm::opt::OptTable::ParseArgs` method

As I briefly mentioned in the previous article in this series, the swift::Driver initializer calls the createSwiftOptTable function in order to get a reference to the llvm::opt::OptTable subclass SwiftOptTable, storing it in an ivar named Driver::Opts:

swift/lib/Driver/Driver.cpp

64  Driver::Driver(StringRef DriverExecutable,
65                 StringRef Name,
66                 ArrayRef<const char *> Args,
67                 DiagnosticEngine &Diags)
68    : Opts(createSwiftOptTable()), Diags(Diags),

Then, the Driver has the command-line arguments to the swift executable parsed, in order to create and configure an instance of swift::Compilation. To parse the arguments, it calls the llvm::opt::OptTable::ParseArgs method:

swift/lib/Driver/Driver.cpp

 841  std::unique_ptr<InputArgList>
 842  Driver::parseArgStrings(ArrayRef<const char *> Args) {
 ...
 854    ArgList = llvm::make_unique<InputArgList>(
 855        getOpts().ParseArgs(Args, MissingArgIndex, MissingArgCount,
 856                            IncludedFlagsBitmask, ExcludedFlagsBitmask));
 ...
 888    return ArgList;
 889  }

The InputArgList returned by the Driver::parseArgStrings method is used throughout libswiftDriver. For example, the previous article in this series showcased the following code, which produced a warning if the -incremental and -whole-module-optimization arguments were used in the same invocation:

swift/lib/Driver/Driver.cpp

529  bool Incremental = ArgList->hasArg(options::OPT_incremental);
530  if (ArgList->hasArg(options::OPT_whole_module_optimization)) {
531    if (Incremental && ShowIncrementalBuildDecisions) {
532      llvm::outs() << "Incremental compilation has been disabled, because it "
533                   << "is not compatible with whole module optimization.";
534    }
535    Incremental = false;
536  }

The code above uses the llvm::opt::InputArgList::hasArg method to check for the OPT_incremental and OPT_whole_module_optimization arguments. Remember, these were cases that were added to the swift::options::ID enum by including Options.inc in stage two above! It all comes toghether here.

Besides checking for the existence of certain options, InputArgList also has methods to grab the values specified by those methods. Here's how libswiftFrontend grabs the value from the popular -warn-long-function-bodies= command-line option, by using the llvm::opt::InputArgList::getLastArg and llvm::opt::Arg::getValue methods:

swift/lib/Frontend/CompilerInvocation.cpp

 178  if (const Arg *A = Args.getLastArg(OPT_warn_long_function_bodies)) {
 179    unsigned attempt;
 180    if (StringRef(A->getValue()).getAsInteger(10, attempt)) {
 181      Diags.diagnose(SourceLoc(), diag::error_invalid_arg_value,
 182                     A->getAsString(Args), A->getValue());
 183    } else {
 184      Opts.WarnLongFunctionBodies = attempt;
 185    }
 186  }

Sorry, no magic here

I enjoy looking into the details of the Swift compiler because – and maybe this sounds silly – it helps me better understand that it's "just a program".

Because I was unfamilar with the LLVM TableGen utility, and with the C/C++ macros that the Swift compiler uses to define its options, it seemed like magic to me that modifying the Options.td file would result in changes to Swift's command-line options. But it's not magic – as this article described, it's a four-stage process in which:

The Options.td file is transformed by TableGen.
The transformed file, Options.inc, is included such that it defines a large enum with all the Swift options as values.
The transformed file is included again, this time to intiialize an LLVM OptTable. This class is capable of searching command-line arguments for option values.
The rest of the Swift compiler codebase uses the LLVM OptTable class to check for arguments as necessary.

Reading and Understanding the Swift Driver Source Code

2017-11-25T05:00:00+00:00

In the previous article of this running series on the Swift compiler, I introduced the Swift compiler driver. To recap: the driver is the executable that is run when users invoke swift or swiftc on the command line. It's responsible for splitting up a single swift or swiftc invocation into multiple child jobs. These jobs are command line invocations of swift -frontend, ld, or other command-line utilities.

This article explains the details of how the Swift driver executable and libswiftDriver accomplish this. Specifically, I'll explain the code that is executed when I run swiftc hello.swift on the command line.

I've found that understanding this process has made it easier for me to fix Swift JIRA bugs labeled "Driver". Please let me know if this article helps you do the same!

The six stages of the Swift driver

I think of the driver as having six main stages, in which it:

Checks whether it needs to be split up: The driver checks whether the current invocation of swift or swiftc should be split up into jobs, or whether it's currently being run as one of those jobs. In other words, it checks whether it's being run as swift or swiftc (and needs to be split up), or as swift -frontend (and so has already been split up). In our case, swiftc hello.swift is not an invocation of swift -frontend, and so the driver will determine it needs to be split up.
Instantiates a swift::Driver: If the invocation needs to be split up into jobs by the driver, a swift::Driver object is instantiated, and its intializer determines the "driver mode" (explained in the previous article).
Instantiates a swift::Compilation: Assuming a driver mode of swiftc, the swift::Driver::buildCompilation method is called. This parses command-line options to create a swift::Compilation object, which encapsulates the information required to build out a set of jobs. One of the most important pieces of information stored at this point is a mapping of inputs to desired outputs.
Splits up the work of a Compilation into a graph of swift::Action objects: The swift::Driver::buildActions method creates a graph of swift::Action objects. These represent smaller unit of work, such as "compile this Swift source file into an object" and "link these objects into an executable." The actions for any given swift invocation can be examined with the -driver-print-actions option.
Instantiates a list of swift::Job based on those actions: Actions contain all the information necessary for a swift::Compilation to instantiate a swift::Job, which represents the actual command-line invocation used to run a child job. To translate actions into jobs, the swift::Compilation::buildJobs method is invoked. As explained in the previous article, the jobs for any given swift invocation can be exampled with the -driver-print-jobs option.
Executes each of the list of swift::Job: Finally, the swift::Compilation::petformJobs method is called. This method runs each of the jobs within a task queue.

The apple/swift repository includes a document named Driver Design & Internals that describes the Swift 2.0 driver as being composed of four stages. I prefer to think of it as six stages, because I like to break the stages down more granularly – my stages one through three correspond to that document's stage one.

Stage 1: Checking whether the current `swift` invocation needs to be split up by the driver

I'll begin where the swift executable (and every C-based program) begins: the main function.

It's important to note that I'll elide many lines of source code that aren't immediately relevant to the six stages of the driver. For example, the very first thing the driver does is call the INITIALIZE_LLVM macro. The Swift compiler would not function without this line of code, but because it's not crucial to understanding the Swift driver, I won't go into that macro here.

1.1: Checking if the invocation is trying to run a "subcommand"

One of the first things swift does in main is check whether it's being run as a "subcommand":

swift/tools/driver/driver.cpp

111  int main(int argc_, const char **argv_) {
...
124    StringRef ExecName = llvm::sys::path::stem(argv[0]);
125    SmallString<256> SubcommandName;
126    if (shouldRunAsSubcommand(ExecName, SubcommandName, argv)) {

"Subcommands" here to refer to commands like swift build or swift test. The shouldRunAsSubcommand function checks the first two command-line arguments and, if they fit a certain pattern, it returns an executable name based on them. For example, an invocation that begins with swift build would result in this function returning "swift-build":

swift/tools/driver/driver.cpp

 70  static bool shouldRunAsSubcommand(StringRef ExecName,
 71                                    SmallString<256> &SubcommandName,
 72                                    SmallVectorImpl<const char *> &Args) {
 ..
 78    if (ExecName != "swift")
 79      return false;
 ..  
 87    StringRef FirstArg(Args[1]);
 88    if (FirstArg.startswith("-") || FirstArg.find('.') != StringRef::npos ||
 89        FirstArg.find('/') != StringRef::npos)
 90      return false;
...  
103    SubcommandName.assign("swift-");
104    SubcommandName.append(Subcommand);
105  
106    return true;
107  }

There are two things to note here:

The Swift driver only considers invocations that begin with swift as potential subcommand invocations. So swiftc build would not be considered a subcommand.
Invocations whose second argument contains a period, or that start with a dash, as not considered subcommands. So swift hello.swift</span> would not be evaluated as a subcommand.

If the driver determines that it is running a subcommand, such as swift build, it finds an executable named swift-build in the same directory as the swift or swiftc executable currently being run, and it executes it and immediately exits the program by calling the swift::ExecuteInPlace function:

swift/tools/driver/driver.cpp

111  int main(int argc_, const char **argv_) {
...
126    if (shouldRunAsSubcommand(ExecName, SubcommandName, argv)) {
...
129      SmallString<256> SubcommandPath(
130        llvm::sys::path::parent_path(getExecutablePath(argv[0])));
131      llvm::sys::path::append(SubcommandPath, SubcommandName);
...
146      argv.push_back(nullptr);
147      ExecuteInPlace(SubcommandPath.c_str(), argv.data());
...
154    }

This is a fun quirk about how the Swift compiler works: if you invoke it with swift foo or swift hello, it'll interpret those as subcommand invocations. This means the driver will attempt to invoke an executable named swift-foo or swift-hello. You can try this yourself by creating an executable named swift-goodjob next to your built Swift executable:
/path/to/your/swift-macosx-x86_64/bin/swift-goodjob
#!/usr/bin/env bash
echo "Keep up the good work!"
Make the file executable by running chmod a+x /path/to/your/swift-macosx-x86_64/bin/swift-goodjob on the command line, and you'll be able to invoke it as a Swift subcommand:
swift goodjob
Invoking it results in a reaffirming message:
Keep up the good work!

1.2: Checking if the invocation is for "integrated tools" like `swift -frontend`

When I invoke swiftc hello.swift on the command line, it's not evaluated as a subcommand for two reasons:

I'm invoking swiftc, not swift. Only swift <subcommand-name> invocations are considered potential subcommands.
The second argument, hello.swift, contains a period. The code I listed above explicitly discounts invocations whose second command contains a period.

Since swiftc hello.swift isn't considered a subcommand, the driver would continue by checking whether swiftc hello.swift needs to be split up into child jobs. The only case in which this isn't necessary is when swift or swiftc is being invoked as swift -frontend, swift -modulewrap, or swift -apinotes. Each of these are considered to have already been split up.

Therefore one of the first things the Swift driver does, after checking whether it should execute a subcommand, is check whether it should execute the swift -frontend code path:

swift/tools/driver/driver.cpp

111  int main(int argc_, const char **argv_) {
...    // ...checks whether to run a subcommand.
...
158    StringRef FirstArg(argv[1]);
159    if (FirstArg == "-frontend") {
160      return performFrontend(llvm::makeArrayRef(argv.data()+2,
161                                                argv.data()+argv.size()),
162                             argv[0], (void *)(intptr_t)getExecutablePath);
163    }
164    if (FirstArg == "-modulewrap") {
165      return modulewrap_main(llvm::makeArrayRef(argv.data()+2,
166                                                argv.data()+argv.size()),
167                             argv[0], (void *)(intptr_t)getExecutablePath);
168    }
169    if (FirstArg == "-apinotes") {
170      return apinotes_main(llvm::makeArrayRef(argv.data()+1,
171                                              argv.data()+argv.size()));
172    }

The first code path in the if statement, for swift -frontend or swiftc -frontend, calls the performFrontend function. This kicks off everything I think of when I think of "compiling Swift code": parsing the source code, building an abstract syntax tree, type checking, and so on.

I'll cover the frontend in a future article. For now, the driver has determined that my swiftc hello.swift invocation probably needs to be split up, and so it continues to stage two.

Stage 2: Instantiating a `swift::Driver`

The logic behind splitting up my swiftc hello.swift invocation is handled by an instance of swift::Driver, which the code in main proceeds to instantiate:

swift/tools/driver/driver.cpp

111  int main(int argc_, const char **argv_) {
...    // ...checks whether to run a subcommand,
...    // or whether to execute the compiler frontend.
182
183    Driver TheDriver(Path, ExecName, argv, Diags);

The Driver initializer does two things:

It instantiates a table of Swift option flags. (In order to do the explanation justice, I'll describe option tables and option parsing in a future article.)
It determines the "driver mode" it's being run under. Recall that the previous article explained how the name of the executable (and the --driver-mode= option) is used to determine how the driver behaves.

swift/lib/Driver/Driver.cpp

64  Driver::Driver(StringRef DriverExecutable,
65                 StringRef Name,
66                 ArrayRef<const char *> Args,
67                 DiagnosticEngine &Diags)
68    : Opts(createSwiftOptTable()), Diags(Diags),
69      Name(Name), DriverExecutable(DriverExecutable),
70      DefaultTargetTriple(llvm::sys::getDefaultTargetTriple()) {
..
74    parseDriverKind(Args.slice(1));
75  }

Once the Driver is instantiated, the driver executable determines what to do based on its driver mode: a mode of swift-format results in an early exit after calling swift_format_main, for example.

swift/tools/driver/driver.cpp

111  int main(int argc_, const char **argv_) {
...    // ...checks whether to run a subcommand,
...    // or whether to execute the compiler frontend.
182
183    Driver TheDriver(Path, ExecName, argv, Diags);
184    switch (TheDriver.getDriverKind()) {
...
189    case Driver::DriverKind::SwiftFormat:
190      return swift_format_main(
191        TheDriver.getArgsWithoutProgramNameAndDriverMode(argv),
192        argv[0], (void *)(intptr_t)getExecutablePath);
193    default:
194      break;
195    }

If you're interested in how swift-format works, just follow the code in the swift_format_main function to find out. swiftc hello.swift, on the other hand, results in a driver mode of DriverKind::Batch, and so this early return isn't taken. Instead, the driver continues its work of splitting up my invocation. To do so, it'll create a swift::Compilation object.

Stage 3: Instantiating a `swift::Compilation`

The driver uses an instance of swift::Compilation to build up and a store a list of jobs. To begin, it calls the swift::Driver::buildCompilation method:

swift/tools/driver/driver.cpp

111  int main(int argc_, const char **argv_) {
...    // ...checks whether to run a subcommand,
...    // or whether to execute the compiler frontend,
...    // and then instantiates a Driver object and
...    // checks whether to return early for swift-format.
196  
197    std::unique_ptr<Compilation> C = TheDriver.buildCompilation(argv);

3.1: Parsing and validating command-line arguments

The Driver::buildCompilation method parses the command-line arguments passed into the original swift or swiftc invocation, by calling the Driver::parseArgStrings method. It then validates those arguments. For example, the -incremental and -whole-module-optimization options are mutually exclusive, so it prints a warning if they're used together:

swift/lib/Driver/Driver.cpp

504  std::unique_ptr<Compilation> Driver::buildCompilation(
505      ArrayRef<const char *> Args) {
...
510    std::unique_ptr<InputArgList> ArgList(parseArgStrings(Args.slice(1)));
...
529    bool Incremental = ArgList->hasArg(options::OPT_incremental);
530    if (ArgList->hasArg(options::OPT_whole_module_optimization)) {
531      if (Incremental && ShowIncrementalBuildDecisions) {
532        llvm::outs() << "Incremental compilation has been disabled, because it "
533                     << "is not compatible with whole module optimization.";
534      }
535      Incremental = false;
536    }

3.2: Instantiating a `swift::ToolChain`

Once the arguments have been validated, the Driver::buildCompilation method instantiates a swift::ToolChain object. A swift::ToolChain is an object that's capable of translating "actions" (swift::Action) into "jobs" (swift::Job).

swift::ToolChain itself is an abstract base class. Several subclasses exist to create jobs for specific targets. For example, the swift::toolchains::Darwin toolchain knows how to create jobs that would work when compiling for a macOS or iOS target. swift::toolchains::Android knows how to create jobs for Android targets.

An appropriate toolchain is instantiated from within the Driver::buildCompilation method, by calling the makeToolChain function:

swift/lib/Driver/Driver.cpp

504  std::unique_ptr<Compilation> Driver::buildCompilation(
505      ArrayRef<const char *> Args) {
...    // ...parses and validates arguments.
562
563    std::unique_ptr<const ToolChain> TC =
564        makeToolChain(*this, llvm::Triple(DefaultTargetTriple));

The makeToolChain function determines which toolchain to instantiate based on the target triple:

swift/lib/Driver/Driver.cpp

204  static std::unique_ptr<const ToolChain>
205  makeToolChain(Driver &driver, const llvm::Triple &target) {
206    switch (target.getOS()) {
207    case llvm::Triple::Darwin:
208    case llvm::Triple::MacOSX:
209    case llvm::Triple::IOS:
210    case llvm::Triple::TvOS:
211    case llvm::Triple::WatchOS:
212      return llvm::make_unique<toolchains::Darwin>(driver, target);
213      break;
214    case llvm::Triple::Linux:
215      if (target.isAndroid()) {
216        return llvm::make_unique<toolchains::Android>(driver, target);
217      } else {
218        return llvm::make_unique<toolchains::GenericUnix>(driver, target);
219      }
220      break;
...
232    }
233  }

3.3: Determining the `swift::driver::OutputInfo`

The driver's freshly instantiated ToolChain is capable of translating actions into jobs. But to determine which actions my swiftc hello.swift invocation is composed of, the driver needs to construct an OutputInfo object:

swift/lib/Driver/Driver.cpp

504  std::unique_ptr<Compilation> Driver::buildCompilation(
505      ArrayRef<const char *> Args) {
...    // ...parses arguments and instantiates a toolchain.
...
583    OutputInfo OI;
584    buildOutputInfo(*TC, *TranslatedArgList, Inputs, OI);

The Driver::buildOutputInfo method sets properties on the OutputInfo object, based on the driver mode and the presence of driver arguments like -emit-executable or -emit-library. Among the most important of these is OutputInfo::LinkAction and OutputInfo::CompilerOutputType:

swift/lib/Driver/Driver.cpp

1052  void Driver::buildOutputInfo(const ToolChain &TC, const DerivedArgList &Args,
1053                               const InputFileList &Inputs,
1054                               OutputInfo &OI) const {
....
1090      switch (OutputModeArg->getOption().getID()) {
1091      case options::OPT_emit_executable:
1092        OI.LinkAction = LinkKind::Executable;
1093        OI.CompilerOutputType = types::TY_Object;
1094        break;
1095  
1096      case options::OPT_emit_library:
1097        OI.LinkAction = LinkKind::DynamicLibrary;
1098        OI.CompilerOutputType = types::TY_Object;
1099        break;
....
1334  }

Finally, the driver is ready to build a list of actions, by instantiating a Compilation object, and then calling the Driver::buildActions method:

swift/lib/Driver/Driver.cpp

504  std::unique_ptr<Compilation> Driver::buildCompilation(
505      ArrayRef<const char *> Args) {
...    // ...parses arguments, instantiates a toolchain,
...    // and builds output info.
688
689    std::unique_ptr<Compilation> C(new Compilation(Diags, Level,
690                                                   std::move(ArgList),
691                                                   std::move(TranslatedArgList),
692                                                   std::move(Inputs),
693                                                   ArgsHash, StartTime,
694                                                   NumberOfParallelCommands,
695                                                   Incremental,
696                                                   DriverSkipExecution,
697                                                   SaveTemps,
698                                                   ShowDriverTimeCompilation,
699                                                   std::move(StatsReporter)));
700    // Construct the graph of Actions.
701    SmallVector<const Action *, 8> TopLevelActions;
702    buildActions(TopLevelActions, *TC, OI, OFM.get(),
703                 rebuildEverything ? nullptr : &outOfDateMap, *C);

Stage 4: Instantiating a graph of `swift::Action`

ToolChain objects are capable of translating "actions" – high-level descriptions of the inputs and outputs of a Swift compilation – into concrete "jobs".

The previous article explained that invoking swiftc hello.swift -driver-print-jobs prints these "jobs", which are themselves command-line invocations:

/Users/bgesiak/Source/apple/build/Ninja-ReleaseAssert/swift-macosx-x86_64/bin/swift \
    -frontend -c -primary-file hello.swift \
    -target x86_64-apple-darwin16.7.0 \
    -enable-objc-interop -color-diagnostics -module-name hello \
    -o /var/folders/ry/2ryfdsb56b30092626qprw6d3rb3ss/T/hello-6ab6fd.o

/usr/bin/ld \
    /var/folders/ry/2ryfdsb56b30092626qprw6d3rb3ss/T/hello-6ab6fd.o \
    -lobjc -lSystem -arch x86_64 -L /Users/bgesiak/Source/apple/build/Ninja-ReleaseAssert/swift-macosx-x86_64/lib/swift/macosx \
    -rpath /Users/bgesiak/Source/apple/build/Ninja-ReleaseAssert/swift-macosx-x86_64/lib/swift/macosx \
    -macosx_version_min 10.12.0 -no_objc_category_merging \
    -o hello

Besides jobs, the driver is also capable of printing a list of actions. Invoking swiftc hello.swift -driver-print-actions outputs:

0: input, "hello.swift", swift
1: compile, {0}, object
2: link, {1}, image

This looks similar to the output of swiftc hello.swift -driver-print-jobs, with a few key differences:

There are three actions. These are "translated," by a ToolChain, into two jobs.
The input file hello.swift is represented as an action, but there is no equivalent representation of it as a job.
The actions are high-level descriptions of what needs to happen. For example, 1: compile, {0}, object means "input file hello.swift needs to be compiled into an object." The job that corresponds to that action is a swift -frontend -c invocation with tons of arguments. In other words, the job is the concrete way that the toolchain chose to fulfill the action.

The actions themselves are constructed from within the Driver::buildActions method. This is a long method that's responsible for creating a list of actions based on the OutputInfo instance, the Driver object's DriverKind, and in some cases the target platform. I'll skip over most of it to show just the parts that are relevant to my swiftc hello.swift invocation:

swift/lib/Driver/Driver.cpp

1336  void Driver::buildActions(SmallVectorImpl<const Action *> &TopLevelActions,
1337                            const ToolChain &TC, const OutputInfo &OI,
1338                            const OutputFileMap *OFM,
1339                            const InputInfoMap *OutOfDateMap,
1340                            Compilation &C) const {
....
1349    SmallVector<const Action *, 2> AllModuleInputs;
1350    SmallVector<const Action *, 2> AllLinkerInputs;
1351
1352    switch (OI.CompilerMode) {
1353    case OutputInfo::Mode::StandardCompile: {
....  
1378      for (const InputPair &Input : Inputs) {
1379        types::ID InputType = Input.first;
1380        const Arg *InputArg = Input.second;
1381  
1382        Action *Current = C.createAction<InputAction>(*InputArg, InputType);
1383        switch (InputType) {
1384        case types::TY_Swift:
1385        case types::TY_SIL:
1386        case types::TY_SIB: {
....
1405            Current = C.createAction<CompileJobAction>(Current,
1406                                                       OI.CompilerOutputType,
1407                                                       previousBuildState);
....
1410            AllModuleInputs.push_back(Current);
1411          }
1412          AllLinkerInputs.push_back(Current);
1413          break;
1414        }
....  
1572    if (OI.shouldLink() && !AllLinkerInputs.empty()) {
1573      auto *LinkAction = C.createAction<LinkJobAction>(AllLinkerInputs,
1574                                                       OI.LinkAction);
....
1627    }
1628  }

The gist of the Driver::buildActions method is that it:

Loops over each of the inputs (in my case, that's just hello.swift) and creates an InputAction for each of them. The Compilation object stores a list of its actions internally.
For source code inputs of type .swift, .sil, or .sib, it creates a CompileJobAction, also stored internally to the Compilation object. It also specifies that the result of a CompileJobAction should be treated as an input to a linker action.
If a link action has been requested, and if there are inputs to that link action, the driver creates a LinkJobAction. Not all swiftc invocations result in a linker invocation – for example, had I ran swiftc hello.swift -emit-object, then the OutputInfo::shouldLink method would return false, and no LinkJobAction would have been created.

Stage 5: Instantiating a list of `swift::Job` based on those actions

After constructing the actions, the driver calls the Driver::buildJobs method to translate actions into jobs:

swift/lib/Driver/Driver.cpp

504  std::unique_ptr<Compilation> Driver::buildCompilation(
505      ArrayRef<const char *> Args) {
...    // ...parses arguments, instantiates a toolchain,
...    // and builds output info, instantiates a Compilation,
...    // and builds a graph of actions.
...
713    buildJobs(TopLevelActions, OI, OFM.get(), *TC, *C);
...
752    return C;
753  }

The Driver::buildJobs method loops over each Action and passes any JobAction objects to the Driver::buildJobsForAction method. This filters out inputs, which are represented as InputAction and so are not instances of JobAction:

swift/lib/Driver/Driver.cpp

1673  void Driver::buildJobs(ArrayRef<const Action *> TopLevelActions,
1674                         const OutputInfo &OI, const OutputFileMap *OFM,
1675                         const ToolChain &TC, Compilation &C) const {
....
1702    for (const Action *A : TopLevelActions) {
1703      if (auto *JA = dyn_cast<JobAction>(A)) {
1704        (void)buildJobsForAction(C, JA, OI, OFM, TC, /*TopLevel*/true, JobCache);
1705      }
1706    }
1707  }

The Driver::buildJobsForAction method does a lot. Among other things, it records dependencies between actions, in order to improve Swift's incremental compilation. That's a topic I'll need to write several articles about, so I won't attempt to cover it here.

At its core, though, the Driver::buildJobsForAction method calls through to the ToolChain object that was instantiated in stage three above, in order to translate each Action into a Job. It then adds that Job to a list stored within the Compilation object:

swift/lib/Driver/Driver.cpp

1993  Job *Driver::buildJobsForAction(Compilation &C, const JobAction *JA,
1994                                  const OutputInfo &OI,
1995                                  const OutputFileMap *OFM,
1996                                  const ToolChain &TC, bool AtTopLevel,
1997                                  JobCacheMap &JobCache) const {
....
2291    std::unique_ptr<Job> ownedJob = TC.constructJob(*JA, C, std::move(InputJobs),
2292                                                    InputActions, 
2293                                                    std::move(Output), OI);
2294    Job *J = C.addJob(std::move(ownedJob));
....  
2371    return J;
2372  }

In order to translate an action into a job, the ToolChain base class uses a macro that calls the appropriate ToolChain::constructInvocation method for each action class type. ToolChain classes overload constructInvocation in order to execute separate logic for, say, constructInvocation(const CompileJobAction &, ...) and constructInvocation(const LinkJobAction &, ...):

swift/lib/Driver/ToolChain.cpp

 66  std::unique_ptr<Job>
 67  ToolChain::constructJob(const JobAction &JA,
 68                          Compilation &C,
 69                          SmallVectorImpl<const Job *> &&inputs,
 70                          ArrayRef<const Action *> inputActions,
 71                          std::unique_ptr<CommandOutput> output,
 72                          const OutputInfo &OI) const {
 ..
 75    auto invocationInfo = [&]() -> InvocationInfo {
 76      switch (JA.getKind()) {
 77    #define CASE(K) case Action::K: \
 78        return constructInvocation(cast<K##Action>(JA), context);
 79      CASE(CompileJob)
 80      CASE(InterpretJob)
 81      CASE(BackendJob)
 82      CASE(MergeModuleJob)
 83      CASE(ModuleWrapJob)
 84      CASE(LinkJob)
 ..
 97    }();
...  
120    return llvm::make_unique<Job>(JA, std::move(inputs), std::move(output),
121                                  executablePath,
122                                  std::move(invocationInfo.Arguments),
123                                  std::move(invocationInfo.ExtraEnvironment),
124                                  std::move(invocationInfo.FilelistInfo));
125  }

Each subclass of toolchain – swift::toolchains::Darwin, swift::toolchains::Android, etc. – constructs platform-specific command-line invocations for each action. Some actions, however, are the same on each platform. One example of such an action is a CompileJobAction – jobs for these are invocations of swift -frontend, and these are pretty much the same on any platform. As a result, they're handled in the base ToolChain class.

Here's how the base ToolChain class transforms the compile action for hello.swift into an invocation of swift -frontend -c:

swift/lib/Driver/ToolChain.cpp

200  ToolChain::InvocationInfo
201  ToolChain::constructInvocation(const CompileJobAction &job,
202                                 const JobContext &context) const {
203    InvocationInfo II{SWIFT_EXECUTABLE_NAME};
204    ArgStringList &Arguments = II.Arguments;
205  
206    Arguments.push_back("-frontend");
...
210    switch (context.OI.CompilerMode) {
211    case OutputInfo::Mode::StandardCompile:
212    case OutputInfo::Mode::SingleCompile: {
213      switch (context.Output.getPrimaryOutputType()) {
214      case types::TY_Object:
215        FrontendModeOption = "-c";
216        break;
...
280    }
...
500    return II;
501  }

Unlike compile actions, link actions differ greatly depending on the target platform. As a result, most ToolChain subclasses override the ToolChain::constructInvocation(const LinkJobAction &, ...) method. Here's swift::toolchains::Darwin, which adds object files as inputs, conditionally adds the -dylib option when linking a dynamic library, and specifies the output file with -o:

swift/lib/Driver/ToolChain.cpp

1211  ToolChain::InvocationInfo
1212  toolchains::Darwin::constructInvocation(const LinkJobAction &job,
1213                                          const JobContext &context) const {
....
1228    const char *LD = "ld";
....  
1238    InvocationInfo II = {LD};
1239    ArgStringList &Arguments = II.Arguments;
....  
1250    addInputsOfType(Arguments, context.InputActions, types::TY_Object);
....  
1264    switch (job.getKind()) {
....
1270    case LinkKind::DynamicLibrary:
1271      Arguments.push_back("-dylib");
1272      break;
1273    }
....
1463    Arguments.push_back("-o");
1464    Arguments.push_back(context.Output.getPrimaryOutputFilename().c_str());
1465  
1466    return II;
1467  }

Stage 6: Executing each of the list of `swift::Job`

Recall that this article began in the swift executable's main function, the entry point of the compiler. That function eventually called the Driver::buildCompilation method, in stage three. The Driver::buildCompilation method called Driver::buildActions in stage four, and then Driver::buildJobs in stage five, before returning back to main.

Now back in main, the driver has a Compilation object that has a complete list of the Job objects it needs to execute. Finally, the driver calls the Compilation::performJobs method, which kicks off the execution of those jobs:

swift/tools/driver/driver.cpp

111  int main(int argc_, const char **argv_) {
...    // ...checks whether to run a subcommand,
...    // or whether to execute the compiler frontend,
...    // and then instantiates a Driver object and
...    // checks whether to return early for swift-format,
...    // and finally builds a Compilation object.
...
203    return C->performJobs();
...  
207  }

The Compilation::performJobs method just calls through to the Compilation::performJobsImpl method, which in turn schedules the jobs and executes them on a task queue:

swift/lib/Driver/Compilation.cpp

826  int Compilation::performJobsImpl(bool &abnormalExit) {
827    PerformJobsState State(*this);
828  
829    State.scheduleInitialJobs();
830    State.scheduleAdditionalJobs();
831    State.runTaskQueueToCompletion();
832    State.checkUnfinishedJobs();
...
843    return State.getResult();
844  }

The task queue itself is implemented in one of two ways, depending on whether the driver is running on a Darwin or on a Unix host:

swift/lib/Basic/TaskQueue.cpp

26  #if LLVM_ON_UNIX && !defined(__CYGWIN__) && !defined(__HAIKU__)
27  #include "Unix/TaskQueue.inc"
28  #else
29  #include "Default/TaskQueue.inc"
30  #endif

I won't go into the details of each of these implementations in this article. Suffice it to say that the child job invocations are each executed in order.

Putting it all together: the story of how `swiftc hello.swift` is executed

In summary, here's how an invocation of swiftc hello.swift, when run on my macOS host machine, goes through the six stages of the Swift driver:

The driver determines the invocation isn't a subcommand (those start with "swift"), and it isn't a swift -frontend invocation, so it must need to be split up somehow.
The driver instantiates a swift::Driver object in order to handle splitting up swiftc hello.swift. Because the name of the executable is swiftc, the driver infers that it's being run in DriverKind::Batch mode.
The driver instantiates a swift::Compilation object to store each of the actions and jobs that will be constructed. A swift::toolchains::Darwin toolchain object is instantiated, so that it can be used later to translate actions into macOS-specific jobs. The swiftc hello.swift invocation arguments are parsed in order to construct a swift::driver::OutputInfo object whose LinkAction is LinkKind::Executable, and whose CompilerOutputType is types::TY_Object.
The driver constructs a graph of swift::Action objects based on the OutputInfo. In the case of swiftc hello.swift, an InputAction is created to represent hello.swift, a CompileJobAction is created based on an OutputInfo::CompilerOutputType of TY_Object, and a LinkJobAction is created because the OutputInfo::LinkAction was set to Executable.
The driver invokes the toolchain in order to instantiate a swift::Job for each JobAction – so that's the CompileJobAction and LinkJobAction from the previous stage.
The driver executes these two jobs on a task queue, thereby compiling hello.swift into an object named hello.o, and linking hello.o into an executable named hello.

I hope that sheds some light on how the Swift driver behaves. Despite the length of this article, there's lot I didn't cover. For example, the Swift driver is also responsible for incremental compilation – if it determines that the result of a job is already available from a previous compilation, it won't run that job. I'll try to write more about how this works in a future article.

An Introduction to the Swift Compiler Driver

2017-11-04T05:00:00+00:00

This article contains some content especially for patrons. Although it reads coherently as-is, to read the full article, please consider supporting me on Patreon, or click here if you are already a patron. $10/month gives you access to all content I will ever write on this website.

The first article of this series on the Swift compiler explained that the compiler is just a command-line program, named swift or swiftc. This article explains what that command-line program does.

Specifically, this article explains the Swift "driver". This is the first bit of code that is executed when users invoke swift or swiftc on the command line.

Like any other C, C++, or Objective-C command-line program, swift and swiftc begin by executing an int main() function:

swift/tools/driver/driver.cpp

111  int main(int argc_, const char **argv_) {
...
207  }

The main() function calls out to code in a Swift compiler library, libswiftDriver, in order to break up a single invocation of swift or swiftc into smaller chunks of work. These smaller chunks are called "jobs".

Swift driver jobs by example: compiling Swift source code into an executable

For example, let's say I wanted to compile the following Swift source file, hello.swift, into an executable:

hello.swift

print("Hello!")

To produce an executable named hello, I'll invoke swiftc on the command line:

swiftc hello.swift -o hello

This produces an executable named hello, by first compiling the Swift code into a .o object file, and then linking that file into an executable named hello. I can request to see the exact series of jobs that led to the final hello executable by using the -driver-print-jobs option:

swiftc -driver-print-jobs hello.swift -o hello

This outputs two jobs:

The first is an invocation of swift -frontend that compiles the hello.swift source code file into an object file named hello-6ab6fd.o.
The second is an invocation of the linker, ld, which links the hello-6ab6fd.o object file with the Swift and Objective-C runtimes in order to produce the executable I requested: hello.

Here's the full output:

/Users/bgesiak/Source/apple/build/Ninja-ReleaseAssert/swift-macosx-x86_64/bin/swift \
    -frontend -c -primary-file hello.swift \
    -target x86_64-apple-darwin16.7.0 \
    -enable-objc-interop -color-diagnostics -module-name hello \
    -o /var/folders/ry/2ryfdsb56b30092626qprw6d3rb3ss/T/hello-6ab6fd.o

/usr/bin/ld \
    /var/folders/ry/2ryfdsb56b30092626qprw6d3rb3ss/T/hello-6ab6fd.o \
    -lobjc -lSystem -arch x86_64 -L /Users/bgesiak/Source/apple/build/Ninja-ReleaseAssert/swift-macosx-x86_64/lib/swift/macosx \
    -rpath /Users/bgesiak/Source/apple/build/Ninja-ReleaseAssert/swift-macosx-x86_64/lib/swift/macosx \
    -macosx_version_min 10.12.0 -no_objc_category_merging \
    -o hello

It would be tedious if users of the Swift compiler had to invoke two commands, swift -frontend and ld, every time they wanted to generate an executable from a Swift source code file. The Swift driver exists to make this more convenient: it translates a single invocation of swift or swiftc into a series of swift -frontend and ld jobs that do what the user wants.

My example above may have been too simple to demonstrate just how useful the Swift driver is – it only took two jobs to do what I wanted. However, compiling multiple Swift source files requires a separate swift -frontend job for each one. Producing a .swiftmodule from those source files requires invocations of swift -frontend -merge-modules, as well as other jobs. The Swift driver coordinates all of this work for me, translating a simple invocation such as swiftc -emit-module into what is sometimes dozens of jobs.

If that weren't enough, the Swift driver is also responsible for determining when jobs do not need to be run, because their output is already available and up-to-date. This is known as "incremental building". If you're interested in helping reduce the amount of time it takes for your Swift source code project to compile, you'll want to understand how the driver accomplishes this. It's a topic I plan on writing about at length in future articles.

When I think of the Swift compiler, I think of a program that parses the Swift source code in a file, outputs warnings or errors if that source code isn't syntactically correct, and then translates that source code into binary 0's and 1's that my operating system knows how to execute as a program. However, it's important to make a distinction: it's invocations of swift -frontend that are responsible for all that good stuff.

Invocations of swift and swiftc, without the -frontend argument, invoke the driver, not the frontend. The Swift driver doesn't type-check Swift source code, it generates jobs that invoke swift -frontend. Those invocations of swift -frontend, in turn, do all the cool compiler stuff I think of when I think of compilers, such as type-checking.

A bird's-eye view of the code in the Swift driver

The basic idea of the Swift driver is that it splits up invocations of swift and swiftc into other, more granular invocations of swift -frontend, ld, and other programs. These child invocations are called "jobs". So, how does it accomplish this? What does the source code in the Swift driver look like, at a high level?

The driver source code exists in two primary components: the driver executable C++ code, and the C++ libswiftDriver library code. The driver executable calls into the libswiftDriver library code to perform most of its tasks. libswiftDriver's interfaces are declared in its .h header files, in swift/include/swift/Driver. The implementation of those interfaces is in .cpp implementation files in swift/lib/Driver. The driver executable is implemented in .cpp files in swift/tools/driver, primarily swift/tools/driver/driver.cpp.

The above diagram isn't a complete graph of library dependencies for the swift and swiftc executables. Both the driver executable and libswiftDriver make use of other apple/swift and LLVM libraries, such as libswiftOption (for parsing command-line arguments like -frontend) and libLLVMSupport (which provides utility functions for reading and creating files, among other things). I'll cover these in more detail later.

CMake files, specifically swift/tools/driver/CMakeLists.txt and swift/lib/Driver/CMakeLists.txt, define how the swift and swiftc executables and how the libswiftDriver library are built.

How `swift`, `swiftc`, and libswiftDriver are built (and just what is the difference between `swift` and `swiftc`, anyway?)

I explained the Swift build system and how the swift executable is built in previous articles. To summarize those articles:

The apple/swift build system is defined primarily in CMake. The top-level swift/CMakeLists.txt file is executed by the cmake executable, and that file in turn calls the CMake built-in function add_subdirectory() in order to execute the nested CMakeLists.txt files in other directories.
To understand how a particular set of source files are built, it makes sense to first look at the CMakeLists.txt file that exists within the same directory as the source files.
The CMakeLists.txt file at the same level as the Swift driver executable source file, swift/tools/driver/driver.cpp, is swift/tools/driver/CMakeLists.txt. That CMakeLists.txt file calls the add_swift_host_tool() function, a custom CMake function defined in the apple/swift project, in order to define how the swift executable should be built.

The swift/tools/driver/CMakeLists.txt file doesn't just define how the swift executable is built. It also defines a symlink, swiftc, that points to swift:

swift/tools/driver/CMakeLists.txt

 1  add_swift_host_tool(swift
 2    api_notes.cpp
 3    driver.cpp
 4    autolink_extract_main.cpp
 5    modulewrap_main.cpp
 6    swift_format_main.cpp
 7    LINK_LIBRARIES
 8      swiftDriver
 9      swiftFrontendTool
10    LLVM_COMPONENT_DEPENDS
11      DebugInfoCodeView
12    SWIFT_COMPONENT compiler
13  )
..
34  add_swift_tool_symlink(swiftc swift compiler)
35  add_swift_tool_symlink(swift-autolink-extract swift autolink-driver)
36  add_swift_tool_symlink(swift-format swift editor-integration)

The name of the function add_swift_tool_symlink() is fairly straightforward, and so I can guess that it creates a symlink named swiftc that points to the executable swift. To confirm this guess, I can use the techniques I described in Reading and Understanding the CMake in apple/swift. A git grep for function(add_swift_tool_symlink reveals the function's definition in swift/cmake/modules/AddSwift.cmake. The function calls through to two LLVM CMake functions. Two more git grep invocations, this time in the LLVM repository, can point me to those functions' definitions, in llvm/cmake/modules/AddLLVM.cmake.

As the CMake confirms, the swiftc executable is really just a symlink to the swift executable. But why create a symlink?

Swift "driver modes", or: "A `swift` by any other name"

You may have noticed, in the CMake snippet above, that not only does the apple/swift build system produce a swiftc symlink, it also creates symlinks named swift-autolink-extract and swift-format. Each one of these represents a different "driver mode" or, as it's referenced in the libswiftDriver source code, a swift::driver::Driver::DriverKind:

swift/include/swift/Driver/Driver.h

120    enum class DriverKind {
121      Interactive,     // swift
122      Batch,           // swiftc
123      AutolinkExtract, // swift-autolink-extract
124      SwiftFormat      // swift-format
125    };

Each of these driver modes:

Accepts a different set of arguments. For example, swiftc accepts the argument -dump-ast, but swift -dump-ast is invalid.
Performs a different set of tasks. For example, swift-format tidies up the whitespace and indentation in Swift source code files. It does not perform any of the compilation that swiftc does.

The "driver mode" is determined by libswiftDriver, which contains code that checks the name of the executable that is currently being run:

swift/lib/Driver/Driver.cpp

 79  void Driver::parseDriverKind(ArrayRef<const char *> Args) {
...
 94    Optional<DriverKind> Kind =
 95    llvm::StringSwitch<Optional<DriverKind>>(DriverName)
 96    .Case("swift", DriverKind::Interactive)
 97    .Case("swiftc", DriverKind::Batch)
 98    .Case("swift-autolink-extract", DriverKind::AutolinkExtract)
 99    .Case("swift-format", DriverKind::SwiftFormat)
100    .Default(None);
...
106  }

That is, when I invoke swift on the command line, that causes libswiftDriver to run in DriverKind::Interactive. swiftc results in DriverKind::Batch. In other parts of libswiftDriver, the current DriverKind is referenced when deciding which command-line options to accept as valid, or which actions to perform.

So while swiftc executes the exact same driver code as swift, the fact that it's named swiftc causes libswiftDriver to behave differently.

This is actually a fun quirk in how Swift works: you can't invoke swift through a symlink the driver code isn't aware of. For example, I tried creating a symlink to my swift executable, like so:

# Change directories to where my 'swift' executable lives:
cd ~/local/Source/apple/build/Ninja-ReleaseAssert/swift-macosx-x86_64/bin/

# Create an alias: 'my-swift-alias' points to 'swift':
ln -s swift my-swift-alias

# Attempt to print the help text for 'swift', by invoking
# my alias:
my-swift-alias -help

Sure enough, this results in an error. The Swift driver doesn't recognize the my-swift-alias driver mode:

<unknown>:0: error: invalid value 'my-swift-alias' in '--driver-mode='

As the error message hints at, I can override the Swift driver behavior that infers the driver mode from the name of the executable by using the --driver-mode= command-line option:

# Print out help text as if I were invoking
# the 'swift' executable:
my-swift-alias --driver-mode=swift -help

Another look at the driver executable and libswiftDriver source code, and next steps

To recap:

The CMake code for the driver executable not only defines an executable named swift, it also defines symlinks pointing to that executable, such as swiftc and swift-format.
The driver determines the mode in which it's being run in, in part based on the name of the executable or symlink its being run with, and then it parses its command-line options based on that driver mode.
Based on its mode and the arguments passed to it, the driver executable calls out to libswiftDriver to build a series of "jobs". These jobs are invocations of swift -frontend, ld, or other executables. It's swift -frontend that performs the actual type-checking and compilation of Swift source code.

Parsing command-line arguments and splitting up a driver invocation into child jobs involves a lot of different classes, most of which are defined in libswiftDriver. To name just a few, there's swift::Driver, swift::Compilation, swift::driver::ToolChain, swift::Action, swift::JobAction, swift::Job, and more. I'll go over the interactions between these classes in more detail in the next article in this series, and then I'll start writing about incremental compilation and other advanced features in libswiftDriver.

In the meantime, try exploring this treemap of the source code in the Swift driver. Each rectangle is sized proportional to the number of source lines of code it contains. Click or tap on a rectangle to view it in GitHub.

The majority of the source code we'll cover next time is in three files:

Reading and Understanding the CMake in apple/swift

2017-09-16T17:00:00+00:00

In my last post, I invoked cmake in order to build apple/swift. When I invoked cmake, I gave it the path to a source code directory. cmake then executes the CMakeLists.txt file in that directory.

In the case of apple/swift, that means the swift/CMakeLists.txt CMake script file is executed.

swift/CMakeLists.txt, in turn, calls the CMake function add_subdirectory() to include its child directories. The CMakeLists.txt files in those subdirectories then include their children, and so on:

~/local/Source/apple/swift/CMakeLists.txt
    lib/CMakeLists.txt
        Driver/CMakeLists.txt
            Action.cpp, Compilation.cpp, ...
        Frontend/CMakeLists.txt
            Frontend.cpp, FrontendOptions.cpp, ...
        ...
     tools/CMakeLists.txt
        ...
     test/CMakeLists.txt
        ...

Here's an animated "time-lapse" of this process.

This recursive execution of build system code is referred to as the recursive make pattern. The LLVM, Clang, and apple/swift projects all structure their CMake code following this pattern.

By convention, a CMakeLists.txt file in a directory describes how the source code in that directory is built. For example, to understand how the source code in swift/lib/Driver is compiled, I would first read the swift/lib/Driver/CMakeLists.txt file.

In addition, apple/swift defines helper CMake functions in the swift/cmake/modules directory. The top-level swift/CMakeLists.txt file includes these CMake functions by first appending the directory to the CMake module path…

swift/CMakeLists.txt

4  list(APPEND CMAKE_MODULE_PATH
5      "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules")

…and then including them by name by using the CMake include() function:

swift/CMakeLists.txt

15  include(SwiftUtils)        # Loads swift/cmake/modules/SwiftUtils.cmake.
16  include(CheckSymbolExists) # Loads swift/cmake/modules/CheckSymbolExists.cmake.

Including the two files above makes functions defined therein available to all CMake scripts in the project. For example, swift/cmake/modules/SwiftUtils.cmake defines a function named precondition(), which halts CMake with an error at configuration time if a variable is not defined.

I find reading the CMake in apple/swift to be straightforward, because the order in which the CMake is executed is clear. All I have to do is read from the top of swift/CMakeLists.txt to the bottom, following any calls to include() or add_subdirectory().

Reading apple/swift CMake by example: the `swift` executable

The above description should be enough to determine how, for example, the swift compiler executable is configured and built. Here's how I found out:

First, I know that swift is, at its core, just a command-line C++ program, so it must have a main() function I can search for:

git \
    -C ~/local/Source/apple/swift \
    grep "int main("

The apple/swift project contains many C++ programs, and so this search returns several results. Looking at the returned file names and their contents, however, it soon becomes clear that swift/tools/driver/driver.cpp is the file that defines the entry point to the swift executable.

As I mentioned above, I can usually find out how a source code file is compiled by looking for a CMakeLists.txt file in the same directory. Here, swift/tools/driver/CMakeLists.txt is in the same directory as driver.cpp. The first few lines of code in that CMake file are exactly what I'm looking for:

swift/tools/driver/CMakeLists.txt

 1  add_swift_host_tool(swift
 2    api_notes.cpp
 3    driver.cpp
 4    autolink_extract_main.cpp
 5    modulewrap_main.cpp
 6    swift_format_main.cpp
 7    LINK_LIBRARIES
 8      swiftDriver
 9      swiftFrontendTool
10    LLVM_COMPONENT_DEPENDS
11      DebugInfoCodeView
12    SWIFT_COMPONENT compiler
13  )

It appears this add_swift_host_tool() function takes the name of the executable to be built (swift), a list of source files (api_notes.cpp driver.cpp ...), a list of libraries to link (LINK_LIBRARIES swiftDriver ...), a list of LLVM dependencies, and something called a SWIFT_COMPONENT.

Based on its name, I can guess that the add_swift_host_tool() function is not built into CMake, so to understand exactly what it does, I can't consult the CMake documentation. Instead, I can git grep for its definition:

git \
    -C ~/local/Source/apple/swift \
    grep --line-number \
    "function(add_swift_host_tool"

The above command outputs:

cmake/modules/AddSwift.cmake:2104:function(add_swift_host_tool executable)

So it looks like the add_swift_host_tool() function is defined in swift/cmake/modules/AddSwift.cmake.

For add_swift_host_tool() to be called from within swift/tools/driver/CMakeLists.txt, the AddSwift.cmake module must have been included at some point. Sure enough, a git grep for "include(AddSwift)" shows that it is included from within swift/CMakeLists.txt. No magic here!

Here's the definition of add_swift_host_tool():

swift/cmake/modules/AddSwift.cmake

2104  function(add_swift_host_tool executable)
2105    cmake_parse_arguments(
2106       ADDSWIFTHOSTTOOL # prefix
2107       "" # options
2108       "" # single-value args
2109       "SWIFT_COMPONENT" # multi-value args
2110       ${ARGN})
2111  
2112    # Create the executable rule.
2113    add_swift_executable(${executable} ${ADDSWIFTHOSTTOOL_UNPARSED_ARGUMENTS})
....  
2130  endfunction()

First, the add_swift_host_tool() function parses any arguments it has been invoked with, by using the CMake function cmake_parse_arguments(). Then, it calls add_swift_executable() with a subset of those arguments.

Like add_swift_host_tool(), the add_swift_executable() function is also defined within the cmake/modules/AddSwift.cmake module. It's documented as "add an executable for the host machine."

The "host machine" part of the documentation above is important. apple/swift CMake is responsible for describing how to build not only executables like swift, but also the Swift runtime and standard library. The Swift runtime could be used on the macOS machine I'm using to build apple/swift, or it could be run on an iOS device, or a tvOS device, or even an Android device.

apple/swift CMake refers to the machine I'm using to configure and build apple/swift as the host machine. It refers to the platform I'm building for as the target. macOS, iOS, tvOS, and Android are all valid targets.

add_swift_executable() describes how to build an executable for the host machine. In my case, that's the macOS machine I'm using to configure and build apple/swift:

swift/cmake/modules/AddSwift.cmake

2063   function(add_swift_executable name)
....   
2083     _add_swift_executable_single(
2084         ${name}
2085         ${SWIFTEXE_SOURCES}
2086         DEPENDS ${SWIFTEXE_DEPENDS}
2087         LLVM_COMPONENT_DEPENDS ${SWIFTEXE_LLVM_COMPONENT_DEPENDS}
2088         LINK_LIBRARIES ${SWIFTEXE_LINK_LIBRARIES}
2089         SDK ${SWIFT_HOST_VARIANT_SDK}
2090         ARCHITECTURE ${SWIFT_HOST_VARIANT_ARCH}
2091         ${SWIFTEXE_EXCLUDE_FROM_ALL_FLAG}
2092         ${SWIFTEXE_DONT_STRIP_NON_MAIN_SYMBOLS_FLAG}
2093         ${SWIFTEXE_DISABLE_ASLR_FLAG})
2094   endfunction()

All add_swift_executable() does is parse arguments and call another function, _add_swift_executable_single(). When it does so, it sends the argument SDK ${CMAKE_HOST_VARIANT_SDK}.

The term "SDK" here is misleading; it means the target for which to build the executable. In this case, its being set as the value of the variable CMAKE_HOST_VARIANT_SDK, which is set to "OSX" at a higher level of the recursive CMake.

How can I be sure the value of CMAKE_HOST_VARIANT_SDK is being set to "OSX"?

There are two ways to confirm the CMAKE_HOST_VARIANT_SDK that's being passed as an argument to _add_swift_executable_single(). The first is by using the CMake function message():
cmake/modules/AddSwift.cmake
++++     message(
++++       STATUS
++++       "Adding Swift executable '${name}' for "
++++       "SDK '${SWIFT_HOST_VARIANT_SDK}'")
2083     _add_swift_executable_single(
2084           ${name}
2085           ${SWIFTEXE_SOURCES}
....   
Adding the above results in the following output when I run cmake to configure the project:
-- Adding Swift executable 'swift' for SDK 'OSX'
Another way is to run cmake using the --trace-expand option. This prints to stderr every single line of CMake that is executed, along with the values of the variables on those lines. That's a ton of output, but I can use grep to filter it for the specific line that swift is added:
cmake --trace-expand \
    ... 2>&1 | grep "_add_swift_executable_single(swift "
That outputs the following:
/Users/bgesiak/local/Source/apple/standalone/swift/cmake/modules/AddSwift.cmake(2083):
  _add_swift_executable_single(
      swift 
      api_notes.cpp;driver.cpp;autolink_extract_main.cpp;modulewrap_main.cpp;swift_format_main.cpp
      DEPENDS
      LLVM_COMPONENT_DEPENDS DebugInfoCodeView
      LINK_LIBRARIES swiftDriver;swiftFrontendTool
      SDK OSX
      ARCHITECTURE x86_64
  )
I can see in both of the outputs above that SDK OSX is set.

The _add_swift_executable_single() function is the one that finally calls into the built-in CMake function add_executable() in order to define the swift executable target:

swift/cmake/modules/AddSwift.cmake

1825   function(_add_swift_executable_single name)
....   
1907     add_executable(${name}
1907         ${SWIFTEXE_SINGLE_EXCLUDE_FROM_ALL_FLAG}
1907         ${SWIFTEXE_SINGLE_SOURCES}
1907         ${SWIFTEXE_SINGLE_EXTERNAL_SOURCES})
....   
1937     target_link_libraries("${name}" ${SWIFTEXE_SINGLE_LINK_LIBRARIES} ${SWIFTEXE_SINGLE_LINK_FAT_LIBRARIES})
....   
1942   endfunction()

You may recall that add_executable() is the same function I used to describe the hello executable, in my toy CMake example from my last post. The function is how to describe, in CMake, that an executable is to be built.

In addition, _add_swift_executable_single() then calls target_link_libraries(). This is another built-in CMake function. It specifies that the swift executable depends upon the swiftDriver and swiftFrontendTool libraries.

A recap of how apple/swift CMake describes the `swift` executable

In summary, the story of how apple/swift CMake describes the swift executable is as follows:

When cmake is run to configure the project, swift/CMakeLists.txt is executed. It adds swift/cmake/modules to the CMake module path, and it includes swift/cmake/modules/AddSwift.cmake, which defines the add_swift_host_tool() function.
swift/CMakeLists.txt then calls add_subdirectory() on each of its subdirectories, including swift/tools. Each of these subdirectories contains a CMakeLists.txt file that includes their subdirectories in turn. swift/tools/CMakeLists.txt is no exception, of course, and it calls add_subdirectory() on swift/tools/driver.
swift/tools/driver/CMakeLists.txt contains the CMake code that describes how to build the swift executable. It invokes the add_swift_host_tool() function. This eventually calls CMake's built-in functions, add_executable() and target_link_libraries(), in order to describe a swift executable that is linked to swiftDriver and swiftFrontend.

The challenge for a contributor who hopes to improve Swift's CMake isn't that the CMake behaves in a magical way, or does unintuitive things. Rather, the challenge is simply that there is a lot of CMake code: _add_swift_executable_single() alone is nearly 100 lines long.

The entire apple/swift project has 189 CMake files, with over 8,000 significant lines of code. It also makes use of CMake functions defined in LLVM, which itself has 347 CMake files that contain over 10,000 significant lines!

As a result, it's not practical for me to describe how every single component in apple/swift is configured. However, if you're interested in how, for example, the sil-opt executable is configured, this post has hopefully shown how you can find out for yourself.

In an upcoming post, I'll use the methods above to learn and write about how some of the main components of the apple/swift project are configured:

The Swift compiler libraries that are used by the swift executable. I'll explore how LLVM tools such as tablegen are used when, for example, defining swift command-line options.
The Swift runtime and standard library. I'll write about the handle_swift_sources() function, which describes how Swift source files should be compiled and included in the finished library.
The Swift test suite. I'll explore how a Python script executes the unique set of tests used by the project.

Afterwards, I'll also write about specific CMake configurations that allow me to build just what I'm working on, in order to iterate quicker on Swift compiler development.

The Swift Compiler's Build System

2017-09-10T17:00:00+00:00

My last post explained that invoking swift/utils/build-script compiles the C++ source code in the apple/swift project and produces a swift compiler executable.

swift/utils/build-script is a Python script that invokes another script, swift/utils/build-script-impl. That, in turn, invokes cmake, in order to configure and then compile the source code in apple/swift.

I'll refer to this process as the Swift build system, and I'll explain it in detail below.

There are many benefits to having a deep understanding of the build system:

When making changes to the apple/swift project, I typically edit source code, then compile that code and run its tests. By understanding the various ways to invoke the build system, I'm able to compile and test the subset of the project that's related to my changes. It takes several minutes to run the entire Swift test suite, but only a fraction of a second to run just the tests I care about.
The build system has a great deal of room for improvement, so I found it to be a very easy place to start contributing to Swift. At the point that I received commit access to apple/swift, over half of my pull requests had been related to the build system.
Improvements to the build system are impactful: a small change could result in faster builds of the Swift compiler, which in turn would speed up the work of the core team, other contributors, and CI. This recent change to the LLVM build system is a good example: it reduced the number of actions performed by the build by several thousand actions, just by modifying seven lines of build system code.

I'll begin with the build system's core: CMake.

Using CMake: An example

The cmake executable reads script files that describe how a software project builds. For example, here's how I'd use it to build a simple C++ program, composed of a single source file, /tmp/src/hello.cpp:

/tmp/class/hello.cpp

#include <iostream>

int main() {
  std::cout << "Hello!" << std::endl;
  return 0;
}

I can describe how to build this program using the CMake language, in a file named src/CMakeLists.txt:

/tmp/src/CMakeLists.txt

# CMake version 3.2 or greater must
# be used to configure this project.
cmake_minimum_required(VERSION 3.2)

# The name of the project is 'Hello'.
project(Hello)

# When built, create an executable
# named 'hello', by compiling the
# source file 'hello.cpp'.
add_executable(hello
               hello.cpp)

Running cmake on the command line generates build files that are used to actually build the project:

# Read the CMakeLists.txt in the '/tmp/src' directory,
# and generate build files at '/tmp/build'.
cmake -H/tmp/src -B/tmp/build

I can then compile the hello executable by executing the build files in /tmp/build. By default, CMake generates a Makefile, which I can execute like so:

make -C /tmp/build

Running the above command compiles and links the executable /tmp/build/hello. Running that program outputs the expected text, "Hello!".

It's important to note that CMake itself doesn't compile and link the hello executable. CMake generates build files, and those build files compile and link hello. I can instruct CMake to generate different kinds of build files, including an Xcode project:

cmake -H/tmp/src -B/tmp/build-with-xcode -G Xcode

The above command generates /tmp/build-with-xcode/Hello.xcodeproj. I can open Hello.xcodeproj in Xcode and build hello by clicking the "Run" button:

I can also build hello via the command line, by invoking xcodebuild:

xcodebuild -project /tmp/build-with-xcode/Hello.xcodeproj

Notice how, when I didn't specify a build file generator using the cmake -G argument, cmake generated a Makefile, that I could build by invoking make. The Xcode project, on the other hand, was built by invoking xcodebuild. Remembering which invocation to use is tiresome, so cmake provides a convenient way to build using whichever files were generated:

# Build using the Makefile in the
# /tmp/build directory.
cmake --build /tmp/build

# Build using the .xcodeproj in the
# /tmp/build-with-xcode directory.
cmake --build /tmp/build-with-xcode

CMake has many more features than just the three lines from my example above. I'll cover additional concepts below, as we encounter them in apple/swift's CMake code. To learn about CMake itself, try LLVM's CMake primer, or the latest official CMake documentation.

Why does apple/swift use CMake?

If you're an iOS developer, you've likely used Xcode project files to specify how your iOS application is built. So why couldn't the apple/swift project just use Xcode, too? The simple hello example above demonstrates several reasons why:

Xcode is a macOS application; it can't be used to build apple/swift on Linux machines. CMake, on the other hand, can generate one set of build files on Linux, or an Xcode project on macOS, or even a Visual Studio "solution" file on Windows. CMake's versatility allows it to generate the best build files for any platform.
Xcode project files are massive XML files that contain a large number of automatically generated ID strings. If you've worked on a small team of even two or three iOS developers, you know how hard it is to resolve multiple sets of changes to such files. Now just imagine resolving those conflicts across the 500+ contributors to apple/swift! Because CMake uses a plain text scripting language, it's easy to read and modify.
Some of CMake's generators build projects faster than Xcode can. For example, try reconfiguring the hello example above to use cmake -G Ninja. This generates Ninja build files, which are incredibly fast.

Building apple/swift with CMake

The apple/swift project is essentially the same as my "Hello" project above. Yes, it is a larger project with more CMake code, but it can be configured and built in the same way "Hello" can. Doing so is a useful exercise in understanding the apple/swift build infrastructure.

First, I'll clone the source code from the apple/swift project, as well as the three projects it depends upon:

apple/swift-cmark: A Markdown parsing library, used by apple/swift when parsing Markdown documentation and comment blocks. This is Apple's fork of commonmark/cmark.
apple/swift-llvm: apple/swift uses LLVM in a variety of ways. The most prominent is as a "backend": a library that generates the 0's and 1's that are capable of running on the target machine, whether that be an iOS armv7 device, a macOS x86_64 device, or even Android armv7. But apple/swift also uses LLVM utilities such as lit and FileCheck, two tools used to run the apple/swift test suite.
apple/swift-clang: apple/swift makes use of Clang, a C and C++ compiler, as part of its C and Objective-C interoperability.

I'll clone these projects into a directory named ~/local/Source/apple/standalone:

git clone https://github.com/apple/swift-cmark.git \
    ~/local/Source/apple/standalone/swift-cmark

git clone https://github.com/apple/swift-llvm.git \
    ~/local/Source/apple/standalone/swift-llvm

# A bug in Swift's CMake requires this repository
# to be named 'clang', not 'swift-clang'.
# https://bugs.swift.org/browse/SR-5778
git clone https://github.com/apple/swift-clang.git \
    ~/local/Source/apple/standalone/clang

git clone https://github.com/apple/swift.git \
    ~/local/Source/apple/standalone/swift

Before I can configure and build apple/swift, I need to build its dependencies. First is apple/swift-cmark. I'll generate Ninja build files because they're faster:

# First configure.
cmake \
    -H~/local/Source/apple/standalone/swift-cmark \
    -B~/local/Source/apple/standalone/swift-cmark-build \
    -G Ninja

# Then build.
cmake --build \
    ~/local/Source/apple/standalone/swift-cmark-build

Next up is apple/swift-llvm and apple/swift-clang. LLVM"s CMake defines cache entry settings that allow users to build both LLVM and Clang at once. As an example of a cache entry setting, consider the following, defined in swift-llvm/CMakeLists.txt:

swift-llvm/CMakeLists.txt

124  set(LLVM_ENABLE_PROJECTS "" CACHE STRING
125          "Semicolon-separated list of projects to build (${LLVM_ALL_PROJECTS}), or \"all\".")

Users are able to set this value by invoking cmake -DLLVM_ENABLE_PROJECTS="foo" when configuring apple/swift-llvm. The empty string "" is the default value used when the user does not specify -DLLVM_ENABLE_PROJECTS= on the command line.

A full explanation of LLVM and Clang's CMake build system is outside of the scope of this article – perhaps I'll write about it more next year – but the following invocations configure and build apple/swift-llvm and apple/swift-clang:

# First configure.
cmake \
  -H~/local/Source/apple/standalone/swift-llvm \
  -B~/local/Source/apple/standalone/swift-llvm-build \
  -G Ninja \
  -DLLVM_ENABLE_PROJECTS=clang

# Then build.
cmake --build \
    ~/local/Source/apple/standalone/swift-llvm-build

# Finally, create a symlink from the build
# directory to the C++ headers included with Xcode.
# If you're following along, make sure the
# path to your Xcode beta is the same as below.
ln -s \
    /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++ \
    ~/local/Source/apple/standalone/swift-llvm-build/include

Now, I can finally configure and build apple/swift itself. Like LLVM, apple/swift CMake defines many settings; for example, CMAKE_BUILD_TYPE is used to determine whether the swift executable is built with debugging symbols or not. I'll go into more of these options below, but for now the following invocations configure and build apple/swift:

# First configure.
cmake \
  -H~/local/Source/apple/standalone/swift \
  -B~/local/Source/apple/standalone/swift-build \
  -G Ninja \
  -DCMAKE_BUILD_TYPE="Debug" \
  -DSWIFT_PATH_TO_CMARK_SOURCE=$HOME/local/Source/apple/standalone/swift-cmark \
  -DSWIFT_PATH_TO_CMARK_BUILD=$HOME/local/Source/apple/standalone/swift-cmark-build \
  -DSWIFT_PATH_TO_LLVM_SOURCE=$HOME/local/Source/apple/standalone/swift-llvm \
  -DSWIFT_PATH_TO_LLVM_BUILD=$HOME/local/Source/apple/standalone/swift-llvm-build \
  -DSWIFT_PATH_TO_CLANG_SOURCE=$HOME/local/Source/apple/standalone/clang \
  -DSWIFT_PATH_TO_CLANG_BUILD=$HOME/local/Source/apple/standalone/swift-llvm-build

# Then build.
cmake --build \
    ~/local/Source/apple/standalone/swift-build

Once the build finishes, the built swift compiler executable is located at ~/local/Source/apple/standalone/swift-build/bin/swift. It works exactly as well as the executable I built using swift/utils/build-script in my last post.

What do `swift/utils/build-script` and `swift/utils/build-script-impl` do?

The apple/swift utils/build-script is a Python script that performs the exact same actions I did above:

Use CMake to configure and build apple/swift-cmark.
Use CMake to configure and build apple/swift-llvm and apple/swift-clang, as well as perform post-build configuration like symlinking C++ headers.
Use CMake to configure and build apple/swift.

swift/utils/build-script accepts command-line arguments to specify how the project is built. The logic in the Python script translates these arguments into arguments to swift/utils/build-script-impl, a shellscript that in turn invokes cmake. For example, when a user invokes:

~/local/Source/apple/swift/utils/build-script --release

This in turn calls swift/utils/build-script-impl:

~/local/Source/apple/swift/utils/build-script-impl \
    --cmark-build-type=Release \
    --llvm-build-type=Release \
    --swift-build-type=Release

When invoked with these arguments, swift/utils/build-script-impl invokes cmake -DCMAKE_BUILD_TYPE=Release to configure and build apple/swift-cmark, apple/swift-llvm (and thus simultaneously apple/swift-clang), and apple/swift.

The shellscript swift/utils/build-script-impl is an unnecessary complication. Ideally, the Python script swift/utils/build-script would invoke cmake directly. A Swift bug report exists to get rid of swift/utils/build-script-impl, but that's easier said than done: the script contains over 3,000 lines of code.

swift/utils/build-script places the build products for each project in a separate directory. The full path of the directory is based upon the options used; for the invocation swift/utils/build-script --release --debug-swift, the products are placed at:

~/local/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/
    cmark-macosx-x86_64/ # swift-cmark build products.
    llvm-macosx-x86_64/  # swift-llvm and swift-clang build products.
    swift-macosx-x86_64/ # swift build products.

It's clear that, compared to invoking cmake multiple times to configure and build three separate projects, one invocation of swift/utils/build-script is much simpler for new apple/swift contributors:

Users don't need to know how to use cmake on the command line.
Users don't need to know the particular CMake settings for each project.

In addition, some projects related to apple/swift, such as apple/swift-corelibs-foundation, do not include CMake files that describe how to build the project. swift/utils/build-script and swift/utils/build-script-impl take care of building these as well.

However, the simplicity comes at a cost:

In order to add a new build setting, contributors to apple/swift need to add the setting to three places: the CMake, the shellscript swift/utils/build-script-impl, and the Python script swift/utils/build-script. Even if swift/utils/build-script-impl were finally deleted one day, contributors would still need to modify both CMake and Python in order to add a new setting.
In cases where the documentation from swift/utils/build-script --help is not completely clear, users need to trace how an option is translated between Python, shellscript, and CMake. In addition, some options are only accessible via CMake; for example, an option to build static libraries for sourcekitd, SOURCEKITD_BUILD_STATIC_INPROC, is only available in CMake.
swift/utils/build-script performs three configure and build actions each time it is invoked, regardless of whether the user has modified any files in apple/swift-cmark, apple/swift-llvm, or apple/swift-clang. Although most generators are smart enough to do nothing if no files have changed, the extra configuration steps makes swift/utils/build-script slower than invoking cmake --build directly.

This is a frequent stumbling block for new contributors to apple/swift.

Many experienced contributors recommend newcomers invoke ninja -C ~/local/Source/apple/build/swift-macosx-x86_64 directly in order to perform faster incremental builds of apple/swift. But a new contributor who doesn't know what swift/utils/build-script does will have a lot of questions:

"What is ninja?"

"What is the swift-macosx-x86_64 directory?"

"I used swift/utils/build-script --xcode in order to generate an Xcode project; attempting to invoke ninja results in an error. Why?"

Contributors familiar with the interaction between swift/utils/build-script and CMake know that swift/utils/build-script --xcode eventually calls cmake -G Xcode when configuring the apple/swift project, and so it stands to reason that invoking ninja on the configured project would not work. A slight improvement would be to recommend newcomers use cmake --build instead.

One last trick: "In-tree" builds of apple/swift

Configuring and building apple/swift using direct invocations of cmake gave me a new appreciation of the work done by swift/utils/build-script and swift/utils/build-script-impl. But it also made me notice that configuring and building Clang did not require a cmake invocation of its own.

The recommended way to build the Clang project is slightly different than the way I built apple/swift with CMake. Whereas building apple/swift required me to build LLVM first, Clang is built as part of the LLVM project. LLVM's CMake automatically detects when Clang is present at llvm/tools/clang. If it is, then LLVM's CMake includes Clang in the build.

It turns out that apple/swift can be built in this way as well:

# Clone apple/swift-llvm.
git clone https://github.com/apple/swift-llvm.git \
    ~/local/Source/apple/intree/swift-llvm

# Clone cmark, clang, and swift
# into swift-llvm/tools.
git clone https://github.com/apple/swift-cmark.git \
    ~/local/Source/apple/intree/swift-llvm/tools/cmark
git clone https://github.com/apple/swift-clang.git \
    ~/local/Source/apple/intree/swift-llvm/tools/clang
git clone https://github.com/apple/swift.git \
    ~/local/Source/apple/intree/swift-llvm/tools/swift

Configuring and building apple/swift "in-tree" requires fewer options to be specified on the command-line, because the paths to the apple/swift-llvm, apple/swift-cmark, and apple/swift-clang source and build directories are inferred by Swift's CMake code:

# First configure.
cmake \
    -H~/local/Source/apple/intree/swift-llvm \
    -B~/local/Source/apple/intree/build \
    -G Ninja

# Then build: first Clang, and then Swift.
cmake --build \
    ~/local/Source/apple/intree/build \
    -- clang swift

Compared to the "standalone" build I first tried, in-tree builds of apple/swift don't require me to remember to recompile apple/swift-llvm. If I make any changes to the apple/swift-llvm source code, they will be recompiled the next time I invoke cmake --build.

However, in-tree builds are not the official documented way to build apple/swift; if I encounter an error related to the build system, it's on me to file a bug and maybe even fix it myself.

For newcomers to apple/swift, I would recommend either using swift/utils/build-script, or the standalone setup.

Summary

apple/swift's build system has three main components: the CMake that describes how the project is built, and the build-script and build-script-impl that are responsible for invoking cmake.
Because CMake generates build files, but does not build the project itself, it's possible to build with your tool of choice: ninja, Xcode, or something else.
The LLVM family of projects frequently references two kinds of builds: "in-tree" builds, in which projects reside within the llvm/tools directory, and "standalone" builds, which have the projects reside elsewhere. "In-tree" builds of llvm/tools/swift are more uncommon; the recommended way to build is "standalone". Builds that use swift/utils/build-script are standalone.

Next Sunday morning, I'll post a "deep dive" into apple/swift's CMake code: how it uses a "recursive make" pattern, examples of how I read the code to see what it does, and some summaries of how things like the compiler itself, the standard library, and the test suite are built.

Getting Started with Swift Compiler Development

2017-08-23T17:00:00+00:00

Over the next few months, I'll be writing a guide to each section of the Swift compiler source code: swift/lib/Driver, swift/lib/Frontend, swift/lib/Parse, and so on. First, here's how I set up a workspace for building and making patches to the Swift compiler.

Being able to modify, experiment with, and attach a debugger to my own copy of the Swift compiler has been crucial to my learning process. If you want to get started contributing, try following along with the setup below.

Step 0: Minimum disk space and hardware

The apple/swift README recommends at least 20GB of free disk space for build products. In addition, a powerful CPU will reduce the amount of time it takes to build the apple/swift project. I use a MacBook Pro.

No matter the CPU, I always spend the majority of my time reading the code. Chances are your machine at home is perfectly capable of building apple/swift. If not, use a $20/month Digital Ocean "droplet" running Ubuntu 16.04. It can compile apple/swift in a few hours.

Step 1: Clone apple/swift

The instructions in the apple/swift README have everything anyone would ever want to know. Based on that, I went on the command line and entered the following three commands:

# 1. Create the directory where apple/swift
#    and its accompanying repositories will
#    live.
mkdir ~/local/Source/apple

# 2. Clone apple/swift within the directory
#    created in step (1).
git clone \
    https://github.com/apple/swift.git \
    ~/local/Source/apple/swift

# 3. Run a script inside the apple/swift
#    repository that automatically downloads
#    the other repositories you'll need.
~/local/Source/apple/swift/utils/update-checkout --clone

Step 2: Build and invoke my own copy of the Swift compiler

In order to build a copy of the Swift compiler that I can attach a debugger to, I run the following command:

~/local/Source/apple/swift/utils/build-script \
    --release \
    --debug-swift

If the command fails, I double- and triple-check that I have all the prerequisites listed in the apple/swift README installed. The most common cause of failure for me, on macOS, is because I don't have the latest beta version of Xcode installed.

The utils/build-script executable builds the libraries and executables in the apple/swift project. Passing it the combination of the --release and --debug-swift options makes it build the Swift compiler with debugging symbols. Debugging symbols take up a lot of extra hard drive space, and the build takes a lot longer with them included, but having them allows me to attach the lldb debugger to the Swift compiler to learn what it's doing. That's invaluable to me.

The build script will produce a ton of build products, including a Swift compiler executable, swiftc. The location of this executable depends on the options passed to utils/build-script. Having used the options above, I can invoke my swiftc executable at the following path:

~/local/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swiftc --version

The command above should output version information such as the following:

Swift version 4.1-dev (LLVM 260a9efcdb, Clang 6af0830132, Swift 539894568f)
Target: x86_64-apple-darwin16.7.0

And of course, I can use my built swiftc to compile and run Swift source code. For example, I created a new file named hello.swift and typed the following:

hello.swift

let greeting: String? = "Hello, Swift!"
print(greeting)

I can compile and run this file using my compiler, like so:

# 1. Compiles the Swift source code to produce an
#    executable named 'hello'.
#    (This also outputs some warnings -- more on
#    these in the next section.)
~/local/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swiftc \
    hello.swift

# 2. Executes the program. This prints
#    'Optional("Hello, Swift!")' to the console.
./hello

Step 3: Modify the Swift compiler

Compiling our program with swiftc hello.swift above produced an executable, but the compiler also output some warnings:

hello.swift:2:9: warning: string interpolation produces a debug description for an optional value; did you mean to make this explicit?
print("\(greeting)")
        ^~~~~~~~~~
hello.swift:2:10: note: use 'String(describing:)' to silence this warning
print("\(greeting)")
        ~^~~~~~~~~
         String(describing:  )
hello.swift:2:10: note: provide a default value to avoid this warning
print("\(greeting)")
        ~^~~~~~~~~
                  ?? <#default value#>

Just for fun, instead of suggesting the programmer provide a default value with ?? <#default value#>, I'll modify the compiler so that it suggests using a force unwrap ! instead.

To do so, I'll open swift/include/swift/AST/DiagnosticsSema.def and swift/lib/Sema/MiscDiagnostics.cpp with a text editor and make the following changes:

~/local/Source/apple/swift/include/swift/AST/DiagnosticsSema.def

2732  NOTE(default_optional_to_any,none,
2733       "provide a default value to avoid this warning", ())
++++  NOTE(force_optional_yolo,none,
++++       "force unwrap the value, because why not? YOLO!!", ())
2734  NOTE(force_optional_to_any,none,
2735       "force-unwrap the value to avoid this warning", ())

~/local/Source/apple/swift/lib/Sema/MiscDiagnostics.cpp

3381          // Suggest inserting a default value. 
3382          TC.diagnose(segment->getLoc(), diag::default_optional_to_any)
++++          TC.diagnose(segment->getLoc(), diag::force_optional_yolo)
3383            .highlight(segment->getSourceRange())
3384            .fixItInsert(segment->getEndLoc(), " ?? <#default value#>");
++++            .fixItInsertAfter(segment->getEndLoc(), "!");

Now, if I rebuild apple/swift and compile the source code again, I'll see the new warnings I added:

# 1. Rebuild apple/swift, by invoking build-script again.
~/local/Source/apple/swift/utils/build-script \
    --release \
    --debug-swift

# 2. Use the freshly build 'swiftc' to compile the source code
#    in hello.swift again.
~/local/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swiftc \
    hello.swift

This should produce a new set of warnings:

hello.swift:2:9: warning: string interpolation produces a debug description for an optional value; did you mean to make this explicit?
print("\(greeting)")
        ^~~~~~~~~~
hello.swift:2:10: note: use 'String(describing:)' to silence this warning
print("\(greeting)")
        ~^~~~~~~~~
         String(describing:  )
hello.swift:2:10: note: force unwrap the value, because why not? YOLO!!
print("\(greeting)")
        ~^~~~~~~~~
                  !

If you've been following along with me, then congratulations! You've just made your first change to the Swift compiler. A change like this one could be sent to apple/swift as a pull request. If accepted, it would be shipped to each and every Swift user in the next Apple release.

You could also distribute your modified Swift compiler as an Xcode toolchain. I'll write more about that, and about the apple/swift build system in general, in my next post.

If you found this post useful, and stay tuned for my next post, on the apple/swift build system: a deep dive into its CMake, shellscript, and Python guts.

Bonus step 4: Debug the Swift compiler

I build apple/swift with debugging symbols so that I can attach the lldb debugger to it. As an example, here's how I'd debug the command I executed above: first, I'll use the -### option in order to have the Swift compiler to print out its sub-commands:

~/local/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swiftc \
    hello.swift \
    -###

This prints the following two sub-commands:

/Users/modocache/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swift \
    -frontend \
    -c \
    -primary-file /Users/modocache/local/Source/tmp/hello.swift \
    -target x86_64-apple-darwin16.7.0 \
    -enable-objc-interop \
    -color-diagnostics \
    -module-name hello \
    -o /var/folders/ry/2ryfdsb56b30092626qprw6d3rb3ss/T/hello-f994f3.o
/usr/bin/ld ... -o hello

The first sub-command is an invocation of the Swift compiler with arguments to produce an object file.
The second sub-command is an invocation of the system linker. This invocation takes the object file produced by the first sub-command, then generates an executable from that file.

I'm interested in debugging the Swift compiler, so I attach lldb to the first sub-command, by copy-pasting the command from the -### output above, but prefixing it with lldb --, like so:

lldb -- /Users/modocache/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swift \
    -frontend \
    -c \
    -primary-file /Users/modocache/local/Source/tmp/hello.swift \
    -target x86_64-apple-darwin16.7.0 \
    -enable-objc-interop \
    -color-diagnostics \
    -module-name hello \
    -o /var/folders/ry/2ryfdsb56b30092626qprw6d3rb3ss/T/hello-f994f3.o

On macOS environments with Xcode installed, lldb should be available on the command line as /usr/bin/lldb. If it's not, double-check that you have the latest version of Xcode installed. If you're using Ubuntu Linux, you should be able to install lldb from here.

The above command attaches lldb to the process and displays a prompt where I can enter commands:

(lldb)

I commonly want to see how the Swift compiler determines whether to print a warning. For example, here's how I would do that for the string optional warning I modified above: first, I set the breakpoint on that file and line number:

(lldb) b MiscDiagnostics.cpp:3377
Breakpoint 1: where = swift`diagnoseUnintendedOptionalBehavior(swift::TypeChecker&, swift::Expr const*, swift::DeclContext const*)::UnintendedOptionalBehaviorWalker::walkToExprPre(swift::Expr*) + 2931 at MiscDiagnostics.cpp:3378, address = 0x0000000101fce333

Then, I run the Swift compiler. It runs until it hits the breakpoint:

(lldb) run
Process 57027 launched: '/Users/bgesiak/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swift' (x86_64)
hello.swift:2:9: warning: string interpolation produces a debug description for an optional value; did you mean to make this explicit?
print("\(greeting)")
        ^~~~~~~~~~
hello.swift:2:10: note: use 'String(describing:)' to silence this warning
print("\(greeting)")
        ~^~~~~~~~~
         String(describing:  )
Process 57027 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x0000000101fce333 swift`diagnoseUnintendedOptionalBehavior(this=0x00007fff5fbf40b0, E=0x00000001178d4a18)::UnintendedOptionalBehaviorWalker::walkToExprPre(swift::Expr*) at MiscDiagnostics.cpp:3378
   3375                .fixItInsert(segment->getEndLoc(), ")");
   3376
   3377              // Suggest inserting a default value.
-> 3378              TC.diagnose(segment->getLoc(), diag::force_optional_yolo)
   3379                .highlight(segment->getSourceRange())
   3380                .fixItInsertAfter(segment->getEndLoc(), "!");
   3381            }
Target 0: (swift) stopped.
(lldb)

The debugger is now stopped just before the point where the Swift compiler would output the warning I modified. To see how it got here, I can use lldb's bt command, which stands for "backtrace":

(lldb) bt
 thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x0000000101fce333 swift`diagnoseUnintendedOptionalBehavior(this=0x00007fff5fbf40b0, E=0x00000001178d4a18)::UnintendedOptionalBehaviorWalker::walkToExprPre(swift::Expr*) at MiscDiagnostics.cpp:3378
    frame #1: 0x00000001027ee90a swift`(anonymous namespace)::Traversal::doIt(this=0x00007fff5fbf4078, E=0x00000001178d4a18) at ASTWalker.cpp:1032
    frame #2: 0x00000001027f3ec0 swift`(anonymous namespace)::Traversal::visitImplicitConversionExpr(this=0x00007fff5fbf4078, E=0x0000000117201890) at ASTWalker.cpp:552
    frame #3: 0x00000001027f17b8 swift`swift::ASTVisitor<(anonymous namespace)::Traversal, swift::Expr*, swift::Stmt*, bool, swift::Pattern*, bool, void>::visitErasureExpr(this=0x00007fff5fbf4078, E=0x0000000117201890) at ExprNodes.def:139
    frame #4: 0x00000001027ef788 swift`swift::ASTVisitor<(anonymous namespace)::Traversal, swift::Expr*, swift::Stmt*, bool, swift::Pattern*, bool, void>::visit(this=0x00007fff5fbf4078, E=0x0000000117201890) at ExprNodes.def:139
    frame #5: 0x00000001027ef036 swift`(anonymous namespace)::Traversal::visit(this=0x00007fff5fbf4078, E=0x0000000117201890) at ASTWalker.cpp:98
    frame #6: 0x00000001027ee940 swift`(anonymous namespace)::Traversal::doIt(this=0x00007fff5fbf4078, E=0x0000000117201890) at ASTWalker.cpp:1037
    frame #7: 0x00000001027f2d00 swift`(anonymous namespace)::Traversal::visitIdentityExpr(this=0x00007fff5fbf4078, E=0x00000001172018c0) at ASTWalker.cpp:461
    frame #8: 0x00000001027f06e8 swift`swift::ASTVisitor<(anonymous namespace)::Traversal, swift::Expr*, swift::Stmt*, bool, swift::Pattern*, bool, void>::visitParenExpr(this=0x00007fff5fbf4078, E=0x00000001172018c0) at ExprNodes.def:90
    frame #9: 0x00000001027ef37c swift`swift::ASTVisitor<(anonymous namespace)::Traversal, swift::Expr*, swift::Stmt*, bool, swift::Pattern*, bool, void>::visit(this=0x00007fff5fbf4078, E=0x00000001172018c0) at ExprNodes.def:90
    frame #10: 0x00000001027ef036 swift`(anonymous namespace)::Traversal::visit(this=0x00007fff5fbf4078, E=0x00000001172018c0) at ASTWalker.cpp:98
    frame #11: 0x00000001027ee940 swift`(anonymous namespace)::Traversal::doIt(this=0x00007fff5fbf4078, E=0x00000001172018c0) at ASTWalker.cpp:1037
    frame #12: 0x00000001027f13fd swift`(anonymous namespace)::Traversal::visitTupleShuffleExpr(this=0x00007fff5fbf4078, E=0x0000000117201908) at ASTWalker.cpp:592
    frame #13: 0x00000001027ef6c4 swift`swift::ASTVisitor<(anonymous namespace)::Traversal, swift::Expr*, swift::Stmt*, bool, swift::Pattern*, bool, void>::visit(this=0x00007fff5fbf4078, E=0x0000000117201908) at ExprNodes.def:132
    frame #14: 0x00000001027ef036 swift`(anonymous namespace)::Traversal::visit(this=0x00007fff5fbf4078, E=0x0000000117201908) at ASTWalker.cpp:98
    frame #15: 0x00000001027ee940 swift`(anonymous namespace)::Traversal::doIt(this=0x00007fff5fbf4078, E=0x0000000117201908) at ASTWalker.cpp:1037
    frame #16: 0x00000001027f3ab1 swift`(anonymous namespace)::Traversal::visitApplyExpr(this=0x00007fff5fbf4078, E=0x00000001178d4aa0) at ASTWalker.cpp:713
    frame #17: 0x00000001027f12a8 swift`swift::ASTVisitor<(anonymous namespace)::Traversal, swift::Expr*, swift::Stmt*, bool, swift::Pattern*, bool, void>::visitCallExpr(this=0x00007fff5fbf4078, E=0x00000001178d4aa0) at ExprNodes.def:121
    frame #18: 0x00000001027ef600 swift`swift::ASTVisitor<(anonymous namespace)::Traversal, swift::Expr*, swift::Stmt*, bool, swift::Pattern*, bool, void>::visit(this=0x00007fff5fbf4078, E=0x00000001178d4aa0) at ExprNodes.def:121
    frame #19: 0x00000001027ef036 swift`(anonymous namespace)::Traversal::visit(this=0x00007fff5fbf4078, E=0x00000001178d4aa0) at ASTWalker.cpp:98
    frame #20: 0x00000001027ee940 swift`(anonymous namespace)::Traversal::doIt(this=0x00007fff5fbf4078, E=0x00000001178d4aa0) at ASTWalker.cpp:1037
    frame #21: 0x00000001027ee8a0 swift`swift::Expr::walk(this=0x00000001178d4aa0, walker=0x00007fff5fbf40b0) at ASTWalker.cpp:1664
    frame #22: 0x0000000101fa8db0 swift`diagnoseUnintendedOptionalBehavior(TC=0x00007fff5fbf6730, E=0x00000001178d4aa0, DC=0x00000001178d48d0) at MiscDiagnostics.cpp:3391
    frame #23: 0x0000000101fa8a3b swift`swift::performSyntacticExprDiagnostics(TC=0x00007fff5fbf6730, E=0x00000001178d4aa0, DC=0x00000001178d48d0, isExprStmt=true) at MiscDiagnostics.cpp:3406
    frame #24: 0x0000000102039c85 swift`swift::TypeChecker::typeCheckExpression(this=0x00007fff5fbf6730, expr=0x00007fff5fbf6320, dc=0x00000001178d48d0, convertType=TypeLoc @ 0x00007fff5fbf4390, convertTypePurpose=CTP_Unused, options=(Storage = 33), listener=0x0000000000000000, baseCS=0x0000000000000000) at TypeCheckConstraints.cpp:1818
    frame #25: 0x00000001021d4855 swift`(anonymous namespace)::StmtChecker::visitBraceStmt(this=0x00007fff5fbf6460, BS=0x00000001178d4ad0) at TypeCheckStmt.cpp:1234
    frame #26: 0x00000001021d438c swift`swift::ASTVisitor<(anonymous namespace)::StmtChecker, void, swift::Stmt*, void, void, void, void>::visit(this=0x00007fff5fbf6460, S=0x00000001178d4ad0) at StmtNodes.def:43
    frame #27: 0x00000001021d250a swift`bool (anonymous namespace)::StmtChecker::typeCheckStmt<swift::BraceStmt>(this=0x00007fff5fbf6460, S=0x00007fff5fbf6448) at TypeCheckStmt.cpp:372
    frame #28: 0x00000001021d2448 swift`swift::TypeChecker::typeCheckTopLevelCodeDecl(this=0x00007fff5fbf6730, TLCD=0x00000001178d48a0) at TypeCheckStmt.cpp:1620
    frame #29: 0x000000010221d299 swift`swift::performTypeChecking(SF=0x0000000117003c00, TLC=0x00007fff5fbf7180, Options=(Storage = 0), StartElem=2, WarnLongFunctionBodies=0, WarnLongExpressionTypeChecking=0, ExpressionTimeoutThreshold=0) at TypeChecker.cpp:720
    frame #30: 0x00000001019329b5 swift`swift::CompilerInstance::performSema(this=0x0000000114818600) at Frontend.cpp:525
    frame #31: 0x00000001000e20de swift`performCompile(Instance=0x0000000114818600, Invocation=0x00007fff5fbfd0a0, Args=ArrayRef<const char *> @ 0x00007fff5fbf9398, ReturnValue=0x00007fff5fbfbec4, observer=0x0000000000000000, Stats=0x0000000000000000) at FrontendTool.cpp:571
    frame #32: 0x00000001000de812 swift`swift::performFrontend(Args=ArrayRef<const char *> @ 0x00007fff5fbfc1d0, Argv0="/Users/bgesiak/Source/apple/build/Ninja-ReleaseAssert+swift-DebugAssert/swift-macosx-x86_64/bin/swift", MainAddr=0x000000010000a260, observer=0x0000000000000000) at FrontendTool.cpp:1381
    frame #33: 0x000000010000b573 swift`main(argc_=13, argv_=0x00007fff5fbff870) at driver.cpp:160
    frame #34: 0x00007fffcd3e8235 libdyld.dylib`start + 1

Wow, that's a deep backtrace! Skimming through it from the bottom up, I can see that the Swift compiler starts in main, calls performFrontend, then performCompile, and so on, until it begins type checking, and eventually it invokes diagnoseUnintendedOptionalBehavior, which is the function that outputs the warning in question.

The wall of text from the backtrace is intimidating at first, but no worries: in upcoming posts, I'll explain what each portion of the Swift compiler does. In the end, the Swift compiler is just a command-line program; there's nothing magical or scary about it.

modocache.io

How the Swift Compiler Emits Diagnostics, Part 2: Swift's Wrappers of LLVM Abstractions

uhoh.swift

Step 1: Creating an llvm::MemoryBuffer for each input source file

Step 2: Registering the llvm::MemoryBuffer with the swift::SourceManager

Step 3: Recording the locations of lexed tokens

Step 4: Printing the diagnostic

Summarizing the last three articles

How Swift and Clang Use LLVM to Read Files into Memory

Reading a file into memory using C++

read.cpp

Reading a large file into memory using mmap(2)

read.cpp

read.cpp

The LLVM implementation of read.cpp: llvm::MemoryBuffer::getFileOrSTDIN

Per-platform implementations of system calls in LLVM

build/include/llvm/Config/config.h

Opening a file on Unix

Reading the file into an llvm::WritableMemoryBuffer

Mapping the file into an llvm::MemoryBufferMMapFile

What I learned

How the Swift Compiler Emits Diagnostics, Part 1: LLVM Abstractions

hello.swift

uhoh.swift

Representing source text and locations, and emitting diagnostics, using libLLVMSupport

libLLVMSupport-Example.cpp

libLLVMSupport-Example.cpp

libLLVMSupport-Example.cpp

libLLVMSupport-Example.cpp

libLLVMSupport-Example.cpp

How exactly do llvm::SourceMgr and llvm::SMLoc work?

Getting Started with the Swift Frontend: Lexing & Parsing

hello.swift

Parsing a Swift source file

The first six stages of the Swift frontend: lexing & parsing

Stage 1: Parsing the -frontend argument

Stage 2: Instantiating a CompilerInstance based on command-line arguments

2.1: Parsing frontend command-line arguments

2.2: Instantiating the ASTContext via the CompilerInstance::setup member function

Stage 3: Kicking off libswiftParse (and libswiftSema)

Stage 4: Opening a Swift.swiftmodule bitstream cursor and kicking off the parsing loop

4.1: More on the Lexer and tokens

4.2: Priming the Lexer to form a token for the identifier "print"

Stage 5: The lex & parse loop

5.1: A quick note on custom allocators in C++

5.2: Parsing the print(...) expression

Stage 6: Reaching the end of the file

Recap: lexing and parsing in the Swift compiler

Option Parsing in the Swift Compiler

An introduction to LLVM TableGen

Stage 1: Swift's CMake instructs TableGen to transform Options.td into Options.inc

Stage 2: The libswiftOption headers declare an enum containing a case for each of the options

Stage 3: libswiftOption instantiates an llvm::opt::OptTable, with an llvm::opt::OptTable::Info element for each of the options

Step 1: Creating an `llvm::MemoryBuffer` for each input source file

Step 2: Registering the `llvm::MemoryBuffer` with the `swift::SourceManager`

Reading a large file into memory using `mmap(2)`

The LLVM implementation of `read.cpp`: `llvm::MemoryBuffer::getFileOrSTDIN`

Reading the file into an `llvm::WritableMemoryBuffer`

Mapping the file into an `llvm::MemoryBufferMMapFile`

How exactly do `llvm::SourceMgr` and `llvm::SMLoc` work?

Stage 1: Parsing the `-frontend` argument

Stage 2: Instantiating a `CompilerInstance` based on command-line arguments

2.2: Instantiating the `ASTContext` via the `CompilerInstance::setup` member function

Stage 4: Opening a `Swift.swiftmodule` bitstream cursor and kicking off the parsing loop

4.1: More on the `Lexer` and tokens

4.2: Priming the `Lexer` to form a token for the identifier `"print"`

5.2: Parsing the `print(...)` expression

Stage 1: Swift's CMake instructs TableGen to transform `Options.td` into `Options.inc`

Stage 2: The `libswiftOption` headers declare an enum containing a case for each of the options

Stage 3: `libswiftOption` instantiates an `llvm::opt::OptTable`, with an `llvm::opt::OptTable::Info` element for each of the options