Architecture

x2c translates one .x file at a time into a .c file, a .h file, a .xi unit interface, and a .d dependency file. A C compiler then builds the C. Everything the language adds on top of C is decided during that translation: Var values, immutable Lists and Strings, interpolation, pattern matching, defer, and lambdas. What comes out is plain C calling into the runtime archive. Each compiler phase transforms the previous phase’s output.

The compiler and the runtime are themselves x2c programs. A compiler written in the language it compiles cannot be built from source in one pass, so the repository carries pre-generated C to start from, then builds the compiler several times over and compares the results. Most of the repository layout supports that self-host build.

For the modules that implement a particular language feature, see the implementation map. For the language being implemented, see the language reference. For the runtime these phases target, see the standard library overview. The compiler API reference lists the functions and types exposed by each compiler module.

Repository topology

bin/        the bootstrap compiler binary
bootstrap/  checked-in portable C for the compiler and the runtime
builds/     self-host stages 0 through 3
commands/   external commands built against the current compiler
include/    links to runtime sources and stage-0 generated headers
lib/        x2c runtime sources
src/        x2c compiler sources
unittest/   runtime suites, compiler fixtures, probes, and benchmarks
examples/   curated executable and build-checked programs
docs/       this book: guide, reference, library, and internals
agents/     repository-facing documentation, contracts, and project skills
plans/      active plans-as-logs and archived execution records
etc/        shared Makefile rules, build configuration, built-in macros,
            and the compile-time Lisp environment and SDK
tools/      documentation and stage-comparison checkers

Three of those directories hold generated files. builds/ holds whole staged builds and is not tracked. include/x2c/ holds only symlinks: the runtime .x sources from lib/, and the .h files stage 0 generated from them. That is where cc -iquote include/x2c finds runtime headers. The C under bootstrap/ is tracked, and it is re-emitted from a known-good stage 0 instead of edited by hand. The compiler under development is builds/0/x2c; see building the compiler for how it is built.

The compiler pipeline

The compiler lives under src/. src/main.x dispatches commands and runs the translation loop; src/cli/cli.x parses the command line. src/frontend.x owns configured source stages and isolated unit lifetimes shared with internal tools. Nested interning pools keep batch translation close to single-file peak memory. For each input file the translator runs one pipeline:

source
  -> tokenizing
  -> shallow parse and global environment discovery
  -> full parse and type annotation
  -> protocol conformance forms
  -> lowering
  -> generation and cache staging
  -> C token emission
  -> formatting and output (.c, .h, .xi, then .d)

Most phases have a flag that prints their output and stops, so the same file can be printed after each phase; the flags are in compiler options. The example below follows one file through the phases.

A worked example

String greet(String name) {
  return %"hello, $name";
}

Saved as greet.x in the repository root and translated into a scratch directory,

mkdir -p /tmp/x2c-arch
./builds/0/x2c translate --out-dir /tmp/x2c-arch greet.x

it produces greet.h:

/* auto-generated by x2c.  Do not edit! */

#pragma once

#ifndef __GUARD_0x10E936D3__
#define __GUARD_0x10E936D3__

#include "x2c.h"
String greet(String name);


#endif /* __GUARD_0x10E936D3__ */

and greet.c:

/* auto-generated by x2c.  Do not edit! */

#define __GUARD_0x10E936D3__
#include "x2c.h"

static String _0;

static int _init_guard_ = 0;

__attribute__((constructor, noinline, cold)) static void _file_init_(void);

__attribute__((constructor, noinline, cold)) static void _file_init_(void){
  x2c_initialize_protocols();
  if(_init_guard_) return;
  _init_guard_ = 1;
  _0 = String_new("hello, ");
}

String greet(String name){
  if(! _init_guard_) _file_init_();
  return String_join(NULL, cons(String_var(_0), cons(String_var(name), NULL)));
}

The interpolation became a String_join over a cons list. The constant "hello, " became a file-static String built once by a generated initializer. The sections below show where each change happens.

Tokenizing

The tokenizer reads source text and produces a positioned token array. Compiler.tokenize in src/compiler/parsing/tokens.x reads the file, hands the text to Tokenizer.new from lib/tokenizer.x, and scans it with the allocation-free character recognizers in lib/scan.x. Whitespace, comments, and preprocessor lines all become tokens. Later phases can therefore report positions in the original file, and directives can be re-emitted where they were written. src/preprocess.x classifies those directive tokens and turns the tokens of a conditional arm that C never takes into comments.

./builds/0/x2c translate --dump-tokens greet.x

(   1, 1  )	ident               String
(   1, 7  )	space               ␠
(   1, 8  )	ident               greet
(   1, 13 )	(                   (
(   1, 14 )	ident               String
(   1, 20 )	space               ␠
(   1, 21 )	ident               name
(   1, 25 )	)                   )
(   1, 26 )	space               ␠
(   1, 27 )	{                   {
(   1, 28 )	space               ␤␠␠
(   2, 3  )	return              return
(   2, 9  )	space               ␠
(   2, 10 )	%"                  %"
(   2, 12 )	segment             hello,␠
(   2, 19 )	$                   $
(   2, 20 )	ident               name
(   2, 24 )	"                   "
(   2, 25 )	;                   ;
(   2, 26 )	space               ␤
(   3, 1  )	}                   }
(   3, 2  )	space               ␤

␠ and ␤ stand for space and newline. %" and segment are already token classes. The scanner recognizes interpolation; no later pass rescans string contents.

Shallow parse and global environment discovery

This phase reads the token stream and the declarations the file can see, and produces a Map from global name to type. This phase exists because x2c’s typed lowering has to know the type of every name in scope, including names that come from C headers the compiler never parses in full.

The default path does not run the C preprocessor. Every unit starts from the implicit runtime prelude, the declarations of lib/x2c.x and everything it includes. src/collect/collect.x shallow-parses source text directly, splicing quote-includes inline at their include points in the same order the preprocessor would have produced them, resolving each through the same search order (the including file’s directory, ., src/, lib/, then the -I chain), and overlays what it finds on the prelude. Source includes install their public declarations and compile-time definitions. Native runtime headers already covered by the prelude add no declarations. Unresolved angle includes are system headers the generated C re-includes; unresolved quote includes are driver errors. Each file is spliced once by real path, so include cycles terminate.

Collection also hashes this unit’s function bodies and initialized file-static values while it skips them. Full parsing records each definition’s referenced names in the same definition metadata. Linked-meta generation pairs those names with the hashes. An unchanged forward definition can therefore use its shipped reference list to check later dependencies before their bodies are parsed. Included source bodies belong to their own units and do not enter this unit’s hash map.

Linked providers retain one source hash and their direct dependency hashes. Imported functions refer to that provider record. The collector checks the dependency graph once per translation process, using its existing source hashes. A changed provider or dependency requires rebuilding its meta code.

The public x2c_meta_definition_hashes() query still returns String hashes for definitions parsed so far. The private Lisp query _x2c.meta.definition.calls reads their existing reference lists for linked generation; it adds no public library callable or separate stored inventory.

Unit interfaces

A file’s contribution is collected once per process. Translating a unit also writes it beside the generated C as a unit interface, <stem>.xi: the identity of the compiler that wrote it, the ordered declaration maps, include placeholders, the source content hash, function definitions, and the macro, Lisp, and embedded-text files the walk read with their hashes. Each declaration map holds its public rows: nonstatic declarations and complete type families needed by public declarations. Source-position rows also retain public macro and keyword definitions, Lisp effects, and provider meta-function identities. A translated unit’s map also records each ordinary external conformance whose adoption it declares, as (conformance BASE PARTICIPANT FORWARD REVERSE (bindings (NAME TYPE)...) (MEMBER STATUS SOURCE)...) keyed by base and participant. It lists each type variable’s binding and each member that did not resolve as no-member; the expected and template types follow from the protocol record. A unit that replays the map installs that conformance when it first uses it, instead of resolving the adoption. When a cold walk collects a file that declares adoptions, the unit resolves them in a compiler that gathers only the prelude’s and that file’s declarations and records the same rows in the file’s entry. A file that imports a package records none. Cold collection and interface replay use the same ordered parts, while the full parser installs included compile-time definitions at their include positions. Macro names become visible immediately; their templates are decoded into the current unit only on first use. The existing semantic write log restores that decode if its expansion rolls back. Local declaration productions stay in their owning unit’s Context until full parsing consumes them. Their syntax, macro frames, and binding facts are not exported. Static helpers remain in their provider. Each provider has one function-hash table. Its meta advertisements identify the provider; collection and replay install the table before advertisements. The file holds one (interface 6 "compiler" "path" "hash" (PARTS...) (DEFINITIONS...) (DEPENDENCIES...) (INCLUDE-DIRS...)) form in %() List syntax, with bare Atoms for its structural words and Strings for identifiers; the reader reads that one form without evaluating it. The API reference reads --dump-definitions output and joins authored prose to those definitions. Before walking a file’s source, collection looks for its interface in the output directory, then in the directory that mirrors the file’s home-relative path under the compiler’s stage directory (or under an installed home), then in a package’s builds/. An interface is used only when its recorded compiler, path, include search roots, and every hash still match; otherwise collection walks the source. Include search roots retain their order and distinguish home-relative, absolute, and cwd-relative spellings. Search candidates with cwd-relative spellings are checked in the current working directory. The compiler identity is a digest of the executable’s contents, so a changed compiler never replays rows an earlier build collected. The prelude is the runtime x2c.xi interface from the library batch, so a stage build produces the prelude the next batch and the next stage replay when the stage compilers are byte-identical, as they are once the bootstrap matches the source. No tracked file is both an input and an output of a build. An installation writes its runtime interfaces with the compiler it installs. x2c env prelude prints the prelude interface this compiler wrote; an empty value means the compiler walks the source of lib/x2c.x, which costs about a quarter of a second per process. Replay must match the walk it replaces: make proof-cold-collection retranslates lib/ and src/ with the hidden --no-interfaces option, which reads no interface, and requires the stage 1 C, headers, and interfaces byte for byte.

--cpp-symbols and --live-symbols discover symbols through the host C preprocessor: the toolchain force-loads lib/x2c.x and runs cc -E -P as a child process. Paths stay separate argv elements, and stdout, stderr, and the real child status come back independently. A second Compiler shallow-parses that output instead.

Either way this is a discovery pass. Shallow parsing (Compiler.shallow_parse) skips function bodies, and full parsing, source diagnostics, and emitted C all come from the original positioned token stream. --no-cpp skips symbol discovery and is a diagnostic aid for C-only source. --dump-symbols prints the symbol table the compiler ends up with; the implicit runtime contributes a few thousand rows, so grep the output.

Full parse and type annotation

The full parse reads tokens and the global environment and produces the parsed AST: immutable Lists, tagged by node kind, annotated with resolved types. Compiler.full_parse in src/compiler/parsing/parse.x drives a recursive-descent grammar split across four modules – src/parse/parse.x for declarations and top-level forms, including initializer recognition and the function-body boundary, src/expressions/expressions.x for the C precedence ladder plus dotted method sugar, src/statements.x for control flow, match, try/catch, and defer, and src/literals/literals.x for collection, interpolated String, and lambda literals. src/meta-group/meta-group.x stages a unit’s bodied meta functions as native code, and src/stage.x carries values across that boundary. The built-in source macro in src/macros/macros.x, etc/builtin-macros.x, and the algorithms in src/builtins.x expand foreach during this pass. src/type.x owns the List-backed Type representation those modules consult; src/ast.x owns sequence placement and binding helpers; src/symtab/symtab.x owns lexical scopes and symbol lookup from inner to outer; src/compiler/names.x owns generated names; src/compiler/parsing/tokens.x raises <malformed> for a token mismatch, and src/compiler/parsing/parse.x holds the filtered recovery boundary that synchronizes after a parse diagnostic; src/diagnostics.x collects positioned diagnostics in order.

Macro definitions use that same recursive-descent parser in macro mode. Each semantic entry point accepts a typed parameter or compile-time Lisp slot and otherwise follows the ordinary grammar, so the stored template is an ordinary AST List. An invocation parses its arguments through those entry points, matches their canonical capture rows once, replaces the template once, and hands the result to the ordinary recursive syntax binder. Nested invocations and Lisp slots are expanded when that binder reaches them; there is no second macro grammar or post-expansion validation pass.

./builds/0/x2c translate --dump-ast greet.x

(function ("String")
 (bind (binding 1 "greet")
  ((fnmod (params (param ("String") (bind (binding 2 "name") ()))))))
 (block
  (at 1 (return ("String")
         (expr ("String")
          (segments (cache 0)
           (segvar (expr ("String") (ident (binding 2 "name"))))))))))

Types are attached as the tree is built. The function, its parameter, the return, and the interpolated expression all carry ("String") already. Identifiers have become numbered bindings, and each statement is wrapped in an (at N ...) node. That number indexes the compiler’s table of recorded source positions, so a later phase can still report the line the statement came from. Source preprocessor lines survive as preproc nodes, kept in source order at the top level and inside compound statements. The implicit runtime prelude is not a source token or AST node.

The parser has already cached the literal. The literal segment "hello, " holds no dynamic references, so src/literals/literals.x recorded it as a cacheable constant and the AST refers to it by slot:

./builds/0/x2c translate --dump-cache greet.x

0	==>	(string (expr ("String") (literal ("String") "hello, ")))

Lowering

Lowering turns the parsed AST into a lowered AST that the emitter can walk without knowing about x2c. src/transform/transform.x normalizes each node until its identity stops changing, because lowering one construct can expose another. src/cleanup/cleanup.x places function cleanup within the shared driver. src/components/lambda-functions.x lowers each lambda the driver meets; see builtin components. src/callables/callables.x converts callable values. It adds an adapter when a receiving callback type differs from a function’s signature. A direct function gets one reusable file-static Func handle, and a function pointer is copied into a new Func context. A public inline function reaches these helpers through a generated bridge that also runs the owning unit’s initializer. The helpers are queued as early declarations, normalized by the same driver, and appended to the unit.

Cleanup lowering completes each function after its body is normalized. A try first takes the landing form that src/components/try-catch.x returns; see builtin components. A program’s rule on try applies here as well. Binding and lowering its replacement keep the already lowered parts of the try as they are. The pass then assigns a name to the runtime record of every defer region and landing frame, builds the statements that leave each region, and runs them wherever control leaves it: the region’s own end, a return – after saving the value, since cleanup may change what the expression read – a break or continue that leaves the construct that bounds it, and an outward goto. The pass rejects a jump into a region, including the region a static local’s runtime initializer opens over the rest of its block. The pass also keeps the locals and parameters a landing’s parts write valid across sigsetjmp, as C requires: it qualifies them volatile unless their address escapes. Transfer rewriting follows expression and lambda normalization within the function.

./builds/0/x2c translate --dump-transforms greet.x

(function ("String")
 (bind (binding 1 "greet")
  ((fnmod (params (param ("String") (bind (binding 2 "name") ()))))))
 (block
  (at 2 (return
         (expr ("String")
          ("String_join(NULL, "
           (expr ("List")
            (cons (expr ("Var")
                   (call (expr ((func (("String"))) "Var")
                          (ident (binding 3 "String_var")))
                    (args (expr ("String") (cache 0)))))
             (cons (expr ("Var")
                    (call (expr ((func (("String"))) "Var")
                           (ident (binding 3 "String_var")))
                     (args (expr ("String") (ident (binding 2 "name"))))))
              (nil)))) ")"))))))

The interpolation is gone, replaced by a String_join call over a cons list of boxed Var values, each String_var call naming its callee as a resolved binding carrying that function’s type. src/components/interpolation.x wrote the join; see builtin components. The return no longer carries a type annotation; nothing downstream needs it. Some children are now raw C text. Lowering may produce emitter-ready fragments, so "String_join(NULL, " sits in the tree as a String instead of a call node.

Generation and cache staging

Generation splits the lowered AST for one unit into two ASTs, one per output file, and adds the generated initialization that makes cached constants work. src/generate/generate.x owns the sequence: drop whitespace and comment nodes, partition declarations and functions into header and source halves, ask src/cache.x to materialize the cache slots as file statics with an initializer, fold every file-level initialization block into one guarded _file_init_ function, synthesize static prototypes, restore vertical spacing, add the unit’s own header as its primary include, and patch main when the unit has one. Generation synthesizes already-lowered nodes, so it runs no further transform pass.

src/cache.x creates _0 in the worked example. Parsing and lowering intern each constant through Compiler.cache in src/compiler/literal-cache.x, which assigns its slot; src/cache.x materializes those slots between lowering and emission: immutable String, boxed Var, and canonical List/cons graphs share generated storage, while mutable collections are copied at their use sites, so caching never changes identity semantics. An ordinary C literal promoted to String enters this same cache without changing its C escape spelling. Parenthesized and conditional raw-string expressions distribute promotion to their literal leaves; dynamic leaves retain their per-evaluation conversion and conditional evaluation still selects only one arm.

Generation computes the transitive cache graph used by each output region. Source-resident values keep the compact _N statics in the .c file. A public inline body instead gets deterministic, filename-hashed static slots in the generated header, so each consuming C translation unit remains self-contained. If the same key occurs in both regions, each region has a slot, but String and List canonicalization makes the resulting value identity the same.

The generated C provides two ways to run the guarded initializer. __attribute__((constructor)) runs the source or TU-local header initializer at load time where the host supports it. Every nonstatic source function except the protocol bootstrap functions checks the file’s guard. In a file whose only initialization is cache setup, only those that reach a cache, directly or through calls, check it. Every header function that reads a header cache checks the header’s guard. Therefore initialization also runs on hosts without constructor support. These immutable values are allocated eagerly and retained for the process lifetime, so an allocation failure happens during load-time initialization instead of at the literal’s source expression.

C token emission

Emission turns the lowered AST into a flat List of C tokens. src/emit/emit.x does this with one stack-local Emitter per translation unit, which holds the current function’s name and static objects. Emission is therefore reentrant, and state from a failed unit does not reach the next unit. Lowering has already placed every cleanup statement, so emission prints frames, records, and statements where the AST puts them. Preprocessor nodes are re-emitted here too, with .x include targets rewritten to the generated .h they correspond to.

./builds/0/x2c translate --dump-code greet.x

String greet(String name){
  return String_join(NULL, cons(String_var(_0), cons(String_var(name), NULL)));
}

That is the emitted unit on its own. The include lines, the _0 static, and _file_init_ are added afterwards by generation. The final greet.c defines the header’s guard macro and includes x2c.h itself, and greet.h holds the declaration.

Formatting and output

Formatting turns the C token List into text. src/format.x walks the tokens without reordering them: parenthesis depth suppresses statement breaks inside expressions, preprocessor tokens occupy their own lines, and a Buffer builds the final String. The result is valid C, with spacing determined by the emitter. src/generate/generate.x then writes the .h, .c, and .xi files together, named from the input basename inside the --out-dir directory. A write failure replaces none of them and becomes a diagnostic carrying the target path and the host error. src/main.x then writes the .d file through src/deps.x.

unittest/compiler-fixtures/ pins the output of each phase. It stores expected tokens, AST, transformed AST, symbol tables, generated C, and exit status per fixture, so a change in any phase shows up as a diff.

Module ownership

Each module under src/ and commands/ has these responsibilities. src/components/ groups the builtin components. It is a directory, not a module block, so its files share no private scope.

• the builtin components, described in builtin components: src/components/bracket-access.x (bracket access), src/components/try-catch.x (try, catch, and finally), src/components/delegate-members.x (delegate fields), src/components/collection-literals.x (Array and Map literals), src/components/printf-format.x (printf formats), src/components/dynamic-operators.x (dynamic operators), src/components/raise.x (raise), src/components/pattern-match.x (match), src/components/interpolation.x (interpolation), src/components/destructuring.x (destructuring), src/components/lambda-functions.x (lambdas), src/components/protocol-adapters.x (protocol members and descriptors), and src/components/string-switch.x (String switches);
• src/operator-ledger.x – operator precedence, compound assignment, and protocol-member mappings from one compile-time ledger;
• src/cli/cli.x, src/main.x – option metadata, parsing, and dispatch;
• src/translation.x – synchronous translation, inspections, worker scheduling, ordered diagnostics, and signal forwarding;
• src/frontend.x – configured source stages, process translation support, and sequential unit Context/Type lifetimes;
• src/meta-project.x – the project meta build: the helper program that runs a project’s meta functions, built and cached before translation;
• src/meta-helper-client.x – the compiler’s side of that helper: one request and one reply per call, with its timeout and failures;
• commands/repl/ – terminal interaction, inline editing and history, persistent submissions, and named inspection over the compiler session API;
• src/project/project.x, src/toolchain.x – shared manifest membership, target relationships, and host compile/archive/link primitives;
• commands/build/ – application build and run orchestration, incremental state, script executable caching, locking, and execution;
• src/report.x – dependency-free terminal progress and stable completion receipts on standard error;
• commands/package/ – package installation, removal, and listing under the x2c home, project lockfile preparation, and source-package builds through the matching compiler; fetch, digest, and extraction use host tools;
• src/deps.x – x2c dependency parsing and atomic depfile publication;
• src/compiler/compiler.x – shared compiler state, with members under src/compiler/ for parsing (tokens, collection, the full parse, top-level definitions, and scripts), declaration production, generated names, lifecycle, binding facts, the literal cache, match patterns, pending support declarations, and source files and locations;
• src/fields.x – macros that copy or set named fields and transfer a segment’s compile-time state;
• src/symtab/symtab.x – the symbol table: scopes, bindings, lookup, typedef resolution, and semantic transactions;
• src/preprocess.x – preprocessor directive lines, conditional arms, and the directives before each form;
• src/parse/parse.x, src/expressions/expressions.x, src/statements.x, src/literals/literals.x – grammar and AST construction;
• src/parse-reports.x, src/expressions-reports.x – package and alias diagnostics, and expression typing diagnostics;
• src/initializers/initializers.x – brace initializer conversion by C’s initialization order;
• src/lambdas.x – lambda parsing and the capture resolution shared by source and constructed lambdas;
• src/macros/macros.x – compile-time macro definitions, imports, Lisp lifting, hygiene, and expansion;
• src/meta-sdk/meta-sdk.x – the compiler’s answers to lib/meta.x operations and the context of the running compile-time call;
• src/meta-native/meta-native.x – meta functions, explicit meta calls, native meta bindings and their lifetimes, and native modules;
• src/stage.x – the arguments and results that cross between meta code and program code;
• src/meta-group/meta-group.x – a unit’s meta group, its emission as C, and in-process staging;
• src/builtins.x, src/linked-meta.x – the built-in macro algorithms and the shipped compile-time code compiled into the compiler;
• src/ast.x – AST sequence placement and binding helpers;
• src/ast-rewrite.x – AST child rewriting and iterative tree traversal;
• src/grammar.x – source-form patterns and parsed-node accessors used by lowering;
• src/type.x, src/protocol/protocol.x – type representation, conversions, protocol declarations, conformance, and native aliases, with the conformance form that presents each conformance’s adapters to src/components/protocol-adapters.x;
• src/type-ledger.x – the Var tag lookups projected from the tag ledger, kept in their own unit so the units that include type.x do not load it;
• src/collect/collect.x – global environment discovery, unit interfaces, and the splicing of module blocks;
• src/utils.x – repository discovery, the driver’s fatal error line, and forked translation workers;
• src/transform/transform.x – lowering to emitter-ready AST;
• src/callables/callables.x – typed callback adapters and Func conversions lowered to C helpers;
• src/adapter-memo.x – macros that cache compiler adapters and reuse each result by key;
• src/cleanup/cleanup.x – which exits leave a cleanup region and which locals an error transfer preserves;
• src/regions/regions.x – per-function region summaries and the warnings for a value that can outlive the region that allocated it;
• src/cache.x – cached constants and their generated initialization;
• src/generate/generate.x – header/source partitioning, unit initialization, include guards, and output writes;
• src/emit/emit.x – AST to C tokens;
• src/format.x – C tokens to text;
• src/diagnostics.x – recorded diagnostics;
• src/sourceview.x – request-owned source overlays and logical file paths;
• commands/editor/editor.x – one-request diagnostics, definition, and hover transport over the configured frontend and source overlays.

The internal frontend is shared by the CLI, editor, and source graph tool. A Frontend borrows a configured CliRequest; its ParsedUnit retains the compiler, AST, diagnostics, and preprocessor output until explicit close. The stages are start/tokenize, collect, and parse. Failed stages return to the caller with readable diagnostics. Adapters choose printing and process exit; the default frontend collects without printing. Closing a unit releases its owned Lisp session, Type unit, and isolated Context before the next unit opens.

Process type/header caches and generated-name state still require sequential units. The frontend is not a concurrent or stable public compiler-library API. A user macro that aborts or exits still ends the process.

The optional editor worker in tools/x2c-editor/ uses the same frontend and project configuration. SourceView supplies immutable unsaved text under each file’s canonical path; an empty overlay remains a present file. The request owns overlays, while a parsed unit owns disk text and semantic results.

Declaration metadata follows the actual symbol-map contribution and key through collection and imports. The primary parser associates resolved bindings with physical token spans for definition and hover queries. It does not infer a source location for constructed syntax that has no physical token. Each editor request runs in a fresh process and collects source declarations without reading .xi interfaces, which do not carry their physical spans. This isolates process caches and macro failures from the editor service.

The VS Code adapter under etc/vsc-extension/ converts compiler UTF-8 byte offsets to editor positions and discards responses after document changes or cancellation. It executes semantic requests only in trusted local workspaces. Syntax highlighting remains available independently of the native worker.

The runtime boundary

The compiler is an x2c program and uses the same runtime that generated programs use. There is no compiler-private collection library, so every runtime weakness the compiler hits is one a user program can hit. The main divisions:

• lib/scope/scope.x owns allocation lifetime, and lib/pool/pool.x the nested interning pools layered over it;
• lib/var/var.x, lib/varconvert.x, lib/varops.x, and lib/dispatch.x own tagged values, cause-raising conversion, operators, and dynamic dispatch;
• lib/string/string.x, lib/symbol.x, lib/atom.x, and lib/list/list.x own canonical immutable values;
• lib/block.x, lib/buffer.x, lib/array.x, and lib/map.x own mutable storage;
• lib/iter.x owns status-bearing traversal;
• lib/match/match.x owns list-pattern matching; lib/match-plan.x compiles patterns to plans, lib/match-cache.x retains them, and lib/match-machine.x executes them over the wordcode and state definitions in lib/machine.x;
• lib/lisp/lisp.x owns the embedded Lisp reader, session, and evaluator; the compiler uses it to read .xi interfaces and to run compile-time macros;
• lib/func.x owns generic native calls through generated adapters;
• lib/tokenizer.x and lib/scan.x own tokenization, so the compiler’s first phase is library code;
• lib/error/error.x owns failure records, handlers, and policy;
• lib/exception.x owns Error transfer and cleanup frames, while lib/file.x and lib/logger.x provide file I/O and logging;
• lib/common.x supplies shared representation and initialization support;
• lib/lib.x contains the standalone DisjointSet utility.

lib/lisp/lisp.x evaluates recursively with tail calls; see production execution. Pattern matching executes compiled Match plans on MatchMachine.

lib/x2c.x is the generated source definition of the implicit runtime prelude. The generator leaves out the optional x2c system modules; they are built with the runtime and need an explicit source include. Runtime component headers keep selective includes while they build the aggregate. Every other generated header includes x2c.h.

The self-host build

Because src/*.x and lib/*.x are x2c, building them requires an x2c compiler, and the only way to get the first one is to start from C that an earlier x2c compiler emitted. That C is tracked under bootstrap/. It has to stay portable, since a C toolchain is all you have at that point.

bootstrap/  -- cc -->  bootstrap compiler
                              |  translates lib/ and src/
                              v
                          builds/0  ->  builds/1  ->  builds/2  ->  builds/3

generated C compared byte for byte:
  stage-diff-0   bootstrap/  against  builds/0
  stage-diff-1   builds/0    against  builds/1
  stage-diff-2   builds/1    against  builds/2
  stage-diff-3   builds/2    against  builds/3

Each arrow is one full translation of lib/ and src/ followed by a C build, and each stage directory holds both halves: the C that the previous compiler in the chain emitted, and the binary built from it. So builds/0 is the bootstrap compiler’s output, builds/1 is stage 0’s output, and so on. Stage 0 is the compiler everything else uses. Stages 1 through 3 show that it reproduces itself, and make stage-3 builds them.

Staging matters because a change to src/ changes two things at once: the compiler’s behavior, and the program that compiler is asked to compile. Stage 0 shows only that the old compiler could translate the new source. Stage 1 is the first build where the new compiler compiles the new source, and stage 2 is the first build where a compiler produced by the new compiler does. Some bugs need both halves of the change to appear, such as a lowering that emits code its own new parser mishandles. Those surface at stage 1 or 2 and nowhere earlier.

The comparison is byte-exact. tools/check-generated-stages.sh, behind the make stage-diff-0 through make stage-diff-3 targets, compares the generated .c/.h file sets of two neighbouring stages and then compares every file byte for byte. stage-diff-0 asks whether the bootstrap compiler re-emits the C it was built from; the rest ask whether each stage emits what the stage before it emitted. Once two adjacent stages agree, the chain has reached a fixed point.

From stage 1 on, a stage build also fails on a diagnostic. The stage Makefile passes the hidden --fatal-warnings option, which fails a unit that reports any warning or region finding, and compiles the generated C with -Werror. Stage 0 stays lenient, because the checked-in bootstrap compiler translates it and a source install compiles it with the host C compiler.

A byte comparison can catch problems that behavioral tests miss. An iteration order that depends on addresses, or a generated-name counter that carries across translation units, shows up here while every suite still passes. The same phase run twice has to produce the same bytes.

Builtin components

src/components/bracket-access.x registers bracket reads, stores, and updates, the $indexed and $x2c.collection.store forms, through ordinary macro patterns and quotations. lib/rewrite.x supplies the decorator. Array and Map mutations call their typed helpers; any other protocol participant calls its getter, setindex, updateindex, or postfixindex member through Code.call_in_order, which evaluates each operand once, in order. Binding, index admission, evaluation order, and cleanup remain compiler responsibilities.

src/components/try-catch.x registers the $caught and $tried source forms from src/grammar.x through the same decorator, both on the try head. Its translator returns a landing form, (landing CODE ROWS): the frame and catch-site template, the finalizer, the exits that close the catch site, claim the finalizer, and leave the frame, and the body and each catch arm as regions. The finalizer is also a region. The cleanup walk described under lowering lowers the rows in order.

src/components/delegate-members.x registers a member call pattern whose receiver hole is %(expr <delegate> *), a field keyword where a type would go. The parser marks each delegate field of an aggregate; a method call that ordinary lookup misses tests the rules registered for the marks its receiver’s aggregate declares, after those keyed by the receiver’s own type. The translator searches the delegate fields with Type.marked_fields and Type.resolve_member, reports an ambiguous path or a cycle, and returns the call through the field path as constructed syntax, which ordinary call resolution binds.

src/components/collection-literals.x registers the Array and Map literal forms, $array_value and $map_value from src/grammar.x, on the literal family keyed by the array and map heads. Each translator returns the counted constructor, Array.update_n or Map.update_n on a new container, with each element, or each key and value, converted to Var in source order; an empty brace among them is an empty Map, and an empty literal is the bare Array.new() or Map.new(). Each part runs once, left to right: once two or more parts read state, the transform moves each part that reads state, through the last one that may change state, into a temporary ahead of the call, as “literal order” in src/transform/literals.x describes.

src/components/printf-format.x registers one call pattern for each printf-family member, on the call family keyed by the callee’s spelling, so a call to any other function only looks its spelling up. The translator reads the static format with Code.format, walks it once, and has each Var value a conversion consumes converted with Code.convert to the C type that conversion reads, or displayed through Var.str for %s. The redundant .str() warning for those values stays with the conversion checks in src/expressions/expressions.x.

src/components/dynamic-operators.x registers its translators with Match patterns typed Var whose operator positions name a set, (!or OP...), on the binary family for binary operators and compound assignments and the unary family for increments, decrements, and unary operators, keyed by each operator and Var. The transform dispatches an operation to those rules only when an operand has Var identity, after protocol operators resolved, presents it typed Var, and normalizes the replacement as it would its own lowering. A binary operator returns a Var.binary call through Code.lowered, so an operator chain does not rebind its remaining terms at each level; an update returns the lowered call of its storage’s update helper with the target’s address, the form the cleanup walk recognizes as a write. Truthiness stays a conversion the transform applies at condition positions, and slices, comparisons, and a participant’s direct compound update remain in the transform. When a rewrite lowers its replacement, the source expressions the replacement holds lower outside the active rule, so a store nested in another store’s value applies the same rule again. A rule whose pattern was derived without the subject’s bindings matches without collecting them.

src/components/raise.x registers raise on the statement family keyed by the raise head. Its translator converts the code to a Symbol and each detail key and value to a Var with Code.convert, promotes a C string literal value with Code.promoted, and reports a value whose type is not immutable with x2c_diagnostic_error_at, which returns so the translation goes on. It returns the converted raise as a lowered carrier, which binding leaves as it is. The compiler lowers that raise’s parts in source order, moving them into temporaries as literal parts move, and builds its literals at run time.

src/components/pattern-match.x registers match on the statement family keyed by the match head. Its translator returns a lowered carrier of verbatim C text around the subject, the patterns, and the arm bodies: a block that converts the subject to a List once with Code.convert and switches on the bits of the subject’s head, with a label on the first arm of each head. Code.pattern_value gives each arm’s head and binders. A cached pattern of literals, unique binders, typed captures, nested Lists, and a final * is tested in place by nested if tests on cursors over the subject, each binder a local read from its cell. A case on a visible macro whose derived pattern has that form is tested the same way after Macro_case_root starts it on the subject’s view; another macro-valued case calls Macro_case_capture_at, and any other pattern calls the Match runtime through the capture buffer, at a static site when Code.is_static_pattern holds. The transform and the cleanup walk step into the nodes the text holds, and the parts of the match lower outside its rule, so a nested match applies it again. The generated switch is what an arm’s break leaves.

src/components/string-switch.x registers switch on the statement family keyed by the switch head and declines a selector whose type is not String or an alias of it. Its translator replaces each label the switch owns by the label’s index, leaving nested switches and expressions alone, and reports a label that is not a string literal with x2c_diagnostic_error_at. It returns a lowered carrier of the integer switch on a call of a static helper that returns the selector’s index among the labels, or 0. The helper is placed once per selector type and label list under (unit-support KEY), named by x2c_binding_for with the same key, so switches with the same labels share it. A program’s switch rule applies to source switches only, not to the integer switch the component writes.

src/components/interpolation.x registers a %"..." literal on the literal family keyed by the segments head. The transform first converts each $ insertion to a String as the parser converts its own, and a constructed String of one constant text reuses the parsed literal’s cache slot. The translator joins the parts with String.join over a List chain of them: cached text, a converted insertion, or String.new of the raw text that constructed syntax spells. The compiler boxes the chain’s parts and evaluates them once each, left to right, moving them ahead of the join when they move.

src/components/destructuring.x registers destructuring declarations on the statement family keyed by the dstrdecl head, an assignment statement keyed by stmnt, and an assignment expression on the literal family keyed by dstrasgn. Its translators answer bound carriers: a List local read from the source with Code.convert, and each target given its element through the conversions lowering applies to an assignment or initialization. The expression form keeps the source in a result local as its value. The parser has already declared the targets, so binding leaves their declarations alone and they keep their scope and identity; the locals are new-name effects.

src/components/lambda-functions.x registers both lambda forms on the literal family keyed by the lambda head, which the transform offers each lambda it meets. A noncapturing lambda returns a bound carrier: the name of a static function placed among the unit’s support declarations, whose body is the lambda’s bound body as a lowered carrier, so it is not bound again. The function lowers with the unit’s other support declarations, so it prepares its own cells and takes its own entry placements. An expression body carries the origin of the item that held the lambda, which x2c_enclosing(<statement>) returns while lowering, so a diagnostic in it points there. A capturing lambda’s function also takes each captured binding as a parameter, a reference as a reference, so its body is not rewritten. The rule places a context record and an adapter, which reads the arguments through Code.func_parameters and calls the function with them and the record’s fields, and returns the statement expression that evaluates the captures left to right and makes the Func. The component also registers a function rule for every definition: a local or parameter that a lambda in the body shares by reference moves into a Scope cell, so the binding outlives the call. A parameter is copied into its cell at entry, a declarator becomes the cell’s declaration, a destructuring declaration copies its target after it, and every read goes through the cell; the replacement is bound again. Capture bindings have separate identities from their source bindings, so moving a source into a shared cell cannot retarget a sibling snapshot.

src/components/protocol-adapters.x registers on the conform head. Before the transform, the compiler presents each conformance whose forward converter the unit defines, and whose participant is not numeric, as a form (conform BASE PARTICIPANT FORWARD REVERSE BINDER DESCRIPTOR (ROW...)). The compiler has already chosen each member’s owner and storage, reported collisions and missing conversions, published the external forwarding members’ signatures, and bound the name of every function it asks for and of the method table. Each row is (MEMBER STATUS SOURCE EXPECTED TEMPLATE PART...), whose parts are (default NAME STORAGE) for a forwarding member, (inherited NAME EXPECTED) for the prototype of a member another conformance defines, and (thunk NAME ERASED) for a descriptor slot. A function that completes a documented prototype keeps its prose. DESCRIPTOR is (METHODS NAME VAR-TAG AREA [TAG]) for a Var participant with its own descriptor, and empty otherwise. The translator places each function and the table with %(unit-support), and the table’s assignment and the registration with %(unit-init AREA), and returns (seq). A rule on the head returns file-scope code, which joins the unit’s support. When no rule returns, the translation ends with no rule lowers this protocol conformance. Native aliases stay in the compiler, because they enter at the participant typedef’s header boundary.

The components and src/grammar.x, whose source forms they recognize, are compiler prelude sources.

A compiler runs its own linked copies of the components’ translators, as it runs its own kernel. An edited component therefore changes the compiler built from it, not the compiler that translates the edit.