The language for building languages

Course 5 · Part 0What are we building?

The final result

A toolkit, langfac, for defining small domain-specific languages — and three real languages built with it. First, an interpreter for a tiny configuration language, built from ordinary functions. Then a finance DSL with its own validation and a small type checker, first as a macro-based extension of Racket, and finally as a genuine standalone language: files that begin #lang finance and run, with no quotation marks around "language" needed. A robot-control DSL follows, built as a reusable interpreter library rather than a one-off. The course closes by compiling that same finance language to Racket code directly, instead of interpreting it, and measuring the difference — and by building one final, capstone language for describing simple games, using every piece of the toolkit at once.

$ cat budget.finance
#lang finance

account checking balance: 2400.00
account savings  balance: 8000.00

rule "emergency fund"
  when savings.balance < 6 * monthly-expenses
  alert "top up savings"

$ racket budget.finance
checking: $2,400.00
savings:  $8,000.00
rule "emergency fund": ok (savings covers 6.0x monthly expenses)

That file runs with the ordinary racket command — no special interpreter to invoke, no wrapper script. By Milestone 9, #lang finance is a real, first-class language exactly as far as the operating system, your editor, and Racket's own module system are concerned.

Why this project is interesting

Every course so far has, at some point, reached the edge of what its host language's syntax could express directly and built something to work around it: Ruby's course built an internal DSL — Ruby code that reads like a pipeline description, using blocks and method_missing to blur the line between data and executable code, but is still, underneath, ordinary Ruby method calls. Perl's course parsed text formats that were never Perl at all. This course asks a different, more radical question: what if the tool for describing "a language that reads like configuration, but is checked and compiled" were not a clever use of an existing language's syntax, but an actual, first-class mechanism the language gives you on purpose? Racket's answer is macros powerful enough to add real syntax, phase separation precise enough to run code at compile time safely, and a module system open enough that "a new #lang" is a supported, ordinary thing to build — not a hack, not a wrapper, not a subset of an existing parser bent sideways.

Why Racket in particular

Three properties, found together nowhere else in this curriculum.

Code is data, structurally, not just in spirit. A Racket program is, at the syntax level, a tree of parenthesized expressions — an s-expression — which is also exactly the shape Racket uses to represent data. (+ 1 2) is a valid piece of code and, quoted, a valid list containing the symbol + and two numbers. This is not a party trick: it is what makes it possible to write a program that manipulates other programs using the same list-processing tools you already know from Section 2.2, rather than a separate parser and AST library bolted on afterward.

Macros run at compile time, with hygiene guaranteed. A macro is a function from syntax to syntax, run by the compiler before your program executes, and Racket's macro system guarantees — checked by the language, not by convention — that a macro cannot accidentally capture a variable from the code that uses it, or vice versa. Ruby's method_missing-based DSLs, met in Course 2, run at ordinary run time and cannot add new syntax at all — a Ruby DSL is still legal Ruby, parsed by Ruby's one fixed grammar. A Racket macro can add genuinely new syntactic forms, checked before the program ever runs, and Milestone 5 shows exactly what "hygiene, guaranteed" is protecting you from by deliberately trying to break it.

#lang is not a special case. Racket itself — the language you have been learning is called #lang racket — is implemented using the same mechanism this course teaches you to use for your own languages. There is no privileged, built-in-only way to make #lang whatever work that your own #lang finance cannot also use. By Milestone 9 you are not approximating a real language, you are building one, with the same standing as any other.

Architecture we are building toward

          text (a .finance or .robot file, or Racket-hosted DSL code)
                              │
                    ┌─────────────────┐
                    │  Reader          │  turns raw text into syntax objects
                    │  (Milestone 9)   │  (source locations attached)
                    └────────┬─────────┘
                              │  syntax objects (code AND data, same shape)
                    ┌─────────────────┐
                    │  Macros          │  syntax → syntax, at compile time
                    │  (Milestones     │  (syntax-parse, syntax classes,
                    │   5, 6, 7)        │   compile-time validation)
                    └────────┬─────────┘
                              │  expands into
                    ┌─────────────────┐
                    │  Interpreter, OR │  Milestones 4, 10: walk an AST
                    │  Compiler        │  Milestone 11: macro-expand straight
                    │  (Milestones      │   into executable Racket, no AST walk
                    │   4, 10, 11)       │   at run time at all
                    └────────┬─────────┘
                              │
                          a running program

Two genuinely different strategies for "make this DSL run," both built in this course, on purpose: interpretation (an AST, a function that walks it, an environment mapping names to values — Milestone 4's config language and Milestone 10's robot DSL) and macro-based compilation (the DSL's syntax expands directly into ordinary Racket code, which Racket's own compiler then optimises exactly as if you had hand-written it — Milestone 11). Neither is universally better; Milestone 11 measures the difference on the same finance language both ways.

The twelve milestones

What you will know afterwards

Why "code is data" is a structural property of s-expressions and not a metaphor; how to write a macro that cannot accidentally capture or be captured, and why that guarantee is checked rather than a matter of discipline; the difference between run time and compile time well enough to know which one a given piece of code you write actually executes in; how an interpreter and a compiler answer the same "make this DSL run" question differently, with different costs; and what it actually takes, mechanically, to turn a language design into a #lang that racket itself will run.

Course 5 · Part 1Install and first program

What Racket is

Racket is a descendant of Scheme — itself a dialect of Lisp — developed since the mid-1990s (originally as PLT Scheme) with language creation as an explicit, central design goal rather than a byproduct. This course was verified on Racket 8.7 [cs] — the [cs] marks the "Chez Scheme" backend, Racket's current default implementation strategy, compiling through Chez Scheme rather than Racket's older custom virtual machine.

Racket is not "just Scheme" for this course's purposes, for one specific reason: its macro system (syntax-parse, syntax classes, module-level phase separation) and its #lang mechanism go considerably further than standard Scheme's syntax-rules, and that additional machinery — not the base language's parentheses — is this entire course's actual subject.

Installing

Linux

sudo apt install racket        # Debian/Ubuntu — often slightly behind
                                # the latest release; fine for this course
# or the official installer, for the newest version:
# https://download.racket-lang.org

racket --version
# Welcome to Racket v8.7 [cs].

macOS

brew install --cask racket

Windows

:: the official installer from racket-lang.org is the straightforward path
winget install Racket.Racket

raco, Racket's tool

raco pkg install <name>   # install a package, like gem install or go get
raco make file.rkt         # compile a file (and cache the result — makes
                           # repeated runs faster, the same idea as Go's
                           # build cache)
raco test file.rkt         # run rackunit tests found in a module
raco exe file.rkt           # produce a standalone executable
raco demod file.rkt          # show a fully macro-expanded module — the
                             # single most useful debugging tool once
                             # Milestone 5 starts, because it shows you
                             # exactly what your macro actually expanded to

raco demod deserves to be called out this early even though nothing until Milestone 5 needs it: the entire back half of this course is macros transforming code into other code, and the single best debugging technique for "my macro did something I did not expect" is looking at exactly what it expanded into, rather than guessing.

DrRacket, and why this course still uses a plain text editor

DrRacket, bundled with every Racket install, is a full IDE built specifically for this language, with a genuinely excellent macro-stepper (Milestone 5 onward, Macro Stepper under the Languages menu shows a macro expanding one step at a time, visually) and syntax-aware structure editing. It is worth opening at least once for that stepper alone. This course's own examples are shown as plain .rkt files runnable from any editor plus racket file.rkt at a terminal, because that is the more transferable skill and the shape every milestone's code in this document takes — but "open this exact file in DrRacket and use the macro stepper" is a standing, implicit suggestion for anything in Milestones 5 through 8 that feels confusing on the page.

Hello, factory

#lang racket

(displayln "hello, language factory")

$ racket hello.rkt
hello, language factory

Every line, explained

• #lang racket is not a comment and not decoration — it is the single most important line in the file, and this whole course is, in a real sense, about what comes after the words #lang. It tells Racket's own module system which language the rest of the file is written in, which determines how the reader turns the following text into syntax, and which determines what identifiers like displayln even mean. By Milestone 9, your own files will start #lang finance instead, and this line is exactly what makes that meaningful rather than cosmetic.
• (displayln "hello, language factory") is a function call: parenthesis, then the function, then its arguments, no commas, no special call syntax — this is the entirety of Racket's function-call grammar, and Section 2.1 explains why having only one call syntax at all is not a limitation.

A real script takes real arguments

racket file.rkt is a bare invocation with no arguments — most of this course's own example scripts stay that way, deliberately, to keep the code on the page focused on whatever that milestone is actually teaching. A script meant to be run repeatedly from a shell, the way budget.finance will be from Milestone 9 onward, usually wants real command-line arguments instead of a hardcoded value, and racket/cmdline is the standard library for that — Racket's equivalent of Go's flag package or Python's argparse.

#lang racket
(require racket/cmdline)

(define name "language factory")

(command-line
 #:program "hello"
 #:once-each
 [("-n" "--name") n "Who to greet" (set! name n)]
 #:args () (void))

(displayln (format "hello, ~a" name))

$ racket hello.rkt
hello, language factory
$ racket hello.rkt --name "budget.finance reader"
hello, budget.finance reader
$ racket hello.rkt --help
usage: hello [ <option> ... ]

<option> is one of

  -n <n>, --name <n>
     Who to greet
  --help, -h
     Show this help
  --
     Do not treat any remaining argument as a switch (at this level)

--help is generated for you, from the same #:once-each clause that declares --name — the one-line description "Who to greet" is what shows up next to it, which is the same "declare it once, get the documentation for free" idea syntax-parse's syntax classes bring to macros in Milestone 6, applied here to an ordinary script's flags instead.

The REPL

$ racket
Welcome to Racket v8.7 [cs].
> (+ 1 2)
3
> (define (square x) (* x x))
> (square 5)
25
> (require racket/list)
> (first '(a b c))
'a

racket with no file argument drops you into a REPL exactly like erl or irb — definitions and expressions typed directly, evaluated immediately. (require racket/list) pulls in one of Racket's many small, focused libraries; racket the language you get from #lang racket already includes a large, convenient standard set, and #lang racket/base — a smaller, faster-loading language this course's own toolkit code prefers once performance starts to matter in Milestone 11 — includes much less, requiring you to require things explicitly.

Documentation

$ raco docs                      # opens the full local documentation in a browser
> (require racket/list)
> ,doc first                       # inside the REPL: jump straight to a
                                   # function's documentation entry

Racket's documentation (locally installed, searchable, and the same content as docs.racket-lang.org) is unusually thorough by the standards of this curriculum — every function's contract, every macro's grammar, and, critically for this course, a complete guide to macros (raco docs, then "Macros and Languages") that this course's Milestones 5 through 9 draw on directly rather than duplicate.

Project layout and testing, from day one

langfac/
├── info.rkt              package metadata (like a gemspec or go.mod)
├── main.rkt               re-exports the toolkit's public API
├── interp/
│   └── config.rkt           Milestone 4's config-language interpreter
└── tests/
    └── config-tests.rkt

#lang racket
(require rackunit)

(check-equal? (+ 1 2) 3)
(check-equal? (+ 1 2) 4)     ;; deliberately wrong, to see real failure output

$ raco test scratch.rkt
--------------------
FAILURE
name:       check-equal?
location:   scratch.rkt:5:0
actual:     3
expected:   4
--------------------
1 success(es) 1 failure(s) 0 error(s) 0 test(s) skipped

rackunit ships with Racket — no dependency to add. raco test finds every check-* call in a module (there is no separate "test function" naming convention to learn; any check-equal?, run at module load time, counts) and reports failures with the exact location, the actual value, and the expected one — deliberately shown failing once here, immediately, so the very first thing you see from this course's testing tool is what a real failure looks like.

Style and formatting

Racket ships no equivalent of gofmt — there is no single official formatter that every .rkt file is expected to already agree with, and no build step in this course's own langfac project runs one. Style here is convention-driven: the Racket Style Guide (linked from raco docs) documents indentation, naming, and module-organisation conventions the community broadly follows, and DrRacket's own structural editor auto-indents new code to match them as you type, which is where most Racket programmers get consistent formatting in practice — by writing inside an editor that already knows the convention, rather than by running a separate formatting pass afterward. A third-party formatter, fmt (raco pkg install fmt, then raco fmt file.rkt), exists and is worth knowing about if a project wants an enforceable, CI-checkable formatting rule the way gofmt -l gives Go — but it is a community package, not part of the language distribution, and this course's own code was formatted by hand against the Style Guide's conventions rather than by running it.

Checkpoint

1. What does the #lang line at the top of a file actually determine?
2. What is the difference between #lang racket and #lang racket/base, and why might a library prefer the smaller one?
3. What does raco demod show you, and why is it specifically useful once you start writing macros?

Course 5 · Part 2Language crash course

2.1 S-expressions: one grammar for everything

Every Racket expression is either an atom (a number, a string, a symbol, a boolean) or a parenthesized list whose first element says what to do with the rest. There is no operator precedence to memorise, no distinction between a function call's syntax and a special form's syntax at the level that matters for this course — (+ 1 2), (if a b c), and (my-macro x y) are, before anything decides otherwise, the identical shape: a list starting with an identifier.

> (+ 1 2 3)
6
> (+ 1 (* 2 3))
7
> '(+ 1 2)
'(+ 1 2)
> (eval '(+ 1 2))
3

The quote (') is the whole of Section 2.1's point made concrete: (+ 1 2) is code, evaluated immediately; '(+ 1 2) is the identical text, but quoted — told "do not evaluate this, hand it to me as data" — and what comes back is an ordinary three-element list, containing the symbol + and two numbers, inspectable and manipulable with the same list functions Section 2.2 covers. eval closes the loop: handed that list back, it runs it as code. Code and data are, literally, the same representation, and quoting is the switch between treating a given piece of text as one or the other.

(require racket/list)

(define (operators expr)
  (cond
    [(not (list? expr)) '()]
    [(empty? expr) '()]
    [else (cons (first expr)
                (append-map operators (rest expr)))]))

2.2 Lists, pairs, and higher-order functions

> (define xs (list 1 2 3 4 5))
> (map (lambda (x) (* x x)) xs)
'(1 4 9 16 25)
> (filter even? xs)
'(2 4)
> (foldl + 0 xs)
15
> (cons 0 xs)
'(0 1 2 3 4 5)
> (first xs)
1
> (rest xs)
'(2 3 4 5)

A Racket list is, underneath, a chain of two-element pairs (cons cells) — (cons 0 xs) builds a new pair whose first half is 0 and whose second half is the existing list xs, and a "list" is precisely a chain of these ending in the empty list '(). map, filter and foldl need no introduction if Course 2's Ruby blocks or Course 1's Go closures are still fresh — the shape (a function, a collection, a new collection or a single accumulated value) is the same idea in a fourth syntax.

> (map + (list 1 2 3) (list 10 20))
map: all lists must have same size
  first list length: 3
  other list length: 2
  procedure: #<procedure:+>

(define (total-of pred xs) (foldl + 0 (filter pred xs)))

2.3 Recursion

(define (sum-list xs)
  (cond
    [(empty? xs) 0]
    [else (+ (first xs) (sum-list (rest xs)))]))

(define (sum-tail xs [acc 0])
  (cond
    [(empty? xs) acc]
    [else (sum-tail (rest xs) (+ acc (first xs)))]))

[acc 0] in the parameter list is a default argument — (sum-tail xs) and (sum-tail xs 10) are both valid calls. Racket's compiler recognises and optimises tail calls, the same guarantee Erlang's course relied on for unbounded recursion depth, though in ordinary Racket code you will reach for for, for/list, and the higher-order functions from Section 2.2 far more often than hand-written recursion — they exist precisely so you do not have to write sum-tail-shaped functions by hand for every single loop.

(define (deep-sum xs)
  (cond
    [(empty? xs) 0]
    [(list? (first xs)) (+ (deep-sum (first xs)) (deep-sum (rest xs)))]
    [else (+ (first xs) (deep-sum (rest xs)))]))

2.4 match: pattern matching over any shape

(define (describe v)
  (match v
    [(list a b) (format "pair: ~a and ~a" a b)]
    [(? string?) "a string"]
    [(? number? n) #:when (negative? n) "a negative number"]
    [(? number?) "a number"]
    [_ "something else"]))

> (describe (list 1 2))
"pair: 1 and 2"
> (describe "hi")
"a string"
> (describe -5)
"a negative number"
> (describe 5)
"a number"

match is Racket's general-purpose destructuring and dispatch tool, playing the same role Erlang's function clauses and guards played in Course 4 — a list shape, a predicate ((? string?)), a predicate with a bound name and a guard clause, and a wildcard, tried top to bottom. Milestone 3 extends this to match on struct shapes directly, and Milestone 4's interpreter is built almost entirely out of one large match over AST node types.

> (describe 42)
match: no matching clause for 42

2.5 Structs

(struct point (x y) #:transparent)

(define p (point 3 4))
(point-x p)          ; 3
(point? p)            ; #t
(struct-copy point p [x 10])   ; point with x replaced, y unchanged

struct declares a new data type with named fields, a constructor (point itself, callable), automatic accessors (point-x, point-y), and a predicate (point?) — all generated from one declaration, the same convenience Erlang's maps and Go's structs each provide differently. #:transparent matters specifically for this course: without it, two structurally-identical struct instances print opaquely and are not equal? to each other by value, which makes testing (Section 1's check-equal?) and debugging output far less useful — every struct in this course's own code is transparent unless there is a specific reason to hide its contents.

(define (midpoint p1 p2)
  (point (/ (+ (point-x p1) (point-x p2)) 2)
         (/ (+ (point-y p1) (point-y p2)) 2)))

2.6 Contracts: specifications the runtime checks for you

#lang racket
(provide (contract-out
  [account-balance (-> account? real?)]
  [withdraw (-> account? (and/c real? positive?) account?)]))

> (withdraw checking -50)
withdraw: contract violation
  expected: a number strictly greater than 0
  given: -50
  in: the 2nd argument of
      (-> account? (and/c real? positive?) account?)
  contract from: (module account)
  blaming: (module account)
   (assuming the contract is correct)
  at: account.rkt:7:5

A contract on provide is an executable specification, checked automatically at the module boundary every time an outside caller uses the function — not a comment describing what should be true, a runtime check that catches exactly the class of "I called this with the wrong kind of value" bug. Notice how much of the report is about where responsibility lies, not just what went wrong: expected is stated in plain English rather than echoing the contract expression verbatim (and/c real? positive? becomes "a number strictly greater than 0"), and blaming names which module's code is at fault for the violation — for a contract this simple the caller is obviously to blame, but for a contract built from several composed pieces across several modules, knowing exactly which one to blame is often the entire value of the error message.

;; acct.rkt
(define (withdraw balance amt) (- balance amt))
(define (withdraw-unchecked balance amt) (withdraw balance amt))  ; internal call, no check
(provide (contract-out [withdraw (-> real? (and/c real? positive?) real?)])
         withdraw-unchecked)

> (withdraw-unchecked 100 -50)   ; from another module -- contract never fires
150
> (withdraw 100 -50)              ; from another module -- contract fires
withdraw: contract violation
  expected: a number strictly greater than 0

2.7 A first taste of macros

You will not write a real macro until Milestone 5, but the shape is worth seeing once now:

(define-syntax-rule (my-unless condition body)
  (if condition (void) body))

(my-unless #f (displayln "this prints"))
(my-unless #t (displayln "this does not"))

define-syntax-rule declares a macro by pattern: wherever (my-unless condition body) appears in your code, the compiler replaces it, before your program runs, with (if condition (void) body), substituting whatever you wrote for condition and body. Compare this to a function: a function receives already- evaluated values; a macro receives un-evaluated syntax and produces more syntax. That distinction — value at run time versus syntax at compile time — is the entire subject of Milestone 5 onward, and Section 2.1's "code is data" is precisely what makes writing the right-hand side of a macro feel like ordinary list manipulation rather than a separate skill.

2.8 Errors and exceptions

(define (safe-divide a b)
  (with-handlers ([exn:fail? (lambda (e) (displayln (exn-message e)) #f)])
    (/ a b)))

> (safe-divide 10 2)
5
> (safe-divide 10 0)
/: division by zero
#f

with-handlers is Racket's try/catch, matching an exception predicate (exn:fail? catches ordinary errors; more specific predicates exist for narrower cases) to a handler function. error raises one: (error 'my-function "bad input: ~a" val) — the leading symbol names the raising context, which shows up in the message exactly as eval-expr did in the interpreter you will build in Milestone 4.

(define (safe-first xs default)
  (with-handlers ([exn:fail? (lambda (e) default)])
    (first xs)))

2.9 Modules, require, and provide

;; geometry.rkt
#lang racket
(provide point distance)   ;; only these are visible outside this module

(struct point (x y) #:transparent)
(define (distance p1 p2)
  (sqrt (+ (sqr (- (point-x p2) (point-x p1)))
           (sqr (- (point-y p2) (point-y p1))))))

;; main.rkt
#lang racket
(require "geometry.rkt")

(distance (point 0 0) (point 3 4))   ; 5

provide is Racket's export list, playing the same role as Erlang's -export or Go's capitalised names — anything not listed is private to the module. (require "geometry.rkt") with a quoted relative path pulls in a local file; (require racket/list) without quotes pulls in an installed collection or package. Every langfac module from Milestone 4 onward follows this shape: a focused module, an explicit, deliberately narrow provide list.

Part 2 checkpoint

1. What does quoting a piece of Racket code actually do, precisely?
2. Why does #:transparent matter for testing a struct with check-equal??
3. What is the fundamental difference between what a function receives and what a macro receives?
4. What does a contract on provide check, and when — at definition time, or at every call from outside the module?

What is next

Milestone 1 turns the shell experiments above into a real langfac project. Milestones 2 and 3 build the data structures and validated boundaries the rest of the toolkit needs. Milestone 4 is where "code is data" stops being a slogan and becomes a working interpreter for a real, if small, language.