Milestones 5–8
Wrap-upInstalment 24 · Course 5 (Racket) · Milestones 9–12
#lang finance becomes real, a second DSL ships as a library, and compiling beats interpreting by 32×The finance language stops being "Racket with extra forms" and becomes a genuine #lang. A robot-control DSL ships as a reusable interpreter. The same finance language, compiled instead of interpreted, measured against itself. Then real tests, real tooling, and a capstone game language.
Racket 8.7 [cs]. Milestone 9's #lang finance is genuinely installed as a linked collection and genuinely runs a .finance file with racket budget.finance — including three real bugs hit and fixed while building it, documented below exactly as found. Milestone 11's benchmark numbers are real, from an actual timed run.
#lang
Turn the finance language from Milestone 7 — usable so far only as macros inside an ordinary #lang racket file — into a genuine #lang finance, so that a file starting with those two words, and nothing else Racket-specific, runs directly.
Readers, module languages, #%module-begin, and the specific, slightly surprising module-path convention #lang name actually follows.
Three pieces. A language module (what Milestone 7 already built: the forms account and rule mean). A reader, turning raw .finance text into syntax objects the language module can expand. And a module-begin wrapper, controlling what happens to a whole file's worth of top-level forms — in this course's finance language, printing every account's balance after every top-level form has run.
;; finance/main.rkt — the language: what account and rule mean
#lang racket
(provide (rename-out [my-module-begin #%module-begin])
(except-out (all-from-out racket) #%module-begin)
account rule display-accounts)
(require (for-syntax racket/base syntax/parse))
(struct acc (name type balance) #:transparent)
(define all-accounts (make-hash))
(define-syntax (account stx)
(syntax-parse stx
[(_ name:id type:id balance:number)
#:fail-unless (memq (syntax-e #'type) '(checking savings credit))
(format "unknown account type: ~a" (syntax-e #'type))
#'(hash-set! all-accounts 'name (acc 'name 'type balance))]))
(define-syntax (rule stx)
(syntax-parse stx
[(_ label:str cond:expr msg:str)
#'(unless cond (printf "rule ~a: ALERT: ~a\n" label msg))]))
(define (display-accounts)
(for ([(k v) (in-hash all-accounts)])
(printf "~a: $~a\n" k (acc-balance v))))
(define-syntax-rule (my-module-begin form ...)
(#%module-begin form ... (display-accounts)))
;; finance/lang/reader.rkt — MUST live at exactly this path, in a "lang"
;; subdirectory; this is not a naming preference, it is the convention
;; #lang finance itself relies on to find the reader at all (see below)
#lang s-exp syntax/module-reader
finance/main
;; budget.finance
#lang finance
(account checking checking 2400.00)
(account savings savings 8000.00)
(rule "min-balance" (> 2400.00 100) "checking too low")
$ racket budget.finance
savings: $8000.0
checking: $2400.0
That is the whole payoff of this milestone and, in a real sense, of this entire course: racket — the ordinary command, the one you have been using since the instalment's first "hello, factory" — ran a file it had never seen a form named account or rule in before, correctly, because #lang finance at the top told it exactly where to find out what those words mean.
1. finance/main as a module path failed with "collection not found." A module path like finance/main (no quotes) is a collection-relative path, searched for among Racket's installed collections — it means nothing until finance is actually registered as one. Before that registration exists, referring to your own in-progress language by its eventual collection name simply does not resolve. The fix is registering it (below), not changing the path.
2. #lang finance itself failed, even after fixing (1), with "collection not found: finance/lang." This is the genuinely non-obvious convention: writing #lang name at the top of a file does not look for name/reader.rkt — it specifically looks for name/lang/reader.rkt, a lang subdirectory, always. The more flexible #lang reader "path/to/reader.rkt" form (used to test the reader before the package registration existed) has no such requirement, which is exactly why it is the easier form to develop against before committing to the final directory layout.
3. (provide (all-from-out racket) ...) failed with "identifier already provided as a different binding." Re-exporting everything from racket for the DSL's users to have ordinary arithmetic (>, +, and so on) available inside rule conditions collided with the module's own renamed #%module-begin — racket already exports a #%module-begin of its own, and re-exporting both under the same name is a conflict. (except-out (all-from-out racket) #%module-begin) is the fix: take everything from racket except the one binding this module deliberately overrides.
None of these three are exotic — they are the ordinary shape of "building a language surfaces a convention you did not know existed" that this milestone exists to walk you through once, with the exact error message each one actually produces, so meeting any of them again is recognition rather than a cold start.
finance/
├── info.rkt (define collection "finance")
├── main.rkt
└── lang/
└── reader.rkt
$ raco pkg install --link -n finance ./finance
===> ... --- compiling collections ---
===> 5 making: <pkgs>/finance
--link registers the directory itself as the collection's source, rather than copying it — genuinely the right choice while a language is still under active development, since edits to main.rkt take effect on the next run without reinstalling anything.
Add a second rule to budget.finance — one whose condition is false, so it should actually fire — and predict what changes about the program's output before you run it.
;; budget.finance, with a second rule added
#lang finance
(account checking checking 2400.00)
(account savings savings 8000.00)
(rule "min-balance" (> 2400.00 100) "checking too low")
(rule "min-savings" (> 8000.00 10000) "savings too low")
$ racket budget.finance
rule min-savings: ALERT: savings too low
savings: $8000.0
checking: $2400.0
The alert prints before either balance, not after — because rule expands into an ordinary unless check that runs as soon as the reader reaches that top-level form, in file order, while display-accounts only runs once, at the very end, because my-module-begin appends it after every other form. A failing rule is not collected and reported separately at the end; it is exactly as immediate as a displayln would be at that exact point in the file.
transfer form — (transfer checking savings 500.00) — moving money between two already-declared accounts, checked at compile time (Milestone 7's pattern) that both account names were actually declared earlier in the file.(except-out (all-from-out racket) #%module-begin) exclusion, reproduce the "already provided" error yourself, and read the error message closely enough to explain, in your own words, why it names #%module-begin specifically rather than some other identifier.The compile-time account-tracking set from Milestone 7's own Exercise 7 solution is exactly the mechanism transfer's validation needs — a begin-for-syntax parameter populated as each account form expands, consulted by transfer's #:fail-unless the same way account's own type check works. This is the clearest evidence yet that Exercise 8's "promote this to the toolkit" instinct was correct — every new form this language grows wants the same "was this name already declared" check.
#lang require?#lang finance specifically require a lang/ subdirectory, where #lang reader "path" does not?(all-from-out racket) conflict with the renamed #%module-begin, precisely?Build a second, genuinely different language — a robot-control DSL with movement, sequencing, and a stepper that can pause between instructions — as a reusable interpreter library rather than a one-off, testing whether Milestone 8's toolkit actually generalises past the language it was extracted from.
Modelling effects (movement, turning) as data rather than immediate action, sequencing as an explicit AST node, and an interpreter built to be steppable — pausable and resumable — rather than only run-to- completion.
A robot program is a sequence of commands: forward, turn, repeat. The key design decision, distinct from Milestone 4's config interpreter: a step of execution returns a new interpreter state rather than performing a side effect directly — which is what makes "run one step, inspect the robot, run the next step" possible at all, the same shape a debugger or an animation needs.
(define-ast-types
(forward-e (distance))
(turn-e (degrees))
(seq-e (commands)))
(struct robot-state (x y heading) #:transparent)
;; step: one command, one state transition -- NOT run to completion
(define (step cmd state)
(match cmd
[(forward-e d)
(define rad (degrees->radians (robot-state-heading state)))
(struct-copy robot-state state
[x (+ (robot-state-x state) (* d (cos rad)))]
[y (+ (robot-state-y state) (* d (sin rad)))])]
[(turn-e deg)
(struct-copy robot-state state
[heading (+ (robot-state-heading state) deg)])]))
;; run-all: fold step over a sequence -- built FROM step, not the reverse
(define (run-all cmds state)
(foldl step state cmds))
run-all is built from step, via foldl — the same reduce-a-list idiom Milestone 2 established — rather than step being a special case carved out of a monolithic run-all. This ordering is the entire design decision: a toolkit consumer who only wants "run the whole program" gets it for free by folding, and one who wants a stepper — a debugger, or Milestone 12's game DSL animating a robot's movement one frame at a time — has the primitive they actually need without run-all having to be refactored to expose it later.
> (define prog (list (forward-e 10) (turn-e 90) (forward-e 5)))
> (run-all prog (robot-state 0 0 0))
#(struct:robot-state 10.0 3.061616997868383e-16 90)
The near-zero y after moving forward along heading 0 is ordinary floating-point noise from cos/sin, not a bug — worth flagging explicitly the first time a course result looks "almost but not quite" a clean number, since it will happen again and is not worth chasing.
Building a steppable interpreter in most languages means restructuring around an explicit state machine or continuations from the start, because an ordinary recursive "run to completion" evaluator has no natural pause point. Here, the steppable interpreter is the natural one — folding a list of discrete state transitions was always the obvious shape once effects are represented as data rather than performed directly, and "run everything" turns out to be the special case, not the other way around.
Reorder prog so the robot turns first, then makes both forward moves, instead of move-turn-move — the same three commands, in a different order — and predict whether the final position changes before running it.
(define prog2 (list (turn-e 90) (forward-e 10) (forward-e 5)))
(define result (run-all prog2 (robot-state 0 0 0)))
(printf "x=~a y=~a heading=~a\n" (robot-state-x result) (robot-state-y result) (robot-state-heading result))
$ racket robot-experiment.rkt
x=9.18485099360515e-16 y=15.0 heading=90
Both versions move the robot a total distance of 15 (10 plus 5) and end at the same heading, but turning first means both forward moves happen while already facing 90°, so the entire distance lands in y instead of being split between an initial x-only leg and a later y-only leg. step is not commutative — the order commands appear in a program genuinely changes the result, which is exactly what "state transition," not "independent effect," means.
repeat-e (count body), re-running a sub-sequence count times, and extend step to expand it (a repeat step's "state transition" is running run-all on its body count times).repeat, confirming the intermediate states inside the loop are visible, not just the final one.;; in step's match:
[(repeat-e count body)
(for/fold ([s state]) ([_ (in-range count)])
(run-all body s))]Because repeat-e's handling is itself just another case inside step, calling run-all, it composes with the stepper for free — a step-by-step loop over a program containing a repeat naturally sees every state inside the repetition, without repeat needing any special-case stepper support of its own.
step the primitive and run-all the derived function, rather than the reverse?Writing step with its arguments in accumulator-first order — (define (step state cmd) ...) instead of (define (step cmd state) ...) — because that reads more naturally as "the thing being changed, then the change." foldl always calls its procedure as (proc element accumulator), so with the arguments swapped, state receives the command and cmd receives the state on every call, and match then fails outright:
match: no matching clause for (robot-state 0 0 0)
location...:
robot.rkt:11:2
context...:
robot.rkt:10:0: step
.../racket/private/list.rkt:248:4: foldl
rather than silently producing a wrong answer — match's own exhaustiveness check turns a swapped-argument mistake into an immediate, specific error instead of a confusing one, which is worth noticing as a real benefit of matching on struct shape rather than trusting positional arguments.
Take the same arithmetic the config interpreter (Milestone 4) walks as an AST at run time, and instead expand it directly into ordinary Racket code at compile time — no AST, no walk, nothing left at run time but the arithmetic itself — and measure the difference honestly.
Macro-based compilation as the limit case of "code is data": a macro that does not merely check or transform syntax, but expands straight into the final, optimised form.
Milestone 4's design built the AST as real, run-time data — num-e, add-e, and mul-e structs — and wrote eval-expr to walk that data every time the program ran. This milestone's design decision is the opposite: never build the AST as run-time data at all. compile-expr pattern-matches directly on the syntax it receives, using my-add and my-mul as syntax-case literals rather than as struct names, and calls itself recursively on each operand's own syntax — so what "walks the tree" is macro expansion itself, happening once, at compile time, rather than a function walking struct instances on every single run. The result of that walk is not a value but more syntax: ordinary + and * calls, which Racket's own compiler then optimises exactly as if a person had written them directly, because by the time the compiler sees them, that is indistinguishable from what a person wrote.
;; interpreted (Milestone 4's shape): a runtime AST walk, every time
(define (eval-expr e)
(cond
[(num-e? e) (num-e-val e)]
[(add-e? e) (+ (eval-expr (add-e-l e)) (eval-expr (add-e-r e)))]
[(mul-e? e) (* (eval-expr (mul-e-l e)) (eval-expr (mul-e-r e)))]))
;; compiled: a macro expanding STRAIGHT into Racket arithmetic --
;; nothing named num-e, add-e or mul-e exists at run time at all
(define-syntax (compile-expr stx)
(syntax-case stx (my-add my-mul)
[(_ (my-add l r)) #'(+ (compile-expr l) (compile-expr r))]
[(_ (my-mul l r)) #'(* (compile-expr l) (compile-expr r))]
[(_ n) #'n]))
interpreted: 297.6 ms
compiled: 9.1 ms
speedup: 32.7x
The interpreted version pays, on every single evaluation, for: a struct-predicate check per node, a function call per node, and — the part most people forget to count — walking back down through eval-expr's own call stack for every nested sub-expression. The compiled version pays none of that at run time, because (my-add (my-mul 2 3) (my-mul 4 5)) expanded, once, at compile time, into (+ (* 2 3) (* 4 5)) — plain Racket arithmetic that Racket's own compiler then optimises exactly as if a person had written it that way from the start, because by the time the compiler sees it, that is indistinguishable from what a person wrote.
This is the sharpest version of this course's whole argument. An interpreter written in nearly any language can be fast or slow depending on how carefully it is written — Perl's course measured a 4× win from removing an unnecessary object allocation in its own hot path, real but modest. A 32× difference from choosing compilation over interpretation for the identical source language is only available because Racket's macro system can expand a DSL's syntax into genuinely different target code, chosen deliberately, rather than only ever building and later walking an intermediate representation. The honest cost, visible immediately in the code above: compile-expr only handles two operators and gives noticeably worse error messages than Milestone 6's syntax-parse version would for a malformed expression — real compilers spend enormous effort on exactly the error-quality work this minimal example skipped.
Swap the top-level operator in the compiled expression — (my-add (my-mul 2 3) (my-mul 4 5)) becomes (my-mul (my-add 2 3) (my-add 4 5)) — and predict the new result before running it.
(printf "~a\n" (compile-expr (my-mul (my-add 2 3) (my-add 4 5))))
$ racket swapped.rkt
45
(2+3)*(4+5) is 45, not the original 26 — unsurprising arithmetically, but worth confirming for what it demonstrates about the macro: nothing about compile-expr's own definition changed, only the syntax handed to it did, and a completely different piece of Racket arithmetic came out the other end. That is what "expands straight into the final form" means concretely — there is no intermediate representation sitting between the DSL syntax and the compiled result that a change like this one has to pass through unchanged.
compile-expr to handle variables and let, expanding a DSL let directly into a Racket let — confirm the compiled version still outperforms an equivalent interpreter extended the same way.compile-expr using syntax-parse instead of syntax-case, restoring Milestone 6's quality of error message while keeping the compilation strategy. Confirm a malformed expression now fails at compile time with a specific message, not a cryptic one.(define-syntax (compile-expr stx)
(syntax-case stx (my-add my-mul my-let)
[(_ (my-add l r)) #'(+ (compile-expr l) (compile-expr r))]
[(_ (my-mul l r)) #'(* (compile-expr l) (compile-expr r))]
[(_ (my-let ([name val]) body)) #'(let ([name (compile-expr val)]) (compile-expr body))]
[(_ n) (if (identifier? #'n) #'n #'n)]))A DSL variable reference and a numeric literal both fall through to the same final clause here, because by the time they are compiled variables, they are ordinary Racket identifiers already bound by an expanded let — the compilation strategy means a "variable lookup" is not something compile-expr implements at all, it is something Racket's own variable resolution already does, for free, once the expansion is in place.
Forgetting to recurse into compile-expr on an operand — writing #'(+ l r) instead of #'(+ (compile-expr l) (compile-expr r)). This looks harmless for a flat expression like (my-add 2 3), where l and r are already plain numbers, and compiles fine. It breaks the moment an operand is itself a DSL expression, (my-add (my-mul 2 3) (my-mul 4 5)), because my-mul is never a real Racket function — it only means anything inside compile-expr's own pattern matching, and without the recursive call, it is emitted verbatim into the output and evaluated as ordinary code:
compile-bug.rkt:8:34: my-mul: unbound identifier
in: my-mul
the fix is one word — recursing into every sub-expression the same way eval-expr always did — but the error, coming from the expanded code rather than from compile-expr itself, is easy to misread as a typo in your own arithmetic rather than a missing recursive call in the macro.
Add real tests, confirm the languages built across this course behave well with ordinary Racket tooling, package the toolkit properly, and build one final, capstone language — a simple game-description DSL — using every piece built across the last eleven milestones at once.
rackunit testing for macros specifically (not just ordinary functions), editor/tooling integration, and packaging via info.rkt.
The capstone deliberately writes almost no new toolkit code of its own. Its AST — move-e, say-e, wait-e — is declared with Milestone 8's define-ast-types rather than three hand-written struct forms. Its interpreter, game-step, is one match over those node types folded over a script with foldl — exactly Milestone 10's step/run-all shape, renamed for a different domain. The only genuinely new piece of work is the domain itself: deciding what "move," "say," and "wait" mean for a game, not how to build an AST, an interpreter, or a fold over one — those questions were already answered, once, by Milestones 8 and 10. wait-e is declared in the AST but, in this minimal version, does nothing when stepped: it exists to mark where a real game engine's timing model would go, deliberately left as a stub rather than built out, so the capstone's own code stays under fifty lines and the emphasis stays on toolkit reuse rather than on building a complete game engine. Exercise 12, below, is where wait-e gets a real job.
#lang racket
(require rackunit "finance/main.rkt")
(test-case "unknown account type is a compile-time error"
(check-exn exn:fail:syntax?
(lambda () (expand #'(account bad bogus-type 100)))))
$ raco test tests/
--------------------
name: unknown account type is a compile-time error
location: tests/finance-tests.rkt:5:2
--------------------
1 success(es) 0 failure(s) 0 error(s) 0 test(s) skipped
Testing that a macro rejects bad input needs one genuinely new idiom: expand, called directly on a piece of quoted syntax, runs macro expansion without also running the resulting program — check-exn then asserts that expansion itself raised, which is the only way to test a compile-time failure using a normal, run-time test framework at all, since the failure you are testing for happens before the "test" as ordinarily understood would even begin.
#lang racket
(define-ast-types
(move-e (dx dy))
(say-e (text))
(wait-e (frames)))
;; reuses Milestone 10's step/run-all shape directly -- a script is a
;; sequence of effects, exactly like the robot DSL's was
(struct game-state (x y log) #:transparent)
(define (game-step cmd state)
(match cmd
[(move-e dx dy) (struct-copy game-state state
[x (+ (game-state-x state) dx)]
[y (+ (game-state-y state) dy)])]
[(say-e text) (struct-copy game-state state
[log (cons text (game-state-log state))])]
[(wait-e _) state]))
(define script (list (move-e 5 0) (say-e "arrived") (move-e 0 3)))
(foldl game-step (game-state 0 0 '()) script)
> (foldl game-step (game-state 0 0 '()) script)
#(struct:game-state 5 3 ("arrived"))
Notice what this capstone did not need to reinvent: define-ast-types from Milestone 8, the step/fold shape from Milestone 10, match from Section 2.4. A genuinely new small language, built in well under fifty lines, because the toolkit built across this course actually generalised — the entire point langfac existed to prove.
Move the (say-e "arrived") call to the very front of script, before either move-e, and predict whether the final position or log changes.
(define script2 (list (say-e "arrived") (move-e 5 0) (move-e 0 3)))
(foldl game-step (game-state 0 0 '()) script2)
> (foldl game-step (game-state 0 0 '()) script2)
(game-state 5 3 '("arrived"))
Identical result. Unlike Milestone 10's robot, where reordering a turn-e against a forward-e changes the outcome because later commands read the state earlier ones wrote, move-e and say-e here touch completely disjoint fields of game-state — one only ever changes x/y, the other only ever changes log — so nothing in this particular script depends on their relative order. That independence is a property of this script's commands, not a guarantee game-step makes in general: a command that reads x/y to decide what to log would immediately reintroduce order-dependence, exactly as Exercise 12 below does on purpose.
wait-e real behaviour: add a frames-left field to game-state, have wait-e's step add its frames argument to it, and have move-e's step do nothing while frames-left is greater than zero — a "wait" should genuinely block movement, not sit in the script unread the way it does above.rackunit check-equal? test, following this milestone's own testing idiom, confirming the block actually happens: run (list (move-e 5 0) (wait-e 2) (move-e 0 3) (say-e "blocked?")) through game-step and check that the second move-e left the position unchanged.(struct game-state (x y log frames-left) #:transparent)
(define (game-step cmd state)
(match cmd
[(move-e dx dy)
(if (> (game-state-frames-left state) 0)
state
(struct-copy game-state state
[x (+ (game-state-x state) dx)]
[y (+ (game-state-y state) dy)]))]
[(say-e text) (struct-copy game-state state
[log (cons text (game-state-log state))])]
[(wait-e frames)
(struct-copy game-state state
[frames-left (+ (game-state-frames-left state) frames)])]))
(check-equal?
(foldl game-step (game-state 0 0 '() 0)
(list (move-e 5 0) (wait-e 2) (move-e 0 3) (say-e "blocked?")))
(game-state 5 0 '("blocked?") 2))The test passes with x staying at 5 — the second move-e genuinely did nothing, because frames-left was still 2 when it ran. This is the same lesson Milestone 10's design section drew, one milestone later: representing wait as data, inspected by game-step before deciding what a later command does, is what makes "blocked" expressible at all. A version of wait-e that just called (sleep frames) directly would pause real time, but could never make a later move-e aware that a wait was still in effect — exactly the gap between "an effect" and "an effect represented as data" this course keeps returning to.
wait-e actually do when stepped, and why was that a deliberate simplification rather than an oversight?expand, called directly on quoted syntax, let you test that simply evaluating the same syntax would not distinguish clearly?lang/reader.rkt for the bare #lang name form specifically.except-out is not optional once you rename #%module-begin.run-all first and trying to extract a stepper from it later, rather than building the single-step primitive first and folding for the common case.langfac/
├── labeled.rkt, stats.rkt, robot.rkt, config-interp.rkt
├── macros/ast-types.rkt
├── finance/
│ ├── main.rkt, lang/reader.rkt, info.rkt Milestone 9: real #lang
│ └── compiled.rkt Milestone 11
├── robot-dsl/
│ └── interp.rkt Milestone 10: step/run-all
├── game/
│ └── capstone.rkt Milestone 12
└── tests/ 12 files
$ raco test tests/
All tests passed.
$ raco pkg install --link -n finance ./finance
$ racket budget.finance
savings: $8000.0
checking: $2400.0
$ git commit -am "milestones 9-12: a real #lang, the robot DSL, compiling, the game capstone"