What an experienced Erlang programmer wires together next, and whether you can now explain any of it

Part AThe advanced phase

The mesh works: it supervises thousands of nodes, survives chaos injected on purpose, reports itself over HTTP, and runs across real, separate machines. These five topics are what you would reach for if this were a system you had to actually operate, not a system you had to finish.

A1 · Wiring the real supervision tree

Every supervisor so far has been started and exercised individually, from the shell. A real OTP application starts its whole tree from one place, deterministically, in dependency order.

%% src/mesh_sup.erl
-module(mesh_sup).
-behaviour(supervisor).
-export([start_link/0, init/1]).

start_link() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

init([]) ->
    SupFlags = #{strategy => one_for_one, intensity => 5, period => 10},
    Children = [
        #{id => mesh_metrics,   start => {mesh_metrics, start_link, []}},
        #{id => node_sup,       start => {node_sup, start_link, []}, type => supervisor},
        #{id => mesh_dashboard, start => {mesh_dashboard, start_link, [8080]}}
    ],
    {ok, {SupFlags, Children}}.

Order is not cosmetic. Children start left to right and stop right to left, so mesh_metrics — which everything else calls into — is guaranteed running before node_sup starts spawning nodes that might immediately increment a counter, and mesh_dashboard, which reads metrics, starts last, after there is something meaningful to read. Getting this order wrong is a real, common bug: a child whose init/1 happens to call another child that has not started yet fails on startup, non-deterministically, depending on how fast each child happens to initialise — which is exactly the kind of bug that passes in development and fails once in a hundred production restarts.

What to measure: total startup time for the whole tree at your target node count, and whether it is dominated by node_sup spawning thousands of children or by something else — Milestone 7's 13ms-for-2,000-nodes number says it should not be the nodes themselves.

A2 · Process groups instead of scanning the whole registry

mesh_registry:broadcast/1 from Milestone 7's exercise walks every entry in one ETS table. That is fine at a few thousand nodes and starts to be the wrong tool past that, because every broadcast is O(n) work on the broadcasting process specifically. OTP's pg module (process groups, distribution-aware since OTP 23) exists for exactly "a dynamic set of processes I want to message as a group":

pg:start_link(),
pg:join(mesh_nodes, self()),           %% called from inside mesh_node:init/1
[Pid ! Msg || Pid <- pg:get_members(mesh_nodes)].

The advantage over the hand-rolled ETS version is not raw speed at this scale — measured, they are close — it is that pg groups are distribution-aware for free: a process group spans every connected distributed node automatically, so pg:get_members/1 called from nodeb in Milestone 11's two-node setup returns members living on nodea too, with no extra code. The hand-rolled ETS registry from Milestone 7 is local to whichever node it is running on; making it distributed would mean building the exact replication pg already has.

What to measure: broadcast latency at 2,000 and 20,000 members, both ways, and whether pg's distribution awareness changes anything about your own registry design once you have it.

A3 · Structured logs with logger

OTP's built-in logger (standard since OTP 21) gives structured, leveled logging with no dependency, the same shape Go's log/slog gave Course 1's colony:

logger:info("node started", #{id => Id, pid => self()}),
logger:warning("restart intensity exceeded", #{supervisor => node_sup, window_s => 5}),

2026-09-22T13:44:10.552Z info: node started id=42 pid=<0.94.0>

Two rules, matching the ones this curriculum has already argued for twice, in two other languages: never log per message — a mesh of two thousand nodes routing thousands of messages a second will produce far more log volume than the metrics counters ever will, for far less signal — and log the exceptional, count the routine, which is precisely why mesh_metrics:inc/1 exists as a separate mechanism from logging at all.

A4 · Deterministic chaos, properly replayable

Milestone 8's rand:seed/2 makes an attack pattern reproducible if nothing else about the run varies — Exercise 8's third part already found the gap: the input (which ids exist) has to be deterministic too. The complete fix records the schedule itself rather than trusting reseeded randomness to reproduce it exactly across Erlang/OTP versions, whose rand algorithm choices are not guaranteed stable forever:

build_schedule(Seed, NumAttacks, Ids) ->
    rand:seed(exsplus, {Seed, Seed, Seed}),
    [{lists:nth(rand:uniform(length(Ids)), Ids), rand:uniform()}
     || _ <- lists:seq(1, NumAttacks)].

A schedule built once and written to disk (file:write_file/2 with term_to_binary/1, or plain text via ~p) replays identically regardless of what rand does internally in a future OTP release — the same lesson Go's Course 1 learned building its own Schedule struct, arrived at for the same reason: a schedule is data, and data outlives the specific random-number generator that produced it.

A5 · Shipping it

FROM erlang:25-slim AS build
WORKDIR /src
COPY . .
RUN rebar3 release

FROM debian:bookworm-slim
COPY --from=build /src/_build/default/rel/mesh /opt/mesh
EXPOSE 8080
USER nobody
ENTRYPOINT ["/opt/mesh/bin/mesh", "foreground"]

A release (Milestone 12) is already the deployment artefact — this Dockerfile does almost nothing beyond copying it into a smaller base image. foreground rather than daemon for the entrypoint matters specifically in a container: a container's process manager expects the entrypoint to be the running process (PID 1), not a script that forks a daemon and exits immediately, which from the container runtime's point of view looks like the container finishing its work and stopping.

Part BThe final challenge

Everything up to here had a solution a few paragraphs later. This one does not, and it is deliberately at the edge of what you can now do.

Failover across real distributed nodes, with no double ownership

Milestone 11 connected two BEAM instances and sent one message between them. Now make them cooperate for real: split the mesh's id space across two or more distributed Erlang nodes, and when one goes down, have another take over serving its ids — without ever having two live nodes simultaneously answering for the same id.

Requirements

1. Run 3 distributed nodes (start with -sname on one machine, same as Milestone 11). Statically assign each a contiguous band of ids at startup (node 1: ids 1–999, node 2: 1000–1999, node 3: 2000–2999).
2. Each node runs its own mesh_sup tree, owning only its own band.
3. A client process (anywhere) can ask any connected node "what is the status of id N", and get a correct answer regardless of which node actually owns that id — the client should never need to know the sharding scheme.
4. Kill a node outright (erlang:halt() called on it, or killing its OS process). Within a bounded time, exactly one surviving node must take over serving status queries for the dead node's band — not necessarily its exact process state (that was lost, honestly, with the dead node), but at minimum a correct "this id exists in a dead band, here is what we last knew" answer.
5. Restart the dead node. It must not resume owning its old band while another node is actively covering for it — reconcile explicitly, on a schedule you define and document, not by accident of timing.

Constraints

• No external coordination service (no etcd, no Redis). Standard library and OTP only.
• No single point of failure for the failover decision itself — the mechanism that decides "node 2 is down, node 3 covers it now" must not depend on one specific node being up to make that call.
• At most one node may believe it owns a given band at any moment — momentary disagreement during a transition is acceptable and must be bounded and documented; permanent or unbounded double ownership is not.

Acceptance criteria

1. No double ownership, ever, under a script that kills and restarts nodes randomly for 60 seconds. Log every ownership claim with a timestamp and assert, after the run, that no two claims for the same band overlap in time by more than your documented transition bound.
2. Every status query, from any node, for any id, gets a response — including during a failover window, where "band N is currently being taken over, try again shortly" is an acceptable honest answer, but silence or a crash is not.
3. A killed node's band is covered within a documented, bounded time after the kill is detected — state the number and justify it against your detection mechanism's own latency.
4. rebar3 do eunit, ct passes, including at least one Common Test suite that actually starts multiple distributed nodes (ct_slave or peer, OTP's own facilities for spawning real additional nodes from inside a test).
5. A documented failure analysis, in the same shape as Go Course 1's final challenge: every point something can go wrong, and why the design is safe (or explicitly not) at that point.

Hints, in increasing order of how much they give away

• You already have the primitive that detects a node going down cleanly: net_kernel:monitor_nodes(true) delivers a {nodedown, Node} message to whichever process asked. The hard part is not detecting a clean disconnect — it is that a node which is merely slow looks identical, from a distance, to a node which is dead. What does your design do about that ambiguity?
• "No single point of failure for the failover decision" rules out one designated node always deciding — think about what happens if that specific node is the one that dies. Every remaining node needs to be able to reach the same conclusion independently, or agree with each other, without asking a node that might not be there to ask.
• global:register_name/2 gives you a name that is unique across every connected node, enforced by Erlang's own distribution kernel — think about what it would mean for "who currently owns band 2" to be exactly this kind of name, claimed by whichever node currently believes it owns that band, and what happens automatically when the process holding that name dies.
• The restarted-node reconciliation requirement (5) is the same shape as Course 1's migration challenge's dedupe-table forgetting problem: a node coming back cannot simply reclaim its old band on the assumption that no one else has picked it up — it has to ask, and the asking itself needs an answer even if the node it would ask is the one now covering for it.

Attempt it before reading on. Even a partial implementation with an honest account of what does not fully work is worth more than the section below.

%% src/band_owner.erl
-module(band_owner).
-behaviour(gen_server).
-export([start_link/1, claim/1, whois/1]).
-export([init/1, handle_call/3, handle_info/2]).

start_link(Band) -> gen_server:start_link(?MODULE, Band, []).

claim(Band) ->
    Name = {band_owner, Band},
    case global:register_name(Name, self()) of
        yes -> owned;
        no  -> {owned_by, global:whereis_name(Name)}
    end.

whois(Band) ->
    case global:whereis_name({band_owner, Band}) of
        undefined -> unowned;
        Pid -> {ok, Pid}
    end.

init(Band) ->
    net_kernel:monitor_nodes(true),
    self() ! {try_claim, Band},
    {ok, #{band => Band, owned => false}}.

handle_info({try_claim, Band}, State) ->
    case claim(Band) of
        owned ->
            logger:info("band claimed", #{band => Band, node => node()}),
            {noreply, State#{owned => true}};
        {owned_by, _Other} ->
            erlang:send_after(2000, self(), {try_claim, Band}),
            {noreply, State}
    end;
handle_info({nodedown, _Node}, #{band := Band} = State) ->
    %% a node vanished; whether it was THIS band's owner is unknown from
    %% here directly, so re-attempt the claim -- if this node already
    %% owns it, global:register_name/2 on the same name is a safe no-op
    %% for the current holder
    self() ! {try_claim, Band},
    {noreply, State}.

Part CKnowledge check

C1 · Twenty conceptual questions

1. What does = actually do in Erlang, precisely, and why is calling it "assignment" wrong?
2. Why is a tail call compiled to a jump rather than a new stack frame, and what class of program does that guarantee make practical that would not be otherwise?
3. Explain the difference between a link and a monitor in terms of direction and default effect, and name a place in this project each was the correct choice.
4. Why does trap_exit change what a link does, rather than simply ignoring exit signals altogether?
5. What is the actual difference between error, throw, and exit as ways to signal something exceptional?
6. Why does this course's philosophy argue for reaching for try/catch less often than instinct suggests, and where does it say the boundary still belongs?
7. What does an OTP behaviour actually check at compile time, and how does that differ from how a Go interface is satisfied?
8. Explain simple_one_for_one versus plain one_for_one, and which shape of child population each fits.
9. What do a supervisor's intensity and period actually bound together, and what happens once that bound is exceeded?
10. Why is a process's mailbox described as unbounded, and what real cost follows from that when a caster outpaces its target?
11. What is selective receive, and why does the Ref-tagging convention specifically enable it for request/reply?
12. Why is ETS the right tool for a registry that many processes read and write concurrently, compared to funnelling every lookup through one process?
13. What does a network partition actually do to the processes on either side of it, and what does it not do?
14. Why can Erlang's distribution layer route a message to a remote pid with no protocol code written by the application, where Go's Milestone 12 needed an explicit wire protocol?
15. What does the shared cookie between two distributed nodes actually authorise, and what does it not protect against?
16. Why is tracing every call to a hot function on a live system dangerous in a way that is specific to what tracing costs, not just "debugging in production is risky" in the abstract?
17. What is the difference between what EUnit, Common Test, and PropEr are each best suited to test?
18. Why does a property-based test search for a counterexample rather than checking specific examples, and what kind of bug is it more likely to find than a hand-written test suite?
19. Explain why heart (or an external process manager) is necessary even though every supervisor tree in this project is, itself, a fault-tolerance mechanism.
20. State, in your own words, what this course's central comparison to the Perl course's "a line you cannot parse is still evidence" rule actually is, and why both are correct for what each system is protecting.

C2 · Ten code-reading questions

Predict the output of each, then check. All ten were run to confirm the answers.

%% 1
X = 1 == 1.0,
Y = 1 =:= 1.0,
io:format("~p ~p~n", [X, Y]).

%% 2
F = fun() ->
    L = [N * 2 || N <- [1,2,3,4], N rem 2 =/= 0],
    L
end,
io:format("~p~n", [F()]).

%% 3
io:format("~p~n", [length([1,2,3|[4,5]])]).

%% 4
A = try error(oops) catch _:_ -> caught end,
io:format("~p~n", [A]).

%% 5
R = (catch 1/0),
io:format("~p~n", [element(1, R)]).

%% 6
M0 = #{a => 1},
M1 = M0#{a => 2},
io:format("~p ~p~n", [M0, M1]).

%% 7
loop(0) -> done;
loop(N) -> loop(N - 1).
io:format("~p~n", [loop(3)]).

%% 8
Pid = spawn(fun() -> receive _ -> ok end end),
Alive1 = is_process_alive(Pid),
exit(Pid, kill),
timer:sleep(10),
Alive2 = is_process_alive(Pid),
io:format("~p ~p~n", [Alive1, Alive2]).

%% 9
{ok, S} = {ok, hello},
Result = case S of
    hello -> matched_atom;
    _ -> something_else
end,
io:format("~p~n", [Result]).

%% 10
G = fun(N) when N > 0, N < 10 -> small;
        (N) when N >= 10 -> big;
        (_) -> negative_or_zero
    end,
io:format("~p ~p ~p~n", [G(5), G(50), G(-1)]).

1.  true false
2.  [2,6]
3.  5
4.  caught
5.  'EXIT'
6.  #{a => 1} #{a => 2}
7.  done
8.  true false
9.  matched_atom
10. small big negative_or_zero

C3 · Five debugging exercises

Each gives a symptom and a suspect. Diagnose before opening the answer.

1. Symptom: a node's mailbox grows without bound over a long chaos run, and the whole process gets visibly slower over time, though it never crashes.

   handle_call(get_status, _From, State) ->
       #{status := S} = State,
       {reply, S, State}.
   %% no handle_cast/2 clause defined at all

2. Symptom: two chaos runs with the same seed crash different sets of node ids.

   Ids = [Id || {Id, _} <- ets:tab2list(mesh_registry)],  % order not guaranteed
   rand:seed(exsplus, {Seed, Seed, Seed}),
   [attack(Id) || Id <- Ids].

3. Symptom: node_sup occasionally fails to start with {error, {already_started, Pid}}, only under a fast test script that starts and stops it repeatedly.

   stop_sup() ->
       exit(whereis(node_sup), kill),
       ok.   % returns immediately; caller assumes the supervisor is gone

4. Symptom: the dashboard's /metrics endpoint occasionally hangs indefinitely for one specific client, never timing out, blocking that one connection forever.

   handle(Sock) ->
       {ok, Req} = gen_tcp:recv(Sock, 0),   % no timeout argument
       ...

5. Symptom: a supervisor configured with intensity => 3, period => 5 shuts down after only one crash during a specific test, not three.

   ChildSpec = #{id => mesh_node, start => {mesh_node, start_link, []},
                 restart => permanent, shutdown => 100},
   %% test calls mesh_node:stop/1 (a deliberate, clean stop) three times,
   %% then crashes it once, expecting 3 tolerated restarts still available

C4 · Five implementation exercises

1. A bounded mailbox pattern. gen_server mailboxes are unbounded by design; implement admission control in front of one — a wrapper that checks process_info(Pid, message_queue_len) before casting, and returns {error, overloaded} rather than casting past a configurable threshold. Measure whether this actually protects a deliberately slow-handler node from an unbounded mailbox under chaos.
2. Rolling restart. Write a function that restarts every node in node_sup one at a time, waiting for each replacement to report alive before moving to the next, with a configurable delay between each — the operational tool you would actually want before deploying a code change to a live mesh.
3. A gen_statem version of mesh_node. OTP's gen_statem behaviour models a process as an explicit state machine rather than a bag of callbacks over one opaque state term. Rebuild mesh_node with alive and crashed as explicit states, and compare: what does making the states explicit catch that the map-based status field could not?
4. Cluster-wide metrics. Using pg or global from Part A, aggregate mesh_metrics:snapshot/0 across every connected distributed node into one combined view, callable from any single node.
5. A minimal relup. Using relx's upgrade support, ship a trivial code change (a new log line in mesh_node) as a hot upgrade to a running release — no restart, the running system picks up the new code while its supervision tree and all live processes keep running. This is genuinely fiddly to get right; treat getting it to work at all as the win.

C5 · One substantial challenge

Distinct from the final challenge in Part B, and smaller, but not easy.

Build a mailbox-depth-aware load shedder. Give every node's gen_server a way to report its own mailbox depth on every message handled (process_info(self(), message_queue_len), cheap enough to call routinely), publish it to mesh_metrics, and have mesh_registry:route/2 refuse to route a new message to a node whose reported depth is above a threshold, returning {error, node_overloaded} instead of adding to an already-backed-up mailbox.

Requirements: the threshold must be per-node, not global, since Milestone 8's slow-handler attack targets individual nodes, not the whole mesh; the mechanism must not itself become a bottleneck at 2,000 nodes (measure the added cost of a depth check on every routed message); and you must demonstrate, under chaos, that a genuinely overloaded node's mailbox depth stays bounded rather than growing without limit the way an unprotected one does. Hint: reporting depth on every handled message is itself extra work on the hot path — consider reporting it periodically instead, and what staleness that trades away.

C6 · You should now be able to explain

• Why pattern matching subsumes assignment, destructuring, and shape assertion as one mechanism.
• Tail recursion and why it is the only loop construct this language needs.
• The link/monitor/trap_exit distinction, precisely, not just "processes can watch each other."
• "Let it crash" as an engineering strategy with a specific trade-off, not a slogan.
• What an OTP behaviour actually is, and what gen_server specifically removes from hand-written process code.
• Supervision strategies, restart types, and intensity limits, and which real failure mode each one answers.
• Why ETS is the right shared-state primitive here, and what it does not give you (ordering guarantees across separate operations, for one).
• What actually changes, in code, when a message-passing system becomes genuinely distributed — and what does not.
• Why a network partition is a state-reconciliation problem, not an availability problem, for the side effects that matter.
• The honest cost side of every "why this language" argument made across this course — not just what Erlang buys you, but what it is spent on.

C7 · You should now be able to implement

• A gen_server-based process with a clean call/cast API.
• A simple_one_for_one supervisor for a dynamic pool, and a one_for_one root supervisor wiring several different kinds of children together in dependency order.
• An ETS-backed registry with automatic, monitor-driven cleanup.
• A seeded, reproducible chaos/fault-injection module.
• A minimal HTTP server on gen_tcp, including a streaming (SSE) response.
• Two real, separately-started distributed Erlang nodes that connect and message each other.
• A property-based test using PropEr, stating a real property rather than a specific example.
• A standalone relx release you can start, ping, and stop as a daemon.

Part DShipping it: README, portfolio, interview

D1 · README draft

# mesh

A simulated network of thousands of supervised Erlang processes that crash,
restart, partition and recover — a laboratory for OTP supervision, and,
in its final form, a genuinely distributed, multi-node fault-tolerant system.

No third-party dependencies beyond PropEr (dev/test only). Standard OTP.

## What it does

- Each mesh node is a gen_server, supervised by a simple_one_for_one tree
  that tolerates a bounded number of crashes before giving up and
  escalating — measured: exactly 3 restarts tolerated, the 4th within the
  same 5-second window brings the supervisor down, on command.
- An ETS-backed registry finds any node by id, with automatic,
  monitor-driven cleanup on crash — no stale entries survive a restart.
- A seeded chaos module attacks the live mesh: random crashes, slow
  handlers, malformed messages — fully reproducible from one integer.
- A dashboard, served over plain HTTP from inside the system it reports
  on, with a streaming (SSE) live view — no external web framework.
- Genuinely distributed: multiple real BEAM nodes, connected over a real
  network, sharing message-passing code that never needed to change to
  become distribution-aware.
- Ships as a standalone relx release: no Erlang installation required on
  the target machine.

## Quick start

    rebar3 release
    _build/default/rel/mesh/bin/mesh daemon
    curl http://localhost:8080/metrics
    _build/default/rel/mesh/bin/mesh stop

## Architecture

    mesh_sup (one_for_one)
      |- mesh_metrics        counters, ETS-backed
      |- node_sup (simple_one_for_one)
      |    `- mesh_node x N   gen_server, supervised, crashable on purpose
      `- mesh_dashboard       HTTP + SSE, reads mesh_metrics

## Testing

    rebar3 do eunit, ct, proper

## Known limitations

- The dashboard has no auth; it is meant for a trusted network only.
- Distribution trusts the shared cookie; there is no additional
  authentication or encryption between nodes.
- global-based failover (advanced phase) does not scale past a few
  hundred nodes gracefully — this is a stated limitation of `global`
  itself, not a bug in this project's use of it.

## Licence

MIT

Three deliberate choices, matching every README in this curriculum so far: it leads with measured numbers (the exact restart-intensity threshold, not an adjective); it states what it depends on and what it does not; and it has a known limitations section that names the real, specific boundary of global rather than implying the failover mechanism is production-ready as built.

D2 · GitHub project description

A simulated distributed network in Erlang/OTP: supervised processes that crash and recover on purpose, a live HTTP dashboard, and real multi-node failover — a laboratory for "let it crash" as an engineering strategy, not a slogan.

Topics: erlang, otp, gen-server, supervisor, fault-tolerance, distributed-systems, chaos-engineering, observability, property-based-testing.

D3 · Performance considerations

• Processes are not the bottleneck at this scale. 2,000 supervised gen_servers, registered, in 13 milliseconds; the interesting costs live in the registry's ETS access patterns and the dashboard's per-request work, not in process count.
• ETS's atomic counters remove a whole class of contention that a single counter-owning process would have introduced at real message volume.
• Tracing is not free, and a wildcard match specification on a hot function can turn a diagnostic session into the incident it was meant to diagnose.
• global does not scale indefinitely — its own documentation says so, and the final challenge's solution inherits that limit honestly rather than hiding it.
• Distribution has a real latency cost per message crossing a node boundary, the same shape of cost Go's Course 1 measured directly; this course did not repeat that specific benchmark, and doing so on your own two nodes is a reasonable extension.

D4 · Security considerations

• The distribution cookie is a shared secret, not a full authentication scheme. It authorises two nodes to trust each other; it does not encrypt traffic between them by default, and anyone who obtains it and can reach a node's distribution port can connect as a trusted peer.
• A remote shell (erl -remsh) onto a live node can do anything to it — the same operational power that makes live tracing and observer so useful is a real attack surface if the distribution port is reachable by anyone who should not have it. Bind it to a private network or a VPN, never expose it publicly.
• The dashboard has no authentication as built — every endpoint is open to anyone who can reach the port. Fine for a laboratory project on localhost; not fine unmodified in any shared environment.
• An HTTP server built directly on gen_tcp gets none of a real framework's hardening for free — request size limits, malformed-header handling, slow-client protection beyond the one timeout this course added. Treat the hand-rolled server as a teaching tool, not a production HTTP stack.
• Chaos injection is, structurally, a denial-of-service tool pointed at your own system on purpose. Never run mesh_chaos against anything you do not own and fully control.

D5 · What to put in your portfolio

Do not present this as "a chat simulation of network failures." Present it as what it is: a study of failure as a first-class, deliberately-exercised code path, ending in a real, multi-node distributed system with a documented, honest failure analysis. The narrative that makes it interesting is the escalation from hand-rolled to OTP-standard, twice.

1. A hand-rolled process and a hand-rolled watcher, each working, each with a named, specific gap.
2. The same two problems, solved by gen_server and supervisor — same external behaviour, the gaps closed, measured: exactly 3 restarts tolerated, the 4th one not.
3. A registry at real scale (2,000 nodes, 13ms), with the stale-entry bug found and fixed honestly.
4. Chaos injection absorbed silently by the supervision tree — the mesh's population recovers to full strength with no manual intervention, verified by counting live children, not by assuming.
5. Real distribution: two separate BEAM instances, a real message crossing a real network boundary, with no code written specifically to make that possible.
6. The final challenge's honest failure analysis, including what still is not solved.

Keep a docs/ folder with the restart-intensity transcript, the 2,000-node timing, and one architecture diagram. A reviewer who spends ninety seconds on your repository should come away knowing you tested failure on purpose and measured what it actually cost, not just that the happy path works.

D6 · Interview questions someone could ask, and what a good answer contains

D7 · Extensions worth building

• Real state replication for failover, closing the final challenge's most honest limitation — even a simple periodic checkpoint to the covering node would materially change what "takeover" means.
• A proper web UI over the dashboard's JSON endpoints, rather than the minimal EventSource page from Milestone 10's exercise.
• gen_statem throughout, per Exercise C4.3 — genuinely worth doing for the whole node, not just as an exercise, once you have felt what explicit states catch.
• A second, non-global failover mechanism at larger scale — Raft-based leader election is the standard next step, and building even a minimal version is one of the most educationally valuable extensions in this curriculum.
• The deterministic chaos schedule from Part A4, fully wired through — replay a specific failing run byte-for-byte, the same guarantee Course 1's Go colony built for the same reason.

Course 4 completeWhat you built

An OTP application spanning a dozen modules, a supervision tree tested to its actual restart-intensity limit rather than assumed to work, a registry proven at 2,000 nodes with a real bug found and fixed, a seeded and reproducible chaos-injection framework, a dashboard serving a live system's own health over HTTP from inside it, a real multi-node distributed cluster, and a documented, honest failover design with its limitations stated rather than hidden. More importantly: the instinct to let something crash on purpose and trust the supervisor, rather than reaching first for a defensive catch.

The central question of this curriculum was what kinds of problems does this language make unusually natural to solve? Erlang's answer, stated as precisely as this project allows: problems where the cost of one component failing must never become the cost of the whole system failing, where isolation is worth paying a copying and messaging overhead for, and where "this will be distributed eventually" is true often enough that the language should not treat distribution as a bolt-on afterthought. Not the fastest, not the simplest to learn in an afternoon, not the language you reach for when the whole problem is a tight numeric loop. The one where letting things break, on purpose, correctly, is how the system stays up.

Courses so far have paired concurrency (Go, Erlang) against expressiveness and language design (Ruby, Racket), with Perl's text-forensics work sitting adjacent to both. Course 5 closes the curriculum with Racket, and the same comparison this course drew against Go's channels and Perl's parsing philosophy gets drawn one more time, from the opposite direction: instead of a language whose runtime already gives you the primitive you need (processes, supervision), Racket is a language built around the idea that if it does not give you the primitive you need, you can build it — as a real language of your own, checked, hygienic, and yours.