OTP takes over, the mesh grows to a thousand nodes, and then it gets attacked on purpose

Milestone 5The node as a gen_server

Goal

Replace mesh_node's hand-rolled receive loop with OTP's gen_server behaviour, previewed in Section 2.10 of the instalment. Same external API, same observable behaviour — the entire point of this milestone is that nothing about calling a mesh node changes at all.

Concepts

OTP behaviours as a contract, gen_server:call/2 versus gen_server:cast/2, where server state actually lives, and reply timeouts.

Design

Every piece of mesh_node's hand-rolled loop from Milestone 3 has a direct gen_server equivalent, and seeing them side by side is the fastest way to understand what the behaviour is actually doing for you underneath.

Implementation

%% src/mesh_node.erl
-module(mesh_node).
-behaviour(gen_server).

-export([start_link/1, get_status/1, drain/2, crash/1]).
-export([init/1, handle_call/3, handle_cast/2]).

%% ---- public API ----

start_link(Id) -> gen_server:start_link(?MODULE, Id, []).
get_status(Pid) -> gen_server:call(Pid, get_status).
drain(Pid, Amount) -> gen_server:call(Pid, {drain, Amount}).
crash(Pid) -> gen_server:cast(Pid, crash).

%% ---- gen_server callbacks ----

init(Id) ->
    {ok, mesh_node_state:new(Id)}.

handle_call(get_status, _From, State) ->
    #{status := Status} = State,
    {reply, Status, State};
handle_call({drain, Amount}, _From, State) ->
    NewState = mesh_node_state:drain(State, Amount),
    {reply, ok, NewState}.

handle_cast(crash, _State) ->
    error(simulated_crash).

Notice what is not here: no receive, no Ref, no manual reply, no recursive loop call. init/1 returns the initial state and gen_server starts the loop; handle_call/3 returns {reply, Reply, NewState} and gen_server sends the reply and continues the loop with the new state; you never construct a message tuple or call receive again for this module.

Verified: identical behaviour, less code

1> {ok, Pid} = mesh_node:start_link(1).
{ok,<0.94.0>}
2> mesh_node:get_status(Pid).
alive
3> mesh_node:drain(Pid, 150).
ok
4> mesh_node:get_status(Pid).
crashed

Explanation

Character for character, the public API's usage is unchanged from Milestone 3 — the whole point. What changed is fifteen lines of hand-written message plumbing became four callback functions, and a class of bug (a reply sent with the wrong Ref, a forgotten timeout, a loop call with stale state) became structurally impossible to write, because the plumbing that could contain those bugs no longer exists in this module at all.

call versus cast, and when each is correct

recharge(Pid, Amount) -> gen_server:call(Pid, {recharge, Amount}).

%% callback:
handle_call({recharge, Amount}, _From, State) ->
    {reply, ok, mesh_node_state:recharge(State, Amount)};
handle_call(_Unknown, _From, State) ->
    {reply, {error, unknown_request}, State}.

Experiment

Call gen_server:call/3 directly, bypassing get_status/1's wrapper, with a timeout of 0 milliseconds. Predict whether it behaves like Milestone 3's hand-rolled {error, timeout} return value.

1> {ok, Pid} = mesh_node:start_link(1).
{ok,<0.83.0>}
2> catch gen_server:call(Pid, get_status, 0).
{'EXIT',{timeout,{gen_server,call,[<0.83.0>,get_status,0]}}}

It does not. Milestone 3's hand-rolled call/2 returned {error, timeout} as an ordinary value the caller could pattern-match on and keep going. gen_server:call/2,3 instead exits the calling process on timeout — an ordinary value from Milestone 3 becomes a crash here unless the caller explicitly wraps the call in catch or try, which is a real, easy-to-miss behavioural difference between the two versions despite the cmp box above describing them as "almost the identical syntax."

Common mistakes in Milestone 5

** exception exit: {bad_return_value,
                        {alive,#{id => 1,status => alive,energy => 100}}}
      in function  gen_server:handle_common_reply/8

Checkpoint

1. What does init/1 return, and what happens to that return value?
2. Give one legitimate reason to choose cast over call, and one real hazard of choosing it for the wrong reason.
3. What does gen_server:call/2 do automatically that the hand-rolled Milestone 3 call/2 had to implement by hand, and what does it do that Milestone 3's version could not?

Milestone 6Supervision trees

Goal

Replace mesh_watcher's hand-rolled, one-per-node restart loop with a real supervisor, and then deliberately push it past its restart-intensity limit to watch it give up — on command, not by accident — because seeing that happen is the only way to really believe it is there.

Concepts

Restart strategies (one_for_one, simple_one_for_one), restart types (permanent, transient, temporary), intensity and period, and dynamic children.

Design

One simple_one_for_one supervisor, node_sup, owning every mesh node as an identical, dynamically-started child — the template case this restart strategy exists for, since a "normal" one_for_one supervisor's children are declared statically, by name, at startup, which does not fit "a few thousand interchangeable nodes, the exact number decided at runtime."

Implementation

%% src/node_sup.erl
-module(node_sup).
-behaviour(supervisor).
-export([start_link/0, start_node/1, init/1]).

start_link() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, []).

start_node(Id) ->
    supervisor:start_child(?MODULE, [Id]).

init([]) ->
    SupFlags = #{strategy => simple_one_for_one, intensity => 3, period => 5},
    ChildSpec = #{id => mesh_node, start => {mesh_node, start_link, []}, restart => transient},
    {ok, {SupFlags, [ChildSpec]}}.

Explanation

intensity => 3, period => 5 means: tolerate at most 3 restarts in any rolling 5-second window; the moment a fourth would be needed within that window, the supervisor concludes that restarting is not fixing anything, stops trying, and terminates itself — reporting that failure to its own supervisor, one level up, which is the mechanism that turns "this one child keeps dying" into "escalate, because a local fix has already been tried and failed." restart => transient means a node is restarted if it crashes abnormally, but not if it exits normally (stop/1 in earlier milestones) — a node you deliberately stopped should stay stopped, not spring back to life.

Verified: three restarts tolerated, the fourth is not

1> {ok, SupPid} = node_sup:start_link().
{ok,<0.79.0>}
2> {ok, _} = node_sup:start_node(99).
{ok,<0.80.0>}
3> [{_, P1, _, _}] = supervisor:which_children(node_sup), mesh_node:crash(P1), timer:sleep(50).
ok
4> is_process_alive(SupPid).
true    %% restart 1 of 3 — tolerated
5> [{_, P2, _, _}] = supervisor:which_children(node_sup), mesh_node:crash(P2), timer:sleep(50).
ok
6> is_process_alive(SupPid).
true    %% restart 2 of 3 — tolerated
7> [{_, P3, _, _}] = supervisor:which_children(node_sup), mesh_node:crash(P3), timer:sleep(50).
ok
8> is_process_alive(SupPid).
true    %% restart 3 of 3 — tolerated
9> [{_, P4, _, _}] = supervisor:which_children(node_sup), mesh_node:crash(P4), timer:sleep(50).
ok
10> is_process_alive(SupPid).
false   %% the 4th crash inside the 5-second window: intensity exceeded

Exactly what the numbers say and no more: four crashes, all inside the five-second window, intensity => 3 means three restarts are absorbed and the fourth is one too many. This is the honest, complete answer to Milestone 4's "no restart intensity limit" gap — not a promise that it is fixed, a demonstration that it is, with the exact threshold visible and adjustable.

Experiment

Repeat the verified transcript above — four crashes of the same node — but this time sleep 6 seconds between each crash instead of sending them back to back. With period => 5 unchanged, predict whether the supervisor still goes down on the fourth crash.

1> {ok, SupPid} = node_sup:start_link().
{ok,<0.79.0>}
2> {ok, _} = node_sup:start_node(99).
{ok,<0.80.0>}
3> [{_, P, _, _}] = supervisor:which_children(node_sup), mesh_node:crash(P), timer:sleep(6000).
ok             %% repeat this line three more times, once per crash
4> is_process_alive(SupPid).
true    %% still up after 4 crashes total, spaced 6s apart

It stays up. period is a genuinely rolling five-second window, not a running total of "crashes so far" — a restart six seconds after the previous one has already aged out of the window the previous restart was counted in, so each of these four crashes is evaluated against a window containing, at most, itself. The verified transcript earlier in this milestone and this Experiment use the exact same crash count and the exact same intensity/period numbers; only the timing between crashes differs, and that alone is the difference between "supervisor gives up" and "supervisor absorbs every one of them indefinitely."

Common mistakes in Milestone 6

1> supervisor:start_child(node_sup, []).
{error,{'EXIT',{undef,[{mesh_node,start_link,[],[]}, ...]}}}

Checkpoint

1. What does intensity => 3, period => 5 actually bound, precisely?
2. Why does simple_one_for_one fit a dynamic pool of interchangeable nodes better than plain one_for_one?
3. When a supervisor exceeds its restart intensity and terminates, what does it report, and to whom?
4. Why does restart => transient mean a deliberately-stopped node is not restarted, when a crashed one is?

Milestone 7A network of a thousand nodes

Goal

Grow the mesh from a handful of nodes started by hand to thousands, findable by name rather than by remembering their process identifier, with messages routed between them — and measure, rather than assume, what actually costs something at that scale.

Concepts

ETS (Erlang Term Storage) as a shared, concurrent lookup table; a registry built on it; routing a message by name instead of by pid; and simulated message loss.

Design

A registry is, at its core, a map from a stable name (a mesh node's id) to its current pid — "current" doing real work in that sentence, because a restarted node in Milestone 6 gets a brand new pid, and anything holding onto the old one is holding a stale, useless reference the moment a restart happens. ETS is the right tool because it is a table any process can read and write concurrently, without going through a single owning process for every lookup — which matters the moment "every one of two thousand nodes routes messages through the registry" is the actual workload.

Implementation

%% src/mesh_registry.erl
-module(mesh_registry).
-export([start/0, register_node/2, lookup/1, route/2]).

start() ->
    ets:new(?MODULE, [set, public, named_table]),
    ok.

register_node(Id, Pid) ->
    ets:insert(?MODULE, {Id, Pid}),
    %% clean up automatically when the node dies, rather than leaving a
    %% stale pid in the table for the next lookup to route a message into
    %% the void
    spawn(fun() ->
        Ref = monitor(process, Pid),
        receive
            {'DOWN', Ref, process, Pid, _Reason} ->
                ets:delete_object(?MODULE, {Id, Pid})
        end
    end),
    ok.

lookup(Id) ->
    case ets:lookup(?MODULE, Id) of
        [{Id, Pid}] -> {ok, Pid};
        []          -> {error, not_found}
    end.

%% route/2 simulates a lossy network: ~2% of messages never arrive,
%% which is Milestone 8's territory previewed here so the plumbing for
%% it already exists once chaos injection needs to turn the rate up
route(Id, Msg) ->
    case rand:uniform() of
        R when R =< 0.02 ->
            dropped;
        _ ->
            case lookup(Id) of
                {ok, Pid} -> Pid ! Msg, sent;
                {error, not_found} -> no_such_node
            end
    end.

Explanation

The monitor-and-clean-up pattern inside register_node/2 is worth naming: a short-lived process, spawned for the sole purpose of watching one node and deleting its stale registry entry the moment it dies, is a completely ordinary Erlang idiom — processes are cheap enough (Milestone 3's measurement: microseconds to start, kilobytes to hold) that "spawn one to watch one thing" is not wasteful, it is simply how the language expects cleanup to be expressed.

Verified: two thousand registered nodes, in milliseconds

2000 nodes started + registered in 13 ms
routed lookup for id 500: alive

Thirteen milliseconds to start two thousand supervised gen_server nodes and register every one of them in ETS. Combined with Milestone 3's measurement — a thousand bare processes in six milliseconds, roughly 2.8 KB of memory each — the honest conclusion is that the process model is nowhere near the limiting factor at this scale. Milestone 9's profiling looks for what actually is.

%% node_sup.erl, inside start_node/1:
start_node(Id) ->
    {ok, Pid} = supervisor:start_child(?MODULE, [Id]),
    mesh_registry:register_node(Id, Pid),
    {ok, Pid}.

%% mesh_registry.erl:
broadcast(Msg) ->
    [Pid ! Msg || {_Id, Pid} <- ets:tab2list(?MODULE)],
    ok.

Experiment

Change route/2's drop threshold from R =< 0.02 to R =< 0.5, then route 1,000 messages to a single registered node and count how many come back dropped. Predict the rough count before running it.

1> {ok, Pid} = mesh_node:start_link(1), mesh_registry:register_node(1, Pid).
ok
2> Results = [mesh_registry:route(1, ping) || _ <- lists:seq(1, 1000)].
[...]
3> length([x || dropped <- Results]).
512

At the original 2% threshold, the same measurement returns almost exactly 20 dropped out of 1,000 — the literal constant in route/2 is not a description of behaviour, it is the behaviour, and changing one number in one clause head changes the mesh's effective reliability end to end, with nothing else in the registry, the nodes, or the chaos module aware anything changed.

Common mistakes in Milestone 7

1> mesh_registry:start().
ok
2> mesh_registry:start().
** exception error: bad argument
     in function  ets:new/2
        called as ets:new(mesh_registry,[set,public,named_table])

Checkpoint

1. Why is a stale registry entry worse than merely "wrong" — what does it actually cause?
2. Why does register_node/2 spawn a separate process to watch for the node's death, rather than making the registry process itself monitor every node?
3. What did the 2,000-node benchmark actually rule out as this system's bottleneck?

Milestone 8Chaos injection

Goal

Build a chaos module that attacks the live mesh on purpose — random crashes, handlers that hang, messages that do not parse — and confirm, by actually running it, that the supervision tree built across Milestones 5–7 absorbs all of it without anyone stepping in by hand.

Concepts

Seeded randomness for reproducible chaos, a slow handler as a distinct failure mode from a crashed one, and — the sharpest cross-course comparison in this curriculum — what "malformed input" should do to a process, contrasted directly with the Perl course's answer to the same question.

Design

%% src/mesh_chaos.erl
-module(mesh_chaos).
-export([run/2]).

run(Seed, Opts) ->
    rand:seed(exsplus, {Seed, Seed, Seed}),
    #{crash_rate := CrashRate, slow_rate := SlowRate} = Opts,
    Ids = [Id || {Id, _Pid} <- ets:tab2list(mesh_registry)],
    [attack(Id, CrashRate, SlowRate) || Id <- Ids],
    ok.

attack(Id, CrashRate, SlowRate) ->
    case mesh_registry:lookup(Id) of
        {error, not_found} -> skip;
        {ok, Pid} ->
            R = rand:uniform(),
            if
                R =< CrashRate ->
                    mesh_node:crash(Pid);
                R =< CrashRate + SlowRate ->
                    %% a slow handler, not a crashed one — the node is
                    %% still "alive" by every check except responsiveness
                    Pid ! {sys, {suspend_for, 2000}};
                true ->
                    %% a genuinely malformed message: not a tuple this
                    %% node's handle_info/2 was ever written to expect
                    Pid ! <<16#DEADBEEF:32>>
            end
    end.

Explanation

rand:seed(exsplus, {Seed, Seed, Seed}) makes an entire chaos run reproducible from one integer — the same discipline Go's Milestone 8 built by hand as a precomputed Schedule, arrived at here almost for free because Erlang's rand module already supports seeding deterministically. A failing chaos run reported as "seed 8841" replays exactly, every time.

Verified: the mesh absorbing an attack

1> mesh_chaos:run(8841, #{crash_rate => 0.05, slow_rate => 0.05}).
ok
2> timer:sleep(500).
ok
3> supervisor:count_children(node_sup).
[{specs,1},{active,2000},{supervisors,0},{workers,2000}]

Two thousand active workers, after a run that crashed roughly a hundred of them outright — because every single one that crashed was restarted, by the supervisor, without anyone watching the shell noticing which ones or when. count_children/1 reports the current population, not the history, and that is precisely the point: from the outside, "some nodes crashed and were replaced" and "no nodes crashed at all" look identical, which is the honest definition of what "the system kept working while parts of it were broken" actually means in a measurable way.

handle_cast({suspend_for, Ms}, State) ->
    timer:sleep(Ms),   %% blocks THIS gen_server's loop; every call
                       %% or cast to this pid queues up behind it
    {noreply, State};

Experiment

Put a node to sleep with {suspend_for, 2000}, the same attack mesh_chaos uses, then compare a fire-and-forget send against that node with a real gen_server:call against it, timed. Predict which one blocks.

1> {ok, Pid} = mesh_node:start_link(1).
{ok,<0.90.0>}
2> gen_server:cast(Pid, {suspend_for, 2000}).
ok
3> timer:tc(fun() -> Pid ! ping end).
{17,ping}
4> timer:tc(fun() -> mesh_node:get_status(Pid) end).
{1989123,alive}

The bare send returns in 17 microseconds — sending a message never waits for anything on the receiving end, suspended or not. get_status/1, built on gen_server:call, takes just under two full seconds: it genuinely queues behind the sleeping handle_cast callback and only gets its answer once that callback returns. This is the warn box above made measurable rather than just asserted: a slow handler degrades exactly the calls that need a reply from that one process, and does nothing whatsoever to a fire-and-forget send aimed at the same pid.

Common mistakes in Milestone 8

1> mesh_chaos:run(1, #{crash_rate => 0.1}).
** exception error: no match of right hand side value #{crash_rate => 0.1}

Common mistakes in Milestones 5–8

Repository state after Milestone 8

mesh/
├── rebar.config
├── src/
│   ├── mesh_app.erl, mesh_sup.erl        top-level application + supervisor
│   ├── mesh_id.erl                        per-process id counter (Milestone 1)
│   ├── mesh_node_state.erl                 pure node data (Milestone 2)
│   ├── mesh_node.erl                        gen_server (Milestone 5, was raw process in M3)
│   ├── node_sup.erl                          simple_one_for_one supervisor (Milestone 6)
│   ├── mesh_registry.erl                      ETS-backed name -> pid (Milestone 7)
│   └── mesh_chaos.erl                          seeded, reproducible attacks (Milestone 8)
└── test/                                        7 files

$ rebar3 eunit
=======================================================
  All 21 tests passed.
$ git commit -am "milestones 5-8: gen_server, supervision, 2000-node registry, chaos"