Watching it, speeding it up, and moving the world to another machine

Milestone 9Metrics, and a profiler you can attach to a running colony

Goal

Counters, gauges and latency histograms for everything the colony does, exposed over HTTP in a format a scraper understands, with pprof mounted alongside so you can profile a live process without restarting it.

Concepts

Atomic counters versus owner-owned counters, cheap histograms without allocation, the Prometheus text format, net/http servers and graceful shutdown, mounting net/http/pprof deliberately, and reading a CPU profile.

Design

We already have two kinds of counter, and the distinction is worth making explicit before adding twenty more.

The ledger must be exactly right, because the conservation invariant depends on it, and it is cheap to keep right because one goroutine owns it. Metrics are written from everywhere and need to be approximately right and very cheap. Do not merge these two ideas: a ledger behind atomics invites someone to read two counters that were never consistent with each other, and a metric behind the owner adds a message round trip to every observation.

For histograms, the naive approach (keep every observation, sort, take a percentile) allocates unboundedly and is useless in a hot path. The standard answer is bucketing, and there is a trick that makes it nearly free: use powers of two as bucket boundaries, and find the bucket with a single bit operation.

Implementation

internal/metrics/metrics.go

// Counter is a monotonically increasing number.
type Counter struct{ v atomic.Int64 }

func (c *Counter) Inc()         { c.v.Add(1) }
func (c *Counter) Add(n int64)  { c.v.Add(n) }
func (c *Counter) Value() int64 { return c.v.Load() }

// Gauge is a number that goes up and down.
type Gauge struct{ v atomic.Int64 }

func (g *Gauge) Set(n int64)  { g.v.Store(n) }
func (g *Gauge) Value() int64 { return g.v.Load() }

Explanation

Wrapping atomic.Int64 in a named type with two methods looks like pointless ceremony until you notice what it prevents: nobody can accidentally read a counter with = instead of .Load(), and the type name documents whether a number is expected to go down. go vet also refuses to let you copy a struct containing an atomic, which catches a whole class of "my metrics stopped moving" bugs.

// Histogram records durations in power-of-two microsecond buckets. Bucket i
// holds observations in [2^(i-1), 2^i) microseconds, which covers 1µs to
// about 4 seconds in 24 buckets for the cost of one atomic add per
// observation and no allocation at all.
type Histogram struct {
	buckets [24]atomic.Int64
	count   atomic.Int64
	sumUS   atomic.Int64
}

func (h *Histogram) Observe(d time.Duration) {
	us := d.Microseconds()
	if us < 0 {
		us = 0
	}
	h.count.Add(1)
	h.sumUS.Add(us)
	h.buckets[bucketFor(us)].Add(1)
}

func bucketFor(us int64) int {
	if us <= 0 {
		return 0
	}
	// bits.Len64 gives the position of the highest set bit, which is
	// floor(log2)+1: exactly the bucket index we want.
	i := bits.Len64(uint64(us))
	if i > 23 {
		i = 23
	}
	return i
}

math/bits.Len64 compiles to a single CPU instruction on every architecture Go supports. So one observation is three atomic adds and one instruction, with no branching on bucket boundaries, no allocation, and no lock. The whole histogram is 26 machine words and can live inline in a struct.

// Quantile returns the upper edge of the bucket holding the qth quantile.
// It is approximate by construction: the answer is only ever accurate to a
// factor of two, which is usually enough to tell "fine" from "on fire".
func (h *Histogram) Quantile(q float64) time.Duration {
	total := h.count.Load()
	if total == 0 {
		return 0
	}
	target := int64(q * float64(total))
	seen := int64(0)
	for i := range h.buckets {
		seen += h.buckets[i].Load()
		if seen >= target {
			return time.Duration(int64(1)<<uint(i)) * time.Microsecond
		}
	}
	return time.Duration(int64(1)<<23) * time.Microsecond
}

Be honest in the doc comment about accuracy. A p99 of "8 ms" here means "somewhere between 4 and 8 ms", and that is fine for alerting and useless for a latency SLO quoted to three decimal places. Real systems use HDR histograms or t-digests when they need better; the point of showing the cheap version is that you now know what the expensive ones are buying.

Note also that Quantile reads 24 atomics one at a time while other goroutines are writing, so the snapshot it returns never existed at any single instant. For a monitoring endpoint that is completely acceptable and worth knowing.

The metric set, and why it is a struct rather than a map

// Set is every metric the colony publishes. Named fields rather than a map
// of strings: the compiler checks the names and no lookup happens in the
// hot path.
type Set struct {
	Delivered     Counter
	FailedPickups Counter
	Lost          Counter
	Restarts      Counter
	Dropped       Counter
	Timeouts      Counter
	Requests      Counter

	AntsAlive  Gauge
	QueueDepth Gauge

	RoundTrip Histogram
	Decide    Histogram
}

Most metrics libraries hand you counter("antfarm_requests_total").Inc(), which needs a map lookup, a string hash and usually a mutex or a sharded cache on every observation. A struct of named fields costs a pointer offset, is checked by the compiler, and shows up in your editor's autocomplete. You lose dynamic metric names, which for an application (as opposed to a library) you rarely want anyway.

Exposition

// WriteProm renders the set in the Prometheus text exposition format, which
// is simple enough to produce by hand and is what most scrapers speak.
func (s *Set) WriteProm(w io.Writer) {
	counter := func(name, help string, c *Counter) {
		fmt.Fprintf(w, "# HELP %s %s\n# TYPE %s counter\n%s %d\n", name, help, name, name, c.Value())
	}
	...
	counter("antfarm_food_delivered_total", "Food units delivered to the nest.", &s.Delivered)
	gauge("antfarm_ants_alive", "Ants currently running.", &s.AntsAlive)
	hist("antfarm_round_trip", &s.RoundTrip)
}

Local closures as helpers inside a function is a very Go thing to do. They capture w, they are not visible outside, and they turn eleven repetitive blocks into eleven readable lines. The naming convention is worth copying: _total suffix for counters, base units (seconds, bytes) rather than milliseconds, and a # HELP line that says what the number means.

Mounting pprof, carefully

// Handler builds the observability endpoints. We mount pprof explicitly
// rather than relying on its init() registering itself on the default mux,
// because exposing profiles by accident is a real security problem.
func (s *Set) Handler() http.Handler {
	mux := http.NewServeMux()

	mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintln(w, "ok")
	})

	mux.HandleFunc("GET /metrics", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
		s.WriteProm(w)
	})

	mux.HandleFunc("GET /debug/pprof/", pprof.Index)
	mux.HandleFunc("GET /debug/pprof/cmdline", pprof.Cmdline)
	mux.HandleFunc("GET /debug/pprof/profile", pprof.Profile)
	mux.HandleFunc("GET /debug/pprof/symbol", pprof.Symbol)
	mux.HandleFunc("GET /debug/pprof/trace", pprof.Trace)

	return mux
}

Wiring it in

The engine gets an embedded metrics.Set, and the interesting instrumentation is two lines in the request path:

func (e *Engine) handle(r request) {
	e.M.Requests.Inc()
	e.M.QueueDepth.Set(int64(len(e.reqs)))
	...
}

len() on a channel returns how many values are buffered in it right now. It is a genuinely useful gauge (queue depth is the single best early warning that a consumer is falling behind) and it is a terrible basis for logic, because by the time you act on it the number has changed. Measure with it, never branch on it.

func (c *antClient) roundTrip(ctx context.Context, r request) (response, error) {
	start := time.Now()
	defer func() { c.e.M.RoundTrip.Observe(time.Since(start)) }()
	...
}

And the HTTP server in main.go, with the shutdown discipline from Milestone 7:

// serve starts an HTTP server and returns a function that shuts it down
// without dropping in-flight requests.
func serve(addr string, h http.Handler, name string) func() {
	srv := &http.Server{
		Addr:              addr,
		Handler:           h,
		ReadHeaderTimeout: 5 * time.Second, // never accept a slow-loris header
	}
	go func() {
		if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
			log.Printf("antfarm: %s server: %v", name, err)
		}
	}()
	return func() {
		ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
		defer cancel()
		if err := srv.Shutdown(ctx); err != nil {
			log.Printf("antfarm: %s server shutdown: %v", name, err)
		}
	}
}

ListenAndServe always returns a non-nil error, and it returns http.ErrServerClosed when you shut it down on purpose, so the errors.Is check is how you avoid logging a scary message during a clean exit. srv.Shutdown stops accepting, waits for in-flight handlers, and respects the context as a deadline. ReadHeaderTimeout is there because the zero-value http.Server has no timeouts at all, which means one slow client can hold a connection open indefinitely.

What it looks like

$ antfarm -ants 1500 -grid 96x96 -food 8 -food-per-source 400 \
      -duration 6s -metrics 127.0.0.1:9090 -view 127.0.0.1:8080 \
      -chaos crash=0.0005,drop=0.002
colony: 1500 ants, 96x96 grid, 8 food sources of 400, behaviour trail, chaos {...}
metrics: http://127.0.0.1:9090/metrics  pprof: http://127.0.0.1:9090/debug/pprof/
view:    http://127.0.0.1:8080/

$ curl -s 127.0.0.1:9090/metrics
# HELP antfarm_food_delivered_total Food units delivered to the nest.
# TYPE antfarm_food_delivered_total counter
antfarm_food_delivered_total 955
# HELP antfarm_ant_restarts_total Ants restarted after a crash.
# TYPE antfarm_ant_restarts_total counter
antfarm_ant_restarts_total 792
# HELP antfarm_messages_dropped_total Requests discarded by chaos injection.
# TYPE antfarm_messages_dropped_total counter
antfarm_messages_dropped_total 6141
# HELP antfarm_request_timeouts_total Requests that never got a reply.
# TYPE antfarm_request_timeouts_total counter
antfarm_request_timeouts_total 5865
# HELP antfarm_requests_total Requests handled by the world owner.
# TYPE antfarm_requests_total counter
antfarm_requests_total 3034578
# HELP antfarm_ants_alive Ants currently running.
# TYPE antfarm_ants_alive gauge
antfarm_ants_alive 1500
# HELP antfarm_queue_depth Requests waiting for the world owner.
# TYPE antfarm_queue_depth gauge
antfarm_queue_depth 0
# TYPE antfarm_round_trip_seconds summary
antfarm_round_trip_seconds{quantile="0.5"} 0.000002
antfarm_round_trip_seconds{quantile="0.9"} 0.001024
antfarm_round_trip_seconds{quantile="0.99"} 0.008192
antfarm_round_trip_seconds_count 3033172

Three million requests in about four seconds, and look at the latency distribution: a median of 2 µs, a p90 of 1 ms, a p99 of 8 ms. That is a spread of four thousand to one between the typical case and the bad case, and it is the signature of a queue. When the owner is free your request is answered immediately; when 1,500 ants arrive at once you wait behind them. Averages would have hidden this completely: the mean here is around 100 µs, a number that describes nobody's experience. Always look at the tail.

Profiling a live process

$ curl -o cpu.prof "127.0.0.1:9090/debug/pprof/profile?seconds=10"
$ go tool pprof -top -nodecount=14 cpu.prof
File: antfarm
Type: cpu
Duration: 10.10s, Total samples = 9970ms (98.69%)
      flat  flat%   sum%        cum   cum%
    1030ms 10.33% 10.33%     3230ms 32.40%  runtime.selectgo
    1000ms 10.03% 20.36%     1000ms 10.03%  runtime.nanotime (partial-inline)
     820ms  8.22% 28.59%      820ms  8.22%  runtime.lock2
     720ms  7.22% 35.81%      790ms  7.92%  runtime.unlock2
     560ms  5.62% 41.42%      560ms  5.62%  time.Now
     350ms  3.51% 44.93%      520ms  5.22%  runtime.casgstatus
     320ms  3.21% 48.14%      320ms  3.21%  runtime.duffcopy
     220ms  2.21% 50.35%      800ms  8.02%  runtime.sellock
     200ms  2.01% 54.56%     6120ms 61.38%  sim.(*Engine).runAnt
     180ms  1.81% 56.37%      770ms  7.72%  runtime.mallocgc
     120ms  1.20% 60.68%      450ms  4.51%  sim.(*antClient).roundTrip.func1

That is a real profile of our real program, and it is damning in the most useful way. Read the columns first: flat is time spent in that function itself, cum is time in it plus everything it called. So runAnt has 2% flat and 61% cumulative: it does nothing itself and everything beneath it.

Now read the names. selectgo (32% cumulative) is the runtime implementing select. lock2, unlock2 and sellock are the locks the runtime uses inside channels. nanotime and time.Now together are 15%. casgstatus is goroutine state transitions, which is scheduling. Add it up: the overwhelming majority of the CPU is channel machinery, timers and scheduling, and almost none of it is ant simulation.

Not one line of Decide, senseAt or applyAction appears in the top fourteen. Our program is not computing; it is communicating about computing.

type SampledHistogram struct {
	Histogram
	n    atomic.Int64
	rate int64 // observe one in every rate
}

func (h *SampledHistogram) Maybe(start time.Time) {
	if h.n.Add(1)%h.rate != 0 {
		return
	}
	h.Observe(time.Since(start))
}

	var start time.Time
	sampled := c.e.M.RoundTrip.Should()     // one atomic add, no clock
	if sampled {
		start = time.Now()
	}
	defer func() {
		if sampled {
			c.e.M.RoundTrip.Observe(time.Since(start))
		}
	}()

Experiment

Remove the ReadHeaderTimeout: 5 * time.Second line from serve's http.Server and rebuild. Open a raw connection to the metrics port (nc 127.0.0.1 9090, or a small Go program that dials and never writes) and leave it sitting there; it stays open indefinitely, because the zero-value http.Server has no timeouts of any kind. Open a few hundred of these in a loop and watch the process happily hold every one open forever. Put the line back, repeat, and confirm each connection is dropped after five seconds. That single struct field is the entire difference between a server that resists a slow-loris attack and one that does not.

Common mistakes in Milestone 9

Checkpoint

1. Why keep the ledger as plain ints owned by one goroutine when atomics are available?
2. How does bits.Len64 produce a histogram bucket, and why is that cheap?
3. Your p99 reads 8 ms. What do you actually know about the 99th percentile?
4. What is the danger in import _ "net/http/pprof"?
5. In the profile, runAnt is 2% flat and 61% cumulative. Explain both numbers.
6. Why did adding a latency histogram show up as time.Now in the profile, and what would you do about it?

Milestone 10A live picture, in the terminal and the browser

Goal

See the colony. An ANSI-redrawn terminal view for a quick look, and a browser page fed by server-sent events for a good one. Neither may ever slow the simulation down.

Concepts

Aggregating at the source, snapshot values as an interface between subsystems, ANSI escape codes, server-sent events, http.Flusher, request contexts as the client's lifetime, and dropping frames as a policy.

Design

The naive viewer asks for the whole world and draws it. At 512×512 with pheromone that is half a megabyte per frame, serialised to JSON, sixty times a second, and all of it built on the owner's goroutine while 50,000 ants wait. The viewer would become the bottleneck of the simulation it is supposed to observe.

So the rule is: aggregate at the source. The owner builds a small, fixed-size Frame, downsampled to whatever resolution the client asked for, and that is all that ever leaves the simulation.

owner goroutine                  viewer(s)
───────────────                  ─────────
holds 512x512 grids    ──►  Frame: 96x96 buckets, ~3 slices
builds one Frame            terminal renderer  → ANSI text
per request                 SSE handler        → JSON over HTTP
                            (either, both, or none)

A second rule follows from the first: the viewer may never block the simulation. If a frame cannot be produced in time, the viewer skips it. A dropped frame is invisible to a human; a stalled colony is not.

Implementation

The frame

// Frame is a downsampled picture of the colony, built by the owner and
// small enough to send many times a second. Aggregating at the source
// rather than shipping the whole grid is what keeps the viewer cheap.
type Frame struct {
	Cols int       `json:"cols"`
	Rows int       `json:"rows"`
	Food []int     `json:"food"` // Cols*Rows, summed per block
	Pher []float64 `json:"pher"` // Cols*Rows, summed per block
	Ants []int     `json:"ants"` // Cols*Rows, ants per block
	Nest [2]int    `json:"nest"`
	Stats Stats    `json:"stats"`
}

func (e *Engine) frameNow(cols int) Frame {
	if cols <= 0 || cols > e.cfg.Width {
		cols = min(64, e.cfg.Width)
	}
	rows := max(1, e.cfg.Height*cols/e.cfg.Width)

	f := Frame{ /* ... allocate cols*rows slices ... */ }

	for y := range e.cfg.Height {
		for x := range e.cfg.Width {
			p := world.Position{X: x, Y: y}
			food, pher := e.world.Food.At(p), e.world.Pher.At(p)
			if food == 0 && pher == 0 {
				continue
			}
			i := (y*rows/e.cfg.Height)*cols + x*cols/e.cfg.Width
			f.Food[i] += food
			f.Pher[i] += pher
		}
	}
	for _, p := range e.positions {
		f.Ants[(p.Y*rows/e.cfg.Height)*cols+p.X*cols/e.cfg.Width]++
	}
	f.Stats = e.statsNow()
	return f
}

Two things to notice.

The struct tags (`json:"cols"`) control JSON field names. They are a string literal attached to a field, read by encoding/json through reflection at run time. go vet checks them for you, which is how I caught a duplicate tag while writing this: struct field Rows repeats json tag "cols". A typo in a string that silently produces the wrong wire format is exactly the kind of thing that should be checked, and it is.

The continue on empty cells is not micro-optimisation, it is the difference between scanning and doing work: most of a colony's grid is empty most of the time, so the loop touches 262,144 cells but does arithmetic on very few. The scan itself is the remaining cost, and Exercise 10 asks what to do about it.

Where do ant positions come from? The owner does not own ants, so it cannot read their positions. But it sees every move request, so it can remember them:

	case ActMove:
		np := e.world.Clamp(r.pos.Add(r.act.Dir))
		if r.id >= 0 && r.id < len(e.positions) {
			e.positions[r.id] = np
		}

A slice indexed by ant ID, written only by the owner. The viewer gets ant positions with no extra messages and no synchronisation, because the information was already flowing past. Deriving a view from a message stream you already have is the cheapest kind of observability, and it generalises: event sourcing is this idea taken seriously.

The terminal renderer

// shades map a density to a character, densest last.
var shades = []rune{' ', '.', ':', '-', '=', '+', '*', '#', '%', '@'}

// Terminal redraws the colony in place using ANSI escape codes.
func Terminal(ctx context.Context, w io.Writer, e *sim.Engine, cols int, every time.Duration) error {
	ticker := time.NewTicker(every)
	defer ticker.Stop()

	fmt.Fprint(w, "\x1b[?25l")       // hide the cursor
	defer fmt.Fprint(w, "\x1b[?25h") // and always put it back

	for {
		select {
		case <-ctx.Done():
			return nil
		case <-ticker.C:
			f, err := e.Frame(ctx, cols)
			if err != nil {
				continue // a dropped frame is not worth stopping for
			}
			fmt.Fprint(w, "\x1b[H\x1b[2J") // home, then clear
			fmt.Fprint(w, Render(f))
		}
	}
}

ANSI escape codes are a tiny language your terminal speaks: \x1b is the escape character, [H moves the cursor to the top left, [2J clears the screen, [?25l and [?25h hide and show the cursor. No library needed for this much. The defer that restores the cursor matters: a program that exits with the cursor hidden leaves the user's terminal broken, and "put back what you changed" applies to terminals as much as to files.

Render is a separate pure function taking a Frame and returning a string, so it can be tested without a terminal, a simulation or a goroutine. Splitting "produce the bytes" from "write the bytes to a live device" is a habit worth having; it also makes the renderer reusable by the web view if you ever want an ASCII mode.

func Render(f sim.Frame) string {
	var b strings.Builder
	b.Grow(f.Rows*(f.Cols+1) + 256)
	...
}

strings.Builder with Grow is the right way to assemble a string in Go: building with s += ... in a loop is O(n²) because strings are immutable and every concatenation copies. Grow pre-sizes the buffer so the builder never reallocates.

Server-sent events

For the browser we need a push channel. The options are WebSockets (bidirectional, needs a library or a hand-rolled handshake), long polling (awkward), or server-sent events (one-way, plain HTTP, built into every browser as EventSource, about fifteen lines of server code). We only push, so SSE wins easily.

func stream(w http.ResponseWriter, r *http.Request, e *sim.Engine, cols int, every time.Duration) {
	flusher, ok := w.(http.Flusher)
	if !ok {
		http.Error(w, "streaming unsupported", http.StatusInternalServerError)
		return
	}
	w.Header().Set("Content-Type", "text/event-stream")
	w.Header().Set("Cache-Control", "no-cache")

	// r.Context() is cancelled when the browser tab closes, which is how
	// this goroutine learns to stop.
	ctx, cancel := context.WithCancel(r.Context())
	defer cancel()

	ticker := time.NewTicker(every)
	defer ticker.Stop()

	enc := json.NewEncoder(w)
	for {
		select {
		case <-ctx.Done():
			return
		case <-ticker.C:
			f, err := e.Frame(ctx, cols)
			if err != nil {
				continue // drop this frame rather than stall the simulation
			}
			fmt.Fprint(w, "data: ")
			if err := enc.Encode(f); err != nil {
				return
			}
			fmt.Fprint(w, "\n")
			flusher.Flush()
		}
	}
}

Explanation

• The SSE wire format is the whole protocol: data: then a payload then a blank line. json.Encoder.Encode conveniently appends a newline, so the extra Fprint(w, "\n") completes the pair.
• w.(http.Flusher) is a type assertion: http.ResponseWriter is an interface, and the concrete value behind it may or may not support flushing. Without Flush, Go buffers the response and the browser receives nothing until the handler returns, which for a stream is never. This is the classic SSE bug.
• r.Context() is cancelled when the client disconnects. That is how this goroutine finds out the tab was closed. Without it, every closed tab would leave a goroutine ticking forever, and a page that the user opens and closes fifty times leaks fifty goroutines.
• The continue on error is the "never block the simulation" rule. If Frame times out because the owner is saturated, we skip that frame and try again in 100 ms.

The browser client is 25 lines of dependency-free JavaScript in a Go string constant: an EventSource, a canvas, and a loop writing RGBA pixels straight into an ImageData. Ants go in the red channel, food in green, pheromone in blue and a little green, and the canvas is scaled up with image-rendering: pixelated. No build step, no framework, and the binary still has nothing to ship alongside it.

	case reqFrame:
		if time.Since(e.lastFrameAt) < e.cfg.FrameInterval {
			r.reply <- response{frame: e.lastFrame}
			return
		}
		e.lastFrame = e.frameNow(r.id)
		e.lastFrameAt = time.Now()
		r.reply <- response{frame: e.lastFrame}

Experiment

Comment out the flusher.Flush() call in stream and rebuild. Open the browser view: the connection succeeds, curl -N -i 127.0.0.1:8080/stream even shows the right Content-Type: text/event-stream header, but no frame ever arrives, because Go's net/http buffers the response until either the buffer fills or the handler returns — and a stream handler never returns. The canvas just sits black. Put Flush() back and watch frames appear within the tick period. This is the single most common bug in a hand-rolled SSE handler, and it produces no error anywhere: the connection is open, the headers are correct, and the browser is simply waiting for bytes that Go is holding onto.

Common mistakes in Milestone 10

Checkpoint

1. Why does the owner downsample instead of returning the grid?
2. How does the owner know where the ants are, given that it does not own them?
3. What does http.Flusher do and what happens without it?
4. How does an SSE handler learn that the browser tab closed?
5. Why is Render a separate pure function?
6. The viewer must never block the simulation. Where in the code is that policy expressed?

Milestone 11Making it fast, guided by the profile

Goal

Take the profile from Milestone 9 seriously and rewrite the hot path. Sensing becomes a lock-free read with no message at all, moves and drops become one-way messages, and writes are sharded across several owners. Then measure honestly.

Concepts

Atomics for concurrent reads with single-owner writes, fixed-point arithmetic because Go has no atomic float, one-way messages, sharding by region, closing channels to drain them, and a genuine correctness bug found by a conservation test.

Design

The profile said: channel operations, timers and scheduling dominate; simulation logic is invisible. So the optimisation strategy writes itself. Do fewer channel operations. Where do they come from?

So roughly 99% of the traffic is sensing and moving, and both can largely be eliminated. Three changes:

1. Reads become atomic loads. If every grid cell is an atomic.Int64, any goroutine may read it race-free at any time with no message. Writes stay single-owner, so there is no compare-and-swap loop and no contention between writers.
2. Moves and drops become one-way messages. No reply channel, no waiting, no timer. The ant computes its own new position and fires a deposit at the world if it is carrying.
3. Writes are sharded. N goroutines, each owning a horizontal band of rows. An operation goes to the shard that owns its row. There is no longer a single serialisation point.

Milestone 5 engine                    Milestone 11 engine
──────────────────                    ───────────────────
sense  → message → owner → reply      sense  → atomic loads (no message)
move   → message → owner → reply      move   → local compute
                                             + one-way deposit if carrying
pickup → message → owner → reply      pickup → message → shard → reply
drop   → message → owner → reply      drop   → one-way message → shard

1 owner, ~2 msgs/tick/ant             N shards, ~0.05 msgs/tick/ant

Implementation

The atomic grid, and the missing atomic float

// pherScale turns float concentrations into fixed-point integers. Go has no
// atomic float64, so we store thousandths and convert on read.
const pherScale = 1000

// AtomicGrid is a grid that any goroutine may read without synchronisation
// and that only the owner of a row band may write. Reads use atomic loads,
// so they are race-free; writes stay single-owner, so no compare-and-swap
// loop is needed.
type AtomicGrid struct {
	W, H  int
	cells []atomic.Int64
}

func (g *AtomicGrid) At(p Position) int64 {
	if !g.InBounds(p) {
		return 0
	}
	return g.cells[p.Y*g.W+p.X].Load()
}

func (g *AtomicGrid) AtFloat(p Position) float64 {
	return float64(g.At(p)) / pherScale
}

// TakeOne removes a unit if there is one, reporting whether it succeeded.
func (g *AtomicGrid) TakeOne(p Position) bool {
	if !g.InBounds(p) {
		return false
	}
	c := &g.cells[p.Y*g.W+p.X]
	if c.Load() <= 0 {
		return false
	}
	c.Add(-1)
	return true
}

Explanation

Go has no atomic float64, and the reason is that the atomic instructions operate on integers. The standard workaround is fixed point: store thousandths as an integer and divide on read. You give up range and precision (a pheromone of 0.0004 is zero here) and gain a data type that atomic.Int64 can hold. The alternative, math.Float64bits with a CAS loop, gives you exact floats at the cost of a retry loop; fixed point is simpler and good enough for a concentration.

TakeOne is load-then-add rather than a compare-and-swap loop, and that is safe only because a single shard owns every write to this row. Two writers would race between the Load and the Add and could both take the last unit, driving the cell negative. The race detector would not catch it, because both operations are atomic; the food conservation test would. Atomics prevent torn reads and writes, not logical races. Write the ownership rule in a comment on the type, as this code does, because nothing else enforces it.

Shards

// shardFor maps a row to the goroutine that owns it.
func (e *FastEngine) shardFor(p world.Position) chan writeOp {
	i := p.Y * len(e.shard) / e.cfg.Height
	...
	return e.shard[i]
}

// runShard owns one band of rows: it is the only writer of those cells.
func (e *FastEngine) runShard(ctx context.Context, i int) {
	lo, hi := e.bandOf(i)
	ticker := time.NewTicker(e.cfg.EvaporateEvery)
	defer ticker.Stop()

	for {
		select {
		case <-ctx.Done():
			return
		case <-ticker.C:
			e.pher.EvaporateBand(lo, hi, e.cfg.Evaporation)
		case op, open := <-e.shard[i]:
			if !open {
				return // every ant has stopped and the queue is drained
			}
			switch op.kind {
			case opDeposit:
				e.pher.AddFloat(op.pos, e.cfg.Deposit, e.cfg.PherMax)
			case opPickUp:
				op.reply <- e.food.TakeOne(op.pos)
			case opDrop:
				e.M.Delivered.Inc()
			}
		}
	}
}

Each shard now also evaporates its own band, which turns Milestone 6's single O(cells) pass on one goroutine into N parallel passes of O(cells/N). The horizontal-band split is chosen because ants move locally: an ant spends most of its life in one or two bands, so cross-shard traffic is low. A hash of the position would spread load perfectly and destroy that locality.

The sense path now needs no shard at all:

// senseFast reads the world directly. No message, no reply, no waiting.
func (e *FastEngine) senseFast(pos world.Position) Sense {
	s := Sense{Pos: pos, Nest: e.nest, Food: int(e.food.At(pos))}
	for i, d := range directions[:8] {
		n := pos.Add(d)
		s.Adjacent[i] = int(e.food.At(n))
		s.Pher[i] = e.pher.AtFloat(n)
	}
	return s
}

Seventeen atomic loads and a struct return. Measured: 29 nanoseconds, against a median of 2 µs for the message-based version. Roughly seventy times faster for the single most frequent operation in the program.

Fire and forget, with shedding

// send is fire-and-forget with load shedding: if the owning shard is
// saturated, the deposit is dropped rather than stalling the ant.
func (e *FastEngine) send(ctx context.Context, op writeOp) {
	select {
	case e.shardFor(op.pos) <- op:
	case <-ctx.Done():
	default:
		e.M.Dropped.Inc()
	}
}

This is backpressure policy expressed in four lines, and the choice it encodes is: a lost pheromone deposit is better than a stalled ant. That is correct for pheromone, which is statistical and evaporates anyway, and it would be catastrophic for the pickup path, which is why pickups do not use this function. Shedding policy is per-message-type, not per-system, and deciding it message by message is the whole discipline.

The bug the conservation test found

The first version passed -race cleanly and failed this:

--- FAIL: TestFastEngineConservesFood (0.74s)
    fast_test.go:22: food not conserved: 359 != 500
        (ants 500 carrying 182 delivered 81 food left 96)

141 units of food had ceased to exist. No race, no panic, no error in any log. Instrumenting the ledger showed 336 successful pickups, 216 drops, 90 ants carrying: 30 units taken from the ground that no ant held and nobody delivered.

The cause was two shutdown bugs, both of them the kind that only appear at the boundary:

	// BUG 1: when ctx is cancelled while we wait for the pickup answer,
	// select may choose Done even though the reply is ready. The shard
	// already took the food. We just threw it away.
	select {
	case ok := <-reply:
		a.Carrying = ok
	case <-ctx.Done():
		return
	}

Remember that select picks randomly among ready cases. At cancellation time, a couple of dozen ants were mid-pickup; for each, the world had already removed a unit of food, and half of them discarded the answer. The fix is to insist on collecting an answer you have already paid for:

		case <-ctx.Done():
			// The shard may already have taken a unit on our behalf.
			// Collect the answer anyway, or that food vanishes from
			// the ledger.
			timer := time.NewTimer(e.cfg.RequestTimeout)
			defer timer.Stop()
			select {
			case ok := <-reply:
				a.Carrying = ok
			case <-timer.C:
				e.M.Lost.Inc()
			}
			return

Bug two was queued drops being discarded when the shards were cancelled with work still in their channels. The fix uses the one safe way to close a channel:

	<-ctx.Done()
	antWG.Wait() // every sender has stopped, so closing is now safe
	for i := range e.shard {
		close(e.shard[i])
	}
	shardWG.Wait()

Closing a channel that senders might still use panics, which is why Milestone 5 forbade it. Here it is not only safe but exactly right, because antWG.Wait() has already established that every sender has stopped. A closed channel keeps delivering its buffered values before reporting closed, so op, open := <-ch drains the queue and then exits. Close-after-senders-finish is the idiomatic way to say "process everything remaining, then stop".

Two lessons worth more than the speedup. First, a race detector cannot find a logic bug: every operation here was properly synchronised and the program was still wrong. Invariant tests find what sanitisers cannot. Second, shutdown is where work gets lost, in this project and in every queueing system, because that is the one moment when the pipeline is asked to stop while it is still full.

Measuring

Food delivered in a fixed two-second window, same seed, same world, single-owner engine versus sharded:

  500 ants: owner   827 delivered | sharded  1180 delivered | 1.4x
 2000 ants: owner   811 delivered | sharded  1458 delivered | 1.8x

(an earlier run on a quieter machine: 1.4x and 2.9x)

Be careful about what this shows. My container has one core, so none of this gain comes from parallelism; it is all reduced overhead, which is exactly what the profile predicted. On a multi-core machine you should see considerably more, because sharded writes and parallel evaporation can then genuinely run at the same time. Run it yourself and compare with GOMAXPROCS set to 1, 2, 4 and 8; unlike the Milestone 4 mutex version, this one should improve rather than degrade.

Notice also that the gain grows with the number of ants (1.4× at 500, 1.8–2.9× at 2,000). That is the signature of removing a serialisation point: with few clients the owner was never the bottleneck, so removing it changes little.

Experiment

The conservation test passes because every pickup is serialised through its shard's channel. Break that deliberately: change the pickup path so an ant calls e.food.TakeOne(pos) directly instead of sending opPickUp to the shard, and set a single food source with enough ants that several routinely stand on the same cell. Run go test -race: clean, because every access is a correctly-used atomic. Now run the conservation test by itself with -count=50: it fails intermittently, with more food taken from the ground than was ever delivered or lost. TakeOne's load-then-add is not atomic as a sequence, and this is what "atomics prevent torn reads and writes, not logical races" looks like when you trigger it instead of taking it on faith.

Common mistakes in Milestone 11

Checkpoint

1. Why can reads be atomic while writes stay single-owner, and what breaks if two goroutines write?
2. Why fixed point instead of float64 for pheromone?
3. Which operations kept their reply, and what is the rule for deciding?
4. Why does horizontal banding beat hashing for this workload?
5. Explain why select's random choice caused food to disappear at shutdown.
6. When is closing a channel the right shutdown signal, and when is it a panic?
7. The speedup grew with ant count. What does that tell you about what was removed?

Milestone 12The world moves to another process

Goal

Run the world in one process and ants in another, talking over TCP. Kill the world, watch the ants wait and reconnect, and see what the network costs.

Concepts

Interfaces as the seam between local and remote, encoding/gob, one goroutine per connection, connection ownership and why the client is not thread-safe, socket deadlines as the network form of a timeout, reconnection with jittered backoff, and a shutdown bug that only a network can give you.

Design

This is the milestone that Milestone 5 was secretly building toward. Because the local protocol was already values in and values out, the change is mostly mechanical: give it a name as an interface, implement that interface twice.

// WorldService is everything an ant needs from the world. Both the local
// engine and the network client implement it, which is what lets an ant run
// unchanged against a world in another process.
type WorldService interface {
	Sense(pos world.Position) (Sense, error)
	Act(id int, pos world.Position, carrying bool, act Action) (ActResult, error)
}

Two methods. The local engine satisfies it with atomic reads and channel sends; the network client satisfies it with gob over a socket. Notice that both methods return an error, including the local one where nothing can fail. Designing the interface for the failing implementation is what makes the seam work: had Sense returned only a Sense, the remote version could not have reported a broken connection without panicking or lying.

  process A: ants                        process B: the world
  ───────────────                        ────────────────────
  ant goroutine ─► netsim.Client ──TCP──► netsim.Server ─► FastEngine
       ▲               (one per ant)         (one goroutine      (shards)
       │                                      per connection)
  same loop as milestone 5

Implementation

The wire format

// Req and Resp are the wire format. They contain only values, which is
// exactly why the local protocol from milestone 5 could become a network
// protocol without redesign.
type Req struct {
	Kind     Kind
	ID       int
	Pos      world.Position
	Carrying bool
	Act      sim.Action
}

type Resp struct {
	Sense sim.Sense
	Pos   world.Position
	Nest  world.Position
	OK    bool
	Err   string // errors do not survive gob; send the text
}

Explanation

encoding/gob is Go's native binary serialisation: self-describing, type-safe between Go programs, and requiring no schema file or code generation. It is a good fit here and a poor fit for anything that must interoperate with another language, where you would reach for protobuf or JSON. gob sends type descriptions once per connection and then compact values, so a long-lived connection is efficient.

Errors do not travel. error is an interface, and gob can only encode concrete types it has been told about, so an error field would fail to encode or arrive as something unusable. Sending the message as a string and reconstructing an error on the far side is the simple, standard answer. The cost is that errors.Is no longer works across the wire, which is a real loss: a remote ErrNoFood arrives as an unrelated error with the same text. Production protocols solve this with numeric error codes, which is worth knowing as the reason gRPC has a status enum.

The server

// Server exposes a sim.WorldService over TCP. Each connection gets one
// goroutine, which is the standard Go server shape and costs almost nothing
// because goroutines are cheap.
func (s *Server) handle(conn net.Conn) {
	defer conn.Close()

	dec := gob.NewDecoder(conn)
	enc := gob.NewEncoder(conn)

	for {
		var req Req
		if err := dec.Decode(&req); err != nil {
			if !errors.Is(err, io.EOF) && !errors.Is(err, net.ErrClosed) {
				log.Printf("netsim: decode: %v", err)
			}
			return
		}

		resp := s.dispatch(req)
		if err := enc.Encode(resp); err != nil {
			return // the client went away; drop the connection
		}
	}
}

Blocking reads in a dedicated goroutine per connection is the whole Go server model, and it is the clearest demonstration of why cheap goroutines matter: this code reads like a single-threaded program from 1995 and scales to tens of thousands of connections. In a language with expensive threads you would be writing an event loop, a state machine per connection, and callbacks. Node, Python and Rust all have async machinery precisely to get back to code that looks like this.

One gob.Decoder per connection, created once and reused: a gob stream is stateful, because type descriptions are sent once, so creating a new decoder per message would be both wrong and slow.

The bug the network gave me

My first version of Close was the obvious one:

func (s *Server) Close() {
	s.ln.Close()   // stop accepting
	s.wg.Wait()    // wait for the connection goroutines
}

The test suite hung until I killed it after 300 seconds. Closing the listener stops new connections and does nothing whatsoever to existing ones: each connection goroutine was parked in dec.Decode(&req), waiting for a request from a client that was still connected and simply not talking. wg.Wait() waited forever.

// Close stops accepting, hangs up on every client, and waits for the
// connection goroutines. Closing the listener alone is not enough: a
// goroutine blocked in Decode on a live connection never notices, and
// Close would wait forever.
func (s *Server) Close() {
	if s.ln != nil {
		s.ln.Close()
	}
	s.mu.Lock()
	s.closing = true
	for conn := range s.conns {
		conn.Close() // unblocks the goroutine parked in Decode
	}
	s.mu.Unlock()

	s.wg.Wait()
}

The fix requires a registry of live connections, and that registry is the one place in this entire project where a mutex is the right tool:

	// mu guards conns. A connection registry is exactly the kind of small
	// shared map a mutex is right for: no ownership to transfer, just a set
	// that several goroutines add to and one goroutine walks at shutdown.
	mu      sync.Mutex
	conns   map[net.Conn]struct{}
	closing bool

There is no ownership to hand over and no flow of work, just a set that accept-time adds to, connection-exit removes from, and shutdown iterates. A channel-based version would be strictly worse. This is what the Milestone 5 comparison meant by "use mutexes for shared state with simple invariants": the mutex is not a failure of the message-passing design, it is the right tool for a different job. The closing flag closes the race where a connection is accepted after shutdown began.

The general lesson is bigger than the fix. Cancelling a goroutine blocked on network I/O requires closing the connection or setting a deadline; a context alone does nothing, because the blocking call is in the kernel. This is the same fact as Milestone 7's "you cannot kill a goroutine", wearing different clothes.

The client, and why it is not thread-safe

// Client is a sim.WorldService backed by a TCP connection. It is NOT safe
// for concurrent use: the protocol is one request and one response in
// order, so each client belongs to exactly one goroutine. That restriction
// is the price of not putting request IDs on the wire.

Say this in the doc comment, because the compiler cannot. A stream protocol with no request IDs matches replies to requests by order, so two goroutines sharing a client would interleave their writes and each read the other's answer. The alternatives are a mutex around the whole round trip (correct, and it serialises every ant on one connection) or request IDs plus a demultiplexing reader goroutine (what gRPC does, and what Exercise 12 asks for). One connection per ant is the simplest thing that works, and it is what we do.

func (c *Client) roundTrip(req Req) (Resp, error) {
	if err := c.connect(); err != nil {
		return Resp{}, err
	}
	// A deadline on the socket is the network equivalent of the timeout we
	// put on the local channel in milestone 5.
	c.conn.SetDeadline(time.Now().Add(2 * time.Second))

	if err := c.enc.Encode(req); err != nil {
		c.drop()
		return Resp{}, fmt.Errorf("%w: send: %v", ErrDisconnected, err)
	}
	var resp Resp
	if err := c.dec.Decode(&resp); err != nil {
		c.drop()
		return Resp{}, fmt.Errorf("%w: receive: %v", ErrDisconnected, err)
	}
	if resp.Err != "" {
		return resp, errors.New(resp.Err)
	}
	return resp, nil
}

Any I/O error drops the connection rather than trying to continue on it, because a gob stream that has lost sync cannot be repaired: the decoder's type state and the byte stream no longer agree. Reconnecting is cheap; resynchronising is impossible. Wrapping with %w keeps errors.Is(err, ErrDisconnected) working, which is how the ant distinguishes "the world is gone, wait and retry" from "the world said no".

// Backoff reports how long to wait before the next reconnection attempt,
// with full jitter so a thousand ants do not reconnect in lockstep.
func (c *Client) Backoff() time.Duration {
	d := 10 * time.Millisecond << min(c.attempts, 6)
	return time.Duration(c.rng.Int64N(int64(d) + 1))
}

Same reasoning as the supervisor backoff in Milestone 8, and now it matters more: when a server restarts, every client discovers it at the same instant, and unjittered retries would arrive as a synchronised wave that knocks it over again.

The ant, almost unchanged

	for ctx.Err() == nil {
		sensed, err := c.Sense(a.Pos)
		if err != nil {
			if errors.Is(err, ErrDisconnected) {
				select {
				case <-time.After(c.Backoff()):
				case <-ctx.Done():
				}
				continue
			}
			return
		}

		act := b.Decide(a, sensed, rng)
		res, err := c.Act(a.ID, a.Pos, a.Carrying, act)
		...
	}

Sense, decide, act. Identical in structure to Milestone 5, with ErrTimeout handling replaced by ErrDisconnected handling. The behaviour code (TrailFollower) is byte-for-byte the same file that ran locally, because it only ever saw a Sense.

Running it

$ go test -race ./internal/netsim/
=== RUN   TestColonyOverTheWire
    net_test.go:94: delivered over TCP: 39, food left 458
--- PASS: TestColonyOverTheWire (0.71s)
=== RUN   TestClientSurvivesServerRestart
--- PASS: TestClientSurvivesServerRestart (0.00s)
PASS

Forty remote ants foraging through forty TCP connections, delivering food to a world in a different object graph, race-clean. The second test kills the server mid-session, confirms the client reports ErrDisconnected, restarts the world at the same address, and confirms the client reconnects on its next attempt with no special handling.

What the network costs

sense latency: local 29ns, over loopback TCP 10.667µs (368x)

That is the number to carry out of this course. The same operation is 368 times slower over a loopback socket than in memory, and loopback is the best case: no switch, no cable, no congestion, no packet loss. Across a data centre add 200 µs or more; across a continent, 50 ms and up, which is five million times the local cost.

This is why "just make it a microservice" is a performance decision and not only an architectural one. Our sharded local engine does about 30 million senses per second per core; over loopback, about 100,000. Distribution buys you fault isolation, independent deployment and horizontal scale, and it costs you three orders of magnitude on every interaction you move across the boundary. The design question is always the same: which interactions cross? Here, the honest answer is that sensing should never cross a network at all, and a real distributed colony would replicate a read-only copy of the local grid to each ant process and only send mutations over the wire. That is a cache with a coherence protocol, and now you are writing a distributed system in earnest.

type MuxClient struct {
	conn net.Conn
	enc  *gob.Encoder

	mu      sync.Mutex              // guards nextID and pending
	nextID  uint64
	pending map[uint64]chan Resp

	writeMu sync.Mutex              // one writer at a time on the socket
}

func (c *MuxClient) roundTrip(ctx context.Context, req Req) (Resp, error) {
	c.mu.Lock()
	c.nextID++
	id := c.nextID
	ch := make(chan Resp, 1)        // buffered: the reader must never block
	c.pending[id] = ch
	c.mu.Unlock()

	defer func() {                  // always reclaim the slot
		c.mu.Lock()
		delete(c.pending, id)
		c.mu.Unlock()
	}()

	req.ID = id
	c.writeMu.Lock()
	err := c.enc.Encode(req)
	c.writeMu.Unlock()
	if err != nil {
		return Resp{}, fmt.Errorf("%w: send: %v", ErrDisconnected, err)
	}

	select {
	case resp := <-ch:
		return resp, nil
	case <-ctx.Done():
		return Resp{}, ctx.Err()
	}
}

// readLoop is the only goroutine that reads the socket.
func (c *MuxClient) readLoop() {
	for {
		var resp Resp
		if err := c.dec.Decode(&resp); err != nil {
			c.failAll(err)
			return
		}
		c.mu.Lock()
		ch, ok := c.pending[resp.ID]
		c.mu.Unlock()
		if ok {
			ch <- resp              // never blocks: buffer 1, one send only
		}
		// unknown ID: the caller gave up. Dropping the response is correct.
	}
}

Experiment

Reuse one plain Client (not the multiplexing MuxClient from the exercise) across two ant goroutines with no synchronisation at all, and run go test -race ./internal/netsim/. It fails immediately and reliably, pointing at concurrent writes through the same gob.Encoder — this is a case where the race detector is exactly the right tool, unlike Milestone 11's shard bug. Fix it the crude way, by wrapping the whole roundTrip call in a mutex, and the race disappears; but now watch throughput with two ants sharing one connection instead of two. Serialising away a race is easy. Getting the throughput back is the rest of Exercise 12.

Common mistakes in Milestone 12

Checkpoint

1. Why does WorldService.Sense return an error even though the local implementation cannot fail?
2. Why is the client documented as unsafe for concurrent use, and what would make it safe?
3. Why did Close hang, and why is a mutex the right fix rather than a channel?
4. Why must a broken gob stream be reconnected rather than retried?
5. What happens to errors.Is(err, world.ErrNoFood) across the network, and what do real protocols do about it?
6. Loopback made one operation 368× slower. What does that imply for which operations should cross a process boundary?
7. A context cannot cancel a goroutine blocked in Decode. What can?

Repository state after Milestone 12

antfarm/                                     2,961 lines of Go, no dependencies
├── go.mod
├── cmd/antfarm/main.go          flags, chaos parsing, signals, servers, wiring
└── internal/
    ├── sim/
    │   ├── ant.go               Ant, Action, ActionKind, Generation
    │   ├── behaviour.go         Behaviour, Forager, TrailFollower
    │   ├── sense.go             Sense, senseAt
    │   ├── engine.go            single-owner engine, supervisor, Frame, Run
    │   ├── fast.go              sharded engine, WorldService, atomic sensing
    │   ├── sim.go               Config, Chaos, Stats, the sequential Sim
    │   ├── concurrent.go        milestone 4's mutex versions, kept for contrast
    │   └── *_test.go            conservation, leaks, chaos, throughput, races
    ├── world/
    │   ├── grid.go              Position, Grid
    │   ├── pheromone.go         FloatGrid
    │   ├── atomicgrid.go        AtomicGrid, fixed-point pheromone
    │   └── world.go             World, food, nest, errors
    ├── metrics/
    │   ├── metrics.go           Counter, Gauge, Histogram, Prometheus output
    │   └── http.go              /metrics, /healthz, /debug/pprof
    ├── view/
    │   ├── terminal.go          ANSI renderer
    │   ├── web.go               SSE stream
    │   └── page.go              the embedded browser client
    └── netsim/
        ├── proto.go             Req, Resp
        ├── server.go            TCP + gob server, connection registry
        ├── client.go            reconnecting client, jittered backoff
        └── net_test.go          colony over TCP, survives a server restart

$ gofmt -l . && go vet ./... && go test -race ./...
ok  	github.com/yourname/antfarm/internal/netsim	0.705s
ok  	github.com/yourname/antfarm/internal/sim	12.426s
ok  	github.com/yourname/antfarm/internal/world	0.001s
$ git commit -am "milestone 12: the world moves to another process"