From a one-liner to something that survives the data

Milestone 1A filter that counts what it reads

Goal

The smallest useful program: read files or standard input, count what you find, report it, and exit with a status that means something. No modules, no objects, no parsing.

Concepts

while (<>), $ARGV and $., exit codes as an interface, warn versus print, heredocs, and why a Unix filter is the right default shape.

Design

Before writing anything, decide what kind of program this is. A Perl data tool is almost always a filter: it reads a stream, writes a stream, says nothing else on standard output, and reports its verdict in the exit code. Getting that shape right at the start means the tool composes with everything else on the machine, and getting it wrong means people wrap your program in a shell script to make it behave.

Three rules, which the whole course follows:

• Results to stdout, diagnostics to stderr. Otherwise strata ... | sort mixes error messages into the data.
• Exit codes follow grep: 0 nothing found, 1 something found, 2 could not run. People already know this convention.
• No arguments means read standard input. One line of Perl gets you this.

Implementation

#!/usr/bin/env perl
use v5.36;

# strata-scan: the smallest useful thing. Count lines, bytes and matches
# across files or standard input, and say something true about the result.

my $pattern = qr/\b(?:ERROR|FATAL)\b/;
my $quiet   = 0;
my @files;

# Option handling by hand for now; Getopt::Long arrives in milestone 10.
while (@ARGV) {
    my $arg = shift @ARGV;
    if    ($arg eq "--pattern") { $pattern = qr/@{[ shift @ARGV ]}/ }
    elsif ($arg eq "--quiet")   { $quiet = 1 }
    elsif ($arg eq "--help")    { print usage(); exit 0 }
    elsif ($arg =~ /^--/)       { die "strata-scan: unknown option $arg\n" . usage() }
    else                        { push @files, $arg }
}
@ARGV = @files;

my ($lines, $bytes, $matched, $empty, $longest) = (0, 0, 0, 0, 0);
my %per_file;

while (my $line = <>) {
    $lines++;
    $bytes += length $line;
    chomp $line;

    $empty++ if $line =~ /^\s*$/;
    $longest = length $line if length $line > $longest;

    if ($line =~ $pattern) {
        $matched++;
        $per_file{$ARGV}++;
    }
}

if ($lines == 0) {
    warn "strata-scan: no input\n";
    exit 2;
}

unless ($quiet) {
    printf "%d lines, %s bytes, %d matched, %d blank, longest %d chars\n",
        $lines, commify($bytes), $matched, $empty, $longest;

    for my $file (sort { $per_file{$b} <=> $per_file{$a} || $a cmp $b } keys %per_file) {
        printf "  %-40s %6d\n", $file, $per_file{$file};
    }
}

exit($matched > 0 ? 1 : 0);

sub commify ($n) { 1 while $n =~ s/^(\d+)(\d{3})/$1,$2/; return $n }

sub usage {
    return <<~"USAGE";
        usage: strata-scan [--pattern REGEX] [--quiet] [FILE...]
               reads standard input when no files are named
               exit 0: no matches   1: matches found   2: no input
        USAGE
}

Explanation

• @ARGV = @files; is the trick that makes hand-rolled options work with <>. The diamond operator reads whatever is left in @ARGV, so we pull the options out first and put the filenames back. Getopt::Long does exactly this for you, which is why it composes with <> too.
• qr/@{[ shift @ARGV ]}/ compiles a user-supplied string into a pattern. @{[ ... ]} is the "baby cart" idiom: it evaluates an expression inside a string or pattern, because interpolation only understands variables. It is ugly; my $p = shift @ARGV; $pattern = qr/$p/; is clearer and you should prefer it. It is here because you will meet it.
• $ARGV is the file currently being read by <>. Note the output below: when reading standard input it is -, which is the Unix convention for "the stream", and which you get for free.
• <<~"USAGE" is an indented heredoc (Perl 5.26+). The tilde strips the leading indentation, so the usage text lines up with the code rather than being jammed against the left margin.
• warn writes to stderr and does not exit; die writes to stderr and exits with a failure status. Neither pollutes stdout.

Running it

$ ./bin/strata-scan share/fixtures/access.log
2004 lines, 211,017 bytes, 0 matched, 1 blank, longest 128 chars
$ echo $?
0

$ ./bin/strata-scan --pattern '" 5\d\d ' share/fixtures/access.log
2004 lines, 211,017 bytes, 398 matched, 1 blank, longest 128 chars
  share/fixtures/access.log                   398
$ echo $?
1

$ head -50 share/fixtures/access.log | ./bin/strata-scan --pattern 'Googlebot'
50 lines, 5,298 bytes, 19 matched, 0 blank, longest 127 chars
  -                                            19

$ ./bin/strata-scan < /dev/null
strata-scan: no input
$ echo $?
2

The same binary, three invocation styles, no code paths for any of them. That is <> earning its place.

Python                                    Perl
──────                                    ────
import sys, re, fileinput                 use v5.36;
pattern = re.compile(r'\bERROR\b')
count = 0                                 while (<>) {
for line in fileinput.input():                $n++ if /\bERROR\b/;
    if pattern.search(line):              }
        count += 1
        name = fileinput.filename()
print(count)                              say $n;
sys.exit(1 if count else 0)

my (@reservoir, $seen);

sub reservoir_add ($line, $n) {
    $seen++;
    if (@reservoir < $n) {
        push @reservoir, $line;
    } else {
        my $j = int rand $seen;          # 0 .. seen-1
        $reservoir[$j] = $line if $j < $n;
    }
}

# histogram
my %hist;
# ... inside the loop:
if (defined $field) {
    my @cols = split ' ', $line;
    $hist{ $cols[$field] // "(missing)" }++;
}

# ... after the loop:
my $width = ($ENV{COLUMNS} || 80) - 30;
my ($max) = sort { $b <=> $a } values %hist;
for my $key (sort { $hist{$b} <=> $hist{$a} || $a cmp $b } keys %hist) {
    my $bar = "#" x int($width * $hist{$key} / $max);
    printf "  %-18s %7d %s\n", substr($key, 0, 18), $hist{$key}, $bar;
}

Experiment

Add print STDERR "$ARGV:$.\n"; right after $lines++; and run strata-scan over two small files at once. $. does not reset at the file boundary: the second file's first line reports as continuing the count from the first, not restarting at 1. Add close ARGV if eof; at the bottom of the loop and run again, and each file now starts back at line 1. Milestone 2's push @first_failures, "$ARGV:$." if ... depends on exactly this number meaning "line within this file", so which behaviour you want is a decision to make deliberately, not a default to inherit by accident.

Common mistakes in Milestone 1

Checkpoint

1. What does <> read from, and how does it decide?
2. Why does the program reassign @ARGV before the loop?
3. What are $ARGV and $., and what is $ARGV when reading a pipe?
4. What do the three exit codes mean and who consumes them?
5. Why is warn preferable to print for the "no input" message?

Milestone 2Reports: one pass, several questions

Goal

Turn the counting filter into something that answers real questions about a log: who is talking most, what is failing, when the spikes were, and what could not be parsed. Still one file, still one pass, still constant memory in the size of the input.

Concepts

Hashes as accumulators, multi-key sorting, list slices, printf report formatting, and the discipline of computing everything in a single traversal.

Design

The instinct from other languages is to read the file into a list and then run four separate passes over it. Do not. Every question you can answer with a counter should be answered during the single pass you were already making, because that is what keeps memory proportional to the number of distinct keys rather than to the size of the input. A log with 200 million lines and 4,000 client addresses needs 4,000 hash entries, not 200 million.

So the shape is: declare one hash per question, update them all in the loop, and do the sorting and formatting afterwards.

Implementation

my %stat = (total => 0, parsed => 0, failed => 0);
my (%hits, %bytes, %errors, %by_minute, %by_class, %by_path, @first_failures);

while (my $line = <>) {
    $stat{total}++;

    unless ($line =~ $APACHE) {
        $stat{failed}++;
        push @first_failures, "$ARGV:$." if @first_failures < 3;
        next;
    }
    $stat{parsed}++;

    my %f = %+;                                   # copy before the next match
    my $bytes = $f{bytes} eq "-" ? 0 : $f{bytes};

    $hits{ $f{ip} }++;
    $bytes{ $f{ip} } += $bytes;
    $errors{ $f{ip} }++ if $f{status} >= 400;
    $by_class{ substr($f{status}, 0, 1) . "xx" }++;
    $by_minute{$1}++ if $f{ts} =~ m{^(\d+/\w+/\d+:\d+:\d+)};
    $by_path{$1}++   if $f{request} =~ m{^[A-Z]+ \s+ ([^?\s]+)}x;
}

Then a reusable top helper, because "the five biggest" appears four times:

# top(n, \%hash) returns the n keys with the largest values, ties broken
# alphabetically so the report is reproducible.
sub top ($n, $href) {
    my @keys = sort { $href->{$b} <=> $href->{$a} || $a cmp $b } keys %$href;
    return grep { defined } @keys[0 .. $n - 1];
}

• The tiebreaker is not decoration. Without || $a cmp $b, two keys with equal counts come out in hash order, which Perl randomises per process. Your report would differ between runs, which makes it useless for diffing and maddening to test.
• @keys[0 .. $n - 1] is a list slice, and indexing past the end yields undef rather than an error, which is why grep { defined } follows. Convenient and a little sloppy; the alternative is @keys[0 .. ($n > @keys ? $#keys : $n - 1)], which nobody writes.
• Passing \%hits and dereferencing with $href->{$b} is the reference rule from Part 2 in daily use: you cannot pass a hash to a sub without flattening it, so you pass a reference.

Running it

$ ./bin/strata-report share/fixtures/access.log

top talkers
  10.14.22.9         1020 hits     23,671,329 bytes   38.9% errors
  10.14.22.31         344 hits      7,822,046 bytes   41.0% errors
  192.168.4.7         324 hits      7,559,417 bytes   37.7% errors
  172.16.0.99         312 hits      7,061,075 bytes   36.9% errors

status classes
  2xx       1028   51.4%
  3xx        197    9.8%
  4xx        377   18.9%
  5xx        398   19.9%

busiest minutes
  12/Sep/2026:13:56             35
  12/Sep/2026:13:36             34
  12/Sep/2026:13:41             34

most requested paths
  /                            361
  /api/export                  342
  /api/search                  336

2,004 lines, 2,000 parsed, 4 unparsed (0.20%)
  first failures: share/fixtures/access.log:2001, ...:2002, ...:2003

Four questions answered in one pass over the file, with the unparsed lines counted and located rather than silently skipped. That last line is the most important one in the output, and Milestone 3 is about taking it seriously.

10.14.22.9 - - [12/Sep/2026:13:44:10 +0000] "GET /api/export HTTP/1.1" 200

pandas                                      Perl
──────                                      ────
df = pd.read_csv(log, sep=None)             while (my $line = <>) {
top = df.groupby("ip").size() \                 $hits{$f{ip}}++;
        .sort_values(ascending=False)           $errors{$f{ip}}++ if $f{status} >= 400;
errs = df[df.status >= 400] \                   $by_class{...}++;
        .groupby("ip").size()                   $by_minute{...}++;
classes = df.status.astype(str).str[0]      }
by_min = df.ts.dt.floor("min") \
        .value_counts()

push @{ $sizes{ $f{ip} } }, $bytes;          # O(n) memory, exact

sub percentile ($values, $p) {
    my @sorted = sort { $a <=> $b } @$values;
    return $sorted[ int($p / 100 * $#sorted) ];
}

# Log-scale buckets: cheap, bounded, accurate to a factor of two.
$size_buckets{ $f{ip} }[ $bytes ? int(log($bytes) / log(2)) : 0 ]++;

use Time::Piece;
my $t = Time::Piece->strptime($f{ts}, "%d/%b/%Y:%H:%M:%S %z");
$by_minute{ $t->epoch - $t->epoch % 60 }++;

$by_class_path{ substr($f{status},0,1) . "xx" }{ $path }++;   # autovivified

for my $class (sort keys %by_class_path) {
    my $paths = $by_class_path{$class};
    say "  $class";
    printf "    %-24s %6d\n", $_, $paths->{$_} for top(3, $paths);
}

Experiment

Delete || $a cmp $b from top's sort block and run strata-report on the same fixture several times in a row. Whenever two clients tie on hits, their order in the "top talkers" table changes between runs, because Perl randomises hash key order per process by design (so that an attacker cannot craft input designed to degrade every hash to worst-case behaviour). Put the tiebreaker back and the order becomes stable again, run after run. A report whose row order depends on the interpreter's random seed is not a bug you find by reading the code; it is a bug you find when two runs over the same input disagree, which is exactly the kind of thing a forensic tool cannot afford.

Checkpoint

1. Why compute every statistic in one pass rather than several?
2. What is memory proportional to in this program, and what is it not proportional to?
3. Why does top need a tiebreaker?
4. Why must %+ be copied into %f immediately?
5. What did the failure count reveal about the fixture, and why does that matter more than the 0.2%?

Milestone 3A parser that treats broken input as evidence

Goal

Extract the regex into a reusable pattern library and a real parser module. Handle both common and combined log formats with one pattern. Distinguish three outcomes: parsed cleanly, parsed with problems, and not parsed at all. Count every problem by kind, and never lose a line.

Concepts

qr// as composable values, regex composition by interpolation, optional groups, a parser as an object with statistics, strict versus lenient modes, and format detection scoring.

Design

Two design decisions carry the whole project.

1. A pattern library, not scattered regexes. The same notions (an IPv4 address, a timestamp, a quoted string) appear in every format we will parse. Defining them once as named qr// values means they can be composed, tested, and fixed in one place.

2. parse always returns a record. Never undef, never an exception. A line that cannot be understood comes back with a problem field and its raw text intact. This sounds like a small API choice and it is the difference between a forensic tool and a reporting script: the lines you cannot parse are frequently the interesting ones. A truncated line marks where the disk filled; garbage in a log marks where something wrote to the wrong file descriptor; an unparsable request line is often an attack.

Three outcomes, not two:

Implementation

lib/Strata/Pattern.pm

package Strata::Pattern;
use v5.36;
use Exporter 'import';

our @EXPORT_OK = qw(pattern compose %PATTERN);

# A library of named, reusable pieces. Each is compiled once with qr// and
# can be interpolated into a larger pattern, which is how you build a big
# regex you can still read a year later.
our %PATTERN = (
    ipv4      => qr/(?:\d{1,3}\.){3}\d{1,3}/,
    ipv6      => qr/(?:[0-9a-fA-F]{0,4}:){2,7}[0-9a-fA-F]{0,4}/,
    host      => qr/[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?(?:\.[A-Za-z0-9-]+)+/,
    email     => qr/[A-Za-z0-9._%+-]+ @ [A-Za-z0-9.-]+ \. [A-Za-z]{2,}/x,
    uuid      => qr/[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}/,
    unix_path => qr{/(?:[\w.\-~%]+/?)*},
    quoted    => qr/"[^"]*"/,

    # 12/Sep/2026:13:44:10 +0000
    ts_apache => qr{\d{2}/\w{3}/\d{4}:\d{2}:\d{2}:\d{2}\s[+-]\d{4}},
    # 2026-09-12T13:44:10.123Z  or  2026-09-12 13:44:10
    ts_iso    => qr/\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?/,
    # Sep 12 13:44:10   (syslog, no year: the format that ruins your day)
    ts_syslog => qr/\w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2}/,

    level     => qr/\b(?:TRACE|DEBUG|INFO|NOTICE|WARN(?:ING)?|ERROR|FATAL|CRIT(?:ICAL)?)\b/,
);

sub pattern ($name) {
    return $PATTERN{$name} // die "Strata::Pattern: no pattern named '$name'\n";
}

# compose("ip", "ts_apache") returns an alternation of named patterns,
# useful for "find any of these anywhere" scanning.
sub compose (@names) {
    my @parts = map { pattern($_) } @names;
    my $alt = join "|", map { "(?:$_)" } @parts;
    return qr/$alt/;
}

1;

Interpolating a qr// into another pattern is the feature that makes this work. A compiled pattern stringifies to something like (?^u:\d{1,3}\.), carrying its own flags, so pasting it into a larger pattern preserves whether it was /x or /i. That is why $PATTERN{email} can be written with /x for readability and still work when composed into a pattern that is not.

Note the comment on ts_syslog. Syslog timestamps have no year, which means correlating a syslog file with anything else requires guessing the year from the file's modification time and handling the December-to-January rollover. Writing that down in the pattern library, where someone will read it, is worth more than the pattern itself. Milestone 8 has to solve it.

lib/Strata/Parser/Apache.pm — one pattern, two formats

my $ACCESS = qr{
    ^ (?<client>\S+) \s+                       # client address (or hostname)
      (?<ident>\S+)  \s+                       # RFC 1413 identity, almost always -
      (?<user>\S+)   \s+                       # HTTP auth user, or -
      \[ (?<ts>[^\]]+) \] \s+                  # [12/Sep/2026:13:44:10 +0000]
      " (?<request>[^"]*) " \s+                # "GET /path HTTP/1.1"
      (?<status>\d{3}) \s*                     # 200
      (?<bytes>\d+|-)?                         # response size, sometimes absent
      (?: \s+ " (?<referer>[^"]*) "
          \s+ " (?<agent>[^"]*) " )?           # the combined-format tail
      \s* $
}x;

Compare that with Milestone 2's one-liner version. Same job, and this one can be read, annotated, and modified by someone who has never seen an Apache log. Four specific improvements:

• (?<bytes>\d+|-)? is optional, which fixes the silent data loss the report revealed. A missing byte count is now a problem on a usable record rather than a discarded line.
• The combined tail is one optional group, so both log formats parse with one pattern and the record records which it was. Two patterns tried in sequence would work and would double the cost on every line.
• [^"]* rather than .*? for quoted fields. Faster (no backtracking) and clearer about intent.
• Every capture is named, so adding a field in the middle changes nothing downstream.

The parser object

sub parse ($self, $line, %ctx) {
    chomp $line;

    my %rec = (
        source => $self->{name},
        file   => $ctx{file} // "-",
        lineno => $ctx{lineno} // 0,
        raw    => $line,
    );

    if ($line !~ /\S/) {
        return $self->_problem(\%rec, "blank");
    }
    unless ($line =~ $ACCESS) {
        return $self->_problem(\%rec, "no_match");
    }

    my %f = %+;                      # copy now: the next match resets %+

    $rec{client} = $f{client};
    $rec{user}   = ($f{user} // "-") eq "-" ? undef : $f{user};
    $rec{ts_raw} = $f{ts};
    $rec{status} = 0 + $f{status};
    $rec{bytes}  = (!defined $f{bytes} || $f{bytes} eq "-") ? 0 : 0 + $f{bytes};
    $rec{format} = defined $f{agent} ? "combined" : "common";

    if (!defined $f{bytes}) {
        push @{ $rec{problems} }, "missing_bytes";
        $self->{problems}{missing_bytes}++;
        return $self->_problem(\%rec, "missing_bytes") if $self->{strict};
    }

    if ($f{request} =~ $REQUEST) {
        my %r = %+;
        @rec{qw(method path proto)} = @r{qw(method path proto)};
        ($rec{path_base}, $rec{query}) = split /\?/, $r{path}, 2;
    } else {
        push @{ $rec{problems} }, "bad_request_line";
        $rec{request_raw} = $f{request};
        return $self->_problem(\%rec, "bad_request_line") if $self->{strict};
    }

    $self->{parsed}++;
    return \%rec;
}

• 0 + $f{status} forces a number. Perl would convert on demand, but storing "200" and storing 200 differ when the value is later used as a hash key, serialised to JSON, or written to SQLite. Normalise types at the boundary, exactly once.
• Normalising - here means nothing downstream ever learns that Apache writes a dash for "nothing". Every parser in this project will absorb its format's conventions so the Record does not leak them.
• strict mode changes policy, not parsing. The same line is imperfect either way; the flag decides whether imperfect is acceptable. That belongs to the caller, because a security audit and a traffic report want different answers.
• The parser accumulates statistics (parsed, failed, problems by kind), so a caller can ask "how well did this go?" without instrumenting the loop itself.

Format detection, cheaply

# A cheap confidence score, used later to sniff an unknown file's format.
# Deliberately not a full parse: it runs on a handful of sample lines.
sub detect ($class, @lines) {
    my $hits = grep { /^\S+ \s+ \S+ \s+ \S+ \s+ \[[^\]]+\] \s+ "[^"]*" \s+ \d{3}/x } @lines;
    return @lines ? $hits / @lines : 0;
}

A score between 0 and 1 rather than a boolean, because Milestone 6 will ask every parser and pick the most confident. Using a looser pattern than the real one is deliberate: detection should recognise the family, and parsing should handle the details.

The tests, which are the real deliverable

subtest "unparsable input is kept, not thrown away" => sub {
    for my $case (
        ["this is not a log line at all", "no_match"],
        ['10.14.22.9 - - [12/Sep/2026:13:44:11 +0000] "GET /trunc', "no_match"],
        ["", "blank"],
        ["   ", "blank"],
    ) {
        my ($line, $expected) = @$case;
        my $r = $p->parse($line, file => "access.log", lineno => 99);
        is $r->{problem}, $expected, "'" . substr($line, 0, 24) . "' => $expected";
        is $r->{raw}, $line, "  raw text is preserved";
        is $r->{file}, "access.log", "  provenance is preserved";
    }
};

subtest "imperfect but usable" => sub {
    my $line = '10.14.22.9 - - [12/Sep/2026:13:44:10 +0000] "GET /api/export HTTP/1.1" 200';
    my $r = $p->parse($line);

    is $r->{status}, 200, "we still got the status";
    is $r->{bytes},  0,   "missing bytes defaults to 0";
    is_deeply $r->{problems}, ["missing_bytes"], "the problem is recorded";
    ok !$r->{problem}, "but the record is still usable";

    my $strict = Strata::Parser::Apache->new(strict => 1);
    is $strict->parse($line)->{problem}, "missing_bytes", "strict mode rejects it";
};

$ prove -l t/20-parser-apache.t
t/20-parser-apache.t .. ok
All tests successful.

Points of technique worth copying. subtest groups related assertions and names them, so a failure says which behaviour broke rather than which line number. A loop over a list of [input, expected] pairs is Perl's table-driven test, and adding a newly discovered broken line is one line. is_deeply compares nested structures, which is how you assert on an arrayref without writing four assertions.

Most importantly: the interesting tests are the failure cases. Six of the assertions concern lines that do not parse, because the whole point of the parser is what it does when the data is wrong.

awk (typical)                                     Perl
─────────────                                     ────
awk '{print $1, $7, $9}' access.log               if ($line =~ $ACCESS) {
  # breaks the moment a quoted field                   my %f = %+;
  # contains a space, e.g. a request                   # $f{request} is the whole
  # line with a spaced-out query string                # quoted field, spaces and all

my $NGINX_ERROR = qr{
    ^ (?<ts>\d{4}/\d{2}/\d{2} \s \d{2}:\d{2}:\d{2}) \s+
      \[ (?<level>\w+) \] \s+
      (?<pid>\d+)\#(?<tid>\d+): \s*
      (?: \*(?<connection>\d+) \s+ )?
      (?<message>.+?)
      (?: , \s+ (?<tail>\w+: \s .*) )? $
}x;

# then, for the tail:
if (defined $f{tail}) {
    while ($f{tail} =~ /(\w+): \s+ ( "[^"]*" | [^,]+ )/gx) {
        my ($k, $v) = ($1, $2);
        $v =~ s/^"|"$//g;
        $rec{"ctx_$k"} = $v;
    }
}

my @mutators = (
    sub ($l) { substr($l, 0, int rand length $l) },              # truncate
    sub ($l) { my $i = int rand length $l; substr($l, $i, 1) = ""; $l },
    sub ($l) { my $i = int rand length $l; substr($l, $i, 0) = '"'; $l },
    sub ($l) { $l . $l },
    sub ($l) { my $i = int rand length $l; substr($l, $i, 1) = "\x{263A}"; $l },
);

for my $line (@good_lines) {
    for my $mutate (@mutators) {
        my $mutant = $mutate->($line);
        my $rec = eval { $p->parse($mutant) };
        ok defined $rec, "survived: " . substr($mutant, 0, 30);
        ok exists $rec->{raw}, "kept the raw text";
    }
}

my $evil = '10.0.0.1 - - [x] "' . ('a" "' x 20) . 'GET / HTTP/1.1" xxx';

Experiment

Change (?<bytes>\d+|-)? in $ACCESS to drop the trailing ?, making the byte count mandatory, and re-parse the line from the Milestone 2 bug report: ... "GET /api/export HTTP/1.1" 200 with no trailing bytes field. It no longer matches at all: the record Milestone 3 exists to rescue — status, method and path preserved, only missing_bytes flagged — goes back to being indistinguishable from no_match garbage. That single ? is the entire difference between "lossy but usable" and "silently gone", worth watching fail once before trusting it in the four parsers still to come.

Common mistakes in Milestone 3

Checkpoint

1. Why does parse never return undef?
2. What are the three outcomes and how does a caller tell them apart?
3. How does one pattern handle both common and combined logs, and why not two patterns?
4. Why does the parser store 0 + $f{status} rather than the string?
5. What does interpolating a qr// into another pattern preserve?
6. Why is detect deliberately looser than parse?

Milestone 4A Record and a Pipeline

Goal

Stop passing bare hashes around. A Record carries fields, provenance, problems and extracted entities, and knows how to render itself. A Pipeline reads a source, parses it, wraps the result, and passes it through a list of stages. Nothing downstream ever learns which format a record came from.

Concepts

bless and accessor methods, closures as pipeline stages, code references in data structures, in-memory filehandles for testing, and the cost of abstraction measured rather than assumed.

Design

The pipeline is the project's spine, so its contract should be as small as possible:

  source (filehandle)
      │  one line
      ▼
  parser->parse(line, file => ..., lineno => ...)   returns a plain hash
      ▼
  Strata::Record->from_parsed(...)                  wraps it
      ▼
  stage 1 -> stage 2 -> stage 3                     each: ($rec, $pipe) -> $rec or undef
      ▼
  counted, and forgotten                            nothing accumulates records

Three consequences of that shape, all deliberate:

• A stage returning undef drops the record, which is how filtering works without a separate mechanism.
• The pipeline never stores records. Anything that needs to accumulate (counts, samples, a database) does it in a stage's own closure, so memory is that stage's responsibility and is visible in its code.
• Opening files is not the pipeline's job. run_handle takes an already-open handle, so gzip, encodings and standard input are somebody else's problem, which is exactly what Milestone 7 needs.

Implementation

lib/Strata/Record.pm

# Build a Record from the plain hash a parser returns. Keeping this in one
# place means parsers stay simple and the Record owns the shape.
sub from_parsed ($class, $parsed) {
    my %copy = %$parsed;
    my %meta;
    $meta{$_} = delete $copy{$_} for qw(source file lineno raw problems);

    return $class->new(
        %meta,
        fields   => \%copy,
        problems => $meta{problems} // [],
    );
}

sub problems ($self) { @{ $self->{problems} } }
sub ok       ($self) { !@{ $self->{problems} } }
sub where    ($self) { $self->{file} . ":" . $self->{lineno} }

sub field ($self, $name, @set) {
    $self->{fields}{$name} = $set[0] if @set;
    return $self->{fields}{$name};
}

• delete in a loop splits one hash into two. $meta{$_} = delete $copy{$_} for qw(...) moves the metadata keys out, leaving only the parsed fields behind. Compact, and it means a parser that adds a new field needs no change here.
• sub field ($self, $name, @set) is a getter and setter in one: pass a value and it sets, omit it and it gets. Using a slurpy @set rather than a defaulted scalar is what lets you set a field to undef deliberately, which matters because "known to be absent" and "not looked at" are different.
• where exists because "$rec->{file}:$rec->{lineno}" would otherwise appear in thirty places, and a record that cannot say where it came from is useless in a forensic tool.

lib/Strata/Pipeline.pm

sub run_handle ($self, $fh, $name) {
    while (defined(my $line = <$fh>)) {
        $self->{stats}{lines}++;
        my $parsed = $self->{parser}->parse($line, file => $name, lineno => $.);
        my $record = Strata::Record->from_parsed($parsed);

        $self->{stats}{problems}++ if !$record->ok;
        $self->{problems_by_kind}{$_}++ for $record->problems;

        my $kept = $record;
        for my $stage (@{ $self->{stages} }) {
            $kept = $stage->{code}->($kept, $self) or last;
        }

        if ($kept) { $self->{stats}{records}++ }
        else       { $self->{stats}{dropped}++ }
    }
    return $self;
}

while (defined(my $line = <$fh>)), not while (my $line = <$fh>). A line containing just "0" with no newline is false in Perl, so the plain form stops early on a file whose last line is a bare zero. Perl special-cases while (<$fh>) to add the defined for you, but only for that exact shape; the moment you write anything else, you need it yourself. This is a genuine bug that appears once every few years and takes an afternoon.

$stage->{code}->($kept, $self) or last — call the code reference, and stop the chain when it returns false. Perl's or has very low precedence, which is exactly what makes this read as "do this, or else stop".

Stages are closures

# counter("status") returns a stage and a hashref it fills in.
sub counter ($field) {
    my %counts;
    my $stage = sub ($rec, $) {
        my $value = $rec->field($field);
        $counts{ defined $value ? $value : "(undef)" }++;
        return $rec;
    };
    return ($stage, \%counts);
}

sub drop_problems () {
    return sub ($rec, $) { return $rec->ok ? $rec : undef };
}

sub sample ($n, $out) {
    my $seen = 0;
    return sub ($rec, $) {
        push @$out, $rec if $seen++ < $n;
        return $rec;
    };
}

Returning both the stage and the hash it fills is the key move. %counts is a lexical variable captured by the closure; the caller gets a reference to it and can read the results after the run, but nobody can reach it except through the stage. That is encapsulation without a class, and it is the most Perl-ish thing in this milestone.

sub ($rec, $) — a signature with an unnamed second parameter. It accepts the pipeline argument and says plainly that this stage ignores it, which is better than accepting a parameter you never mention.

Testing with in-memory filehandles

sub fh_for (@lines) {
    my $text = join "\n", @lines, "";
    open my $fh, "<", \$text or die $!;   # an in-memory filehandle
    return $fh;
}

open $fh, "<", \$string opens a scalar as a file. No temporary files, no fixtures on disk for the small cases, no cleanup, and the tests run in milliseconds. Every language should have this and most make you reach for a library. Use real fixture files for the realistic cases and in-memory handles for the focused ones.

$ prove -l t/
t/20-parser-apache.t .. ok
t/30-pipeline.t ....... ok
All tests successful.
Files=2, Tests=9

Putting it together

$ ./bin/strata ingest share/fixtures/access.log
  share/fixtures/access.log                    2004 lines

2,004 lines, 2,004 records, 4 with problems, 2004 lines/sec

problems
  blank                     1
  missing_bytes             1
  no_match                  2

first offenders
  share/fixtures/access.log:2001  10.14.22.9 - - [12/Sep/2026:13:44:10 +0000] "G
  share/fixtures/access.log:2002  this is not a log line at all
  share/fixtures/access.log:2003  10.14.22.9 - - [12/Sep/2026:13:44:11 +0000] "G

top clients
  10.14.22.9          1021
  10.14.22.31          344
  192.168.4.7          324
  172.16.0.99          312
  (undef)                3

status codes
  (undef)       3
  200       1029
  ...

Compare with Milestone 2: the same file now yields 2,004 records instead of 2,000, because the missing-bytes line is parsed rather than discarded, and the three genuinely unparsable lines are present as records with problems rather than absent. Client 10.14.22.9 gained the request it was previously denied.

The (undef) rows are the new behaviour being honest: those are the unparsable records flowing through counters that expect fields. In a real report you would put drop_problems() before the counters, or have the counter skip records that are not ok. Seeing them is better than not seeing them, and choosing where to drop them is now a one-line decision in the pipeline rather than a property of the parser.

What the abstraction costs

Three programs, the same 200,400-line file, same machine:

./bin/strata-scan                 0.37s   544,525 lines/sec    (count and match only)
./bin/strata-report               1.72s   116,322 lines/sec    (regex + six hashes)
./bin/strata ingest               5.47s    36,668 lines/sec    (Record objects + stages)

And memory, peak resident set on a 21 MB file:

strata ingest (streaming)                     9,036 KB
bare read loop                                4,992 KB
slurping the file into an array              43,292 KB

Two findings, and the second one is uncomfortable.

Streaming works. The pipeline uses 9 MB regardless of file size, while slurping a 21 MB file into an array costs 43 MB, about twice the file, because every line becomes a Perl scalar with its own overhead. On a 21 GB file that is 43 GB and the difference between a tool and an outage.

The object pipeline is three times slower than the flat script, and fifteen times slower than counting. That is the price of one blessed hash and two closure calls per line, and it is real: 36,000 lines a second means a 200 GB log takes hours.

Is it worth it? For now, yes, and I want to be precise about why rather than waving at "clean code". The pipeline buys pluggable formats (Milestone 6), reusable stages, provenance that survives, and a testable seam. A flat script buys none of those and would have to be rewritten to gain any of them. But the number is now on the table, it is the reason Milestone 11 exists, and the honest answer at that point may be that the hot loop gets specialised while the architecture stays. This is exactly the trade the Go course made in reverse: there, the profile said channel overhead dominated and we removed it; here, the profile will say object creation dominates, and we will decide what to do with that evidence rather than guessing now.

shell pipeline                                Perl Pipeline
───────────────                                ─────────────
grep ERROR access.log \                        $pipe->add_stage($match_stage)
  | awk '{print $1}' \                              ->add_stage($counter_stage)
  | sort | uniq -c \                                ->add_stage(drop_problems());
  | sort -rn
                                                $pipe->run_handle($fh, $name);

sub from_parsed ($class, $parsed) {
    my %meta;
    $meta{$_} = delete $parsed->{$_} for qw(source file lineno raw problems);
    $meta{fields} = $parsed;              # reuse, do not copy
    $meta{problems} //= [];
    $meta{entities} = {};
    return bless \%meta, $class;
}

sub pair_with_previous ($forget_after = 10_000) {
    my (%last, %last_seen, $n);

    return sub ($rec, $) {
        my $client = $rec->field("client") // return $rec;
        $n++;

        $rec->field(prev => $last{$client}) if exists $last{$client};
        $last{$client} = $rec;
        $last_seen{$client} = $n;

        # Periodically forget clients we have not seen recently, so a log
        # with a million distinct clients cannot exhaust memory.
        if ($n % $forget_after == 0) {
            for my $c (keys %last_seen) {
                next if $n - $last_seen{$c} < $forget_after;
                delete $last{$c};
                delete $last_seen{$c};
            }
        }
        return $rec;
    };
}

Experiment

Take any stage and replace its final return $rec; with a bare trailing print STDERR "stage ran\n"; — exactly the mistake the common-mistakes box below warns about — and put it first in a chain that has a second stage after it. The second stage does not receive the record: it receives the return value of print, which is 1, and the moment it calls $rec->field(...) on that plain scalar, Perl dies with Can't locate object method "field" via package "1". A stage that happened to be last in the chain would have hidden this entirely, since the pipeline only checks truthiness for counting records versus drops — which is exactly why this mistake is easy to introduce and easy not to notice until stages get reordered.

Common mistakes in Milestone 4

Checkpoint

1. Why does the pipeline take an open filehandle rather than a filename?
2. How does a stage drop a record, and where does the count go?
3. Why does counter return both a stage and a hash reference?
4. What does open $fh, "<", \$string do and why is it useful in tests?
5. Why must the read loop say defined?
6. The object pipeline runs at a third the speed of the flat script. What did that buy, and what will you do about it?

Repository state after Milestone 4

strata/
├── bin/
│   ├── strata            ingest subcommand, wires a pipeline together
│   ├── strata-scan       milestone 1: the filter
│   └── strata-report     milestone 2: one-pass reporting
├── lib/Strata/
│   ├── Pattern.pm        named, composable qr// building blocks
│   ├── Record.pm         fields + provenance + problems + entities
│   ├── Pipeline.pm       source -> parser -> stages, with stats
│   └── Parser/
│       └── Apache.pm     common and combined, lenient and strict
├── t/
│   ├── 20-parser-apache.t   good, imperfect, broken, statistics, detection
│   └── 30-pipeline.t        stages, dropping, provenance
└── share/fixtures/
    └── access.log        2,004 lines, four of them deliberately wrong

$ prove -l t/
All tests successful.  Files=2, Tests=9
$ git commit -am "milestone 4: a record model and a streaming pipeline"