What an experienced Perl programmer reaches for next, and whether you can now explain any of it

Part AThe advanced phase

The tool works: it ingests five formats, survives hostile files, extracts and normalises entities, correlates them into events in SQLite, behaves under a pipe and under Ctrl-C, fuzzes and profiles itself, and walks its own correlations as a graph. These five topics are what you would reach for next if Strata were something you had to operate rather than something you had to finish.

A1 · A plugin architecture for parsers

Strata::Parser::Registry has known every parser by name since Milestone 6. A real deployment wants to drop a new format handler into a directory without editing a core file at all. Perl's answer needs no dependency beyond the standard library:

package Strata::Parser::Loader;
use v5.36;
use File::Find;

# Auto-discover every Strata::Parser::* module under lib/, without a static
# list anywhere. A third party ships a new .pm file; nothing else changes.
sub discover ($class, $lib_dir = "lib") {
    my @found;
    find({
        wanted => sub {
            return unless /\.pm$/ && m{Strata/Parser/};
            (my $mod = $File::Find::name) =~ s{^\Q$lib_dir\E/}{};
            $mod =~ s{/}{::}g;
            $mod =~ s{\.pm$}{};
            push @found, $mod;
        },
        no_chdir => 1,
    }, "$lib_dir/Strata/Parser");

    for my $mod (@found) {
        (my $path = "$mod.pm") =~ s{::}{/}g;
        require $path;
    }
    return sort @found;
}

Verified against a two-file throwaway package tree: discover found and loaded both modules by name, in the order sort put them in, with no registry to keep in sync by hand. The trade-off is honest: a static list in Registry.pm is one file you can read top to bottom to know every format Strata understands; auto-discovery means reading the filesystem to answer the same question. For a tool meant to grow third-party parsers, that trade is usually worth it. What to measure: startup time as the parser directory grows — File::Find walking hundreds of files on every invocation is a real cost a static list does not pay, and the standard mitigation is a cached manifest invalidated by directory mtime.

A2 · Advanced parsing: when a regex stops being enough

Every parser so far matches one line at a time because every format so far is line-oriented, or was made to look that way (Milestone 6's pull-parser for XML). Nested configuration — the kind you would find describing which log sources feed Strata itself — is not: a regex has no memory of "how deep am I", which is precisely what nesting requires.

sub tokenize ($text) {
    my @tok;
    while ($text =~ /\G\s*(\{|\}|=|"[^"]*"|[\w.]+)/gc) {
        push @tok, $1;
    }
    return @tok;
}

sub parse_block ($tokens, $pos) {
    my %node;
    while ($$pos < @$tokens && $tokens->[$$pos] ne '}') {
        my $key = $tokens->[$$pos++];
        if ($tokens->[$$pos] eq '{') {
            $$pos++;
            $node{$key} = parse_block($tokens, $pos);
            die "expected '}'\n" unless ($tokens->[$$pos] // '') eq '}';
            $$pos++;
        } elsif ($tokens->[$$pos] eq '=') {
            $$pos++;
            (my $val = $tokens->[$$pos++]) =~ s/^"(.*)"$/$1/;
            $node{$key} = $val;
        } else {
            die "expected '{' or '=' after '$key'\n";
        }
    }
    return \%node;
}

\G anchors the next match to where the previous one left off rather than to the start of the string, and /gc (global, keep the position on failure) is what makes repeated /\G.../gc matches walk a string left to right without ever re-scanning what came before. This is Perl's built-in tokenizer loop, and most hand-rolled Perl lexers use exactly this idiom. parse_block is then ordinary recursive descent: a block is a sequence of key = value or key { ...nested block... } entries, and the recursion handles arbitrary nesting depth for free, which no single regex — however elaborate — can do, because a regular expression fundamentally cannot count in a way that carries across an unbounded nesting depth.

Verified on a three-level config with a nested block:

host: web-3, max_conn: 100

A3 · Property-based testing for the normaliser

Milestone 8's timestamp tests check specific strings against specific epochs — necessary, and not sufficient, because it only tests the inputs someone thought to write down. A property test instead asserts something that must be true for any input in a category, and lets the computer generate the inputs:

use Time::Piece;

srand(9001);
my $failures = 0;
for (1 .. 2000) {
    my $epoch = 1_600_000_000 + int(rand(200_000_000));       # any epoch in ~2020-2026
    my $iso   = gmtime($epoch)->datetime . "Z";
    my $back  = Time::Piece->strptime($iso, "%Y-%m-%dT%H:%M:%SZ")->epoch;
    if ($back != $epoch) {
        say "MISMATCH: epoch=$epoch iso=$iso back=$back";
        $failures++;
    }
}
say "$failures failures out of 2000 random epochs";

0 failures out of 2000 random epochs

The property here is a round trip: format then parse should return what you started with, for every epoch, not just the six hand-picked ones from Milestone 8. Perl has no built-in property-testing framework as batteries-included as Haskell's QuickCheck or Erlang's PropEr; Test::LectroTest exists on CPAN and is worth knowing about, but the technique above — a seeded random loop with an assertion inside it — captures the essential idea in eight lines and needs nothing installed. Seeding with srand(9001) is what makes a failing run reproducible: rerun with the same seed and you get the exact same 2,000 epochs, in the exact same order, which is the property-testing equivalent of Milestone 8's deterministic chaos schedule.

A4 · Structured logs for a forensic audit trail

A tool whose job is reconstructing what happened should be able to say what it did, precisely, after the fact. A minimal leveled logger, dependency-free:

package Strata::Log;
use v5.36;
use POSIX qw(strftime);

my %LEVEL = (debug => 0, info => 1, warn => 2, error => 3);
my $threshold = $LEVEL{ $ENV{STRATA_LOG_LEVEL} // "info" } // 1;

sub log ($level, $msg, %fields) {
    return if $LEVEL{$level} < $threshold;
    my $ts = strftime("%Y-%m-%dT%H:%M:%SZ", gmtime);
    my $kv = join " ", map { qq{$_="$fields{$_}"} } sort keys %fields;
    say STDERR "$ts level=$level msg=\"$msg\" $kv";
}

sub info  ($msg, %f) { log("info",  $msg, %f) }
sub warn  ($msg, %f) { log("warn",  $msg, %f) }
sub error ($msg, %f) { log("error", $msg, %f) }
1;

2026-09-12T13:44:10Z level=info msg="ingest complete" files="11" records="4035"
2026-09-12T13:44:11Z level=warn msg="unterminated quote, falling back" file="users.csv" lineno="42"

Key-value fields rather than an interpolated sentence, for the same reason Go's advanced phase gave for log/slog: a log aggregator can filter on file="users.csv" without a regular expression. STDERR, not STDOUT — Milestone 10 already made STDOUT the data channel a script might pipe into jq or head; mixing log lines into it would corrupt exactly the output that milestone worked to make well-behaved.

A5 · Shipping it

FROM perl:5.38-slim AS build
WORKDIR /src
COPY cpanfile .
RUN cpanm --notest --local-lib=/deps --installdeps .
COPY . .

FROM perl:5.38-slim
COPY --from=build /deps /deps
COPY --from=build /src /app
ENV PERL5LIB=/deps/lib/perl5
WORKDIR /app
RUN useradd -r -s /usr/sbin/nologin strata
USER strata
ENTRYPOINT ["perl", "bin/strata"]

A two-stage build so the compiler and build-time dependencies DBD::SQLite and XML::LibXML needed to compile their C extensions never ship in the final image, and a non-root user, because there is no reason a tool that only reads log files and writes a SQLite database needs to run as root.

# /etc/systemd/system/strata-ingest.timer
[Timer]
OnCalendar=*-*-* *:00/15
Persistent=true

[Install]
WantedBy=timers.target

A systemd timer rather than a naive cron entry gets you two things cron does not for free: Persistent=true catches up a missed run after the machine was off, and journalctl -u strata-ingest gives you the structured logs from A4 alongside systemd's own record of exit status and duration, in one place.

Part BThe final challenge

Everything up to here had a solution a few paragraphs later. This one does not, and it is deliberately at the edge of what you can now do. Spend real time on it before opening the last section.

Tailing logs that rotate, truncate, and disappear out from under you

Every milestone so far ingests files that already exist and stop changing while Strata reads them. Real logs do not hold still: they grow, get rotated (renamed aside, a fresh empty file created at the old path) or copy-truncated (truncated to zero length in place, same file, same inode, same path) by the logging system while your tool is running, sometimes every hour. Build strata watch DIR...: a live-ingesting mode that follows files as they change, survives both kinds of rotation with no duplicate and no lost record, and can be stopped and restarted without reprocessing anything it already saw.

Requirements

1. strata watch DIR [DIR...] watches every regular file already in the given directories and picks up new files that appear in them later.
2. Bytes appended to a file are ingested incrementally — read from where you left off, not by re-reading the file from the start.
3. Handle rotate-by-rename (the file at a watched path becomes a different file, identified by a change of device+inode) by starting the new file at its own beginning.
4. Handle copytruncate (the same file, same inode, truncated to a smaller size than you had already read) by resuming from the new end of file rather than either re-reading old bytes or crashing on a negative-length read.
5. Every few seconds, run correlation on only the newly ingested occurrences and print any event that was created or extended since the last pass — a live feed, not only a final report.
6. Ctrl-C stops cleanly (reusing Milestone 10's signal discipline) and remembers exactly how far into each watched file it got, in the store itself, so a restart neither reprocesses nor skips.
7. Idle files must not burn CPU. Fifty quiet files and one active one should look, in a CPU profile, overwhelmingly like "waiting", not like "checking".

Constraints

• No new heavy dependency. Linux::Inotify2 is an acceptable optional enhancement if you get the polling version working first; it must not be required.
• Per-file position bookkeeping lives in a watch_state table in the same SQLite database as everything else — not a side file that can drift out of sync with what was actually committed.
• Bounded memory regardless of how long watch has been running or how many rotations a file has been through.

Acceptance criteria

1. Rename-rotation. Append 100 lines, mv the file aside, create a fresh file at the same path, append 50 more. All 150 lines ingested, none duplicated.
2. Copytruncate. Append 100 lines, truncate the file to zero length in place (same inode), append 50 more. All 150 lines ingested, none duplicated — this is a different code path from (1) and must be tested separately.
3. Restart. Watch, ingest some lines, Ctrl-C, restart against the same directories. No re-ingestion of already-seen bytes; lines appended while stopped are picked up on restart.
4. Idle cost. 50 idle files plus one file appended to once a second for 60 seconds: the watcher's own CPU time over that minute stays low, measured with /usr/bin/time -v or equivalent, and does not scale with how long the run has lasted.
5. Live feed. An event appears in the live output within one polling interval of the occurrence that completed it — not only when the process is later stopped.

Hints, in increasing order of how much they give away

• The hard part is not reading new bytes. It is knowing when "the file currently at this path" is no longer the file you were reading a moment ago.
• One stat() call gives you a fact that changes under rename-rotation but not under copytruncate, and a different fact that changes under both. You need both facts, not just one.
• What you persist per watched file needs to be exactly enough to detect both kinds of change on your very next poll, including the first poll after a restart.
• A rename-rotation loses you the last few bytes written to the old file in the instant between your last read and the rename, if you are not also watching for that specific transition. Decide, and say in your write-up, whether you are willing to accept that small window or whether you close it (and how) — tail --follow=name versus tail --follow=descriptor made different, documented choices about exactly this.
• For the CPU constraint: a modest fixed poll interval (hundreds of milliseconds, not microseconds) already satisfies the requirement at the stated scale of fifty files. Say in your write-up at roughly what file count this approach stops being fine, and what you would reach for instead.

Attempt it before reading on. Even a partial implementation with an honest account of what you didn't solve is worth more than the section below.

sub poll_file ($self, $path) {
    my @st = stat($path) or return $self->_forget($path);   # file gone
    my ($dev, $ino, $size) = @st[0, 1, 7];
    my $prev = $self->{state}{$path};

    if (!$prev || $prev->{dev} != $dev || $prev->{ino} != $ino) {
        $self->{state}{$path} = { dev => $dev, ino => $ino, offset => 0 };
        $self->_save_state($path);
        return $self->_read_from($path, 0);
    }

    if ($size < $prev->{offset}) {
        $prev->{offset} = 0;
        $self->_save_state($path);
        return $self->_read_from($path, 0);
    }

    return () if $size == $prev->{offset};
    return $self->_read_from($path, $prev->{offset});
}

CREATE TABLE watch_state (
    path      TEXT PRIMARY KEY,
    dev       INTEGER NOT NULL,
    ino       INTEGER NOT NULL,
    offset    INTEGER NOT NULL,
    updated_at INTEGER NOT NULL
);

my %touched;                                  # (type, value) pairs seen this poll
for my $occ (@new_occurrences) {
    $touched{"$occ->{type}\0$occ->{value}"} = 1;
}

for my $key (keys %touched) {
    my ($type, $value) = split "\0", $key;
    $self->{store}->resessionize_one($type, $value);   # re-sweep just this entity's occurrences
}

Part CKnowledge check

C1 · Twenty conceptual questions

1. State Perl's context rule precisely: what decides whether an expression is evaluated in list or scalar context, and name three operations that behave differently depending on which.
2. Why does my $n = @array give you a count while my ($n) = @array gives you the first element?
3. What does wantarray return in each of the three calling contexts, and why is "void" a context distinct from "scalar"?
4. Explain autovivification. Why does if ($h{a}{b}{c}) create $h{a} and $h{a}{b} even though the condition is false?
5. Why are hash keys always strings, and what follows for a hash keyed by something that looks numeric, like an IP octet or a port number?
6. What is the difference between my and local? Give a case where local is the only correct choice.
7. Why did for (my $i = 1; $i <= 3; $i++) { push @subs, sub { $i } } capture one shared $i across all three closures, while for my $i (1..3) { push @subs, sub { $i } } does not?
8. Explain foreach's aliasing behaviour: why does $_ = uc $_ inside for (@list) { ... } modify @list itself?
9. What does local $/ do, and why does setting it to undef enable "slurp mode"?
10. Why is Perl's sort guaranteed stable since 5.8, and what would break in the sessioniser's sweep if it were not?
11. What is a reference, in terms of what it actually stores, and how does it differ from a C pointer?
12. Explain why Text::CSV or XML::LibXML cannot be replaced by a "good enough" regex, using a specific example from Milestone 6.
13. What does PerlIO::encoding::fallback control, and what does Perl's :encoding layer do to an undecodable byte if you never set it?
14. Why must alarm(0) be called on every exit path of a timeout-protected block, success and failure alike?
15. Explain the difference between eq/== and when using the wrong one produces a bug that only shows up on some inputs.
16. Why does wrapping ten thousand INSERTs in one transaction, rather than autocommitting each one, produce a two-orders-of-magnitude speedup in SQLite specifically?
17. What makes a SQLite index "covering", and why does that matter more for a point lookup than for a full scan?
18. After fork(), what exactly do the parent and child share, and what is copied? Why does that make a data race between them structurally impossible?
19. Why does closing the unused end of a pipe, in both the parent and the child, matter for correctly detecting end-of-file?
20. What does WITH RECURSIVE's plain UNION (rather than UNION ALL) prevent when the underlying graph contains a cycle?

C2 · Ten code-reading questions

Predict the output of each, then check. All ten were run to confirm the answers.

// 1
my @a = (1,2,3);
my $n = @a;
my ($first) = @a;
say "$n $first";

// 2
my %h;
$h{a}{b}{c} = 1;
say exists $h{a} ? "yes" : "no";
say ref $h{a};

// 3
my $s = "9"; $s++;
my $t = "Az"; $t++;
say "$s $t";

// 4
my @list = (3,1,2);
say "@{[ sort @list ]}";
my @nums = (10, 9, 2);
say "@{[ sort @nums ]}";

// 5
sub ctx { return wantarray ? "list" : defined(wantarray) ? "scalar" : "void" }
ctx();
my $x = ctx();
my @y = ctx();
say "$x @y";

// 6
my @subs;
for my $i (1..3) { push @subs, sub { $i } }
say join ",", map { $_->() } @subs;

// 7
my @subs2;
for (my $i = 1; $i <= 3; $i++) { push @subs2, sub { $i } }
say join ",", map { $_->() } @subs2;

// 8
my @names = ("alice", "bob");
for (@names) { $_ = ucfirst $_ }
say "@names";

// 9
my @prices = (19.995, 4.999);
printf "%d %.2f\n", $_, $_ for @prices;

// 10
my @arr = (1,2,3,4);
my $count = () = @arr;
say $count;

1.  3 1
2.  yes
    HASH
3.  10 Ba
4.  1 2 3
    10 2 9
5.  scalar list
6.  1,2,3
7.  4,4,4
8.  Alice Bob
9.  19 20.00
    4 5.00
10. 4

C3 · Five debugging exercises

Each gives a symptom and a suspect. Diagnose before opening the answer.

1. Symptom: a report script that builds closures per row for lazy formatting prints the same value in every row instead of each row's own value.

   my @formatters;
   for (my $i = 0; $i < @rows; $i++) {
       push @formatters, sub { format_row($rows[$i]) };
   }

2. Symptom: a "does this record have a location?" check that should just read data is somehow creating thousands of empty hash entries, visible as memory growth and a much slower keys %index loop later in the same run.

   for my $rec (@records) {
       next unless $rec->{fields}{geo}{country};
       $index{ $rec->{fields}{geo}{country} }++;
   }

3. Symptom: a "top talkers" report that groups by client IP shows the same address listed three separate times with different counts, instead of once with the combined count.

   my %by_ip;
   $by_ip{ $rec->field("ip_octet_str") }++ for @records;   # values like "007", "07", "7"

4. Symptom: a normalising pass that's supposed to build a cleaned-up copy of a list instead corrupts the original list it was only meant to read.

   my @hosts = extract_hosts(@records);
   for (@hosts) { $_ = lc $_ }
   push @clean_hosts, @hosts;   # caller finds @hosts itself is now lowercased too

5. Symptom: a nightly summary occasionally dies with Died within alarm handler or hangs entirely, only ever under heavy load, never in development.

   sub with_deadline ($code, $seconds) {
       $SIG{ALRM} = sub { die "timeout\n" };   # note: not local
       alarm($seconds);
       my $r = $code->();
       alarm(0);
       return $r;
   }

C4 · Five implementation exercises

1. A syslog parser (Milestone 6's own forward reference): both the classic Mon DD HH:MM:SS host tag: msg form and the RFC 5424 <PRI>1 timestamp ... form, in one parser, with a detect that does not falsely claim ordinary prose. Wire it into the Registry and add fixtures.
2. Parallel correlate. Milestone 12 parallelised ingestion; sessionisation still runs single-threaded. Shard by entity (type, value) hash across forked workers (each sessionises a disjoint slice of entities, so there is no shared state to coordinate), and benchmark against the single-process version at 500,000 occurrences.
3. A redaction stage using the keyed-hash pseudonymisation sketched in Milestone 8's exercises: replace extracted emails and card-shaped numbers in raw with stable tokens before a record is ever written to records.raw, so the database itself never holds the original sensitive text.
4. An idle-safe strata watch health check: a --report-every 60s flag that logs (via Milestone A4's logger) a one-line summary — files watched, bytes ingested, events created, current lag if measurable — on a fixed interval regardless of whether anything changed, so an operator watching logs can distinguish "quietly healthy" from "silently stuck".
5. A strata verify subcommand that re-reads every file referenced in records, re-parses each recorded line by (file, lineno), and reports any record whose stored raw no longer matches what is currently on disk at that position — detecting logs that were edited or replaced after ingestion, which is exactly the kind of tampering a forensic tool should be able to notice about its own evidence.

C5 · One substantial challenge

Distinct from the final challenge in Part B, and smaller, but not easy.

Build a differential fuzzer. For any two of the five format parsers, generate inputs that are ambiguous between formats on purpose (a CSV row whose first field looks like a JSON object; a line that is simultaneously a plausible Apache log line and plausible unstructured prose) and assert a specific property: that Strata::Parser::Registry's detect scores never produce a tie between two non-fallback parsers on the same input, or, where a genuine tie is unavoidable, that the registry's tie-breaking rule is deterministic and documented rather than dependent on hash key iteration order.

Requirements: at least 5,000 generated inputs per format pair; any discovered tie or nondeterminism must be saved as a permanent fixture and regression test, the same way Milestone 11's fuzzer turned its one found hang into t/90-fuzz.t; and the generator must be seeded and the seed logged, so a found problem is reproducible on demand. Hint: the interesting bugs are rarely in one parser's detect being wrong in isolation — they are in the comparison between two parsers' scores being close enough that which one wins depends on something that was never meant to be load-bearing, like hash iteration order in the registry's own scoring loop.

C6 · You should now be able to explain

• Perl's context rule, precisely, and at least four operations whose behaviour depends on it.
• Autovivification: what triggers it, why a read can trigger it, and how to avoid it when you only meant to check.
• foreach's aliasing of $_, and the difference between a C-style for loop's one shared variable and for my $x (...)'s fresh one per iteration.
• Why hash keys are always strings, and what that implies for any value used as one.
• The full DBI connect/prepare/execute/placeholder cycle, and why placeholders are not optional the moment any value comes from outside the program.
• Why one transaction beats autocommit-per-row, in terms of what SQLite actually does once per commit.
• How to read EXPLAIN QUERY PLAN, and what "covering index" means.
• The complete rotation/truncation detection logic from the final challenge, and why device+inode and size/offset together are both necessary.
• Why fork() makes data races between two Perl worker processes structurally impossible, and what it costs you in exchange.
• The zombie-process and pipe-buffer-deadlock hazards of fork-and-pipe designs, and what specifically avoids each.
• Why alarm() plus local $SIG{ALRM} is Perl's idiomatic last-resort timeout, and the two places a timeout helper must cancel it.
• The difference between testing specific examples and property-based testing, with a concrete property from this project.
• Recursive descent parsing well enough to say why a regex cannot handle arbitrary nesting on its own.
• Why SIGPIPE's default action is silent termination, and when you would override it.

C7 · You should now be able to implement

• A streaming text pipeline that survives gzip, encoding errors, truncation and adversarial line lengths without unbounded memory growth.
• A pluggable format-detection system using confidence scores rather than a boolean, with a never-refuses fallback.
• An entity extraction pipeline that separates matching, validation and normalisation into distinct steps, driven by context rules rather than the pattern alone.
• A SQLite-backed correlation engine: schema, transactional bulk load, a sort-then-sweep grouping algorithm, and the indexes that make its queries fast.
• A command-line tool with real subcommands, Getopt::Long option parsing, sysexits-style exit codes, and graceful behaviour under both SIGINT and a closed output pipe.
• A deadline-bounded fuzzer using alarm(), and a shared contract test reused across a family of plugin-like modules.
• A profiling session with Devel::NYTProf that finds and fixes a genuine algorithmic hot path, with before-and-after numbers.
• A fork-and-pipe worker pool with correct reaping and no shared mutable state.
• A recursive graph query over SQLite using WITH RECURSIVE.
• A rotation-safe file tailer using device/inode/offset tracking, persisted so it survives a restart.

Part DShipping it: README, portfolio, interview

D1 · README draft

# strata

A text-forensics tool: ingest ugly, heterogeneous log data (Apache/nginx,
CSV, JSON lines, XML, arbitrary unstructured text — even gzipped, mis-encoded
or truncated), extract and normalise entities, correlate them into events,
and query the result as a graph. Built to survive input designed to break it.

## What it does

- Five pluggable formats behind one interface, chosen by confidence-scored
  content sniffing, never by trusting a file extension alone.
- Streams input of any size in bounded memory: gzip, BOMs, bad encodings,
  truncated files and multi-megabyte "lines" are all handled explicitly.
- Extracts and normalises entities (IPs, hosts, paths, timestamps) with a
  match -> validate -> normalise pipeline, not bare regex matching.
- Correlates occurrences into events in SQLite (one transaction per batch:
  ~700x the throughput of naive autocommit-per-row inserts) and links
  related events into a graph, queryable with recursive SQL.
- A real command-line tool: subcommands, --help, sysexits-style exit codes,
  clean shutdown on Ctrl-C, correct behaviour piped into `head`.
- A deadline-bounded fuzzer (found a real infinite loop in under a second)
  and a profiling session with a measured 50x fix, both checked into the
  test suite as permanent regressions.
- `strata watch`: follows growing log files live, correct across both
  rename-rotation and copytruncate, resumable after a restart.

No third-party dependencies beyond DBI, DBD::SQLite, Text::CSV and
XML::LibXML — no framework, no ORM.

## Quick start

    cpanm --installdeps .
    ./bin/strata ingest strata.db share/fixtures/*.*
    ./bin/strata correlate strata.db --gap 60
    ./bin/strata query strata.db --type ipv4 --value 10.0.5.100
    ./bin/strata watch strata.db /var/log/myapp

## Commands

    ingest DB FILES...     parse and store records + entity occurrences
    entities DB FILES...   list extracted entities, ranked
    correlate DB           build events from occurrences (sessionisation)
    query DB [filters]     look up records and events
    graph DB --from ID     walk correlated events as a graph, N hops
    watch DB DIRS...       live-ingest and live-correlate growing logs

## Architecture

    lib/Strata/
    ├── Source.pm, Extract.pm, Normalize.pm    survive and clean the input
    ├── Parser/                                 five formats, one contract
    ├── Store.pm, Correlate.pm, Graph.pm        SQLite: events and edges
    └── Log.pm                                  structured logs, audit trail

## Testing

    prove -l t/                    everything, including the fuzzer
    perl -d:NYTProf bin/strata ...  profile a real invocation

The fuzz suite (`t/90-fuzz.t`) is deadline-bounded: every case gets one
second via `alarm()`, so a hang is reported as a test failure, never as a
CI run that quietly never finishes.

## Known limitations

- `watch` polls; past a few hundred files, an inotify-based backend would
  be needed instead (sketched, not implemented).
- Correlation runs in one process; ingestion parallelises with `fork`,
  sessionisation currently does not.
- No consensus or distribution: one SQLite file, one machine.

## Licence

MIT

Three deliberate choices, matching Course 1's README: it leads with what was measured (the 700× transaction number, the 50× profiling fix, the sub-second fuzzer find) rather than adjectives; it names the fuzzer and profiler as permanent parts of the test suite, not one-off exercises; and it has a known limitations section, because a forensics tool that overstates its own reliability is a bad one.

D2 · GitHub project description

A text-forensics and log-correlation tool in Perl: five formats behind one confidence-scored interface, entity extraction and correlation in SQLite, a fuzzed and profiled parser, and a rotation-safe live tail. Built as a laboratory for Perl's text-processing and Unix-glue strengths.

Topics: perl, log-analysis, text-processing, sqlite, dbi, cli, fuzzing, forensics, data-correlation, parsing, regex.

D3 · Performance considerations

• Transactions dominate every other consideration for bulk writes. The single biggest number in this whole project — roughly 700× — came from wrapping inserts in one transaction, not from anything algorithmic.
• An index earns its keep on the query shape that runs most often, not the one that looks most important. The point lookup, not the full sweep, is what a covering index actually paid for here.
• Measure before optimising the obvious target. The qr//-caching guess that measured as a no-op, next to the grep-versus-hash dedupe that measured as 50×, is the whole argument for profiling over intuition in one pair of numbers.
• A process-per-file design (fork) parallelises CPU-bound parsing cleanly but does not parallelise a single SQLite writer — past the point of write contention, more workers stop helping and start costing.
• Bound everything an adversarial or merely malformed file could make unbounded: line length, decompressed size, and — new in the final challenge — how far a live tail can fall behind before that lag itself needs to be visible.

D4 · Security considerations

• Every SQL statement in this project uses placeholders. Never interpolate a value — especially one taken from an ingested log line — directly into a SQL string; that is exactly how a forensics tool ingesting attacker-controlled log data becomes attacker-controlled itself.
• Never build a shell command from a filename, per Milestone 7's exercise: the list form of open/system (arguments as separate strings, no shell interpolation) is mandatory the moment a filename can come from outside the program, because a file literally named ; rm -rf ~ is a real, if rare, adversarial input.
• Decompression bombs. Milestone 7 bounds line length and peeks a fixed number of bytes, but nothing here bounds total decompressed volume from a gzip file — a small file on disk can still expand to gigabytes while being read. A production deployment should cap total bytes read per file, not merely per line.
• Sensitive data in raw. Ingested log lines can contain emails, card-shaped numbers, or credentials, stored verbatim in records.raw unless the redaction stage from exercise C4.3 runs first. Decide deliberately whether raw text belongs in the store at all for a given deployment.
• The store has no access control of its own. Anyone who can read strata.db can read everything correlated into it; file permissions on the SQLite file are the actual security boundary, and there is no row-level protection layered on top.
• strata watch follows any file it is pointed at, including one that changes ownership or gets replaced by a symlink to something outside the intended log directory. Validate watched paths resolve inside an expected root before trusting what they report.

D5 · What to put in your portfolio

Do not present this as "a log parser". Present it as what it is: a forensics pipeline built to survive hostile input at every stage, with every claim about performance and robustness backed by a number you actually measured. The narrative that makes it interesting is the sequence of real bugs found by testing for them on purpose, not the feature list.

1. A parser that trusted well-formed input, broken by six deliberately hostile fixtures — gzip, BOM, bad encoding, truncation, binary bytes, a giant line — each with the specific fix.
2. Two false positives in entity extraction (a version number mistaken for an IP, a protocol version mistaken for a path) that only a match-validate-normalise pipeline with context rules catches.
3. A ~700× transaction speedup and a ~53× index speedup, both measured, both explained in terms of what SQLite is actually doing differently.
4. A fuzzer that found a genuine infinite loop in under a second, turned immediately into a permanent regression test rather than a one-off finding.
5. A profiling session where the "obvious" optimisation (cache the regex) measured as a no-op and the unglamorous one (a hash instead of grep) measured as 50×.
6. The rotation-safe live tail, and the honest accounting of what it still does not solve (inotify, the rename-instant gap, multi-writer SQLite).

Keep a docs/ folder with the NYTProf HTML report, the EXPLAIN QUERY PLAN output, and one architecture diagram. A reviewer who spends ninety seconds on the repository should come away knowing you test for failure on purpose, not just that the happy path works.

D6 · Interview questions someone could ask, and what a good answer contains

D7 · Extensions worth building

• An inotify backend for watch, falling back to polling automatically when Linux::Inotify2 is unavailable or the platform is not Linux.
• A web UI over query and graph — this project deliberately stayed command-line-only; a small Mojolicious app rendering the graph queries as an actual graph would be a natural, contained extension.
• Streaming JSON Lines output for every command, so strata composes with jq the way a well-behaved Unix tool should, beyond the one flag sketched in Exercise 10.
• A second storage backend (Postgres, via the same DBI interface with different SQL for the recursive query and the pragmas) to see how much of Strata::Store is genuinely portable and how much quietly assumed SQLite.
• The differential fuzzer from C5, which is the most educationally valuable of these by a distance.

Course 3 completeWhat you built

A dependency-light Perl distribution across roughly a dozen modules, five parsers behind one contract, a SQLite-backed correlation and graph engine with measured order-of-magnitude performance differences, a command-line tool that behaves correctly under a signal and inside a pipeline, a fuzzer that found a real bug in under a second, and a live log tailer that survives both ways a log file can change out from under you. More importantly: a habit of writing the hostile fixture before trusting the code that has to survive it, and of measuring a claimed speedup before writing it down.

The central question of this curriculum was what kinds of problems does this language make unusually natural to solve? Perl's answer, stated as precisely as this project allows: problems where the input is real-world messy, the shape of "correct" is "did not corrupt or lose the awkward 10% of records", and the win comes from CPAN's decades of exactly-this-format modules plus a handful of small, sharp built-in idioms — context, autovivification, foreach aliasing, alarm(), fork — that read as strange in isolation and as exactly right once you have needed them once. Not the fastest, not the most structured, not the friendliest first error message. The one where a text file nobody designed on purpose stops being a mystery in an afternoon.

Courses 4 and 5 continue the same comparison — Erlang's processes and supervision trees against Go's goroutines and Perl's fork, and Racket's macros against the recursive-descent parser built in this instalment's advanced phase. The syllabus overview describes what they cover.