Reworking Phases 2–4 When There Is No Test Suite
In the case study, the ledger-events monorepo has a working test
runner (pnpm test --filter <package>) that lets Claude write a
failing regression test before implementing the pagination fix, and
later confirm the fix with a real, re-runnable check.
Now suppose you're debugging the identical symptom — CSV exports intermittently missing rows — in a codebase with no automated test suite at all. There's no test runner to write a failing test against, and no CI to catch a regression later.
- Rework Phase 2 (Exploration): does the plan still make sense to propose, and does it change shape without a test suite to design verification around?
- Rework Phase 3 (the wrong turn): without a failing test to prove the timezone hypothesis wrong, how would you actually discover that it didn't fix the bug? Be concrete about what evidence you'd look for.
- Rework Phase 4 (Implementation): what does "give Claude a way to verify its work" mean when there's no test framework to write tests in? Name at least two concrete substitutes.
1. Reworking exploration:
The plan itself barely changes — you still want the three-subagent fan out (reproduce the symptom, recent history, pagination code path) and you still want plan mode, because the uncertainty that justifies plan mode (unfamiliar code, multi-file change, an unclear root cause) has nothing to do with whether a test suite exists. What does change is what the plan promises as its verification step. Instead of "write a test with fixture rows straddling a day boundary," the plan should promise something concrete but not test-shaped: "reproduce the missing rows against a copy of production-like data by running the actual export job twice under simulated concurrent writes and diffing the output row counts." The plan review step matters more here, not less — with no test suite as a safety net, a wrong or vague verification strategy in the plan is more expensive to discover later.
2. Discovering the timezone hypothesis was wrong, without a test:
You'd need a substitute check that plays the same role a failing test
played in the original narrative: something that returns a pass/fail
signal Claude (and you) can read. Concretely: run the actual export job
against a fixture dataset with known row counts, before and after the
timezone patch, and diff the exported row count against the expected
count. If the patch is applied and the count is still short by the same
number of rows as before, that's the equivalent falsification signal
the failing test provided in the original narrative — it just comes
from a one-off script and a manual diff instead of pnpm test. The
key discipline that transfers unchanged: don't declare victory because
the patch "looks right" or because no error was thrown; insist on
re-running the same reproduction and comparing evidence, in both
versions, before and after.
3. Substitutes for "give Claude a way to verify its work":
A test suite is one implementation of "a check that returns pass/fail," not the only one. Concrete substitutes: (a) a small, disposable verification script — not a permanent test file, just a script that seeds known data, runs the export, and asserts the output row count matches, deleted or promoted to a real test later; (b) a screenshot- or output-diff style check where you paste the customer's actual short CSV and a corrected one and ask Claude to diff row counts programmatically; (c) a build/typecheck exit code as a weaker but still real signal that at least confirms the change compiles and doesn't silently break an interface, useful as a first gate even when it says nothing about correctness. None of these substitute for writing an actual test suite going forward — and one reasonable output of this whole incident, test-suite-less codebase or not, is "we should add pagination-integrity tests," which is itself a CLAUDE.md-worthy lesson from the incident.
Share this question