A survey of what a full-stack developer learned doing AI engineering on a live production system

This is Part 3 of a three-part series. Part 1 is the story of why the system exists; Part 2 is the engineering detail. This part is the distillation: the rules I now follow, each one paid for by a specific mistake¹. I’d bet that this list is far from complete and I will have the opportunity to add to it.

I came to AI engineering with decades of full-stack habits. Some transferred directly, some had to be unlearned, and a few new ones had to be acquired at the cost of production incidents. Here are twelve, in roughly the order the pipeline taught them to me.

1. Regex before AI. If you can write a rule to split possible answers into an identifiable finite set — say, always either a yes or a no — then write it in code. Our Stage 1 runs entirely without AI — regex is cheap, fast, and exact. The AI calls are reserved for the profiles that actually need them.

2. Exit early. Order the stages from cheapest to dearest (if possible), and let every stage have the option to end the process. A dead domain doesn’t need classification; an irrelevant site doesn’t need a written profile. Early exits are simultaneously a cost decision and a quality decision — the expensive calls only ever see input that needs them.

3. Fail loudly. For a while, timeouts and 503 errors from our inference provider were being rescued silently and returned as “irrelevant” — and profiles of legitimate publishers were marked for deletion, quietly, at scale (the full story is in Part 2). An exception that turns into a plausible-looking verdict causes far more damage than a crash, because nothing looks wrong. Let errors surface.

4. Retry explicitly. The corollary of failing loudly: transient errors deserve a deliberate retry policy, not a rescue block that swallows them. Decide how many attempts, with what backoff, and what happens when retries are exhausted — and make that last case visible too.

5. Human-review irreversible actions — until the data says otherwise. Deletion started with a human gate, and that gate is what caught the silent-failure bug. Later, when extended testing showed near-100% accuracy on deletion verdicts, we removed the gate and kept an appeal path instead.

6. Benchmark against your own data, not public benchmarks. Leaderboard scores told me nothing about classifying publisher websites in dozens of languages. Testing model tiers against our own profiles did — and showed that the quality curve flattens far below the frontier for our task, while the cost curve keeps climbing (the arithmetic is in Part 2). The frontier premium may be worth it for your problem. Find out with your data, not someone else’s.

7. Build your eval set from real failures. Every production mistake — the self-published author classified alongside Penguin Random House, the domain registrar that passed the supplier check — is a labelled test case. Collected, they become the yardstick every prompt change and every candidate model has to beat before it touches production.

8. Prompt calibration beats intuition. The service that holds the pipeline’s prompts has 70 commits, and the count still rises. Every version that failed had seemed obviously right when I wrote it; the edge cases only emerged in production. The only way I’ve found to improve a prompt is to run real data through it, read the failures, adjust, and rerun. There’s a sweet spot between too much instruction and too little, and only calibration finds it.

9. Let the code make the final call. When I let the model assign taxonomy tags directly, it invented tags that weren’t in the vocabulary and inferred things the content didn’t support. When I tried to use regex to extract them from the content, I failed miserably. The fix: the model produces hints, and a deterministic pattern matcher makes the decision against the controlled vocabulary. Wherever an AI output feeds something that must be exact, put deterministic code between the model’s output and the final result.

10. Your first pipeline is a draft — consolidation comes after correctness. The pipeline launched with five AI calls per profile because that was the obvious, correct decomposition of the problem. It runs on two today. I could not have designed the two-call version upfront: only production revealed which calls could be eliminated and/or consolidated. Build the version you can reason about, get it correct, and let the running system tell you what to improve, consolidate or drop.

11. Re-test the model market regularly — price and quality both move. Model prices fall and capabilities rise while your pipeline stands still. A decision that was right six months ago can be wrong today. Re-benchmarking isn’t churn; it’s how the pipeline got cheaper and simpler — maybe even better.

12. Pay frontier prices to solve; pay open-weight prices to run. When a task resists a smaller model, take it to a frontier model first and let it wrestle the problem into shape — the framing, the rules, the edge cases. If the frontier models can’t solve it, then either the issue is elsewhere in the system, or the problem may be unsolvable — for now at least. Once the task is well-defined and demonstrably solved, it can be handed down to a smaller model to see if it can run predictably at production prices. Think of the frontier model as a consultant you bring in to crack a problem, and the small model as the staff who run the routine.

If these twelve compress into one sentence, it’s the meta-lesson of the whole project: a system built with domain insight and a modest model will beat a naive use of a frontier one. That has been true for every task in this pipeline. The scarce ingredient was never access to models — it was knowing what counts as a publisher, what a directory owes its members, and where software must not be allowed to guess.

If your production experience contradicts any of these, I’d genuinely like to hear it — almost all of this comes from a single system, and YMMV.