What I learned running an AI verification pipeline over 20,000 publisher profiles

This is Part 2 of a three-part series. Part 1 tells the story of why this system exists. Part 3 collects the rules I now follow. This part is the engineering.

PublishersGlobal runs a verification pipeline over an active directory of ~20,000 publishers and service providers. Every profile needs to be continuously verified and re-verified, because data on the internet degrades very quickly. In the first few months we ran close to 30,000 verification passes. This post recounts some of what I learned building and running the verification system on open-weight models.

TL;DR

• Programmatic checks come before AI calls. Regex is more reliable, cheaper and faster — exhaust the programmatic options first.
• The pipeline launched with five AI calls per profile. Today it runs on only two. Calibration, optimisation and pipeline redesign eliminated the rest, with no measurable loss in quality.
• For most tasks in the middle of the complexity bell curve, the quality curve flattens as you go up in model tier. The frontier premium can be hard to justify on a limited budget.
• Cost varies enormously by tier: the original five-call design priced out at ~$26 to ~$1,552 per 20,000-profile run depending on the model. The current two-call pipeline costs substantially less.
• What quality improvement justifies frontier spend is entirely context-dependent. A 10% improvement means something very different in a medical coding system than in a publishing directory.
• For bounded tasks, prompt calibration can matter more than model tier. And calibration task itself is never really finished.

What the pipeline does

The pipeline runs in two modes:

• it processes new submissions as they come in, and
• it re-verifies existing profiles on a rolling schedule.

Every profile in the database cycles through it periodically. Sites that are dead — or no longer anything to do with publishing — get flagged for removal. Businesses that have changed get their profiles updated.

The architecture has six stages:

1. Website fetch and programmatic checks
2. AI analysis — entity classification
3. Editorial content generation
4. Taxonomy matching against a controlled vocabulary
5. Recommendation — approve, flag for review, or reject
6. Execution

The most important design decision is what happens before the AI ever runs.

Stage 1 is entirely programmatic. The pipeline fetches the website and runs a battery of non-AI checks: is the domain live? Is this a domain-for-sale page? A redirect-only page? Is the content gambling, adult? Is the site blocking automated access? These checks are fast and free.

Many verification runs exit here without a single AI call. Dead DNS, parked domains, gambling sites: these don’t need classification. They need deletion. Running them through a model call would be unnecessary.

This is the first rule I’d give anyone building something similar: don’t use AI where a regex works. It sounds obvious. But when you’re building fast and the model is a function call away, it is tempting to use it anyway.

For profiles that pass Stage 1, the current pipeline makes exactly two AI calls:

1. Entity classification — is this a publisher, a supplier, a self-publishing platform, something irrelevant, or something dangerous?
2. Editorial content generation — write a brief and a full description of the organisation, suggest taxonomy tags

The pipeline did not start this lean. I launched with five separate AI calls per profile verification: language detection, translation for non-English sites, a pre-review triage, and then classification and content generation. Over months of calibration I managed to eliminate three of them. The model was able to classify and describe non-English content directly, which removed the language-detection and translation calls entirely and the triage work folded into other stages. The first pipeline was working fine, but the new one made fewer calls, used substantially fewer tokens and ran much faster with no measurable drop in quality. Looking back, the first pipeline was simply a draft.

Taxonomy matching is where AI and programmatic logic meet. Editorial call outputs loose taxonomy hints alongside the description — “this looks like a book publisher, publishes in English and German.” Those hints are used by a pattern matcher running against a controlled vocabulary to boost confidence. The model’s output is treated as a hint; deterministic code makes the final decision. I tried letting the model do the matching directly, but it would invent tags not in the vocabulary or infer things the content didn’t support. Pattern matching by itself did not work so well either. But pattern matching seeded with AI hints turned out to be accurate and far more predictable.

Each stage can exit early. If classification comes back as irrelevant, content generation is skipped. The expensive calls only run when the cheaper checks are passed.

What it cost — the numbers I can share

I spent time comparing open-weight models at inference providers against frontier models before committing to an architecture. I will show you the math I used to make the decision.

A note on what these numbers describe. The arithmetic below is for the original five-call design — I’ve kept it because the decision was actually made then, and the numbers are real. The current two-call pipeline consumes substantially fewer tokens. The math also assumes every one of the 20,000 profiles gets the full treatment, with no early exits; in production many verification runs exit at Stage 1 with zero AI calls, so actual spend runs lower still. I’ve left both simplifications in to keep the arithmetic easy to understand.

Here’s how I worked it out. (If you don’t care about token arithmetic, skip to the table.) Each profile that went through full analysis made up to five AI calls. Based on the token counts for each call, a typical profile consumed roughly:

• English profile (about 77% of the database): ~15,500 input tokens, ~1,250 output tokens
• Non-English profile (about 23%, requiring a translation pass): ~20,500 input tokens, ~6,250 output tokens (translation was the one call whose output is a large body of text rather than a small structured JSON response, so its output tokens scale with content length)

For a batch of 20,000 profiles, that blends out to approximately 333 million input tokens and 48 million output tokens in total.

The table below shows the approximate cost of three open-weight models at low, mid and high price points (available on Together AI) against a frontier model. Prices as of July 2026.

Note the ordering: price doesn’t track parameter count. The 120B model costs less to run than the 70B one because serving cost depends on architecture — mixture-of-experts models like gpt-oss activate only a fraction of their parameters per token, while a dense model like Llama 3.3 uses all of them.

(Yearly totals are computed from the unrounded per-run costs — the exact math is below — and rounded at the end, so they won’t precisely match the rounded per-run column times six.)

The math for each, using 333M input tokens and 48M output tokens per 20k-profile run:

gpt-oss-20B — 333M × $0.05 = $16.65 input + 48M × $0.20 = $9.60 output = $26.25 per run

gpt-oss-120B — 333M × $0.15 = $49.95 input + 48M × $0.60 = $28.80 output = $78.75 per run

Llama 3.3 70B — 333M × $1.04 = $346.32 input + 48M × $1.04 = $49.92 output = $396.24 per run

Frontier model (illustrative) — 333M × $2.50 = $832.50 input + 48M × $15.00 = $720 output = $1,552.50 per run

The question the table can’t answer — does quality also scale with cost?

The table above shows how cost scales across model tiers. It doesn’t show whether quality scales with it — and that’s the more important question.

In my evaluation, the cost curve is steep — steeper still at the frontier — while the quality curve for our use case flattens quickly. And the trade-off doesn’t hold still: every change to the process shifts it, so the candidate models need re-benchmarking as the pipeline evolves.

A 10% improvement in classification accuracy sounds meaningful in the abstract. Whether it matters in practice depends entirely on what happens to the errors. For a publishing directory:

• A wrong classification (publisher tagged as supplier, or vice versa) was rare and sometimes genuinely debatable — some entities are both, or borderline. Either way, errors get caught programmatically in the taxonomy extraction phase, corrected on the next re-verification pass, or flagged by a human user or the profile’s owner in the meantime.
• If there was a slightly weaker description or brief it was not noticeable — generation/summarisation quality doesn’t reliably track model tier anyway.

Compare that to a context where errors have real consequences: a medical coding system, a legal document classifier, a fraud detection pipeline. There, a 10% quality improvement at 4x the cost might be the easiest business decision you ever make. Here, it wasn’t worth much at all.

Here’s a generalisation I’d defend: for most classification tasks that sit in the middle of the complexity bell curve — structured inputs, bounded outputs, well-defined categories, recoverable errors — the quality difference between a well-prompted mid-tier open-weight model and a frontier model is real but rarely perceptible, and prompt calibration and programmatic checks close most of the gap. For this project, a dollar spent on better prompt calibration consistently outperformed a dollar spent on a larger model. That may not be true for every problem, but it’s worth testing before you commit to a cost structure.

One nuance: the premium that’s hard to justify in production can be worth paying during development. I’ve sometimes let a frontier model wrestle a hard task into shape first, then handed the solved, well-defined version down to a smaller model to run. More on that in Part 3.

I’m not sharing which tier we landed on, beyond this: an open-weight model, not a frontier one. And the decision wasn’t primarily about cost — it was about what the quality difference was actually worth in our specific context, tested against real data, not benchmarks.

Worth repeating: the early-exit design from Stage 1 is a cost lever as much as a quality one. The profiles that die there cost nothing in every pass that follows.

What 70 commits taught me about prompt calibration

The service that holds the pipeline’s prompts — classification, editorial generation, and formerly translation and triage — has 70 commits in the git history, and the count keeps rising: streamlining the pipeline down to two calls added a batch of its own, yet most of them are prompt changes. Clearly not the proudest corner of my git history, but I hadn’t expected calibration to be this hard: it is easy to over-correct in one direction and then the other. Counterintuitively, giving more direction and more examples is not always helpful — it can make the prompt too complex, or simply confusing.

Prompt engineering for classification is empirical: the way to get it right is to run production data through it, read the failures, adjust, and rerun.

Some examples of failure modes, each with its own fix.

Watch out for edge cases — the publisher definition problem

The core question the classification prompt has to answer: what makes an organisation a publisher?

My first version asked: “Do they CREATE or PRODUCE content for readers?” Seemed right. But individual self-published authors create content for readers, and they were being classified as publishers alongside Penguin Random House. Wrong.

Second version: “Do they SELECT, CURATE and PUBLISH content?” Better. SELECT and CURATE add the editorial gatekeeping role — a self-published author didn’t pass. But organisations with active blogs started passing. A SaaS company that publishes blogs or newsletters is not a publisher.

Third version added an explicit rule: publishing must be the core business, not a side channel. Running a newsletter doesn’t make you a publisher. Writing blog posts doesn’t make you a publisher.

Each of those three versions was live in production before the failure mode appeared. You can’t reason your way to the edge cases in advance. They emerge in production, and you have to keep reviewing.

Even with that there was a category that needed an explicit business decision. Is a vanity publisher a publisher or a supplier? In a way it is a publisher, because they publish, but unlike regular publishers they charge the authors instead of paying them. In that sense they are service providers. But then they are not providing services to publishers, like most service providers in the directory, instead they are providing a service to authors. This debate eventually led to a directory structure reorganisation.

The two failure modes of prompt optimisation

Every classification prompt has two failure modes, and optimising against one tends to create the other.

The supplier category — organisations that provide services to publishers — took 15+ prompt iterations. Tighten the definition to exclude general B2B companies, and literary agents fall out of the classification. Broaden it to include literary agents, and domain registrars start passing because they technically “serve publishers.”

The calibration is never really finished — what you’re doing is drawing a boundary in language, and language is imprecise.

The lesson: when a new failure mode appears, understand why the model reasoned its way there before fixing the prompt. The failures are rarely arbitrary — the model is reasoning from the words you gave it. If a domain registrar passes the supplier check, it’s because the prompt contains a test that a domain registrar legitimately passes. Find the test, change it, and check what else you’re about to break.

The translation hallucination

About 23% of the profiles in the database are on non-English websites. The original pipeline translated them to English before analysis — and that step produced my favourite failure.

The first version of the translation prompt had a straightforward instruction: translate this content to English. What I got back was not always a translation. A sparse three-page German publisher website would return a comprehensive English description. Generated from the information the model had about this publisher in its training data. But that training data was dated. The model was hallucinating, and doing it confidently. I needed a translation of the actual content, not stale knowledge the model had about the entity.

The fix was an explicit block at the top of the prompt — capitalised, prominent — forbidding the model from using any knowledge not present in the source content:

⚠️ CRITICAL TRANSLATION RULES ⚠️ You MUST translate the PROVIDED CONTENT ONLY You MUST NOT use your training knowledge…

Even though the translation step no longer exists in the pipeline (the model classifies and describes non-English content directly from the source language — one of the three calls that calibration eliminated), the lesson outlived the step it came from: naming the failure mode made the fix obvious. The difficulty was noticing it in the first place.

The failure that nearly deleted real publishers

Silent failures are worse than loud ones. For a period, API timeouts and 503 errors from the inference provider were being read as entity_type: irrelevant. The pipeline kept running. Some profiles kept getting incorrectly flagged for deletion. By the time I caught it, a number of legitimate publishers had been marked for deletion. It got caught because, at the time, deletion had a human in the loop.

That gate is gone now. After extended testing showed near-100% accuracy on deletion verdicts, we removed the human review — and kept an appeal path, so an entity wrongly marked for deletion can get itself restored.

Fail loudly. Let errors surface. Retry explicitly. And keep a path to undo anything irreversible.

Prompt placement matters more than you’d imagine

A negation rule buried in the middle of a long prompt gets ignored. The same rule at the top and bottom gets followed. Primacy and recency bias consistently affected behaviour in my experiments. If a rule is important, think carefully where to place it. Maybe try both top and bottom.

The pipeline running today is quite different from the one I launched, and it will keep changing. Part 3 collects what all this iteration has taught me — the rules I’d start with if I were doing it again.