Building an AI system that doesn’t make things up
A verified-claims intelligence engine that produces competitive-analysis reports a business can bet decisions on — because nothing reaches the page that wasn’t gathered, verified, and human-approved.
Scroll to run the pipeline. Raw signals enter on the left; a trustworthy report leaves on the right — and in the middle sits a human gate that nothing passes unapproved. Tap any stage for detail.
Prose is generated only over approved facts. That’s what kills hallucination.
Large language models are fluent but unreliable narrators. Ask one to both determine what’s true and write the report, and it will confidently fabricate to fill gaps — an invented statistic, a competitor that doesn’t exist, a number that feels right. For a document a business pays for and makes decisions on, a single made-up figure destroys the entire report’s credibility. Most “AI report” tools ship exactly this failure mode.
Separate the two jobs the model was doing at once. Truth-determination and prose-generation become different stages with a hard wall between them — and a human in the middle. Three immutable layers:
Raw observations
Machine-gathered signals, stored as data — never as prose.
A verified-claims ledger
The trusted middle. A human approves every claim before it can be used.
Prose over approved claims only
The writer model can only speak about facts that already cleared the gate.
Fixing the architecture instead of patching prompts collapsed dozens of hand-written guardrail rules and a stack of prompt templates down to almost nothing. Reliability became a property of the system, not a plea to the model.
A six-stage pipeline, ~13,000 lines of Python, running as a file-based state machine with no hand-editing of any output:
- 01
Signal gathering
Automated collection across roughly a dozen public data sources — search and maps presence, reviews, site health, technical and content signals, AI-search visibility.
- 02
Automated verification
Programmatic checks flag thin, stale, or unverifiable data before a human ever sees it.
- 03
Human approval gate
The operator approves, rejects, or annotates each claim; re-verification invalidates stale approvals. Nothing proceeds without sign-off.
- 04
Grounded analysis
Each report section is written against cited, approved claims only, using structured outputs.
- 05
Independent cross-check
A second frontier model from a different family reviews every section and auto-regenerates it on disagreement, up to a bounded number of cycles.
- 06
Deterministic assembly
Verified sections compile into a finished report. No improvisation at the finish line.
The part that signals seniority — the rules that hold even when they cost the system a finding:
- Failed data collection is never a finding. A blocked scrape is reported as “could not verify” — never spun as a target’s weakness.
- Estimates are labeled as estimates. Used only as relative comparisons, never stated as measured fact.
- Every claim traces to an observation. No orphan assertions.
- Human-in-the-loop is structural. The gate physically prevents unverified prose — not a checkbox.
- Multi-model orchestration. Two independent model families cross-check, because a single model grading its own work is indefensible.
A pipeline that produces genuinely trustworthy competitive intelligence — the reliability coming from system design, not from hoping the model behaves. The transferable lesson, and the reason it belongs here: making AI dependable is a systems problem, and the same discipline that makes a report trustworthy is what makes revenue infrastructure trustworthy.