How Zapier Cut Spec Hallucinations to Under 1% With AI
26% to <1%
Not disclosed
Not disclosed
Zapier's LLM-powered spec-generation system automatically produced OpenAPI specifications from API documentation, but its performance was opaque. The team could not measure accuracy, quantify the impact of pipeline changes, or identify where to prioritize improvements.
Fractional AI built a comprehensive LLM evaluation framework with seven distinct metrics measuring correctness and hallucination rates across endpoint paths and field properties, paired with iterative experimentation to optimize the four-step pipeline.
The pipeline scrapes API docs to markdown, lists endpoints, extracts relevant documentation, and generates OpenAPI specs. Evaluations ran on Braintrust against curated, manually reviewed test data. Winning experiments included targeted prompt engineering, passing extra context (name, HTTP verb, URL) between stages, model selection (GPT-4o default, Claude Sonnet 3.5 for extraction), and prompt caching for cost savings.
Best fit for product teams shipping LLM features who need rigorous, metric-driven evaluation to reduce hallucinations before scaling.
Endpoint hallucinations 26% to <1%
Field type accuracy ~2x
~25% cost reduction per run
Chris Taylor
By building a seven-metric LLM evaluation framework that measured correctness and hallucination rates across the spec-generation pipeline, then running iterative experiments. Targeted prompt engineering, passing extra context between stages, and model selection drove hallucinated endpoint paths from 26% down to under 1%.
The pipeline used GPT-4o by default and Claude Sonnet 3.5 for extraction, with evaluations run on the Braintrust platform against curated, manually reviewed OpenAPI specs and Postman collections. Prompt caching was added to reduce cost.
Hallucinated endpoint paths fell from 26% to under 1%, field-type accuracy improved nearly 2x on the FieldTyper scorer metric, and prompt caching cut cost by roughly 25% (about $6) per run.
A specific timeline was not disclosed. The work was iterative - building the metric suite, then running successive experiments on Braintrust and keeping only the changes that measurably improved accuracy.
It is best suited to product teams shipping LLM features who need rigorous, metric-driven evaluation to reduce hallucinations before scaling.
Have a similar challenge?
