I added cost and token tracing to a small agentic RAG app I've been building to answer one narrow question: is a model-decided agent cheaper than always-on retrieval? I ran a controlled A/B, pulled the aggregate, and the agent came out ~17% cheaper per question. Nice. Ship the conclusion.
Then I looked at the per-question rows, and the 17% fell apart in my hands.
This is a write-up of what the aggregate hid, why the fix was persisting the answer next to the cost rather than just the cost, and the decision rule I took away from it. Upfront caveat so nobody misreads this: this is a personal practice project (ai-eng-prep — Python, OpenAI Responses API, Qdrant, Postgres), and the sample is eight questions. Everything here is directional, not a benchmark. n=8 can show you a mechanism; it cannot prove a rate. Read it as "here's a trap I walked into," not "here are numbers you can quote."
The setup
Two modes answer the same questions:
- RAG (always-on retrieval): embed the query → pull top-k chunks from Qdrant → stuff them into the prompt → generate. It always retrieves, every time.
- Agent (model-decided): an LLM with two tools —
search_docsandcalculate— that decides per query whether to retrieve, compute, or just answer.
The observability layer is the part I actually care about. A contextvar-based usage collector records tokens and cost for both the generation calls and the embedding calls, then persists a row to a Postgres traces table: mode, prompt, model_calls, tools_used, cost_usd, latency_ms, answer, trajectory. Note that last stretch — it stores the answer and the trajectory, not just the spend. That detail is the whole story.
The A/B: 8 matched questions, each run through both modes — 4 project-specific doc-questions (only answerable from my indexed notes), 2 arithmetic, 2 general-knowledge. I let RAG answer everything, not just the grounded ones, so the cost-to-answer comparison was fair rather than rigged toward its strength.
The headline
Grouped by mode:
| mode | runs | avg cost | avg latency | avg model calls |
|---|---|---|---|---|
| agent | 8 | $0.000891 | 7187 ms | 2.0 |
| rag | 8 | $0.001073 | 7654 ms | 2.0 |
Agent ~17% cheaper. And here's the first thing that should have made me suspicious rather than satisfied: the average model-call count is identical — 2.0 and 2.0. If the agent were winning by "deciding to do less work," you'd expect it to make fewer calls. It didn't. So whatever the 17% was, it wasn't about call count. It was about tokens. That's a clue that the averages are blending two different things together.
The autopsy
I sorted the rows by per-question cost gap. One question owned the result:
"What are the five fields of the AssistantResponse schema?"
- agent: 1 model call,
tools_used={}, $0.000812 — it did not retrieve. - rag: 2 model calls, $0.002002 — retrieved and grounded. ~2.5× the cost, ~2× the latency.
That single row's gap is $0.00119. The agent's entire cost advantage across all eight questions is $0.00146 in total. So one question accounts for ~82% of the agent's whole lead. Strip it out and the 17% headline mostly evaporates. My "conclusion" was one lucky row wearing a trench coat.
But "lucky" is doing work in that sentence, and it's worth being precise about why.
The twist: both answers were correct
Here's what makes this more than a sampling gripe. Both modes got the question right.
RAG:
The five fields are: answer, confidence, missing_context, next_actions, source_references.
Grounded in the actual project note.
Agent:
The five fields are: 1) answer (string) 2) confidence (enum: high/medium/low) 3) missing_context (array of strings) 4) next_actions (array of strings) 5) source_references (array of strings).
Also correct — and notice it's more detailed. It added types the source note never specified. That's not retrieval. That's the model reconstructing a plausible schema from its own training knowledge, and doing a convincing job.
The trace makes the distinction non-negotiable: the agent row shows 1 call, tools_used={}. It definitively did not touch the docs. RAG grounded its answer in the source. Two identical-looking correct answers, opposite trust properties. One is checked against the source of truth; the other is a confident guess that happened to land. On a project-specific fact, that gap is the entire ballgame — and it was invisible to the cost column.
That's the confound. The agent didn't look cheaper because it was smarter. It looked cheaper because on that one question it skipped grounding — and skipping grounding is cheaper by construction. The cost delta wasn't measuring efficiency. It was measuring the price of not verifying, and only surviving because the guess was right. Swap in a schema that changed after the model's training cutoff and you get the same low cost, the same confident tone, and a wrong answer that reads exactly like the right one.
The real lesson
Cost comparisons between a grounded system and an ungrounded one are confounded by grounding itself, not just by quality. When an agent skips retrieval, you're not comparing "cheaper agent vs. pricier RAG." You're comparing "answered from memory" vs. "answered from the source" — and pricing the difference as if it were efficiency.
A cost-only tracer would have printed "agent 17% cheaper" and shut up. It never would have surfaced that one of those cheap answers bypassed the docs, because cost cannot see grounding. What caught it was storing the answer and the tool trace right next to the number. The metric was quietly cheating; the trajectory column ratted it out.
So: instrument the output, not just the spend. Persist the answer, the tools used, the trajectory — the things that let you ask "cheaper doing what?" A number you can't interrogate is a number that can lie to you politely.
Two smaller findings that break the easy story
While I was in there, two more of my assumptions didn't survive contact with the rows:
- On arithmetic, RAG was actually cheaper. The agent's
calculatetool adds a round-trip — call to decide, call to answer — so it runs two generations where RAG just answers directly. "The agent saves work by skipping steps" is flatly false for anything tool-requiring; it adds a step. - There's a structural tool-schema overhead. The agent carries both tool definitions in-context on every call — that isn't a measurement, it's just how the API works — so even a direct one-call answer pays to haul them around. It surfaced on "capital of Australia," where the agent's 1-call answer ($0.000610) landed above RAG's 2-call answer ($0.000581). One tiny gap, well inside token-count noise, so take it as an illustration of the mechanism rather than proof of its size — but the overhead itself is real by construction.
And where the agent did retrieve on doc-questions, cost was roughly a wash — RAG's one big-context generation lands close to the agent's two smaller ones.
The takeaway (a rule, not a winner)
I'm not telling you agents are expensive or RAG wins. Eight questions can't crown anything. What it can do is hand me a decision rule:
- Don't compare agent-vs-RAG on cost aggregates. The average blended a grounded answer and an ungrounded one into one tidy, misleading number.
- Treat a cost edge on knowledge questions as a smell, not a win — it may be the system quietly skipping the source of truth, invisible when it guesses right and indistinguishable from a correct answer until the day it guesses wrong.
- Trace the output, not only the spend. The cheapest observability upgrade I made was storing the answer beside the cost. It's the only reason I caught my own metric lying.
The broader version of this is old news — grounded-but-current beats confident-but-ungrounded — but it's easy to nod at in the abstract and still get fooled by a spreadsheet. I got fooled by mine for about ten minutes. The trace store un-fooled me, and that's the part I'd actually build again.