- +18 pts
- fully correct answers vs. standard RAG, same 3,000-token budget
- 4× less
- context than RAG, and still 16 points ahead (75% vs. 59%)
- 90%
- evidence recall for an agent using OpenScaffold, vs. 67% with connectors
Ask an AI agent to do real work for a customer and it needs to know things nobody wrote down in one place: what the customer asked for, what they ruled out three months later, which ticket the promise was made in, and whether the thing marked done was ever confirmed. That history is spread across Slack threads, Linear issues and support tickets, and it changes over time.
The usual answer is retrieval: index everything, pull back the chunks that look most like the question, and hand them to the model. It works for lookups. It struggles when the answer depends on how a requirement evolved, or on two messages in two different tools that only make sense together.
OpenScaffold takes a different approach. It keeps a structured, evidence-backed model of each customer, built from their Slack, Linear and support history, where every fact points back to the exact words someone said, who said it and when, and later messages that replace or correct earlier ones are tracked as such. When an agent asks for context, it gets the facts that matter for the task, with their quotes, instead of the most similar-looking text.
We wanted to know whether that actually helps, so we measured it.
The test
We took a 14-month customer history (about 1,700 records across Slack, Linear and support tickets) and wrote 28 tasks of the kind a team doing work for that customer would face. 22 of the 28 could only be answered by combining information from more than one source. Each task has a written answer key listing the critical points a good answer must cover and the evidence it should rest on.
We scored three things:
- Fully correct: the share of tasks answered fully correctly against the key.
- Critical points covered: how many of the key's critical points the answers covered.
- Evidence recall: how much of the evidence the key cites was found.
We ran two comparisons: what each retrieval method puts in front of the model, and what an agent actually does with it. Every approach used Claude Opus 5.5.
Retrieval: same budget, better context
First, OpenScaffold against standard RAG over the same records, each given the same context budget of 3,000 tokens, then four times that.
| Retrieval method | Fully correct | Critical points covered | Evidence recall | Context tokens |
|---|---|---|---|---|
| Standard RAG | 57% | 70% | 71% | 3,000 |
| OpenScaffold | 75% | 82% | 85% | 3,000 |
| Standard RAG (4× context) | 59% | 79% | 81% | 12,000 |
| OpenScaffold (4× context) | 75% | 89% | 94% | 12,000 |
At 3,000 tokens, OpenScaffold got 75% of tasks fully correct against 57% for RAG, an 18-point gap, and recalled 14 points more of the evidence.
The more telling row is the third one. Giving RAG four times as much context, 12,000 tokens, barely moved it: 59% fully correct, still well below OpenScaffold at a quarter of the size. More context brings in more of the right evidence (RAG's recall rises to 81%), but also more of everything else, and the model still misses how the pieces fit. The bottleneck isn't how much the model sees. It's whether what it sees is the right thing, in the right state.
With the larger budget, OpenScaffold's evidence recall reached 94% and critical-point coverage 89%, while the share of fully correct answers held at 75%.
Agents: the same tasks, end to end
Retrieval scores only tell part of the story; what matters is what an agent does with the context. So we gave Claude Opus 5.5 the same 28 tasks two ways: once with connectors to the customer's sources so it could search them itself, and once with OpenScaffold.
| Metric | Agent + source connectors | Agent + OpenScaffold |
|---|---|---|
| Fully correct tasks | 68% | 82% |
| Critical points covered | 82% | 85% |
| Evidence recall | 67% | 90% |
| Context tokens | ~5,100 | ~5,000 |
With OpenScaffold, the agent got 82% of tasks fully correct against 68% with connectors, and its evidence recall rose from 67% to 90%, using about the same amount of context.
Coverage of critical points was close (85% against 82%). An agent with connectors often finds most of the pieces; where it falls short is putting together every piece a task needs, and grounding its answer in the evidence that supports it. That is the gap between an answer that is mostly right and one a team can act on.
What this means
The results point the same way in both comparisons: for customer work, the quality of the context matters more than its size. A model of the customer that knows which facts are current, where each one came from and how they connect across tools gives an agent more correct answers from the same token budget than searching raw text does, even when the raw search gets four times the room.
Methodology and limits
- Data: one 14-month customer history across Slack, Linear and support tickets, about 1,700 records.
- Tasks: 28 in total; 22 required information from more than one source.
- Model: every approach used Claude Opus 5.5.
- Grading: answers were model-graded against written answer keys; each approach was evaluated once per task.
This is one customer history and 28 tasks, graded by a model, with one run per approach. Treat the numbers as a clear signal rather than a final word.