Announcement
Purpigal’s data agent now scores 64.6 on KramaBench
Up from 57.7 in our first result, and above the result the KramaBench paper itself reports for the same setting.
On 1 November we shared our first benchmark result: 57.7 on KramaBench. Today Purpigal’s data agent scores 64.6 across all 104 tasks, on the same benchmark, setting and model. That is above the 55.8 the KramaBench paper itself reports for this setting,1 and above smolagents Deep Research’s 53.2 on the same model family.2
The setting
As before, we ran KramaBench in Full Input mode, its hardest setting. For each question the agent receives a domain’s entire data folder, which in astronomy is more than 1,500 files. It has to decide which files matter, read formats ranging from multi-sheet workbooks to satellite orbit data, and compute the answer itself. Nothing tells it where the answer lives.
Results
| System | All 104 tasks | The 78 tasks not used in development |
|---|---|---|
| Purpigal data agent | 64.6 ± 0.9 | 63.5 ± 1.7 |
| Purpigal data agent, first result | 57.7 ± 1.9 | 54.3 ± 2.0 |
| KramaBench paper (Claude 3.7)1 | 55.8 ± 3.4 | Not available |
| smolagents Deep Research (same model family)2 | 53.2 | Not available |
Scores are the mean of three independent runs, with the standard deviation across those runs.3 The agent still runs on DeepSeek V4-Flash, a low-cost model; with nine attempts per question it costs under 6 cents per task.
What changed
Three things changed since our first result, all in how the agent reads and answers:
- Nine attempts per question, not three. The agent answers each question nine times independently, and the same fixed, deterministic rule chooses one answer.
- Numbers as they are written. Numbers stored as text, with thousands separators or a dollar sign, are read as numbers, and column names lose stray spaces.
- Readers for messy real-world files. Notes and stray lines in a file no longer get in the way of the figures it holds.
Every answer is auditable
As before, each answer comes with an operation trace: the exact steps the agent took, from the files it opened to the computation that produced the number. 95% of those traces re-run to the same answer without calling a model, and none re-runs to a different one.
What’s next
This is our second KramaBench result. We’ll share the next one when it lands.
Notes
- 1The KramaBench paper’s own Full Input result: smolagents Deep Research on Claude 3.7, the mean of three runs, as published in arXiv 2506.06541 (Table 5). It was measured on a different model from ours, so the comparison is approximate. ↩
- 2The smolagents Deep Research figure is as published in arXiv 2609.32528, self-reported, on a preview of DeepSeek V4-Flash. Our scores use KramaBench’s official scorer, with an LLM judge (GPT-5 mini) for the tasks that need one. Comparisons with published figures are approximate. ↩
- 3Seeds 17, 19 and 23; nine attempts per task at temperature 0, one selected by a fixed rule; KramaBench’s official aggregation. ↩