Announcement
Purpigal’s data agent outperforms smolagents Deep Research on KramaBench
Our first published benchmark result: 57.7 on KramaBench’s hardest setting, with a replayable trace behind every answer.
Today we’re sharing our first benchmark result. KramaBench asks an AI system to answer analytical questions over real data lakes in archaeology, astronomy, biomedicine, environmental science, law and wildfire. On it, Purpigal’s data agent scores 57.7 across all 104 tasks, ahead of smolagents Deep Research’s 53.2 on the same model family.1
| System | Score |
|---|---|
| Purpigal data agent | 57.7 |
| smolagents Deep Research1same model family | 53.2 |
| Purpigal data agentprevious version | 37.5 |
0100 |
The setting
We ran KramaBench in Full Input mode, its hardest setting. For each question the agent receives a domain’s entire data folder, which in astronomy is more than 1,500 files. It has to decide which files matter, read formats ranging from multi-sheet workbooks to satellite orbit data, and compute the answer itself. Nothing tells it where the answer lives.
Results
| System | All 104 tasks | The 78 tasks not used in development |
|---|---|---|
| Purpigal data agent | 57.7 ± 1.9 | 54.3 ± 2.0 |
| smolagents Deep Research (same model family)1 | 53.2 | Not available |
Scores are the mean of three independent runs, with the standard deviation across those runs.2 The agent runs on DeepSeek V4-Flash, a low-cost model, and costs under 2 cents per task.
What changed
Our previous version scored 37.5 on the same benchmark and model. Most of the improvement came from the agent loop itself:
- A typed answer. The agent commits to a value of the type the question asks for.
- A working memory. What the agent has already found is carried from one step to the next.
- Readers for real file formats. The agent can read the formats these data lakes actually contain.
- Three attempts per question. The agent answers each question three times independently, and a fixed, deterministic rule chooses one answer.
Every answer is auditable
Each answer comes with an operation trace: the exact steps the agent took, from the files it opened to the computation that produced the number. 96% of those traces re-run to the same answer without calling a model, so an answer can be checked rather than taken on trust.
What’s next
This is the first of several benchmark results we’ll be sharing. More are coming soon.
Notes
- 1The smolagents Deep Research figure is as published in arXiv 2609.32528, self-reported, on a preview of DeepSeek V4-Flash. Our scores use KramaBench’s official scorer, with an LLM judge (GPT-5 mini) for the tasks that need one. Comparisons with published figures are approximate. ↩
- 2Seeds 7, 11 and 13; three attempts per task at temperature 0, one selected by a fixed rule; KramaBench’s official aggregation. ↩