News

Announcement

Purpigal’s data agent now scores 64.6 on KramaBench

Up from 57.7 in our first result, and above the result the KramaBench paper itself reports for the same setting.

On 1 November we shared our first benchmark result: 57.7 on KramaBench. Today Purpigal’s data agent scores 64.6 across all 104 tasks, on the same benchmark, setting and model. That is above the 55.8 the KramaBench paper itself reports for this setting,1 and above smolagents Deep Research’s 53.2 on the same model family.2

Fig. 1
KramaBench, Full Input, all 104 tasks · Higher is better
SystemScore
Purpigal data agent
64.6
Purpigal data agentfirst result
57.7
KramaBench paper1smolagents Deep Research, Claude 3.7
55.8
smolagents Deep Research2same model family
53.2

The setting

As before, we ran KramaBench in Full Input mode, its hardest setting. For each question the agent receives a domain’s entire data folder, which in astronomy is more than 1,500 files. It has to decide which files matter, read formats ranging from multi-sheet workbooks to satellite orbit data, and compute the answer itself. Nothing tells it where the answer lives.

Results

KramaBench, Full Input: scores on all 104 tasks and on the 78 tasks not used in development
SystemAll 104 tasksThe 78 tasks not used in development
Purpigal data agent64.6 ± 0.963.5 ± 1.7
Purpigal data agent, first result57.7 ± 1.954.3 ± 2.0
KramaBench paper (Claude 3.7)155.8 ± 3.4Not available
smolagents Deep Research (same model family)253.2Not available

Scores are the mean of three independent runs, with the standard deviation across those runs.3 The agent still runs on DeepSeek V4-Flash, a low-cost model; with nine attempts per question it costs under 6 cents per task.

What changed

Three things changed since our first result, all in how the agent reads and answers:

  • Nine attempts per question, not three. The agent answers each question nine times independently, and the same fixed, deterministic rule chooses one answer.
  • Numbers as they are written. Numbers stored as text, with thousands separators or a dollar sign, are read as numbers, and column names lose stray spaces.
  • Readers for messy real-world files. Notes and stray lines in a file no longer get in the way of the figures it holds.

Every answer is auditable

As before, each answer comes with an operation trace: the exact steps the agent took, from the files it opened to the computation that produced the number. 95% of those traces re-run to the same answer without calling a model, and none re-runs to a different one.

What’s next

This is our second KramaBench result. We’ll share the next one when it lands.

Notes

  1. 1
    The KramaBench paper’s own Full Input result: smolagents Deep Research on Claude 3.7, the mean of three runs, as published in arXiv 2506.06541 (Table 5). It was measured on a different model from ours, so the comparison is approximate. ↩
  2. 2
    The smolagents Deep Research figure is as published in arXiv 2609.32528, self-reported, on a preview of DeepSeek V4-Flash. Our scores use KramaBench’s official scorer, with an LLM judge (GPT-5 mini) for the tasks that need one. Comparisons with published figures are approximate. ↩
  3. 3
    Seeds 17, 19 and 23; nine attempts per task at temperature 0, one selected by a fixed rule; KramaBench’s official aggregation. ↩

See it against your own numbers.

Bring the history you already keep. We will show you what it finds in it before you commit to anything.

Book a call

We usually reply within a working day.