Letronna®

Benchmark // Written in Rust

Fast is
a feature.

Letronna is one native binary. No runtime to boot, no interpreter warming up, no garbage collector between you and the answer. Here is what that buys, measured.

  • Cold start

    7.65ms

    Launch the terminal app to a ready prompt, 500 sessions on disk, in a pseudo-terminal

  • Memory at idle

    9.52MB

    Resident set size at that ready prompt

  • Parse a 10k-event stream

    13.2ms

    A long reply, a word at a time, from a local server

Speed // median of 7 runs

Every hot path.

What a person waits for, and what grows with the history: the core's bench and the terminal's, each held to a checked-in baseline, so a slower build fails the bench. No model is called; this is the harness alone.

  1. [ 01 ]Cold startterminal_start_ms7.65msLaunch the terminal app to a ready prompt, 500 sessions on disk, in a pseudo-terminal
  2. [ 02 ]Memory at idleterminal_rss_mb9.52MBResident set size at that ready prompt
  3. [ 03 ]List 500 sessionslisting_500_ms1.47msLoad the session list from its index
  4. [ 04 ]List 5,000 sessionslisting_5000_ms16.0msThe same, with some 90 MB of sessions on disk
  5. [ 05 ]Search 5,000 sessionssearch_5000_ms102msFull-text, a common word and a rare one, top 20 hits
  6. [ 06 ]Open a 10k-message sessionload_10k_ms11.8msRead a long day of tool calls in from disk
  7. [ 07 ]Fit a 200k contextwindow_start_10k_ms0.522msPlace a 200k-token window over those 10,000 messages
  8. [ 08 ]Parse a 10k-event streamstream_10k_events_ms13.2msA long reply, a word at a time, from a local server

Bars on a log scale, 0.1 ms to 1 s. Other agent harnesses will be timed on the same machine with the same data; their rows land here when they are measured, not before.

Harness-Bench // 106 agent tasks

Fast, and it
gets it right.

Speed means nothing if the work is wrong. On Harness-Bench, real tasks from files and git to email triage and code debugging, graded by the paper's own judge, Letronna's best run tops every configuration the paper reports, and beats the paper's average harness on the same model by 15 to 27 points.

Same model · DeepSeek V4 Flash

Paper's average harness57.5
Letronna84.0

+26.5 points

Same model · Claude Sonnet 4.6

Paper's average harness65.3
Letronna80.4

+15.1 points

Ranked with the paper's best configurations

  1. [ 01 ]LetronnaDeepSeek V4 Flash84.0this run
  2. [ 02 ]LetronnaMuse Spark 1.383.6this run
  3. [ 03 ]LetronnaClaude Opus 5.583.4this run
  4. [ 04 ]LetronnaClaude Sonnet 5.583.4this run
  5. [ 05 ]NanoBotGPT-5.481.3paper
  6. [ 06 ]LetronnaClaude Sonnet 4.680.4this run
  7. [ 07 ]CodexGPT-5.480.4paper
  8. [ 08 ]MoltisGPT-5.474.4paper
  9. [ 09 ]HermesGPT-5.474.0paper
  10. [ 10 ]OpenClawGPT-5.469.3paper
  11. [ 11 ]NullClawGPT-5.468.9paper
  12. [ 12 ]ZeroClawGPT-5.454.3paper

Work per task

HarnessScoreTokensthousand / taskModel callsper task
LetronnaDeepSeek V4 Flash84.0112.79.0
LetronnaMuse Spark 1.383.670.18.7
LetronnaClaude Opus 5.583.450.17.1
LetronnaClaude Sonnet 5.583.447.36.4
LetronnaClaude Sonnet 4.680.486.211.3
Codexpaper · mean of 8 models80.486.15.0
NanoBotpaper · mean of 8 models76.268.77.3
Hermespaper · mean of 8 models71.2139.722.6
Moltispaper · mean of 8 models68.8134.98.0
NullClawpaper · mean of 8 models64.4175.112.1
ZeroClawpaper · mean of 8 models61.4133.28.6
OpenClawpaper · mean of 8 models52.482.15.0

Scores 0 to 100, combining the task's outcome check, a security gate and the judge's rubric, as in the paper. Letronna: one run per task, over the tasks the judge graded (106 of 106 on Sonnet 4.6, 105 on DeepSeek V4 Flash). Paper figures are read off its Figure 3 and Table 2. Models differ between rows; the same-model comparison is above.

Task explorer // all 106 tasks

Every task, every note.

Main against base, the build before the harness work. Select a row for the judge's note on both. The raw data is the file this table reads.

Loading the raw data…

SWE-bench Verified // 10 tasks

Tokens it takes
to solve.

The ten tasks OpenHuman's harness benchmark published, graded by the official SWE-bench evaluator, every harness on DeepSeek V4.1 Flash. Lower is better. Letronna solves all ten with the fewest tokens of any harness that does.

Tokens per solved task · thousand

  1. [ 01 ]Codexpublished370k7/10 solved · 1.74¢
  2. [ 02 ]Letronnaround one376k10/10 solved · 0.81¢
  3. [ 03 ]OpenCodepublished390k10/10 solved · 0.77¢
  4. [ 04 ]Claude Codepublished428k10/10 solved · 3.51¢
  5. [ 05 ]OpenClawpublished442k9/10 solved · 1.07¢
  6. [ 06 ]Hermespublished482k10/10 solved · 0.82¢
  7. [ 07 ]DeepSeek harnesspublished744k10/10 solved · 1.29¢
  8. [ 08 ]Letronnabaseline801k9/10 solved · 2.07¢

Letronna across models

ModelSolvedof 10Tokensthousand / solvedModel callsper taskCostcents / solved
DeepSeek V4.1 Flash1074024.01.27
Claude Haiku 5.59645.81.72
Claude Sonnet 5.59726.65.15
Kimi K2.7 Code946725.212.59
GLM-5.3 Flash8869.90.89
GPT-6 Luna714710.11.24

One run per task; a single task can take several times as many calls from one run to the next, and a later build's run came to 740k tokens per solved task (10/10, 1.27¢). Tokens are prompt plus output over all ten tasks, divided by the tasks solved. Cost is at DeepSeek's own prices for the DeepSeek rows, the published harnesses' web search included, and at each model's list price in the model table. The published harnesses ran with internet access; Letronna's runs were offline. Tokenizers differ between model families, so tokens compare the same model best. Every task: swe-bench.json.

Terminal-Bench 2.0 // 89 tasks

Tasks it finishes
in a terminal.

All 89 tasks of Terminal-Bench 2.0 through Harbor, with Letronna as the agent on each model, one trial per task. Scored out of all 89; higher is better.

Tasks passed · Letronna on each model

  1. [ 01 ]LetronnaClaude Sonnet 5.579.8%71/89 passed
  2. [ 02 ]LetronnaGPT-6.1 Sol78.7%70/89 passed
  3. [ 03 ]LetronnaClaude Opus 5.576.4%68/89 passed
  4. [ 04 ]LetronnaGPT-6 Sol70.8%63/89 passed
  5. [ 05 ]LetronnaDeepSeek V4.1 Flash69.7%62/89 passed
  6. [ 06 ]LetronnaClaude Haiku 5.565.2%58/89 passed
  7. [ 07 ]LetronnaMuse Spark 1.353.9%48/89 passed

Run 20 October 2026 with Harbor 0.24. The latest trial per task counts; a run that looked up a task's answers online counts as a fail, and trials lost to infrastructure (docker, provider outages) were run again. Claude models ran at high reasoning. Claude Opus 5.5 lost 9 security and biology tasks to its provider's safety filter, Sonnet 5.5 3 and Haiku 5.5 1; they count as fails. Grok 4.7 is left out until its run finishes.

Method

How it was run.

Machine
Mac mini (Mac16,10), Apple M4 (4P + 6E cores), 16 GB, macOS 26.2
Build
Release build, cargo test --release -- --ignored, nothing else running. Sessions are written for real, some 90 MB of them.
Data
Synthetic sessions with the shape of real ones: five turns of question, tool call, result and answer, plus one 10,000-message day of tool work. The stream comes from a local server, so no network is timed.
SWE-bench
SWE-bench Verified, the 10 tasks of OpenHuman's harness benchmark (tinyhumansai/openhuman-benchmarks), in its 4-CPU, 8 GB task container behind its metering proxy, graded by the official evaluator. Letronna runs headless, letronna -p. Published rows are the rig's run of 2 October 2026.
Harness-Bench
Qihoo360/harness-bench at commit 1025086, all 106 tasks, judged by Claude Sonnet 4.6 as in the paper. Rerun 13 October 2026 on Letronna main (847fa57c). Paper: arXiv 2605.27922.