Swapping the "harness" around a coding agent changes results no more than a rerun does, but Claude Code uses 2–3 times more tokens, a study on open models finds
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
In this test, a complex wrapper around the model made the work more expensive but did not raise the share of tasks solved.
In the paper "What does a harness buy? Mostly tokens," Liu and Han of Shandong University compared three ready-made software wrappers around a model ("harnesses") — Claude Code, OpenCode and mini-SWE-agent — on five models (Qwen3.6, Qwen3.8, DeepSeek-V4-Flash, GLM-5.3-Flash, HY4-Preview), offline: 45 hard SWE-bench Verified tasks plus 447 more. A task's outcome changed in 13% of cases both when the harness was swapped and on a simple rerun; the difference in solved shares is within noise (±5 percentage points). Claude Code's system prompt takes about 16,600 tokens versus under 1,000 for mini-SWE-agent. The result was obtained on open models in offline conditions; the account comes from The Decoder.
Sources 1