Measuring Sleev

It's easy to assume big agent bills are just the cost of doing serious work. But most of what you're paying for isn't new thinking. It's old terminal logs and stale error messages being re-sent over and over. Sleev cleans up that clutter automatically as you work, letting you run longer sessions for a fraction of the cost. Below, we break down our head-to-head benchmarks to show exactly how much you save, and why your agent's coding accuracy stays just as sharp.

DeepSWE

We measure Sleev on DeepSWE 1.1, a benchmark of 113 real-repository coding tasks scored by the benchmark's own tests. We run the baseline DeepSWE and compare it to runs using Sleev as well as other popular context and cost optimization tools.

We first ran the standard DeepSWE 1.1 as published: every task gets a fresh context window, so sessions stay short. However, real agentic use may often run long, and context may build up over hours of work. To measure that, we also ran a modified version that batches six tasks into one continuous context window. While solve rate holds in both and Sleev imposes no penalty on short sessions, we show that its savings grow quadratically with session length.

Standard runs [GPT-5.5 high] one task per session
Batched runs [GPT-5.6 medium] six tasks per session
Baseline Sleev Baseline Sleev
Solve rate pass@1 57.5% ± 5.0 59.1% ± 2.2 61.1% ± 2.3 61.9% ± 1.9
Cost per task mean $3.71 $3.50 $18.08 $3.99
Total cost mean per full run, 113 tasks $419.10 $395.09 $2,042.63 $451.06
Input tokens mean per task 4,380,061 3,253,863 18,566,080 3,491,998
Peak context size median per task · mean per run 102,315 82,225 674,822 141,739
Agent steps mean per task 52.5 57.0 67.7 60.2
Cache hit rate 96.0% 92.5% 99.2% 93.9%
Output tokens mean per task 24,271 25,501 24,911 26,672

Standard runs: one task per context window running GPT-5.5 at high reasoning. Batched runs: six tasks chained through one continuous context window running GPT-5.6 Sol at medium reasoning. Both runs were run 4 times each. The standard runs were completed before the GPT-5.6 family was released.

Cumulative session cost and context growth

Every request an agent makes re-sends the whole conversation. In a short session that is cheap. But every turn makes the conversation longer, so every request gets more expensive than the last, and the session's total cost grows quadratically with its length. Our batched runs show this directly: with six tasks in one context window, the baseline's bill curves upward turn after turn. With Sleev the same session grows linearly.

The chart also compares two other context tools run through the same batched benchmark: Headroom, a compression layer that advertises 15 to 20% fewer tokens for coding agents, and Context Mode, which sandboxes tool output and advertises up to 98% smaller tool results. Over an average session, both ended above the baseline.

Sleev and Headroom both matched the baseline's solve rate. Context Mode did not, solving 54% of tasks against the baseline's 61%.

Cumulative session cost

GPT-5.6 Sol

$50 $100 $150 0 100 200 300 398 Turn number $166.25 $120.22 $107.29 $24.15
Baseline Sleev Headroom Context Mode
Averaged across full batched DeepSWE 1.1 benchmark runs: 4 runs (452 total tasks) each for Sleev and the baseline, and 1 run (113 tasks) each for Headroom and Context Mode, chained 6 tasks per session.

In multi-step agent workflows, context bloat causes more than just quadratic billing: it slows down generation, inflates time-to-first-token, and causes retrieval degradation as hundreds of stale tool outputs dilute the model's attention.

In the batched baseline DeepSWE runs, input context without management climbs continuously throughout the session, reaching an average peak of 495k tokens by turn 398. With Sleev continuously compressing stale execution logs into compact summaries, context plateaus early and peaks at 111k tokens, a 77.6% reduction.

By contrast, Headroom and Context Mode do not limit context growth in the benchmark: Headroom climbs steadily to a 509k peak, while Context Mode reaches 601k tokens as uncompacted tool state accumulates between compaction boundaries.

Note that the newest GPT models maintain unusually stable retrieval and reasoning at large context sizes compared to most architectures. While we do not yet have benchmark evidence showing solve-rate gains from context reduction on other model families, performance degradation at high context is well documented[1–4].

Input context per turn

GPT-5.6 Sol

100k 200k 300k 400k 500k 600k 0 100 200 300 398 Turn number 601k 509k 495k 111k
Baseline Sleev Headroom Context Mode

What about cache?

Prompt caching offers substantial discounts on repeated prefix tokens, creating the intuition that retaining bloated context in the prompt is cheap. In long multi-step agent sessions, however, cache reads compound quickly. In baseline benchmarks, cache read fees account for 88% of total inference spend ($95 out of $107 per session) as the window expands.

Sleev compressions break the prompt cache, requiring a one-time rewrite (an uncached write fee on that turn). But they shrink the prefix permanently, lowering the cost floor of every subsequent request.

The upfront rewrite cost breaks even within 2 to 3 turns in this benchmark, though this depends on model pricing and performance. After breakeven, every subsequent request generates compounding net savings. Additionally, Sleev keeps per-request pricing predictable, avoiding the 7x higher cost volatility seen in the baseline as unmanaged context expands.

Cost per request

Single-turn cache rewrite fee vs. ongoing per-turn savings

$0.60 $0.40 $0.20 $0.00 0 100 200 300 398 Requests in session Baseline Sleev
The baseline cost jump at request 174 is attributed to OpenAI API rate increases when prompt context exceeds 272k tokens.

Methodology

All benchmarks are measured directly against real-repository coding tasks using identical seeds, pinned harness versions, and sandboxed execution environments. Our open source benchmark repository with the full evaluation harness, raw session rollouts, and replication tooling will be released soon.

References

  1. Chroma Research (Context Rot) : Empirical report evaluating retrieval degradation across frontier models as context windows fill.
  2. Stanford "Lost in the Middle" (arXiv:2307.03172) : Academic foundation establishing position-based performance degradation in long contexts.
  3. NVIDIA "RULER" (arXiv:2404.06654) : Benchmark evaluation demonstrating that effective context windows fall far short of advertised maximums.
  4. GAIR "Context Rot in Long-Horizon Search" (arXiv:2606.29718) : Multi-turn agentic study evaluating degradation over extended horizons.

Footnotes

  1. Session curves are traced from 171 GPT-5.6 Codex sessions (76 unmanaged, 57 with Sleev, 19 with Headroom, 19 with Context Mode) and show the average session on each side. The DeepSWE aggregate is a separate experiment run on gpt-5.5.
  2. DeepSWE figures are not comparable to official leaderboard scores: Codex CLI replaces DeepSWE's reference harness. Runs from June 21 to July 2, 2026.
  3. 95% confidence intervals follow DeepSWE convention: 1.96 times the standard deviation of run-level pass@1 over the square root of runs.
  4. Batched-run costs are repriced per request at OpenAI's list (non-promotional) GPT-5.6 Sol rates: $5 / $0.50 / $30 per million input, cached input and output tokens, doubling input and adding 50% to output for any request above 272K input tokens. Sleev's costs include the gateway's own compression calls.