Both studies compare CherryOnTop with Claude Code 2.1.296 using the same model (Claude Haiku 5.5), API-billed, on identical tasks with hidden-test verification. They were run by the CherryOnTop team. They have not been independently replicated.
32 task slots, 2 repeats, 63 of 64 pairs analysed (126 runs), medium effort, $1 cap per run. Pair 45 is excluded because its verifier timed out in all three attempts. Intervals are paired bootstrap over tasks (5,000 resamples).
| Metric | Claude Code | CherryOnTop | 95% CI of difference |
|---|---|---|---|
| Verified success | 44/63 (69.8%) | 45/63 (71.4%) | −4.8 to +8.2 pp |
| Cost per attempted task | $0.0102 | $0.0094 | −$0.0032 to +$0.0019 (includes zero) |
| Total billed tokens per task | 383,522 | 274,477 | −199k to −26k |
| Median wall-clock | 806 s | 505 s | −438 to −143 s |
| Median active agent time | 67 s | 100 s | +4 to +64 s (CherryOnTop slower) |
Success is equivalent within the predeclared −10 pp margin, not better. 15 of 63 pairs fail on both systems (concurrency, performance, API compatibility, Java, Go). Wall-clock is lower partly because Claude Code installs itself per container until the images were pre-seeded identically for both arms. Total billed tokens count cache reads and writes.
12 tasks, 3 repeats, 35 valid pairs, high effort, $0.50 cap per run, 900 s agent timeout. The decision rule was fixed before the run.
| Metric | Claude Code | CherryOnTop | Result |
|---|---|---|---|
| Verified | 28/35 | 30/35 | +5.7 pp, CI 0.0 to +13.9 |
| Cost per attempt | $0.0122 | $0.0091 | ratio 0.770, CI 0.679 to 0.878 |
| Agent time | CherryOnTop 2.0× slower | mostly tool execution and waiting | |
One model, small samples, one team's harness. CherryOnTop's cost advantage concentrates in long, hard and large-repository tasks; on medium tasks it was equal or slightly more expensive. The current build runs on Linux x64 only. Contact: abhay.chourasiya.work1@gmail.com.