Measured,not claimed.
Every number here comes from real runs of the CLI against real git repositories: 86 labeled scenarios, 800 randomized trials checked against an independent oracle, concurrency and stability tests, token counts, and secret-scanner tests on 1,737,973 lines of real code.
resume --check; the full resume prompt is 469Does the gate say theright thing?
86 scenarios, each applying one change after a checkpoint and comparing the verdict with the expected one. I wrote the expectations before running, so this shows the tool behaves as designed, and where it cannot know better.
Scenarios by family
Misses (1)
| Scenario | Expected | Got |
|---|---|---|
| remote_ahead_unfetched teammate pushed to origin; local clone has not fetched | REVIEW_REQUIRED | READY |
An unfetched remote is invisible by design (no network). Everything else, including rewrites, branch moves, merges, nested repos, privacy exclusions and corrupted state, behaves as labeled.
800 randomhistories, one oracle.
Each trial applies 1 to 5 random operations (edit, add, delete, commit, branch, amend, reset, stash, rename, ignored files…). An independent oracle, built from plain git plumbing and content hashes, decides what the verdict should be. Seeds 20261012 and 777, so it is reproducible. The second seed never edits the tool's own notes (see below).
Oracle vs tool
Result
1 disagreement; 0 unexplained. Mean 3.0 operations per trial.
seed 20261012: 399 / 400 agree.
seed 777: 400 / 400 agree (notes excluded from edit targets).
seed 20261012, trial 169: ops commit → env → edit → delete; oracle REVIEW_REQUIRED, tool BLOCKED. contextbridge/graph.json is not valid JSON (Extra data: line 47 column 1 (char 835)); fix or delete it The random 'edit' step appended text to a notes file under contextbridge/ that an earlier `git add -A` had committed. The tool correctly returned BLOCKED (corrupt notes); the oracle only checks that state.json exists. Harness artifact, not a tool bug. Later seeds exclude the notes from edit targets.
What it costs anagent to read.
Counted with real tokenizers (cl100k and o200k via tiktoken) over the actual output of the CLI. Claude's own tokenizer is not available offline, so treat these as close proxies, not exact Claude counts.
Tokens a fresh agent must read
Grey is a typical orientation dump: git status + git log -20 --stat + a diff stat. It is not equivalent information (it has no evidence, decisions or verdict); it shows the size of the cheapest alternative.
Handoff size stays flat as projects grow
Across all 46 scenarios (cl100k tokens)
| Output | Median | p90 | Max | Min |
|---|---|---|---|---|
| resume --check | 65 | 114 | 262 | 30 |
| resume (gate + handoff) | 468.5 | 537 | 584 | 51 |
The maximum is the scenario with 1,000 new files. The gate collapses repeated findings into a count, so even that output stays small.
Fast enough to runbefore every turn.
resume --check latency (median of 5)
Each call is a fresh Python process; about 40 ms of that is interpreter startup. Cost scales with uncommitted files (hashing), not repo size.
Other timings
- save
- 265 ms at 10,000 files, 500 uncommitted
- run overhead
- 192 ms on top of the command
- export / import
- 84 ms / 98 ms with 500 work items (88,474 bytes)
- scanner
- about 3.9 MB/s
- CLI startup
- 37 ms installed from the wheel
What it catches,and what it misses.
1,000 synthetic fake secrets per format (31 formats with dedicated patterns), then 5 formats with none, then how often it fires on real code.
Detected and redacted: 31 formats
Redaction removed the secret and left nothing re-detectable in 100% of detections.
Deliberately not detected: 5 formats
Random hex and base64 strings, UUID-style keys and short passwords are not flagged; doing so would flag hashes and IDs. The scanner is best-effort, and the README says so.
Hits on real code
Lines flagged per million lines scanned. What remains is mostly test fixtures with literal passwords, such as passwd = 'my1pa$$word!', and very long minified lines. Lockfiles produced 0 hits.
Stable, portable,and lossless.
Test-suite stability
36 tests per run, median 12.0 s. No flaky failures.
Python versions (macOS)
| Python 3.11.13 | 3/3 runs passed, 36 tests |
| Python 3.14.6 | 3/3 runs passed, 36 tests |
Packaging
- wheel
- 25,002 bytes, tests excluded: true
- sdist
- 35,062 bytes
- install
- py3.11.13 ok (1.0s, 0.3.3) · py3.14.6 ok (0.9s, 0.3.3)
- dependencies
- 0
Concurrent writers
Several processes run run in the same repository at once. Writers take an exclusive lock around reading and writing the notes, so every executed run is recorded with a unique id and the files stay valid JSON.
Read this beforeyou trust it.
What this does not prove
- It does not measure whether an AI agent finishes tasks better with ContextBridge. That needs many paid agent runs against a baseline.
- I wrote the scenario expectations, so they reflect my idea of correct. The fuzz oracle is independent code, but also mine.
- Token counts use OpenAI-style tokenizers as a proxy for Claude's.
- One machine (Apple M3, macOS). Linux and Windows are covered by CI for functionality, not benchmarked here.
- No other tool was installed or compared.
Reproduce
git clone https://github.com/Abhinay2206/ContextBridge cd ContextBridge-Benchmark python3 bench.py # speed, labeled scenarios, scanner python3 bench2.py # fuzz, tokens, concurrency, matrix python3 merge.py && python3 charts.py
The harness lives in the ContextBridge-Benchmark folder next to the repo; it is not part of the shipped package.