Eval

A context strategy is a claim about accuracy and cost, so it should come with numbers. ContextEval runs a labeled question set through several configurations.

Level 1 makes no model calls. For each configuration and budget it reports recall of the evidence, precision per token, how often distractors and superseded policies get in, tokens per call, and how much of the prompt prefix consecutive calls share.
Level 2 also answers every question with a chat model and has a judge grade the answer against a reference.

Level 1 results

Run on 2026-10-15 with all-MiniLM-L6-v2 embeddings on the Nimbus corpus that ships with astromesh-rag (commit a86bfba): 60 cases at budgets of 750, 1000 and 2000 tokens.

uv run --extra eval astromesh-context eval --level 1 --embedder st --out results/

config	budget	recall	precision/token	distractor rate	superseded rate	noise share	tokens/call	candidate tokens	prefix reuse	stable tokens
everything	750	0.550	0.072	0.350	0.000	0.040	748.700	3150.433	1.000	512.000
everything	1000	0.653	0.068	0.650	0.000	0.074	996.850	3150.433	1.000	512.000
everything	2000	0.750	0.041	1.000	0.033	0.140	1980.383	3150.433	1.000	512.000
fixed_slots	750	0.333	0.047	0.283	0.000	0.031	694.200	3150.433	1.000	512.000
fixed_slots	1000	0.606	0.071	0.467	0.000	0.045	893.833	3150.433	1.000	512.000
fixed_slots	2000	0.725	0.048	0.900	0.033	0.101	1642.433	3150.433	1.000	512.000
scored_knapsack	750	0.550	0.072	0.350	0.000	0.040	748.717	3150.433	1.000	512.000
scored_knapsack	1000	0.653	0.068	0.650	0.000	0.074	996.850	3150.433	1.000	512.000
scored_knapsack	2000	0.750	0.041	1.000	0.033	0.140	1980.383	3150.433	1.000	512.000
scored_knapsack_density	750	0.267	0.030	0.533	0.000	0.057	740.267	3150.433	1.000	512.000
scored_knapsack_density	1000	0.644	0.066	0.733	0.000	0.094	985.683	3150.433	1.000	512.000
scored_knapsack_density	2000	0.750	0.041	1.000	0.033	0.139	1971.417	3150.433	1.000	512.000

everything here is truncated to the budget, and candidate tokens is what the sources proposed before any policy ran, so every budget is a squeeze.

How to read it

One budget filled by calibrated score beats fixed per-source slots at tight budgets. Recall is 0.550 against 0.333 for fixed_slots at 750 tokens, 0.653 against 0.606 at 1000 and 0.750 against 0.725 at 2000. The price is more distractors (0.350 against 0.283 at 750, 0.650 against 0.467 at 1000), and at 1000 fixed_slots has slightly better precision per token (0.071 against 0.068).
Cross-source dedupe by containment, plus the deprecation flag inherited by the copy that stays, keeps superseded policies out of the window: superseded_rate is 0.000 at 750 and 1000 and 0.033 at 2000 for every configuration. Before this change it was 0.15 to 0.52, in the maintainer’s run at commit 170d40a.
prefix reuse is 1.000 on every row by construction: in this eval only the system prompt and the OKF index are stable, and both are pinned. It shows that the ordering keeps the stable block byte-identical across calls, not a cache-hit rate under realistic churn.
scored_knapsack ties everything truncated on recall, precision and distractor rate on every row (tokens per call differ by under 0.1 at 750), because both fill by score order. What the knapsack adds over truncation is caps, minimums and floors per source, not a better ranking.
Weighting by value per token (scored_knapsack_density, beta=0.5) lost at tight budgets on this corpus: recall 0.267 against 0.550 at 750, 0.644 against 0.653 at 1000, and a tie at 2000. That is why the default is beta=0.

This is one corpus, one embedder and 60 cases. It compares policies against each other; it is not a claim about your data. Run it on yours.

Running it

Level 1, no model calls:

astromesh-context eval --level 1 --embedder st --out results/

Level 2, answering and judging with an OpenAI-compatible chat model:

CTX_EVAL_BASE_URL=... CTX_EVAL_API_KEY=... CTX_EVAL_MODEL=... CTX_EVAL_JUDGE_MODEL=... \
  uv run --extra eval astromesh-context eval --level 2 --embedder st --out results/

Both need the eval extra.