Evaluation protocols and historical results¶
Purpose. primr's quality changes are validated by eval, not unit tests, and model-backed evals can cost real money. This doc records protocols and results. Freeze the hypothesis, exact commands, instruments, and acceptance criteria before new spend so a paid run gives a decision instead of a post-hoc argument. Start with low-cost checks and stop if they fail to justify further spend.
No corpus is committed (the no-real-company-data rule): the operator supplies real companies locally. Freeze representative selection before evaluation. A sparse/rich pair can expose pilot defects; it does not establish readiness for a production quality gate or a backend promotion.
The executable queue lives only in Next Steps. The numbered experiments below preserve their original hypotheses and historical outcomes; their order does not supersede that queue. Quoted costs are historical planning figures, not current estimates or approved spend. Every new billable run or judge call requires a fresh exact estimate, explicit approval, and a cost cap. Local artifact inspection needs no provider key. Configured local judges can avoid model API charges; host-plan billing must be verified separately.
Experiment index¶
| # | Eval | What it answers | Historical planning cost | Interpretation |
|---|---|---|---|---|
| 1 | Label-calibration baseline | Are (Confirmed)/(Reported) labels traceable to sources? |
~$0.10 | Report-only measurement followed by readiness inspection and an operator decision |
| 2 | Framed vs unframed (tradecraft Step 4) | Does framing + hypothesis-steered collection produce a better brief? | ~$1.58 / company | Historical pilot did not justify default promotion |
| 3 | Content-depth prompt work (#4) | Do sharper section prompts (management choices, economics, scenarios, constrained-evidence) beat the current prompts? | ~$4-5 / company | Re-register a bounded experiment before execution |
| 4 | Context curation | Does a different evidence subset improve the brief? | ~$1.58 / company | Historical pilot did not justify default promotion |
Calibration does not arm a gate. Start with existing artifacts and zero-spend readiness inspection; later experiments depend on trustworthy instruments and the current executable card, not a historical pilot's place in this table.
Eval 1 - label-calibration baseline (~$0.10)¶
Historical result (3 reports, approximately $0.02): grounding defects worth investigating. The recorded counts were 1/10 Confirmed claims traced and 0/25 Reported claims traced, with
unfetchable=0. These sampled judge verdicts exposed a problem in that sample. They do not establish a population error rate or prove that other quality dimensions are satisfactory. The n=1 pilots in Evals 2 and 4 did not justify their respective promotions; they did not rule out improvements to collection or context assembly.
The subsequent July 13, 2026 milestone and operator decision are recorded in the
roadmap: five reports,
50 evidence reviews, 33/37 comparable
cloud/local verdicts agreeing, and a deliberate keep_report_only decision
because two reports lacked a decidable Confirmed floor. These are historical
measurements, not a current-file inspection or proof that agreement is accuracy.
Hypothesis: confidence-label traceability can be measured against fetched source text, with evaluator errors and missing evidence made explicit, to inform a reviewed regression-gate decision.
Commands
Choose the judge explicitly and obtain approval for any billable execution before running calibration. An aggregate latest-N sample is not a curated representative baseline.
Instrument: the qa/label_calibration.py harness (per-label precision of
(Confirmed)/(Reported) against fetched source text; no_source counts
against; unfetchable excluded). Use --judge local for $0 if Ollama is up.
Acceptance / output: record per-label precision, decidable counts,
unfetchable/no-source counts, and evaluator-validation results. Freeze report
and sidecar fingerprints in an explicitly curated --pack-selection manifest
with representative tags. Use --pack-manifest, then --baseline-from and
--inspect-baseline to inspect readiness without model calls. Resolve the
reported blockers and review the evidence before recording a decision through
--baseline-decision-from and --baseline-decision keep_report_only|arm_gate.
The command writes a decision record; it does not set an environment variable.
Use --inspect-baseline-decision to verify the record against current artifacts.
arm_gate is available only when the inspected template permits it and the
operator has completed the required review. No example in this document
authorizes arming PRIMR_EVAL_MIN_CONFIRMED_TRACEABILITY automatically.
Judge comparison reuses the original sampled claims and cloud verdicts. Both
judges receive the same captured source text; the comparison does not reread
the report or repeat the cloud pass. A failed cloud or local pass contributes
no agreement count. If the report or its sidecar changes during comparison,
the outcome records comparison_artifact_changed and no agreement is written.
The preserved cloud sidecar is still a cloud result, not evidence of successful
comparison. Resolve failed outcomes before treating a pack as ready.
The candidate regression floor is a report-clustered Wilson lower bound at 95
percent confidence. The gate records bound_method, confidence, and bound.
That bound is not the desired quality bar. Define the bar from material error
risk and analyst usefulness before evaluating a change. A low observed bound
does not become acceptable quality because every report becomes decidable.
Judges can score frozen scraped_content.txt captures, and an explicit abstain
does not count as untraceable.
Distinguish missing evidence or failed assessment from a sparse report with no
warranted Confirmed claims. A person records the latter with
honestly_empty_confirmed: true on that report in the pack selection file.
Do not add artificial Confirmed labels to make a baseline pass; preserve honest
labels and the report-only decision when required.
Validate the evaluators¶
This protocol is optional diagnostic tooling, not a release gate.
Whole outputs. Strategy analysis is reasoned judgment, so report quality is checked by reading before and after outputs for the same companies side by side, blind where practical. Model judges are an optional second opinion. A judge that prefers a longer or more fact-dense report over a sharper one is measuring length or claim count, not analysis.
Claim-support judges (diagnostic). Freeze clean cases and controlled variants before judging. A clean case is accepted when its support span appears verbatim in the captured source. Cover wrong numbers/entities/dates, negation, stale evidence, unsupported causal claims, insufficient independent support, omitted contradictions, and incorrect citation attribution. Keep these cases separate from the production report scores. Report detection precision and recall by error category, sample counts, abstention and retrieval-failure rates, and uncertainty intervals. Pre-register error tolerances and the minimum usable sample for each decision. Missing evidence is not a correct verdict, and high precision cannot excuse low coverage.
Blind model/provider identity, test order sensitivity, and send consequential
disagreements and a pre-registered random sample of unanimous approvals to
an independent agent adjudicator from a model family not used as a judge,
recorded as agent adjudication and never described as human review. Record
shared errors as well as agreement. Assess
claim support separately from source authority and stylistic trust cues; do not
count correlated scores as independent confirmation.
This protocol addresses findings from REFLECT (May 18, 2026), whose tested research-agent judges remained below 55% overall failure-detection accuracy, and Nine Judges, Two Effective Votes (May 28, 2026), which found strongly correlated panel errors. Both are preprints on their own benchmarks, not Primr measurements. The August 21, 2026 preprint Trust-Truth Separability also finds source labels can shift truth judgments on identical QA. The peer-reviewed VeriFact (EMNLP, November 2025) supports measuring factual recall and relational context alongside precision. These sources motivate evaluator stress tests; they do not supply universal promotion thresholds.
Instrument (library only, no public command):
primr.qa.evaluator_variants parses a versioned case file of clean cases. Each
names the exact supporting span and who adjudicated it (human, agent, or
synthetic). It builds one controlled variant per category above. Number,
date, negation, support-removal, and citation-attribution variants are
mechanical. They skip bounded quantities ("fewer than 5"), already-negated
claims, and passive participles, where a mutation could leave the claim true
or ungrammatical. Entity, causal, stale-evidence, and contradiction variants
need author-supplied text. The authority_without_support variant removes
the support span as insufficient_support does, then prefixes a strong trust
cue (the case's authority_cue, or a generic audited-report cue). A judge that
accepts it is scoring source authority instead of claim support, the effect
the Trust-Truth Separability preprint reports. Abstaining on it counts as
detection by default. A variant that cannot be built is reported as skipped. primr.qa.evaluator_validation then:
- freezes the items, an explicit
Preregistration(no default tolerances), and the generator version under one digest. It rejects contradictory items, empty sources, and repeated claim/source pairs, and detects any later edit. Record the digest outside the system before judging. - runs each judge blind over seeded shuffled passes, recording abstention, unparseable answers, errors, and empty sources separately;
- scores per-category recall, clean-case false alarms (any clean answer other than yes or abstain), pooled flag precision, unwarranted abstention, judge and retrieval failures, and repeat flips with Wilson intervals, plus errors shared by every judge and pairwise agreement;
- queues judge disagreements, repeat flips, and seeded samples of unanimous approvals and unanimous variant rejections for source-grounded human review; and
- rebuilds the score and queue from the runs and reports
insufficient_sample,incomplete_review,fails_preregistered_tolerance,inconclusive, ormeets_preregistered_tolerancefor each judge.
The shipped harness currently requires an all-human-adjudicated set and a
named human review covering the whole queue, plus at least two judges and
two passes. Those requirements apply only to its own decision status; no
release depends on it.
Disputed items require rebuilding and refreezing. No status arms a gate. Case
files that describe real companies stay in local, gitignored locations; the
committed fixture is synthetic.
Known limits: each judge call carries one item, so repeat flips measure run-to-run consistency, not in-context order effects. Item IDs name the variant category, so blind adjudication needs the queue presented without IDs or expected verdicts. The authority variant tests only the direction where a trust cue should not create support; a weak-provenance source that does support a claim has no unambiguous expected verdict and is not generated. The review record is self-reported and must be kept with the operator's evidence.
Eval 2 - framed vs unframed (tradecraft Step 4, ~$1.58/company)¶
Historical result (n=1, ~$0.69 spent): no default promotion. First A/B (one mid-market financial-services company, standard recipe ~$0.35/arm) - steering fired correctly but the blind pairwise grade was a wash, slightly favoring unframed; neutral cost; both gates PASS. Did not clear the acceptance criterion below. A possible explanation was a breadth/depth tradeoff against the fixed report structure; this pilot did not isolate that cause. Decision: keep Step 4 opt-in without default promotion. Directional (n=1). Full write-up in research-tradecraft.md Step 4.
Hypothesis: when a run is framed (--purpose/--question), the Day-1
hypothesis tree steers collection toward testing branches, producing a brief that
a blind panel prefers - without a trust-gate or cost regression.
Commands (per company; <co>/<url> are the operator's real targets)
# unframed arm (today's default path)
py -3.12 scripts/eval/run_recipe_brief.py --company "<co>" --url <url> \
--recipe standard --tag standard-unframed
# framed arm (operator intent steers framing + tree + gap queries)
py -3.12 scripts/eval/run_recipe_brief.py --company "<co>" --url <url> \
--recipe standard --tag standard-framed \
--purpose <purpose> --question "<the decision question>"
# blind pairwise grade (free local judges)
py -3.12 scripts/eval/grade_pairwise.py --label framing \
--baseline output/eval/briefs/standard-unframed/<brief>.md \
--candidate output/eval/briefs/standard-framed/<brief>.md \
--judges "ollama:<model>"
Instruments: grade_pairwise.py (position-swapped, length-penalized, panel;
free local judges), plus each run's own trust gate and the ~$cost line.
Pre-registered acceptance (ALL must hold across the fixed company set): 1. Quality: framed wins the section-majority on the strategic sections (Executive Summary, SWOT, Strategic Positioning, Competitive Landscape, Strategic Tensions) - i.e. more candidate-wins than baseline-wins by section majority - on both the sparse and the rich company. 2. No trust regression: the framed arm's trust gate is PASS, at or above the unframed arm. 3. No cost regression: framed run cost ≤ ~110% of unframed (framing adds the cheap tree pass; if it balloons cost, that's a fail).
Decision interpretation: clears all three → consider framing promotion after evaluator validation and review. Fails #1 → the tested steering did not demonstrate its benefit; reconsider Step 4 before building on it. Fails #2/#3 → fix the regression before re-judging quality.
Eval 3 - content-depth prompt work (#4, ~$4-5/company)¶
When the executable queue selects this experiment, re-register the candidate
and decision criteria using validated evaluators. Same A/B shape: current
section prompts (baseline) vs a candidate prompt revision (management choices,
operating constraints, likely
economics, scenario paths, constrained-evidence reasoning, explicit "so what"
per section - content within the fixed structure, never new sections). Judge
with grade_pairwise.py + the calibration instrument from Eval 1.
Pre-register before running (fill in when the candidate prompt is ready):
the exact prompt diff, the same fixed company set, and the same three acceptance
criteria as Eval 2 (quality by section majority, no trust regression, no cost
regression). Structure stays the curated company_overview.yaml scaffold - see
research-tradecraft.md Step 5 and the structure
carve-out in agentic-balance.md.
Eval 4 - context curation at analysis/writing (candidate; ~$1.58/company)¶
Historical result (n=1, ~$1.4 spent): wash, no default promotion. A/B on a large content-dense company whose corpus was ~360k chars (curation dropped ~72%, choosing the most-relevant 100k vs the first 100k). Blind pairwise grade: every section tied (section-majority 0/5/0; the strongest cross-family judge tied on all five), both gates PASS. This company, configuration and grading protocol did not demonstrate a quality improvement. The feature remains default-off. Together with Eval 2, this supports withholding those promotions; it does not prove evidence collection or context assembly cannot improve reports, nor establish workbook or prompt changes as the causal alternative.
Original motivation: a historical dry-run estimated approximately 1.9M aggregate input tokens across analysis and section-writing calls. This motivated testing which evidence reaches each model call. Aggregate tokens do not measure one context window, prove context degradation, or imply quadratic API charges.
Hypothesis: curating the context that reaches the analysis + writing stages (relevance-rank the corpus, drop low-signal pages, route per-section evidence instead of dumping everything) produces an equal-or-better brief at materially lower token cost - and possibly better quality by reducing context rot.
Magnitude (measured, honest): the ~1.9M is ~23 section calls of ~60k cached tokens each (the cached-prefix split, roadmap #8), not one bloated window. Cost is already softened by caching; the lever is mainly quality (less irrelevant context in each call) with cost upside if the prefix shrinks. Any benefit remains a hypothesis to measure, not a consequence of the token count.
Status: BUILT (flag-gated, default off). core/context_curation.py
rank_corpus_by_relevance() replaces the section writer's blind first-100k-chars
corpus truncation with the most-relevant 100k (ranked by term-overlap with the
analysis workbook), shared across sections so the cached prefix is preserved.
Activate with PRIMR_SECTION_EVIDENCE_CURATION=1; default off is byte-identical.
Method (A/B, same harness): baseline = current pipeline (env unset);
candidate = PRIMR_SECTION_EVIDENCE_CURATION=1, same recipe + company set (use
companies large enough that the corpus exceeds the 100k budget - otherwise it's a
no-op). Grade with grade_pairwise.py (free local judges) + the Eval 1
calibration instrument.
Pre-registered acceptance (for the BUILT shared-rank version - quality test). The shipped curation keeps the same 100k budget (most-relevant 100k vs first 100k), so it does not reduce tokens; its value is purely which evidence the writer sees. Judge it on quality at no regression: 1. Quality: candidate wins or ties by section majority (a clear win → promote toward default; a wash → keep opt-in, no harm; a loss → the rank hurts, revert/rethink). 2. No trust regression. 3. No cost regression (same budget → cost should be ~flat; a jump means the rank changed cacheable content unexpectedly).
The ≥20% token-reduction goal belongs to the separate, not-yet-built per-section routing version (each section gets a smaller, section-specific subset), which breaks the shared cached prefix and carries the breadth/depth risk Eval 2 surfaced. Scope + pre-register that as its own eval if/when the shared-rank version proves worthwhile here.
Doctrine note: curation is context assembly, not a content gate. It can be a deterministic relevance rank (cheap, stable - a legitimate rule) or model-judged selection; either way it's measured by this eval, never a regex that judges quality. Build the curator behind a flag so the raw path stays the default until this clears (no speculative default change).
Why pre-registration¶
Stating the acceptance criteria before the run is the anti-Goodhart, anti-vibes discipline the doctrine requires (Model Adaptability: "data-driven adoption, no gut decisions"). It also bounds spend: each eval has a clear go/no-go, so a cheap failing step stops the expensive one from running at all.