1.x Completion: Finishing the Excellent Single-Shot Brief¶
Status: ACTIVE - this is the current line of work. ROADMAP anchors: Active Queue #3 (remainder), #4, #9, #23; panel-review high tier (label calibration, evidence-fetching verify).
Motivation¶
1.x is "URL in, consultant-grade artifact out, done well." As of v1.30.0 the engineering backlog behind that promise is largely closed: the artifact shipping contract, failover, budget controls, observability, the QA iteration loop, and runtime robustness all shipped. What remains is concentrated in two places - measured quality (is the analysis actually good, are the labels actually true) and core testability (the pipeline heart is a ~1,900-line function the suite can only test around).
Immediate next slice¶
The next 1.x slice is not broader prompt tuning, and it is not a simplistic
fact-checker. It is evidence-grounded validation: the calibration sidecar and
eval scorecard now carry source-level review dimensions, and the next live
slice is to run a multi-report baseline, compare local and cloud judges on the
same sampled report units, and evaluate whether the measured support,
contradiction, source independence, source authority, reasoning strength, and
uncertainty honesty rates justify any hard gate. In parallel, surface
contradicted --verify claims in the report trust summary so the user sees the
risk in the artifact, not only in sidecar JSON. Any hard
FAIL_CALIBRATION threshold must come from the measured agreement-validated
floor of that broader rubric, never from string overlap or an isolated
citation-presence check.
This v1.40 decision remains report-bound and ledger-independent. The planned
run-scoped epistemic ledger must reuse the measured vocabulary and baseline
later; it does not block, reset, or substitute for the representative corpus.
Its first production phase is shadow-only after the calibration decision. See
run-epistemic-ledger.md.
Workstreams, in dependency order¶
1. Measure the epistemics - SHIPPED (post-1.30.0: PRs #27, #28, refine acceptance guard)¶
All three pieces landed: evidence-based --verify (fetched snippets +
first-party/third-party provenance + deterministic self-corroboration
downgrade, PR #27); the label-calibration harness
(qa/label_calibration.py - deterministic claim sampler, traceability
audit with injectable fetch/judge seams, per-label precision, PR #28); and
the refine-loop anti-Goodhart guard (each iteration audited by the
calibration harness; traceability degradation reverts the iteration).
Remaining within this workstream:
- ~~Wire the metric into the model_eval scorecard~~ - DONE: primr calibrate
"Company" / --calibrate-recent N (with --dry-run judge-call/cost
preview) audits shipped reports and persists <report>.calibration.json
sidecars; the offline eval reads sidecars into per-report
confirmed/reported traceability, a pooled ## Label Calibration
scorecard section, CSV columns, and a FAIL_CALIBRATION decision row.
The hard gate is armed via PRIMR_EVAL_MIN_CONFIRMED_TRACEABILITY
(fraction); it stays unset (report-only) until the baseline below exists.
The harness was also fixed for two artifact-format drifts a free corpus
dry-run caught before any paid pass: the [cite: N] url Sources-appendix
entry form, and standalone block-trailing labels ([paragraphs]
[What to validate] [(Label)]) now associate with their block's prose and
citations instead of sampling bare label lines.
- Run one calibration pass over recent current-format reports (measured by
dry-run: ~164 judge calls ≈ $0.07-0.15) to establish the per-label
baseline, but treat that as only the first slice of the validation rubric.
Agreement-validated scoring fields for support, contradiction, source
independence, source authority, reasoning strength, uncertainty honesty, and
business relevance are now in the calibration sidecar and offline eval
scorecard. Set PRIMR_EVAL_MIN_CONFIRMED_TRACEABILITY only from the measured
floor once the broader rubric says the threshold is meaningful.
The judge can also run on a local OpenAI-compatible server for $0
(--judge auto|local, auto-detected via /v1/models, cloud default,
sidecars stamp judge: {kind, model}); --judge-compare measures
cloud-vs-local agreement on the same claims so a given local setup's
judging is trusted with receipts, not vibes. This is also the first
concrete slice of the 2.0 backend-freedom pillar pulled into 1.x: the
utility-tier local-routing plan (scrape summaries, link selection,
posting triage) follows the same pattern - detect, opt in, validate by
agreement, fail open to cloud - and is gated on these instruments.
- Surface contradicted claims from verification.json in the report's
trust summary (today: JSON + console only)
- ~~v2 calibration check: Estimated/Hypothesis claims must NOT be
verbatim-from-source (the mislabel in the other direction)~~ - DONE:
calibration sidecars, eval scorecards, CSV exports, and baseline artifacts
now surface source-copied inference claims as report-only signals.
- ~~Act on the measurement: downgrade ungrounded labels at ship time~~ -
DONE (opt-in): qa/label_honesty.py + PRIMR_LABEL_HONESTY=1 wire a
pre-ship pass into the trust stage that re-judges each (Confirmed)/
(Reported) claim against its cited source (reusing the calibration
harness's extract/fetch/judge seams) and rewrites the ones that don't
trace to (Estimated). The downgrade is mechanical and fail-safe
(confidence only lowers; no_source/unfetchable/uncertain verdicts keep
the label), writes a _label_honesty.json audit, and never blocks
shipping. Default-off keeps the standard run byte-identical until the
agreement-validated baseline above justifies promotion; a hard gate is
never armed from a lone judge.
Original build spec (kept for reference):
- A claim sampler: extract N labeled claims per eval report with their citations (deterministic, reuses the citation parser)
- A traceability check per label class:
(Confirmed)must trace to a cited source containing the claim's substance;(Reported)to a third-party source;(Estimated)/(Hypothesis)are exempt from traceability but must not be verbatim-from-source (else they're mislabeled Confirmed) - An LLM-judged evidence review step where string matching is insufficient, run against the fetched source text, never titles. The judge must evaluate support, contradiction, source independence, source authority, reasoning strength, and uncertainty honesty, not just whether a phrase appears.
- Surface per-label precision in the eval scorecard; add "Confirmed-claim traceability >= X%" as a HARD gate once a baseline exists
Validation: the harness is free to build (mocked); one calibration pass over the standing eval corpus costs roughly one judge run (~$0.25-0.50).
Evidence-fetching --verify. Today _classify_results judges claim
support from search-result titles, and the claimed company's own domain can
"verify" a claim sourced from itself. Change:
- Fetch top-hit page content (reuse
scrape_external_sources_validated, capped pages, existing SSRF guard) and classify against snippets - Exclude same-registrable-domain sources from "supporting"
- Record contradicted claims in
verification.jsonAND surface them in the report's trust summary
Validation: free with mocked search/scrape; one live --verify run ≈ $0.01.
De-Goodhart the refine loop. primr refine optimizes the artifact-
discipline score, which counts the tokens the regenerator can insert. Gate
acceptance on a check the scorer can't see: a claim spot-check (from the
calibration harness) on regenerated sections, or an LLM-judge delta. Reject
the iteration when discipline rose but independent quality didn't.
2. Refactor the orchestrators (#23) - EXTRACTION COMPLETE¶
Status: all ten stages of perform_fast_research extracted (Batches A-G,
~1,600 lines into eleven stage modules with ~110 hermetic tests); the
orchestrator is a ~295-line coordinator. Remaining: FastRunContext, the
per-module coverage target, the C901 budget, and the
_execute_consulting_research split. Working map:
23-orchestrator-refactor-map.md.
perform_fast_research (~1,900 lines) interleaves I/O, LLM calls, and state
transitions; _execute_consulting_research (~270 lines) similar. Rules:
- No behavior change - seam introduction only; eval scores must be unchanged after the refactor
- Extract, in this order (each is independently shippable): scrape+corpus assembly; external search + gap analysis; workbook stage; the per-section writing loop; cross-validation + repair; strategy generation. Each becomes a function taking explicit inputs and returning structured results, with the LLM boundary injectable
- After the split: raise per-module coverage targets (research_agent ≥ 80%),
then enable the
C901complexity budget repo-wide
This unblocks every later pipeline change (batch API, overlap, routing) from being reviewable diffs instead of edits inside a monster.
3. Consultant-grade strategic writing (#4)¶
One focused prompt-and-eval cycle, judged by the step-1 instruments:
- Section prompts tuned around management choices, operating constraints, likely economics, scenario paths, validation questions
- Fewer brittle section suppressions; constrained-evidence reasoning when direct data is thin
- Dense references concentrated in appendices; body reads as analysis
- Target: sparse-company runs feel substantive; rich-company runs sharp
Validation: this is inherently eval-judged. Budget one corpus pass per prompt iteration (~$4-5 for n=5 at the standard recipe); cap at 2-3 iterations per cycle; pre-register acceptance criteria before the first run (the EVAL_V1_24_0 discipline).
4. Cost levers (#9 batch API, #19 pipeline overlap)¶
Mechanical after step 2. Batch API: --batch-api flag, xAI batch first,
graceful fallback to the executor path, batch pricing in ModelConfig so
estimates stay honest. Overlap: start external search once the homepage is
in; asyncio.gather with completion barriers; progress display shows
concurrent phases. Validation: deterministic tests free; one live batch run
to confirm the discount (~$0.50-0.80).
Also in this band¶
- #3 remainder (live-site dependent, opportunistic): page-access trace
analytics/evals now have the body-free fixture/prediction scoring foundation,
an operator-facing
--eval-page-access-fixturecommand, and a canonical representative sanitized protected-site corpus attests/fixtures/page_access/protected_site_trace_corpus.json. Hiring signals in--premiumremain open; BambooHR/iCIMS public-board adapters are shipped. - Panel medium tier: meeting-brief one-pager output mode,
--jsonoutput for the main run,primr replay(record/replay run transcripts for demos), install extras (primr[browser],primr[ocr]), sample gallery
Exit criteria (1.x done)¶
- Confidence labels and sampled report reasoning have measured calibration across support, contradiction, source quality, reasoning strength, and uncertainty honesty
--verifyverifies against fetched evidence, never titles, and surfaces contradictions in the artifact- The three monster functions are split; pipeline-core coverage ≥ 80%; complexity budget enforced
- A #4 prompt cycle has shipped with pre-registered acceptance met
- Sparse-company and rich-company runs both judged at target by the eval harness; artifacts ship clean (gates at zero) on the standing corpus
Explicitly not in 1.x¶
Anything from the 2.0 pillars (routing layer, memory, control-plane authz); new output formats beyond the one-pager/JSON noted above; new scrape tiers.