Roadmap and compatibility review¶
Primr should close the demonstrated agent-integration defects, finish its bounded orchestration ownership repair, then prove the quality of its reports and evaluators before promoting additional backends. This ordering protects the existing product while turning its strongest promise into measurable evidence: a company name and website produce a useful, defensible long-form Strategic Overview and strategy documents as Word and Markdown artifacts.
This is an evidence snapshot checked on September 13, 2026 against released
baseline v1.39.14, commit 6dc8c92b3eee8d1fc29fc66c229c2c49dfd794f3.
The executable queue remains exclusively in Next Steps.
Recommendations below explain that queue; they do not define another release
schedule, authorize provider spending, or claim universal client support.
Assessment of the README and roadmap¶
The README has the right product focus and is already short enough to be a useful front door. It distinguishes agent-host Primr Zero from the paid terminal path, explains why configured credentials are not permission to spend, and leads with durable artifacts. Retain this structure. Add a direct route to the executable queue, rather than expanding the README into a second architecture or protocol manual.
The roadmap's dependency order is also sound: quality measurement precedes backend promotion; governed job consumption precedes memory; evidence-aware writing must earn promotion through measured improvement. Its weakness is navigation and historical accumulation. The lengthy implementation ledger contains older experiments and plans alongside newer measurements. Readers can mistake historical instructions for current authority, particularly when an old evaluation says to arm a gate that the current baseline deliberately keeps report-only. Correct these contradictions at their source; retain the single current queue instead of adding another summary of every feature.
The product contract provides a strong filter for new work. Protocols, provider choice, ledgers, and adapters matter when they improve report usefulness, reliability, safety, or economics. Their mere availability is not a reason to expand the product. A polished report with unsupported material claims is a failure even when every stage ran successfully; a correct claim inventory without strategic interpretation also falls short of the promised product.
What takes priority and why¶
| Priority | Decision | Evidence and completion standard |
|---|---|---|
| Immediate | Repair existing agent entry points | The HTTP transport accepted an invalid browser Origin; portable plugin state defaults to its installation tree; the VS Code example has the wrong top-level key. Prove the fixes through real transport and installed-layout tests. |
| Next bounded cleanup | Remove one orchestration back edge | fast_run_sections imports its coordinator's writer and coherence functions. The architecture test records an 11-module core cycle. Reduce that component without artifact, estimate, approval, or budget drift. |
| Next product milestone | Validate the corpus and the evaluators | Five reports and judge agreement are useful instruments, but two reports lack a decidable Confirmed floor and agreement does not establish correctness. Complete evidence review and independently audit shared errors. |
| Conditional expansion | Promote a backend only after comparison | Compare identical evidence packets and artifact contracts. Record quality, failures, latency, and actual cost. Retain preview status when the outcome is inconclusive. |
| Later | Tasks, governed memory, claim-aware writing | Integrate through existing job and artifact ownership; require SDK support and measured report lift. Do not duplicate lifecycle storage or make an always-running service part of Primr. |
The architecture repair remains worthwhile, but its value is enabling reliable changes rather than directly improving report quality. The compatibility findings justify moving ahead of it because they affect current advertised entry points and have small, deterministic fixes. That is a bounded correction to the queue, not an invitation to postpone quality behind an endless sequence of refactors.
MCP revision and implementation status¶
The current official core revision is 2026-07-28. It changes the connection
model substantially from 2025-11-25: modern requests carry version and client
capabilities themselves; server/discover replaces initialization as the
discovery mechanism; core HTTP sessions are removed. Results carry
resultType, and cacheable results include freshness and scope metadata.
Primr's SDK v2 adoption addresses this generation of the protocol; the correct
next check is transport behavior, not another version-string replacement.
MCP specification,
revision changes.
The current official Python SDK release at review time is v2.2.0, published September 7, 2026; Primr's lockfile uses 2.0.0 and its declared dependency range permits 2.x. The newer release includes HTTP session limits and authorization fixes. The SDK roadmap still lists redesigned Tasks support as unimplemented. Schedule a separately validated lock refresh, including exported container dependencies. Do not infer that upgrading the SDK implements Primr's Tasks lifecycle or repairs configuration that explicitly omits transport security. Python SDK v2.2.0 release, SDK roadmap.
| Boundary | Baseline evidence | Required assurance |
|---|---|---|
| Current core | SDK v2 handlers, discovery, cache hints, resource templates, and aligned unknown-name errors are present | Test actual HTTP requests with per-request metadata and required headers, not only direct handler calls |
| Older clients | Legacy initialization remains supported by the SDK | Exercise legacy negotiation and subsequent operations separately from modern requests |
| Browser and host validation | StreamableHTTPSessionManager was constructed without security_settings; an invalid Origin reached tools/list |
Enable the SDK guard; test rejected Origin and Host values, local defaults, and explicit trusted ingress |
| Authorization | Primr has bearer credentials, scopes, ownership checks, and estimate-bound approval tokens | Preserve those boundaries; distinguish this configured-token contract from full OAuth discovery and registration |
| Long-running work | Primr has its own job handles, cancellation, persisted journals, and compact resources | Continue using these until a supported Tasks adapter preserves the same approval, ownership, and recovery semantics |
| Optional extensions | Tasks, trace propagation, and other extensions have separate implementation requirements | Advertise only what is implemented and validated; core conformance does not imply extension support |
The Origin issue was reproduced against the actual installed SDK using an ASGI request: an untrusted browser Origin received HTTP 200 and a complete tool listing. Existing focused tests passed because they primarily exercised handlers or replaced the transport. The protocol requires rejection of an invalid present Origin with HTTP 403. This makes a wire-level regression more valuable than adding another mocked constructor test. Streamable HTTP security requirements.
MCP Tasks is now an optional extension with a redesigned contract. Its polling
and input-update model differs from the earlier experimental API. Plan against
the current extension, not an old tasks/result or tasks/list design. Preserve
Primr's existing durable handles as the operational fallback and map a future
adapter onto the same supervisor and journal. The transport's request ID is
not a substitute for durable job identity or an idempotent paid-operation
policy. MCP Tasks extension.
Remote authorization needs an equally precise claim. A configured bearer token can support Primr's current scoped operations. It does not establish that an arbitrary OAuth-only host will discover, register with, and authenticate to the server. That integration requires its own issuer, metadata, audience, and client acceptance work. Keep these requirements visible without building a new identity service into a local-first company-research tool.
Agent Plugins and host configuration¶
Agent Plugins v1.0.0 is published, rather than merely a working draft.
Its fixed package layout covers plugin.json, skills, and optional MCP server
configuration, while installation, permissions, and client-specific behavior
remain the host's responsibility. The live plugin and MCP JSON schemas match
the repository's parsed offline snapshots at review time, and the existing
manifests validate. The schema is not the gap.
Agent Plugins specification,
canonical schemas.
The runtime gap is the working directory. Without an explicit cwd, a
conformant client uses the plugin root. Installed Primr can then create its
output, working, and log directories there. Immutable installations can fail,
and replacing a plugin can put research state at risk. The specification
permits cwd: "${PLUGIN_DATA}" and requires clients to create that writable,
persistent location before launch. Use that contract and return actual artifact
paths to consumers. Shell CLI behavior remains a separate execution context.
Plugin subprocess environment.
A meaningful test places the installed package outside the repository, launches from the resolved persistent directory, and verifies state creation without writes inside the plugin package. Source-checkout tests alone are insufficient: repository-root discovery can hide the installed-mode failure. A real host acceptance check is still needed before claiming that a named application's plugin installer, permission UI, executable lookup, and restart behavior have all been tested.
The portable MCP command is the bare primr executable. A conformant plugin
manifest cannot guarantee that it is installed or visible on the host's PATH.
When the MCP component cannot launch, supported skills can still guide Primr
Zero through a working CLI or host-native fallback. Never substitute a paid
research call simply because a transport component is unavailable.
Native configuration examples must also stay client-specific. In particular,
VS Code's .vscode/mcp.json uses a top-level servers object. Reusing the
Windsurf-style mcpServers example does not configure that file correctly.
Provide a dedicated VS Code snippet and retain a structural regression for its
shape. This is separate from the portable Agent Plugins mcp.json schema.
VS Code MCP configuration.
Quality evidence that changes the plan¶
Primr's recorded baseline contains five distinct reports, 50 evidence reviews,
and agreement on 33 of 37 comparable cloud/local judgments. Its measured
Confirmed floor is 30 percent across three contributing reports; two lack a
decidable floor. The recorded keep_report_only decision is appropriate.
These are historical repository measurements, not a new independent assessment
of today's generated reports. A minimum observed score is a diagnostic
regression baseline, not a sufficient standard for analyst excellence.
Recent evidence supports testing the evaluators themselves. Time to REFLECT introduces controlled research-agent errors and reports below 55 percent overall accuracy for the best tested judges across its evaluation. The task match makes this a useful warning for research reports, but it is a May 2026 preprint and does not measure Primr. The practical recommendation is to include known wrong numbers, entity swaps, unsupported conclusions, and citation misattribution before trusting a judge to gate release. REFLECT.
Nine Judges, Two Effective Votes reports correlated mistakes across nine models from seven families on its natural-language-inference datasets. More judges did not yield the independence their count suggested. This May 2026 preprint does not invalidate Primr's existing cloud/local comparison; it shows why agreement needs an independent correctness check. Audit a pre-registered random sample of unanimous approvals as well as every consequential disagreement. Report shared-error counts alongside agreement. Correlated judge errors.
The August 2026 Trust-Truth Separability preprint finds that changing source attribution on identical question-answer content can shift both trust and truth judgments. Blind provider and model identity in comparative evaluations. Assess source authority and whether a claim follows from its evidence as distinct dimensions; do not count correlated scores as independent support. Trust and truth study.
Precision also needs a coverage counterpart. The peer-reviewed EMNLP 2025 VeriFact work examines missing context and relational facts in long-form factuality evaluation. Its precision-and-recall framing supports measuring omitted material facts and contradictions, rather than rewarding a report solely for making fewer checkable assertions. Primr should preserve both broad company understanding and explicit uncertainty. VeriFact.
These studies motivate measurements, not universal numerical thresholds. The corpus should include sparse and rich public evidence, blocked collection, hiring signals, changing company context, and contradictory or stale sources. Each report must have an explicit evidence state. A sparse report with no justifiable Confirmed claims should remain honest; it must not manufacture high-confidence labels to qualify for a calibration statistic.
Before any quality promotion, freeze the reports, source receipts, model and prompt fingerprints, sampling plan, error taxonomy, human labels, and decision criteria. Then report error-detection precision and recall by category, abstentions, retrieval failures, omitted material facts, shared errors, and sampling uncertainty. Keep structural checks such as citation resolution and DOCX rendering deterministic. Evidence support and strategic usefulness still need source-grounded judgment and review.
Backend freedom, memory, and economics¶
Backend freedom should mean a validated alternative that preserves the artifact and uncertainty contract. It should not mean that every provider slug appears in a menu. The repository already has capability routing, stage metadata, an unpromoted host adapter, and a standing offline source-relevance corpus. The valuable next result is a defensible comparison on frozen evidence, followed by an explicit retain-or-promote decision. Recheck exact model IDs, pricing, retirements, and SDK behavior immediately before a paid comparison.
Current provider documentation reinforces cost provenance rather than a new
default-model recommendation. xAI returns per-request cost_in_usd_ticks
including applicable discounts and server-side tools; preserve those receipts
instead of replacing them with token-price estimates. OpenRouter distinguishes
the charged amount from upstream inference cost, while max_price limits token
pricing rather than total campaign spending. Claude Code's local dollar display
is not an authoritative bill for included subscription usage. These are
different accounting contracts and should remain explicit in route metadata.
xAI cost tracking,
OpenRouter accounting,
OpenRouter provider selection,
Claude Code costs.
Similarly, memory should first prove that a subsequent company engagement improves without importing stale claims or exposing unrelated research. Keep the proposed ledger in shadow mode until a held-out comparison supports its use in writing. Retention, deletion, export, evidence provenance, and freshness are entry requirements. More persistent state is not itself product progress.
Cost controls need a clear scope. A Primr per-run ceiling does not enforce a whole campaign's total across research runs, judges, host agents, and retries. The review's requested ceiling is $10 total. No paid Primr research or provider-judge execution is part of this change. Host-agent dollar usage is not exposed in this environment, so an all-inclusive measured total cannot be asserted. An unrelated deferred Premium evaluation's $25 ceiling is not authorization to spend that amount here.
If later paid evaluation is justified, prepare the exact pilot and fresh estimate first. Reserve already-authorized noninterruptible work, include judges and provider tools, reconcile actual charges, and stop on an instrumentation or protocol defect. A small budget should produce a decision or a clearly documented limit, not a claim that a small sample proves production-wide quality.
Persistent development guidance¶
The development contract now lives in
AGENTS.md, which
CLAUDE.md imports (#264); the operating guide moved to
clients/primr-operator.md. The refinement
adds evidence-specific verification, source and history checks, use of the
existing AST dependency graph, precise status claims, and resumable state in
the existing gitignored .agent/ directory. It preserves the Python stack,
canonical seams, spending gate, local-first boundary, and release policy.
Commands were checked against the manifests, current tool behavior, and CI.
The primary CI environment includes all extras; its five sequential test
shards contribute to one 81 percent branch-coverage gate. Local verification
uses uv run --no-sync to reuse the prepared environment, while dependency
exports continue to use CI's pinned uv version. Mypy has a mixed existing
baseline with module-specific stronger checks, rather than universal strict
coverage; the new MCP configuration boundary joins that stronger set.
uv environment synchronization,
mypy guidance for existing code.
Validation and limits¶
Local checks passed for the real HTTP transport and Azure configuration (68 tests), portable plugin packaging and installed state (10 tests), and the existing architecture ratchets. The generated Bicep template compiles. Ruff, mypy, strict documentation, static security checks, dependency auditing, source-distribution inspection, and the separate recovery floor passed during this review. Dry-run tests exposed local gateway configuration leakage; the fixture now isolates that setting and all 35 handler cases pass.
The local full-suite run completed the protocol, agentic/property, and provider groups. Windows event-loop setup stalled during the core retry, so it is not reported as a complete suite or combined coverage result. The release remains gated on the existing CI matrix, its 81 percent combined branch-coverage floor, and an independent successful run on the exact main commit. The pull request and release workflow carry the final run evidence; baseline CI does not prove this patch.
This review validates primary-source contracts and local implementation behavior. It does not claim a live acceptance run in every named host, production cloud deployment, full OAuth interoperability, Tasks support, or new paid report-quality evidence. Those limits define subsequent acceptance work rather than weakening the requirements for the behavior implemented now.
Source register¶
All external sources were accessed September 13, 2026. Undated living documents are identified as such; an access date is not a publication date.
| Publisher | Source | Publication or revision | Use |
|---|---|---|---|
| Model Context Protocol | Core specification and changes | 2026-07-28 | Current protocol generation and extension separation |
| Model Context Protocol | Transports | 2026-07-28 | HTTP security and transport behavior |
| Model Context Protocol | Tasks | Living extension documentation | Future durable-job adapter boundary |
| MCP Python SDK maintainers | v2.2.0 | 2026-09-07 | SDK currency and implementation limitations |
| Agent Plugins | Specification, schemas | Published v1.0.0; living pages | Package/runtime contract and schema comparison |
| Microsoft | VS Code MCP servers | Living documentation | Native client configuration shape |
| Wang et al. | Time to REFLECT | 2026-05-18; preprint | Controlled research-agent error evaluation |
| Kohli | Nine Judges, Two Effective Votes | 2026-05-28; preprint | Correlated judge mistakes |
| Sun et al. | Trust-Truth Separability | 2026-08-21; preprint | Source-attribution sensitivity |
| Association for Computational Linguistics | VeriFact | EMNLP, November 2025 | Long-form factual precision and recall |
| xAI | Cost tracking | Living documentation | Exact request-level charges |
| OpenRouter | Accounting, routing | Living documentation | Charge provenance and price-ceiling scope |
| Anthropic | Claude Code costs | Living documentation | Host usage estimates versus subscription billing |
| Astral | Locking and syncing | Living documentation | Verification environment behavior |
| Mypy maintainers | Existing codebases | Living documentation | Incremental type-checking boundaries |
Repository evidence: Next Steps, product contract, evaluation plan, Premium evaluation design, architecture cohesion plan, and agent control-plane design.