Primr Architecture¶
This document describes the internal architecture of Primr, a research tool that generates company intelligence briefs using a local orchestration pipeline and provider-backed AI models.
Overview¶
Primr is designed around a core insight: good company research requires both breadth (what's publicly available) and depth (what it means strategically). The architecture reflects this by combining two complementary research engines, each optimized for different aspects of the problem.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β CLI Entry Point β
β primr "Company" https://... β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Research Orchestrator β
β (Mode Selection & Coordination) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βββββββββββββββββββββββββΌββββββββββββββββββββββββ
β β β
βΌ βΌ βΌ
βββββββββββββββ βββββββββββββββ βββββββββββββββ
β Scrape β β Deep β β Complete β
β Mode β β Mode β β Mode β
β (5-10m) β β (10-15m) β β (30-40m) β
βββββββββββββββ βββββββββββββββ βββββββββββββββ
β β β
βΌ βΌ βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Report Generation β
β (DOCX, PDF, TXT Output) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Design Principles¶
1. Epistemic Humility¶
Primr distinguishes between facts, inferences, and hypotheses. The system is designed to surface what we know, what we think, and what we should ask. This is encoded in prompts, output formatting, and the separation between data collection and analysis.
2. Local-First¶
Local execution and local artifact ownership are the defaults. Data leaves the machine only through web retrieval, search, model providers, or gateways the operator explicitly configures. Those may include xAI, Gemini, OpenAI, Anthropic, cloud gateways, or a user-operated OpenAI-compatible endpoint. Primr does not upload reports to its own hosted service.
3. Graceful Degradation¶
Every component has fallbacks. If Playwright fails, try httpx. If httpx fails, try requests. If the AI planning call fails, use default chapter structure. The system should produce useful output even when individual components fail.
4. Separation of Concerns¶
Data collection (scraping, search) is separate from analysis (AI). Analysis is separate from output (document generation). This makes the system testable and allows components to evolve independently.
5. Measured Implementation Choices¶
Primr is Python-first, not Python-only. Python owns orchestration, provider integration, research policy, and report generation because ecosystem leverage and iteration speed dominate there. A different language or runtime is introduced only at a narrow, versioned boundary after an optimized Python baseline and production-shaped profile demonstrate a material end-to-end benefit.
Adoption requires contract and differential tests, supported-platform packaging, equivalent observability and failure semantics, a fallback where practical, and a measured maintenance and security payoff. Rust is the only current in-process accelerator candidate, specifically for deterministic HTML analysis after a parse-once Python baseline. Go is reserved for an independently deployed control-plane boundary after scale and SLO evidence. Mojo is not embedded; MAX or another runtime may compete as an external OpenAI-compatible model server.
The full policy, benchmarks, stop conditions, and packaging rules live in
design/runtime-language-boundaries.md.
Research Modes¶
Canonical production topology¶
This table is the source of truth for report-generation ownership. Class names that remain importable for compatibility are not necessarily active product paths.
| Surface | Active caller and behavior | Status |
|---|---|---|
| Standard provider-backed report | perform_research dispatches to perform_fast_research: collection and workbook analysis, planned section batches with rolling context, cross-validation, trust polish, and a guarded whole-document coherence pass |
Production default when the measured xAI plus Gemini recipe is available |
| Deep/Premium report | perform_deep_research dispatches through ResearchOrchestrator and premium_deep_research_stage to generate_comprehensive_report: Deep Research dossier, sequential Flash section writing, then assembly |
Production Deep Research path |
Single-call DeepResearchOrchestrator.generate_report |
One comprehensive Deep Research interaction | Compatibility and test surface; no internal production caller |
MasterArchitect, ResearchExecutor, ReportAggregator |
Chapter planning, node execution, and aggregation APIs | Compatibility surface; no internal production caller |
accordion_test.py / --test-accordion |
Standalone manual experiment | Experimental CLI compatibility surface |
Standard uses bounded concurrency inside an explicit report plan and follows it with document-level validation. Premium keeps section writing sequential so recent findings can flow forward and provider pressure stays bounded. Neither topology should be changed on latency intuition alone; compare complete report quality on the same evidence and evaluation corpus.
Scrape Mode (Corpus+Insights)¶
Website-focused research using the build_site_corpus workflow with AI-powered insight extraction.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Scrape Mode Pipeline (Corpus+Insights) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β build_site_corpus (fetch_web_content) β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β 1. discover_site_urls β (in_scope_urls, external_urls) β β
β β 2. rank_and_select_urls β selected_urls β β
β β 3. scrape_pages (using scrape_page primitive with tiers) β β
β β 4. build_corpus β cleaned text β β
β β 5. save_raw_scrapes, save_external_links β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
β Outputs: _raw_scrapes/, scraped_content.txt, _external_links.txt β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β scrape_page primitive (9-Tier Orchestrator) β
β β
β Playwright -> Aggressive -> Patchright -> curl_cffi β
β -> DrissionPage Stealth -> DrissionPage -> Vision β
β -> httpx -> requests β
β β
β Features: Sticky tier, Circuit breaker, Cookie handoff, Soft block β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β extract_insights (summarize_scraped_content) β
β (Extract key facts from corpus - LLM-powered) β
β β
β Output: insights.txt β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
When to use: Quick website intel and data collection for downstream use. The current static estimate is about 2-5 minutes and $0.10. This is a billable provider-backed path, not Primr Zero.
Deep Mode¶
Deep mode uses the production Accordion path: Gemini Deep Research gathers a dossier with built-in Google Search, then Gemini Flash writes configured report sections sequentially before assembly. Before the deep phase, the ATS/careers hiring-signal stage runs and its fenced block is threaded into stage-1 context.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Deep Mode Pipeline β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Hiring Signals (ATS / careers) β fenced stage-1 context β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Deep Research Agent (Gemini) β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β 1. Autonomous Planning: Agent decides what to research β β
β β 2. Web Search: Built-in Google Search integration β β
β β 3. Page Analysis: Reads and synthesizes web content β β
β β 4. Citation Tracking: Automatic source attribution β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Sequential Report Expansion β
β (Dossier -> configured sections -> assembly) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
When to use: Broad market analysis, competitive intelligence, industry trends, when you need information beyond the company's own website.
Premium Accordion Path¶
The Deep/Premium path treats Deep Research as a lead researcher that gathers a dossier and Gemini Flash as the sequential report writer. It is intended for greater evidence breadth and section depth than a single Deep Research output, not for a guaranteed page count.
The Accordion design exists because one agent interaction has bounded output. Primr asks Deep Research for evidence and citations, then expands the report through the configured YAML section plan. Generated length still depends on evidence and model behavior and is not a product guarantee.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Complete Mode: Accordion Method β
β (Research -> Write Sections -> Assemble) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Phase 1: Data Collection
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Structured Pipeline β
β (Full website scraping + Google search + AI analysis) β
β β β
β βΌ β
β Context File Generation β
β (Baseline facts uploaded to File Search Store) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
Phase 2: Research Dossier
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Deep Research Agent β
β (deep-research-preview-04-2026) β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Role: Lead Researcher (NOT Writer) β β
β β Input: "Compile a research dossier, NOT the final report" β β
β β Output: Raw facts, data tables, citations, case studies β β
β β Access: File Search Store with Phase 1 context β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
β Deep Research gathers evidence; it does not own final length. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
Phase 3: Section-by-Section Writing
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Sequential Section Writer β
β (configured Gemini Flash model) β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β Process: Write one section at a time, sequentially β β
β β Context: Dossier + bounded excerpts of recent sections β β
β β Output: Evidence-dependent section prose β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β
β Section 1 βββΆ Section 2 βββΆ Section 3 βββΆ configured plan β
β β β β β β
β βββββββββββββββ΄ββββββββββββββ΄βββββββββββββββββββββββ β
β Context flows forward for consistency β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
Phase 4: Assembly (< 1 min)
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Report Assembler β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β - Combine sections with consistent formatting β β
β β - Generate table of contents β β
β β - Preserve section-local citations β β
β β - Apply clean header style (no PART headers) β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Why the Accordion Method?
A single Deep Research interaction has a bounded response. Asking one response to satisfy an arbitrary page target can trade evidence density for repetition and can compound unsupported claims.
The Accordion Method solves this by: 1. Expanding (research): Gather comprehensive facts 2. Structuring (plan): Use the configured YAML section order and contract 3. Expanding (writing): Write each section with the dossier and bounded recent context
Model Usage:
- deep-research-preview-04-2026: Autonomous web research (Phase 2)
- PrimrModels.FLASH_MODEL: Direct section generation (Phase 3)
Rate Limit Strategy: - Sequential section writes (not parallel) avoid 429 errors - Adaptive delays between sections, beginning from the configured safety floor - Adaptive backoff on rate limit detection
Hierarchy of Truth: 1. Company Facts (from Phase 1 scraping): highest authority 2. Research Dossier (from Phase 2 Deep Research): external context 3. Section Writing (Phase 3): synthesis with consistent voice
Core Components¶
Research Orchestrator¶
Location: src/primr/core/research_orchestrator.py
The central coordinator that routes research requests to the appropriate engine based on mode selection. Handles: - Mode dispatch (scrape, deep, complete) - Progress callbacks - Error aggregation - Metrics emission
orchestrator = ResearchOrchestrator()
result = await orchestrator.research(
"Acme Corp",
"https://acme.example",
mode=ResearchMode.COMPLETE
)
Core Module Structure¶
The src/primr/core/ directory contains the research orchestration logic, decomposed into focused modules:
| Module | Responsibility |
|---|---|
cli.py, cli_*.py |
CLI entry point, parsing, command dispatch, preflight, budget, and recovery handlers |
cli_contract.py |
Stable command enum and parsed configuration contract shared by CLI workflows |
research_agent.py |
Shared research entry point and fast/premium pipeline dispatch |
research_orchestrator.py |
Structured and premium Deep Research orchestration used by non-fast paths |
fast_run_*.py |
Ten extracted fast-pipeline stages coordinated by perform_fast_research |
stage_inventory.py |
Canonical capability and promotion requirements for production AI stages |
workspace.py |
Working folder creation, file consolidation, section output |
structured_research.py |
Website scraping pipeline, section-by-section analysis |
vendor_research.py |
Platform AI capabilities research (major providers) |
ai_strategy.py, strategy_*.py |
Strategy generation, prompt assembly, loops, and artifacts |
deep_research_runner.py |
Deep Research execution with preflight validation |
research_framing.py, context_curation.py |
Operator intent and bounded context preparation |
report_models.py |
Shared research and report data structures |
run_state_io.py |
Durable local run-state and event recording |
source_relevance.py |
Routed source selection, including the bounded host-agent pilot |
stage_route_comparison.py, stage_eval_scorecard.py |
Body-free route and quality evidence artifacts |
evidence_bundle_input.py |
Bounded, stable prep snapshots and source identity inputs |
local_draft.py |
Source-chunk mapping, bounded claim selection, and transactional local draft artifacts |
website_summary_eval_inputs.py |
Provider and prep-draft discovery with baseline provenance |
verification_summary.py |
Compact claim-verification summary shaping |
container.py |
Dependency injection |
The top-level cli_entry.py dispatches draft to the import-light
cli_draft.py before loading the operational cloud CLI. Explicit draft model
selection uses live installed runtime metadata and the existing local transport
seam, with proxies and redirects disabled. ai/openai_transport.py owns the
shared SDK client factory without importing the provider registry. Its source excerpt check and Word
renderer preserve inspectable evidence, but do not validate claim entailment.
local_stage_eval.py retains its public input/discovery imports while the
discovery owner carries baseline provenance. Local draft comparisons cannot
enter quality-evidence exports or model recommendations.
Utility compatibility exports resolve through their owning modules on first access, retaining type-checking imports and the eagerly bound public console singleton. The home path does not initialize unrelated async or sanitizer code.
The lightweight entry point also handles the bare home screen and explicit
--banner before cloud imports or credential validation. The banner is static
ANSI cyan with a dim tagline; legacy animated modes resolve to static output.
Routine operational commands do not render it. Estimate requests never enter
interactive key setup. Console progress, warnings, and failures use stderr;
results and structured JSON retain stdout. Live cursor updates follow stderr's
terminal capabilities and clear only a line actually written by progress.
Provider readiness in doctor uses one shared availability snapshot per provider,
including the existing zero-token local endpoint probe. A dummy local auth token
is request compatibility, not configuration or readiness evidence. Key states
and diagnostic details wrap at the terminal width after secret redaction.
Word report presentation preserves claim evidence labels without translating source counts or risk categories into confidence. The legacy confidence formatters accept only explicit Confirmed, Reported, Estimated, or Hypothesis labels; generic high/medium/low scores produce no marks. Recommendation priority uses words in navy. The palette is checked using the W3C relative-luminance contrast formula with a 4.5:1 floor for normal text on each actual report background, including muted metadata on white and alice-blue callouts. Tests inspect saved Word text and explicit run/background colors as well as inherited palette pairs.
The MCP cost-cap policy reads only its explicit enforcement override; server construction and serving do not publish process-global transport policy. The legacy scrape facade re-exports the canonical cache type and delegates cache operations to the orchestrator instead of allocating an unused second cache. Ranked links compare descending scores while retaining record equality and stable input order for distinct tied records. All comparison methods decline unrelated types. The sync bridge awaits through a coroutine wrapper so its Awaitable contract works on every maintained interpreter.
Fallback recovery composition owns fan-out and injects its HTTP callback into PDF and structured-data adapters. Their shared FallbackPage result and HttpGet protocol live in a leaf contract module; the default HTTP adapter delegates to the existing per-hop validated, IP-pinned safe HTTP transport. Recovery adapters never import their composition root. The legacy FallbackPage and HTTP import surfaces remain compatible, and the import-cycle baseline removes the retired three-module component. Fresh-process tests cover both adapter import orders.
Evaluation recipes and profile slots are immutable configuration contracts. The declarative candidate catalog imports only those contracts; the registry loads candidates and protects built-in profiles without loading model_eval, cost estimation, or model execution. The evaluation runner re-exports the same objects and registration functions for compatibility. Routing's recipe type references the leaf contract. Google dispatch and the legacy llm provider seam share the provider registry cache, so neither imports the other's dispatch module. Fresh-process tests verify catalog isolation and shared provider identity.
The fast-mode orchestrator (perform_fast_research) is decomposed into
per-stage modules (roadmap #23; see
design/23-orchestrator-refactor-map.md
for the detailed data-flow contracts). All planned extraction batches are
complete:
| Stage module | Pipeline stage |
|---|---|
fast_run_setup.py |
Model resolution, routing, run identity (frozen FastRunSetup) |
fast_run_collection.py |
Site corpus, external validation pools, and recovery executor |
fast_run_hiring.py |
Hiring-signals gathering + run-state recording |
insights_assembly.py |
Combined-insights / external-sources string assembly (pure) |
fast_run_gaps.py |
Gap analysis and targeted research deepening |
fast_run_workbook.py |
Analysis workbook; constructs the shared reasoning session |
fast_run_sections.py |
Section planning and writing orchestration with injected writer/coherence callbacks |
fast_run_validation.py |
Cross-validation, contradiction handling, and enrichment |
fast_run_trust.py |
Trust polish, citation repair, QA gate (frozen FastTrustResult) |
fast_run_strategy.py |
Strategy generation: budget checkpoint, per-vendor + YAML (frozen StrategyPhaseResult) |
fast_run_summary.py |
Final summary, artifact gating, usage recording |
Artifact assembly remains a deliberately small inline coordinator step rather than a separate stage module.
The section stage owns parallel dispatch, frozen prior-section snapshots, executive-summary ordering, recovery integration, and route accounting. The research coordinator supplies its section-writing and coherence helpers through typed callbacks. The stage does not import the coordinator, and the existing facade helper names remain available to callers and patch-based integrations.
Planned Epistemic Data Flow¶
After the v1.40 calibration decision, Primr will introduce a shadow run-scoped intermediate representation between analysis and prose:
retained evidence
|
v
source receipts + digest-bound evidence anchors
|
v
ResearchFinding records
observations and attributed source claims
|
v
StrategicInference records
explicit premises, assumptions, counterevidence, disconfirming signals
|
v
fixed report scaffold and strategy writers
|
v
shipping artifacts
The initial shadow ledger does not drive the writer or change report bytes. A
later promotion may generate bounded per-section evidence views and collect
used finding/inference IDs, while the curated company_overview.yaml section
structure stays fixed. Referential checks remain deterministic; whether a
finding supports an inference or the prose represents it faithfully remains
agreement-validated model or human judgment.
A small declarative ontology may compile node/relation endpoint legality, reference integrity, symmetry, and acyclicity into a semantic firewall. It is experimental and arrives only with the typed records. It is not an ontology over current report prose, an RDF stack, or a general graph engine.
Within a run, the native source index plus finding/inference JSONL artifacts are
the planned structured snapshot. Across runs, the 2.0 SQLite
claim/proposition store is canonical. OKF remains an export/interchange
projection. See
design/run-epistemic-ledger.md.
Prompt Architecture¶
The src/primr/prompts/ directory contains the externalized prompt system (v1.2.5+):
| Module | Responsibility |
|---|---|
composer.py |
PromptComposer class for building prompts from YAML |
loader.py |
YAML loading utilities and legacy prompt builders |
registry.py |
StrategyModuleRegistry for discovering strategy modules |
schema.py |
Dataclass definitions for prompt configs |
shared_loader.py |
SharedComponentLoader for epistemic rules, formatting, personas |
exceptions.py |
Custom exceptions for prompt configuration errors |
Strategy modules are YAML configs in src/primr/prompts/strategies/ that define different types of strategic analysis (AI, cloud migration, data strategy, etc.).
Each module exposes dataclasses and functions that can be imported directly:
# Direct imports from specialized modules (preferred for new code)
from primr.core.workspace import create_working_folder, WorkspaceConfig
from primr.core.ai_strategy import generate_ai_strategy_sync, CloudVendor
from primr.core.deep_research_runner import validate_preflight, DeepResearchConfig
from primr.core.cli import parse_args
from primr.core.cli_contract import CLIConfig
# Backward-compatible imports (still work, delegate to new modules)
from primr.core.research_agent import main, run_doctor, create_working_folder
Scraping Conceptual Model¶
Primr distinguishes between two levels of scraping:
| Level | Conceptual Name | Implementation | Input | Output |
|---|---|---|---|---|
| Primitive | scrape_page |
ScrapeOrchestrator.scrape_url() |
URL | content + tier + quality + errors |
| Workflow | build_site_corpus |
fetch_web_content() |
base domain | corpus + raw scrapes + external links |
Naming Rules (enforced in docs and code):
- scrape_page always refers to ONE URL (the primitive)
- build_site_corpus always refers to multi-page site workflow
- extract_insights always refers to corpus β structured facts compression (implemented by summarize_scraped_content())
- --mode scrape is "Corpus+Insights mode" (multi-page corpus, not one page)
- Never use "scrape" alone without clarifying page-level or site-level
Key Principle: There is ONE site-to-corpus workflow (build_site_corpus). All modes that need a corpus call this function. No other function should implement a site discovery + scrape loop.
9-Tier Scraping Engine (scrape_page primitive)¶
Location: src/primr/data/scraping/orchestrator.py
The scrape_page primitive uses a tiered fallback system for web scraping, designed for 2026 realities where most sites use JavaScript. Browser-first approach ensures reliable scraping of modern sites.
| Tier | Method | Use Case | Speed |
|---|---|---|---|
| 1 | Playwright | JS-rendered content (default) | Medium |
| 2 | Playwright Aggressive | Content expansion (accordions, lazy load) | Medium |
| 3 | Patchright | Persistent real-Chrome stealth for challenge shells | Slow |
| 4 | curl_cffi | TLS fingerprint impersonation | Fast |
| 5 | DrissionPage Stealth | Maximum stealth with challenge waiting | Slow |
| 6 | DrissionPage | Driverless browser via CDP | Slow |
| 7 | Vision | AI-based screenshot extraction | Slow |
| 8 | httpx | HTTP/2 sites, better headers | Fast |
| 9 | requests | Simple sites, no JS (fallback) | Fast |
Key Features:
- Sticky Tier: Once a tier works for a host, it's tried first for subsequent pages
- Circuit Breaker: Skips failing tiers after 3 consecutive failures per host
- Cookie Handoff: Browser-obtained cookies reused by faster HTTP tiers
- Soft Block Detection: Checks content, not just HTTP status (catches "200 OK" traps)
- Host Positive Marker Learning: Confirmed real first-party pages can persist a bounded, filtered marker set in user data and reuse it as positive evidence for later pages on the same host
- Blocked-Origin Summary: If live scraping and same-site recovery fail, the CLI reports sanitized evidence, recovery candidate count, and public-fallback next action before moving to Wayback, subdomains, EDGAR, and Wikipedia
- TLS Fingerprint Impersonation: curl_cffi mimics real browser TLS signatures
- Driverless Browsers: DrissionPage uses CDP directly, bypassing WebDriver detection
- Content-Type Routing: Automatic detection (HTML, PDF, binary) via headers and magic bytes - PDFs are extracted locally with PyMuPDF by default; Gemini PDF extraction is opt-in via PRIMR_PDF_LLM_MAX_CALLS
- Smart Tier Escalation (v1.2.4+): Stops after 3 consecutive failures of same error type to avoid wasting time on impossible pages
- Adaptive Timeout: 45s max per page (reduced to 25s when best_tier is known for the host)
- Headed Popup Budget: Opt-in counter (env PRIMR_MAX_HEADED_POPUPS, default 0) shared across the Patchright stealth tier and the orchestrator's adaptive Playwright retry. When unset no visible-browser windows ever open; set to N to allow up to N total popups for a single run. External-source validation uses a separate orchestrator that excludes Patchright entirely, so validation scrapes never trigger popups regardless of the budget. On Linux the budget is treated as 0 unless DISPLAY or WAYLAND_DISPLAY is set, so headless runs never attempt a visible launch.
Hiring-Signal Gathering (v1.19.0, expanded v1.27.0)¶
Location: src/primr/data/hiring_signals.py
Runs after the main-site scrape (fast mode) and before Phase 2 research deepening. Extracts strategic signals from a company's open job postings - often the most honest public statement of what they're building right now. Job postings are also the primary input to the skill pack subsystem (primr skills) - see Skill Pack Planning below.
Discovery chain:
1. Explicit career URLs - operator-supplied boards are intentional identity assertions and may merge segmented hiring sites.
2. First-party careers pages - probe the supplied website's careers paths and subdomains, reuse only company-site corpus entries, and follow hosted ATS links or redirects. Canonical Workday URLs in first-party corpus also resolve directly without blind guesses.
3. ATS guesses with identity checks - recon hints precede hostname and name variants, capped at six slugs. Ten public adapters cover Greenhouse, Lever, Ashby, SmartRecruiters, Workday, Workable, Recruitee, Jobvite, iCIMS, and BambooHR. hiring_identity.py requires a company-site posting URL, explicit board-level company homepage, or JobPosting.hiringOrganization.url / sameAs from a bounded posting sample. Employer metadata links only the individual role it describes; arbitrary customer or tool links never promote a board. It ranks only verified boards, keeping unrelated matches from masking a valid tenant. Workday blind discovery retains its existing bounded probe budget.
4. DuckDuckGo search fallback - when the earlier chain yields no postings, search known job-board hosts. Each result must independently pass the employer metadata check before its metadata can contribute roles or counts.
5. Identity receipts - accepted postings carry identity: verified and identity_evidence. Prep excludes unverified rows from its source index and reports identity-linked counts. The verified status records an association basis, including operator assertions and self-reported provider metadata; it does not independently establish that every posting claim is true. first_party_url.py owns normalized host-tree matching and link parsing; sibling brand domains require explicit operator career URLs rather than inferred ownership.
6. LLM triage - small Grok call picks up to 15 signal-rich postings (biased toward senior / engineering / product / data / security / platform roles; down-weights retail / sales SDR / entry-level). Deterministic title-based ranker as fallback when the LLM call fails.
7. Body fetch - ATS postings usually include the body inline. HTML postings fetched in parallel via the external orchestrator.
8. Batched LLM extraction - one Grok reasoning call over the aggregated JD text produces structured JSON: roles & locations, tech-stack frequency, strategic initiatives, culture signals, notable absences, hiring volume, summary.
Outputs:
- <working>/_hiring/hiring_signals.md - human-readable summary
- <working>/_hiring/hiring_signals.json - structured extraction
- <working>/_hiring/postings_index.json - full discovered list before triage
- <working>/_hiring/raw/jd_NNN_*.txt - individual JD bodies with metadata
Wikipedia fallback identity lives in data/wikipedia_identity.py. A bounded
MediaWiki search resolves article redirects and pageprops.wikibase_item, then
checks current Wikidata P856 official website claims against the supplied site
tree. Missing, deprecated, ended, and mismatched claims are omitted. Provider
contracts: MediaWiki page properties,
Wikidata entity API,
official website property,
Greenhouse public boards,
Lever postings, and
JobPosting employer identity. Every added fetch uses
the existing SSRF-safe HTTP seam; identity metadata never authorizes egress.
Integration: the extracted signals are rendered via render_for_prompt into a === HIRING SIGNALS === block and appended to insights.txt plus the raw external-sources bundle. The Phase 2 gap-filling rebuild preserves this block so every downstream phase (workbook, section writing, cross-validation, Phase 6 strategy) can see it.
Fail-open at every stage. No ATS match + no careers page + no web-search hits β source: none, run continues unchanged. Companies that don't publish jobs produce reports as if the phase never ran. Skip entirely with PRIMR_SKIP_HIRING_SIGNALS=1.
Skill Pack Planning (v1.27.0)¶
Location: src/primr/skill_pack/planner.py, industry.py, discovery.py
Job postings are the primary input to the skill pack subsystem; operator-supplied job descriptions / role briefs (--from-jd) are treated as explicit hiring evidence when the operator has a better role artifact than discovery can find. DNS recon and the strategic report are supporting context. The planning step replaces the single-call discover_roles with a structured two-call plan that preserves provenance end-to-end. It also records a non-blocking posting-coverage assessment so enterprise-scale rosters that only see one narrow posting band are marked posting-incomplete rather than treated as complete coverage.
Pipeline:
1. Evidence load - recon (_recon_context.txt), hiring (_hiring/hiring_signals.md plus optional _hiring/operator_role_brief.md from --from-jd), research (insights.txt / report.md / analysis_workbook.md). Fails closed when posting / role-brief evidence and research evidence are empty unless allow_recon_only=True.
2. Industry classification - LLM-only resolution (no heuristics): parse structured fields from a primr strategic report when one is supplied via --from-report, otherwise one cheap LLM call against the evidence inputs. Produces IndustryClassification with business_model / industry_vertical / company_stage / employee_estimate / confidence / cited_evidence / source.
3. Call A - observed roles - LLM extracts roles from the hiring evidence only. Operator role briefs are prepended to the hiring stream and treated as evidence, never instructions. Every role MUST carry at least one verbatim posting or role-brief citation or it's dropped at parse time. Provenance: posting. Confidence: Confirmed.
4. Call B - plausible roles - LLM infers roles from recon + research + the industry classification + the Call A output (to exclude duplicates). Every role MUST carry at least one specific research citation OR an explicit business-model + stage rationale. Common org-shape roles (Marketing, Sales, Customer Success, Finance, HR) become plausible only when company stage is Mid-market or larger. Generic VP / Chief-X titles are forbidden without specific evidence. Provenance: research or industry. Confidence: Inferred or Speculated.
5. Merge and cap - archetype-based dedupe with observed-wins; signal-driven split with no hard ratio; cap at roles_count; overflow goes to gap_flagged. Archetype matching favors exact slugs, normalized aliases, strong display-name matches, and multi-keyword evidence; weak display-name guesses return no archetype so authoring does not inherit the wrong role family.
6. Persist - writes <working>/role_plan.md (human view) and <working>/role_plan.json (machine view, used by --from-plan).
Operator surface (roster curation):
- --plan-only writes the plan and exits before authoring.
- --from-plan PATH skips planning and authors against a saved plan's final_roster verbatim.
- --from-jd PATH sanitizes a local JD / role brief into _hiring/operator_role_brief.md; it can augment a normal run or act as the sole evidence source for a JD-only draft skill pack.
- posting-incomplete is a visibility signal, not a ship block: role planning preserves the observed postings, then points the operator toward repeatable --career-url, --from-jd, --roles-add, --roles-override, or richer report evidence when the discovered posting slice is too narrow for the organization's scale.
- --roles-add "A, B" augments the discovered or saved-plan roster with operator-supplied labels (materialized as provenance: override).
- --roles-skip "X, Y" removes named roles from the discovered or saved-plan roster (matches display_name or kebab-case slug, exact, case-insensitive).
- --roles-override "A, B, ..." bypasses planning entirely; up to MAX_ROLES labels. Mutually exclusive with --roles-add / --roles-skip (override wins, curation warned).
- --allow-recon-only opts in to the degraded path when both posting and research evidence are empty.
Cap-aware merge with operator priority: when curation pushes the roster over MAX_ROLES, plausible roles trim first, then observed, then never operator-added. Trimmed entries flow to gap_flagged for plan-artifact transparency. Name + archetype dedupe between added and discovered roles: existing role wins (preserves citations); operator can force a specific variant with --roles-skip + --roles-add in one shot. Empty roster after curation is a hard error.
Authoring (author_role_skills) branches the prompt on RoleEvidence.provenance: posting-grounded roles emphasize "anchor every skill in the specific responsibilities the postings name"; research-inferred roles emphasize "this role isn't in posting data but is plausible because of [citations]; reference the named practice"; industry-inferred roles emphasize "this role reflects business-model typicality, tuned to the company's named stack where possible"; operator-supplied roles pass through.
Link Discovery (Homepage-First, v1.1.0)¶
Part of the build_site_corpus workflow:
1. Render homepage with Playwright (browser-first for JS-heavy sites)
2. Extract navigation links (current and navigable)
3. Expand section pages (news, blog, press, resources) to get actual articles
4. Common URL guessing only if < 20 links found
5. Sitemap only as fallback if still < 20 links (sitemaps often stale)
6. LLM selects most valuable pages for research (leadership, products, news, investors)
Scope Policy:
- IN-SCOPE: same domain + subdomains (always scraped)
- OUT-OF-SCOPE: external domains (recorded to _external_links.txt, not scraped in scrape mode)
Deep Research Client¶
Location: src/primr/ai/deep_research.py
Integration with Gemini's Deep Research Agent, which autonomously plans and executes multi-step research.
Key features: - Adaptive polling (faster initially, slower as research progresses) - Job persistence for recovery after interruption - Thinking log capture for transparency - Pre-flight validation before expensive API calls
client = DeepResearchClient()
result = await client.research(
"Research Acme Corp's competitive position",
output_format="company_profile",
priority_urls=["https://acme.example"]
)
Compatibility Chapter-Plan APIs¶
Locations: src/primr/ai/report_architect.py,
src/primr/ai/research_executor.py, and
src/primr/ai/report_aggregator.py.
These importable APIs implement JSON chapter planning, node execution, and aggregation. Primr retains them as compatibility and test surfaces, but the active Standard and Premium callers listed above do not use them. New production work must not attach to this stack until its compatibility inventory reaches an explicit keep or deprecate decision.
Its fallback chapter structure is: 1. Executive Summary & Company Snapshot 2. Products, Services & Value Proposition 3. Leadership, Culture & Organization 4. Financial Position & Business Model 5. Target Markets & Customer Segments 6. Competitive Landscape & Market Position 7. Industry Dynamics & External Forces 8. SWOT Analysis & Strategic Assessment 9. Risk Analysis & Mitigation Strategies 10. Strategic Recommendations & Discovery Questions
Quality Grading Agent¶
Location: src/primr/ai/grading_agent.py
Grades each section of a report on a 0-100 scale based on: - Clarity & Readability - Completeness - Insight Depth - Accuracy (vs. scraped website data)
Sections scoring below the threshold (default: 70) trigger additional research refinement.
AI Client¶
Location: src/primr/ai/client.py
Unified interface for all LLM operations with: - Automatic retry with exponential backoff - Model fallback chains - Token usage tracking - Quota exhaustion detection (stops immediately, doesn't retry)
client = AIClient()
response = client.generate(
"Analyze this company",
model_type="research",
thinking_level="high"
)
print(client.get_usage_summary())
Continuous Reasoning Session (default-on)¶
Location: src/primr/ai/grok_client.py (class ContinuousReasoningSession)
Multi-turn Grok session that preserves message history across pipeline stages, so workbook generation (Phase 3) and cross-validation (Phase 5) share working memory instead of each starting from a serialized summary read off disk. Sits alongside the stateless grok_llm helper and reuses the same retry, error-classification, and module-level token-tracking machinery. On by default after the n=3 pilot; pass --no-continuous-reasoning (or set PRIMR_CONTINUOUS_REASONING=0) to revert to the fresh-call topology.
from primr.ai.grok_client import ContinuousReasoningSession
session = ContinuousReasoningSession(
model="grok-4.7",
system_prompt="You are a senior strategic analyst...",
)
workbook = session.send(workbook_prompt, max_tokens=18_000, temperature=0.5)
# ... section writing happens out-of-session (parallel + fresh-call) ...
cross_val = session.send(cross_val_prompt, max_tokens=5_000, temperature=0.2)
# Session retains the corpus + workbook reasoning, so the validator
# can verify the report against the workbook's mandate, not just against URLs.
When the session is constructed. The CLI flag (or env var) is resolved during run setup (fast_run_setup.resolve_fast_run_setup), but the session itself is constructed lazily at the workbook stage (fast_run_workbook.generate_analysis_workbook, which returns it for cross-validation reuse). That lets the workbook's system prompt be passed as a real role:system message at session init - Grok rejects mid-conversation system messages, so this placement matters. An earlier implementation that folded the system prompt into the first user turn measurably degraded workbook quality during the pilot; the lazy construction is the fix.
What stays unchanged. Section writing (Phase 4) is intentionally untouched and remains parallel + fresh-call per section via the existing ThreadPoolExecutor(max_workers=4) pattern. The topology change is targeted at sequential reasoning handoffs, not parallel sub-agents. Strategy generation (Phase 6) and gap analysis (Phase 2) also remain fresh-call.
Status. On by default. n=3 paired-comparison pilot showed measurably better workbooks (3/3 wins on a blind LLM judge), richer cross-validation in 2/3 cases, and ~81% fewer leaked-instruction lines in final reports, at an average ~+12% cost (range β3.7% to +32%). See ROADMAP "Continuous Reasoning Session" entry for the full pilot writeup and the rationale for the default flip.
Data Flow¶
Scrape Mode Data Flow¶
URL Input
β
βΌ
βββββββββββββββββββ
β URL Validation β
β & Normalizationβ
βββββββββββββββββββ
β
βΌ
βββββββββββββββββββ βββββββββββββββββββ
β Cache Check ββββββΆβ Return Cached β
β (LRU + Disk) β β Content β
βββββββββββββββββββ βββββββββββββββββββ
β (miss)
βΌ
βββββββββββββββββββ
β Tier 1-4 β
β Scraping β
βββββββββββββββββββ
β
βΌ
βββββββββββββββββββ
β Content β
β Extraction β
β (BeautifulSoup)β
βββββββββββββββββββ
β
βΌ
βββββββββββββββββββ
β Soft Block β
β Detection β
βββββββββββββββββββ
β
βΌ
βββββββββββββββββββ
β Cache Write β
β (LRU + Disk) β
βββββββββββββββββββ
β
βΌ
βββββββββββββββββββ
β Link Discovery β
β & Scoring β
βββββββββββββββββββ
β
βΌ
βββββββββββββββββββ
β Section β
β Extraction β
β (AI-powered) β
βββββββββββββββββββ
β
βΌ
βββββββββββββββββββ
β Quality β
β Grading β
βββββββββββββββββββ
β
βΌ
Section Results
Complete Mode Data Flow (Accordion Method)¶
Company Name + URL
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Phase 1: Structured Pipeline (Data Collection) β
β βββββββββββββββ βββββββββββββββ βββββββββββββββ β
β β Scraping βββΆβ Section βββΆβ Context β β
β β Engine β β Extraction β β File Gen β β
β βββββββββββββββ βββββββββββββββ βββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β File Search Store Upload β
β (Baseline facts accessible to Deep Research) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Phase 2: Research Dossier (ONE Deep Research call) β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β deep-research-preview-04-2026 β β
β β Role: Lead Researcher (gather facts, NOT write report) β β
β β Output: Raw facts, data tables, citations β β
β β Returns: dossier text and source metadata β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Phase 3: Sequential Section Writing β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β configured Gemini Flash model, direct generation β β
β β β β
β β Section 1 βββΆ Section 2 βββΆ Section 3 βββΆ YAML plan β β
β β β β β β β
β β βββββββββββββββ΄ββββββββββββββ β β
β β Recent bounded excerpts flow forward β β
β β β β
β β Length: evidence-dependent β β
β β Delay: adaptive rate-limit pacing β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Phase 4: Report Assembly β
β βββββββββββββββ βββββββββββββββ βββββββββββββββ β
β β Section βββΆβ TOC βββΆβ Citation β β
β β Combine β β Generation β β Preserve β β
β βββββββββββββββ βββββββββββββββ βββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Report Generation β
β βββββββββββββββ βββββββββββββββ βββββββββββββββ β
β β TXT β β DOCX β β PDF β β
β βββββββββββββββ βββββββββββββββ βββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Module Structure¶
src/primr/
βββ __init__.py, __main__.py # Package and `python -m primr` entry points
βββ types.py # Transport-neutral result and protocol types
βββ job_status.py # Versioned cross-transport lifecycle snapshots
βββ primr_cli.py # Legacy module wrapper; delegates to cli_entry
β
βββ core/ # CLI and research/strategy orchestration
β βββ cli.py, cli_*.py # Command parsing, dispatch, preflight, and budgets
β βββ cli_contract.py # Shared command and parsed-configuration contract
β βββ research_agent.py # Shared run entry and mode dispatch
β βββ research_orchestrator.py # Structured and premium orchestration
β βββ fast_run_*.py # Ten extracted fast-pipeline stages
β βββ stage_inventory.py # Production stage capability requirements
β βββ strategy_*.py # Strategy prompt, loop, generation, and artifacts
β βββ workspace.py, run_state_io.py # Workspace and durable run state
β
βββ ai/ # Inference, providers, routing, and Deep Research
β βββ providers/ # xAI, Gemini, OpenAI-compatible/OpenRouter, Anthropic, Bedrock (boto3 converse), Azure Foundry (OpenAI-SDK) adapters
β βββ routing.py # Legacy role-to-provider routing
β βββ capability_routing.py, stage_routing.py # Stage capability router
β βββ host_agent_runner.py, host_agent_cli.py # Official host-runner seam
β βββ local_inference.py # Local OpenAI-compatible detection and calls
β βββ deep_research*.py # Deep Research client, parsing, polling, execution
β βββ client.py, llm.py # Current and legacy model-call seams
β
βββ data/ # Collection, discovery, hiring, and fallback evidence
β βββ scraping/ # Nine-tier scrape primitive and trace support
β βββ scrape.py # Multi-page site-to-corpus workflow
β βββ hiring_*.py # ATS discovery, extraction, routing, and artifacts
β βββ fallback_sources.py # Public-data fallback fan-out
β βββ search_utils.py # External search seam
β
βββ pipeline/ # Recovery, retry, failover, and model breakers
βββ prompts/ # YAML prompt composition and strategy registry
β βββ shared/ # Epistemic, formatting, and persona components
β βββ strategies/ # Discoverable strategy definitions
β βββ skill_pack/ # Skill-pack planning and evaluation prompts
βββ output/ # MD/TXT/DOCX/PDF rendering and artifact validation
βββ qa/ # Report analysis, calibration, honesty, and scoring
β βββ calibration_runner.py # One report snapshot shared by cloud/local comparison
β βββ calibration_sidecars.py # Sidecar snapshot reads, fingerprints, and report binding
β βββ calibration_judge_agreement.py # Claim-aligned, decidable verdict comparison
βββ agentic/ # Memory, hypotheses, hooks, and research subagents
β βββ subagents/ # Scraper, analyst, writer, QA, and verifier roles
βββ skill_pack/ # Role planning, skill authoring, validation, packaging
βββ config/ # Environment, settings, models, sections, eval profiles
β βββ model_config.py # Shared model metadata, pricing, and provider retirement checks
β βββ model_registry.py # Direct-provider registrations; re-exports ModelConfig
β βββ gemini_models.py # Direct Gemini catalog, history, and dated prices/retirements
β βββ openai_models.py # Direct OpenAI catalog and standard context tiers
β βββ anthropic_models.py # Direct Claude catalog, lifecycle and shared request capabilities
β βββ bedrock_models.py # AWS-verified global and geo Converse candidates
β βββ openrouter_models.py # OpenRouter registrations and published route retirements
βββ utils/ # Shared async, I/O, security, console, and logging seams
β βββ windows_job.py # Owned Windows process trees and native memory limits
β βββ errors/ # Typed error hierarchy and retry helpers
β
βββ mcp_server/ # MCP tools, resources, jobs, approval, and audit policy
β βββ doctor_status.py # Transport-neutral configuration health
β βββ job_process.py # Parent-owned local worker supervisor
β βββ job_process_types.py # Retained handles and cancellation results
β βββ job_worker.py # One-job Python child entry point
β βββ worker_environment.py # Least-privilege worker environment
β βββ worker_protocol.py # Strict versioned JSONL lifecycle contract
β βββ worker_process_control.py # Cross-platform tree signaling
β βββ worker_terminal_policy.py # Exit and terminal-state compatibility
β βββ job_terminal_manifest.py # Atomic worker-exit audit manifest
β βββ job_tools.py # Cancellation authorization and response
β βββ qa_operations.py # Transport-neutral existing-report QA
β βββ research_validation.py # Research execution-shape validation
β βββ server_context.py # Acyclic structural controller contract
β βββ strategy_catalog.py # Standalone strategy dispatch and cost catalog
β βββ strategy_operations.py # Transport-neutral strategy generation
β βββ job_store.py # Journal, timestamps, and controller lease
βββ a2a/ # A2A facade over governed MCP pipeline services
β βββ call_context.py # Trusted loopback and authenticated task ownership
β βββ cancellation.py # Owned-task cancellation workflow
β βββ lifecycle_events.py # Progress and exactly-once terminal ordering
β βββ status_events.py # Shared task-status event construction
β βββ task_store.py # Exact-owner SDK task retrieval and job mapping
βββ api/ # REST scaffold; research submission is not production-wired
This is a concern-level map, not an exhaustive listing of every helper module.
mcp_server.server_context.MCPServerContext is the structural boundary for
shared MCP and A2A consumers. Only mcp_server/cli.py and a2a/cli.py import
the concrete server to construct it. Health, QA, and strategy operations live
in dependency-light leaf modules so transport modules do not route through the
research pipeline merely to reuse one operation.
utils.atomic_io owns atomic replacement and bounded advisory mutation locks
for cooperating local processes. Pending-job persistence and usage-history
migration/saves use the same persistent sibling-lock protocol. Usage saves
reload disk state under the lock before appending unflushed session records;
atomic replacement alone cannot preserve another writer's committed records.
Usage history lives in the durable user data directory and retains original
legacy bytes when importing older installation-local histories. See
CONFIG.md for storage overrides and lock limitations.
The detailed fast-stage contracts live in
design/23-orchestrator-refactor-map.md,
and the capability-routing inventory lives in
design/2.0-backend-freedom.md.
Error Handling Strategy¶
Retry with Backoff¶
All AI operations use exponential backoff with jitter: - Base delay: 1 second - Multiplier: 2x per attempt - Max delay: 60 seconds for rate limits - Jitter: random variation to avoid thundering herd
Quota Exhaustion¶
Daily API quota exhaustion is detected and handled specially: - Immediate stop (no retries) - Clear error message with recovery instructions - Suggestion to check quota status
Graceful Degradation¶
| Component | Failure Mode | Fallback |
|---|---|---|
| Tier 1 scraping | HTTP error | Try Tier 2 |
| Tier 2 scraping | Connection error | Try Tier 3 |
| Tier 3 scraping | Timeout | Try Tier 4 |
| Tier 4 scraping | All failed | Return partial content |
| Chapter planning | API error | Use default chapters |
| Chapter execution | Timeout | Mark chapter failed, continue |
| Grading | API error | Skip grading, use content as-is |
Pipeline Resilience Layer¶
Location: src/primr/pipeline/
The pipeline resilience layer formalizes Primr's retry and recovery logic into three interlocking subsystems. Instead of ad-hoc retry loops scattered across AI clients, every pipeline stage declares a cost-ordered recovery hierarchy - a sequence of actions ranked cheapest-first (e.g., retry β fallback model β skip). A stage classifier labels each stage as foreground (must complete) or background (bail on API overload or budget stress), so background stages like cross-validation and strategy generation never amplify capacity cascades during batch runs. A model circuit breaker tracks consecutive API failures per model and automatically routes to fallback models after 3 failures, with recovery probes after 10 minutes.
The resilience layer sits between the pipeline orchestrator (research_agent.py) and the AI clients (grok_client.py, llm.py). It shares no mutable global state and is fully unit-testable. On successful runs, it adds no observable behavior change (NFR 1).
- Recovery Table (
recovery.py): Declarative mapping from each of the six pipeline stages to its recovery hierarchy. Pure data, serializable to JSON. Dry-run shows a one-line stage count by default;--verboselists actions and the full serialized table. - Stage Classifier (
stages.py): Static foreground/background classification. Foreground: scraping, external search, analysis, section writing. Background: cross-validation, strategy generation. - Model Circuit Breaker (
model_breaker.py): Per-model health tracking with provider-aware fallback chains (e.g., Grok 4.7 β Grok 4.3 β Claude Sonnet 5.5 β GPT-6.1 Terra β Gemini Flash). Verifies API key availability before cross-provider fallback. - Recovery Executor (
executor.py): Integration glue that wraps stage callables, consults the classifier and recovery table on failure, and logs all recovery events to_run_state.json. - Integration Helpers (
integration.py): Thin wrappers connecting the executor to each pipeline stage at the appropriate granularity (per-page for scraping, per-section for writing, per-stage for analysis).
Run primr --dry-run <company> <url> to inspect the recovery table and stage classifications without executing any research.
Execution Ownership and Cancellation¶
Cancelling an asyncio task does not terminate blocking work that has already
started in a thread. A terminal cancelled state is therefore truthful only
after the system has observed that the worker it owns has exited. A remote
provider task may remain non-interruptible unless that provider exposes and
accepts a cancellation operation.
Local MCP and A2A research jobs run in one supervised Python child process per
job. The controller retains the worker handle, validates a strict 1 MiB JSONL
event stream, requests cooperative stop, waits for a bounded interval, then
terminates the owned process tree if necessary. The child must join that
ownership boundary and emit ready before the control surface accepts the run.
The worker receives an explicit research-provider and runtime environment
allowlist. Controller authentication, approval, cloud-identity, telemetry, and
CI credentials are removed. The supervised .env loader parses raw values
without interpolation and rejects interpolation-bearing assignments, so a
blocked value cannot be copied into an allowed provider variable. Before
pipeline imports, the child duplicates the control and event
pipes into non-inheritable private descriptors, replaces ordinary stdin with
DEVNULL, and routes ordinary stdout to the worker log. Native stdout writes
therefore cannot corrupt JSONL, and normal exec-based descendants cannot retain
the private protocol pipes.
The parent is the only writer of canonical job state. It validates the complete snapshot schema, types, ranges, job binding, and event sequence. Worker clock values do not define canonical stage, heartbeat, or completion timestamps: the parent records its own observation time, and terminal snapshots remain provisional until process exit is observed. Terminal states are immutable and repeated cancellation is idempotent.
Exactly one controller may own a journal. MCP, co-hosted A2A, and standalone
A2A enter the same reference-counted controller lifecycle, which acquires an
OS-backed exclusive lease and runs one bounded atomic-write preflight across
the journal, audit, and output persistence boundaries. It securely opens and
reads the actual audit sink before strictly reloading the journal and performing
restart reconciliation. Corrupt journals and unavailable persistence fail
activation without deleting the diagnostic file. Runtime journal or required
worker-artifact write failures close readiness and journal writes roll memory
back to the last durable state. HTTP /healthz remains
a shallow liveness probe; /readyz returns ready only while the lifecycle,
lease, persistence, and audit checks are ready, and otherwise returns body-safe
component states. The process-local control plane must run as exactly one
persistent replica. On final shutdown,
the shielded lifecycle refuses new starts and runs bounded cooperative,
terminate, and kill phases. It releases the lease only after every retained
worker is reaped and descendant-tree cleanup is confirmed. If OS termination
or tree cleanup cannot be confirmed, shutdown fails loudly and keeps both the
worker handle and lease instead of allowing another controller to reconcile a
possibly live worker. Interrupted journals with no retained worker reconcile
to failed/server_restart only after a later controller acquires the lease.
Local no-auth A2A is permitted only on an explicit loopback listener. The
server installs an internal request marker that remote JWT subjects cannot
select. That marker can read only jobs owned by the local a2a identity;
authenticated A2A uses the exact non-reserved token subject. SDK tasks/get
applies the same exact-owner rule and returns the same not-found result for a
missing task and a task owned by someone else.
POSIX workers start in a new session and use process-group signals. Linux also
sets PR_SET_PDEATHSIG during the bootstrap window, then transfers parent-loss
handling to the private control-pipe reader so EOF can kill the whole process
group. Windows uses a named Job Object with KILL_ON_JOB_CLOSE. The POSIX
parent-loss path is best effort when native code holds the GIL long enough to
starve the reader or a descendant deliberately escapes with setsid() or a new
process group; Primr never converts an unconfirmed exit into a terminal claim.
When a retained worker exits as failed or cancelled, the supervisor atomically
writes a worker-exit manifest with the observed exit method. Spawn failures and
restart reconciliation remain journal-only. Provider-side work is recorded as
unknown when the provider offers no cancellation confirmation. Hosted one-job
containers should converge on this same protocol rather than maintain a second
lifecycle vocabulary. See
design/runtime-language-boundaries.md.
Security Architecture¶
Primr underwent comprehensive security review in January 2026. All critical vulnerabilities have been addressed.
Security Principles¶
- Defense in Depth: Multiple layers of validation and protection
- Fail Secure: Invalid inputs rejected, not processed
- Least Privilege: Minimal permissions and access
- Input Validation: All external inputs validated before use
Implemented Protections¶
SSRF (Server-Side Request Forgery) Protection¶
Location: src/primr/utils/validators.py and src/primr/utils/url_security.py
All HTTP requests are validated before execution to prevent internal network access:
def validate_url_for_request(url: str, allow_private_ips: bool = False) -> tuple[bool, str, str | None]:
"""
Validate URL for making external HTTP requests (SSRF protection).
Blocks:
- Internal/private IP addresses (localhost, 10.x, 192.168.x, 169.254.x, 172.16-31.x)
- Loopback addresses (127.x.x.x, ::1)
- Link-local addresses (169.254.x.x, fe80::/10)
- Non-HTTP schemes (file://, ftp://, etc.)
- Invalid URLs
- Hostnames that resolve to private IPs (DNS rebinding protection)
"""
The higher-level scraping tiers call validate_url_for_request() before
network access. Shared fail-open and archived-content recovery helpers use
src/primr/data/safe_http.py:safe_http_get(), while Google grounding citation
resolution uses async_safe_http_head(). Both safe HTTP helpers follow
redirects manually, resolve each hop once, validate every returned address, and
connect to the validated IP literal while preserving the original Host header
and HTTPS SNI. Discovery helpers keep their requests.Response contract and
manually revalidate each redirect hop. The pooled HTTPClient does the same
for GET/HEAD while preserving session and retry behavior. The requests, httpx,
and curl_cffi scraping tiers also follow redirects manually so each tier
validates redirect targets before connecting while preserving its own transport
behavior. The httpx scraping tier connects to the validated IP literal with
original Host and HTTPS SNI. Requests-family egress uses
data.pinned_requests.PinnedHTTPAdapter, which lets pooled HTTPClient
requests and the tiered requests scraper connect through urllib3 to the
validated IP literal while keeping the logical request URL, original Host, and
HTTPS SNI. The curl_cffi scraper tier passes the vetted per-hop address to
libcurl with CurlOpt.RESOLVE, keeps the logical URL so TLS fingerprint
impersonation, Host, and SNI stay aligned, and disables environment proxy trust.
Browser-backed Chromium tiers use data.scraping.browser_egress to translate
the same validated connection artifact into Chromium host-resolver rules for
the initial hostname. They also launch through data.scraping.browser_proxy, a
loopback HTTP proxy that validates each browser-discovered HTTP request or HTTPS
CONNECT target, dials the validated IP literal, and tunnels bytes without
terminating TLS. Chromium is launched with loopback proxy bypass disabled and
QUIC disabled so browser traffic stays on the TCP proxy path. Playwright,
Playwright aggressive, vision, and Patchright also install a request route
guard before navigation so unsafe browser requests are aborted before Playwright
or Patchright continues them. DrissionPage receives the same proxy and
initial-host resolver controls through Chromium startup args.
Protected Functions and Seams:
- src/primr/data/safe_http.py: safe_http_get() for fallback, hiring, and Wayback CDX/replay fetches; async_safe_http_head() for citation redirect resolution
- src/primr/data/pinned_requests.py: PinnedHTTPAdapter for requests-family validated-IP connection pinning
- src/primr/data/http_client.py: HTTPClient.get() and HTTPClient.head()
- src/primr/data/scraping/wayback.py: _fetch() delegates to safe_http_get()
- src/primr/data/scraping/net.py: make_request() and head_exists() for sitemap and URL-existence checks
- src/primr/data/scraping/http_clients.py: scrape_with_requests(), scrape_with_httpx(), scrape_with_curl_cffi()
- src/primr/data/scraping/browser_egress.py: Chromium resolver-rule planning and Playwright-compatible request route guard
- src/primr/data/scraping/browser_proxy.py: local loopback HTTP/CONNECT proxy for dynamic browser-discovered host pinning
- src/primr/data/scraping/browsers.py: scrape_with_playwright(), scrape_with_playwright_aggressive(), scrape_with_drissionpage(), scrape_with_drissionpage_stealth()
- src/primr/data/scraping/vision_browser.py: scrape_with_vision()
- src/primr/data/scraping/stealth_browser.py: scrape_with_patchright() (Kasada / Akamai bypass tier)
XXE (XML External Entity) Protection¶
Location: src/primr/data/scraping/discovery.py
XML parsing uses secure parser with entity expansion disabled:
parser = ET.XMLParser()
parser.entity = {} # Disable entity expansion
parser.parser.SetParamEntityParsing(0) # Disable parameter entities
root = ET.fromstring(content, parser=parser)
Path Traversal Protection¶
Location: src/primr/utils/validators.py
File paths validated to prevent directory traversal:
def validate_file_path(path: str, base_dir: str | None = None) -> tuple[bool, str]:
"""
Validate file path to prevent directory traversal attacks.
Checks for:
- Parent directory references (..)
- Absolute paths outside base directory
- Invalid characters
"""
SQL Injection Protection¶
All database queries use parameterized statements:
# Safe: Uses ? placeholders
cursor.execute("SELECT * FROM cache WHERE url = ?", (url,))
# Never: String concatenation
# cursor.execute(f"SELECT * FROM cache WHERE url = '{url}'") # UNSAFE
Secure Hashing¶
MD5 usage explicitly marked as non-security:
# Safe: MD5 for cache keys only, not security
hashlib.md5(url.encode(), usedforsecurity=False).hexdigest()
Security Testing¶
Location: tests/security/
The security suite is split by behavior rather than held in one compatibility
module. It covers SSRF and redirect-hop egress controls, XXE, path traversal,
input validation, prompt-injection fixtures, and invariant properties. The
legacy tests/test_security.py file is only a compatibility placeholder and
does not collect the suite.
Run the current suite with:
Automated Security Scanning¶
Bandit (Python Security Linter)¶
Configuration: .bandit
CI runs Bandit against src/primr at medium severity and confidence. Run the
same check locally with:
uv run --no-sync bandit -r src/primr -c .bandit --severity-level medium --confidence-level medium -q
Dependency Audit¶
CI and the scheduled dependency-audit workflow run pip-audit against the
locked environment. Current workflow status, not a dated prose snapshot, is
the security result. Run the same audit locally with:
Security Best Practices¶
- API Keys: Never hardcoded, always from environment variables
- Secrets:
.env.exampleprovided without actual secrets - YAML Loading: Always uses
yaml.safe_load(), neveryaml.load() - File Operations: Proper encoding, no unsafe file handling
- Command Execution: No
shell=True, no unsafe subprocess calls - Error Messages: No system information leakage
Security Documentation¶
Current policy and controls: docs/SECURITY.md
Historical audit snapshot: docs/SECURITY_REVIEW_2026-01-21.md
Includes: - Vulnerability findings and fixes - Security test coverage - Automated scanning results - Production readiness assessment - Ongoing security recommendations
Caching Strategy¶
Memory Cache (LRU)¶
- Max size: 100 entries
- Thread-safe with locking
- Evicts oldest entries when full
Disk Cache¶
- Location:
logs/scrape_cache/ - TTL: 24 hours
- Format:
.txtcontent +.metaJSON metadata - Checked after memory cache miss
Cache Key Generation¶
URLs are normalized and hashed to create cache keys: 1. Remove fragments (#) 2. Normalize path (remove trailing slashes) 3. Hash with SHA-256
Observability¶
Structured Logging¶
All components use the unified logger from utils/logging_config.py:
- Module-specific loggers (ai.client, data.scrape, etc.)
- Configurable log levels
- Structured context (correlation IDs, operation names)
Metrics¶
Research operations emit metrics via utils/observability.py:
- Operation duration
- Success/failure status
- Section counts
- Citation counts
- Error types
Token Usage Tracking¶
The AI client tracks token usage for cost monitoring: - Input tokens per call - Output tokens per call - Cumulative totals - Estimated cost calculation
Testing Strategy¶
Unit Tests¶
Location: tests/
- Mirror source structure (
test_ai/,test_data/, etc.) - Mock external dependencies (API calls, file system)
- Focus on edge cases and error handling
Property-Based Tests¶
Using Hypothesis for: - URL normalization - Cache key generation - Content extraction - Type guard validation
Integration Tests¶
- End-to-end scraping (with real sites)
- API integration (with mocked responses)
- Report generation pipeline
Configuration¶
Environment Variables¶
| Variable | Required | Description |
|---|---|---|
XAI_API_KEY |
Recommended | Grok standard reasoning and strategy pipeline |
GEMINI_API_KEY |
Recommended | Gemini writing, utility, premium mode, and scrape summaries |
OPENAI_API_KEY |
Optional | OpenAI provider for routed eval/fallback paths |
ANTHROPIC_API_KEY |
Optional | Anthropic provider for routed eval/fallback paths |
LOCAL_LLM_BASE_URL |
Optional | Primary generic OpenAI-compatible endpoint for local eval and utility paths |
OLLAMA_BASE_URL |
Optional | Ollama-compatible fallback when LOCAL_LLM_BASE_URL is unset |
SEARCH_API_KEY |
Optional | Google Custom Search API key, only with SEARCH_PROVIDER=google |
SEARCH_ENGINE_ID |
Optional | Google Custom Search Engine ID, only with SEARCH_PROVIDER=google |
AI_RESEARCH_MODEL |
No | Override research model |
AI_REPORT_MODEL |
No | Override report model |
VERBOSE |
No | Enable verbose output |
DEBUG |
No | Enable debug mode |
Configuration Classes¶
See docs/CONFIG.md for detailed configuration reference.
Performance Characteristics¶
Typical Durations¶
| Mode | Duration | Output Size | API Calls |
|---|---|---|---|
| Scrape | 2-5 min | Site corpus plus insights | Scrape requests plus routed insight extraction |
| Standard provider-backed | 34-53 min | Evidence-dependent Strategic Overview + strategy | Planned section batches plus validation and coherence stages |
| Deep | 32-62 min with the default strategy; 24-47 min without it | Evidence-dependent Deep Research report plus optional strategy | One Deep Research dossier, 23 sequential Flash section calls, and the default strategy task |
| Premium | 74-132 min with the default strategy | Evidence-dependent report plus strategy | Structured collection, one Deep Research dossier, 23 sequential Flash sections, and the default strategy task |
Resource Usage¶
- Memory: ~200-500MB during scraping (Playwright browser)
- Network: Variable based on site complexity
- Provider calls vary by route; the dry-run lists the priced plan
Rate Limits¶
- Active Deep/Premium report path: one dossier interaction followed by sequential section calls; the configurable concurrent chapter executor is a compatibility surface, not the production topology
- Scraping: Homepage-first pilot, then a bounded 3-worker live corpus path; requests still pass through per-host concurrency, token-bucket, jitter, and backoff controls
- Gemini API: Respects 429 responses with backoff
Concurrency policy and live implementation locations are documented in
CONCURRENCY.md.
Future Considerations¶
The architecture is designed to support future enhancements without major restructuring:
- Research State Persistence: The section-based output format can be extended to include confidence levels and hypothesis tracking
- Iterative Refinement: The grading loop provides a foundation for incorporating user feedback
- Centralized Execution: The orchestrator pattern allows swapping local execution for remote job queues
- Optional Native Acceleration: A versioned HTML-analysis accelerator may ship only after the parse-once Python reference and end-to-end adoption gates in the runtime-boundary decision record pass
See ROADMAP.md for planned features.
Under the Hood - Quick Reference¶
This section provides a quick-reference summary of the retrieval engine, model pricing, and agentic architecture. For full details, see the component sections above.
9-Tier Retrieval Engine¶
Browser-first, falls back automatically:
- Playwright (JS rendering)
- Playwright Aggressive (accordions, lazy load)
- Patchright (persistent real-Chrome stealth)
- curl_cffi (TLS fingerprint impersonation)
- DrissionPage Stealth (challenge waiting)
- DrissionPage (driverless CDP)
- Vision (screenshot + LLM extraction)
- httpx (HTTP/2)
- requests (simple fallback)
Includes sticky tier memory, circuit breakers, cookie handoff, and automatic PDF detection. Playwright tiers now perform adaptive lazy-load scrolling (up to 20 steps by default, stops early when page height stabilizes).
Models & Pricing¶
| Model | Role | Pricing (per 1M tokens) |
|---|---|---|
| Grok 4.7 | Default reasoning in every tier. xAI-only writing and utility use it too, and --grok-tier max writes with it |
$2/$6 (<200k) Β· $4/$12 (>=200k) Β· $0.50 cached below the boundary |
| Grok 4.3 | Registered for historical quotes. Not in the analysis fallback chain | $1.25/$2.50 (<200k) Β· $2.50/$5.00 (>=200k) Β· $0.20 cached below the boundary |
| Grok 4.20 non-reasoning | Registered historical id. New xAI writing requests use Grok 4.7 | See registry / estimator |
| Grok 4.20 reasoning | Registered historical id | See registry / estimator |
| Gemini 3.5 Flash-Lite | Default routed writing path when GEMINI_API_KEY is configured |
$0.30/$2.50 |
| Gemini 3.8 Flash | Default utility (Flash) model, and the last step of the analysis fallback chain | $0.75/$3.75; $1.50/$7.50 from 2027-01-01 |
| Gemini 4 Carbon | Default Pro model | $2/$10 through 2026-11-30; $4/$20 from 2026-12-01 |
| Gemini 4 Argon | Registered Gemini 4 model. Not the Pro default. Same scheduled rates as Carbon | $2/$10; $4/$20 from 2026-12-01 |
| Configured Gemini Flash | Active Deep/Premium sequential section writer | See provider pricing in the estimator |
| Deep Research Agent | --mode deep and --premium: autonomous dossier research |
~$2.50 planning estimate per standard task; actual token and tool billing varies |
Why the defaults are the current flagships¶
Primr originally ran everything through Google's Deep Research API plus Gemini
3.1 Pro. That path remains available via --premium when maximum research depth
justifies the cost. Since 2026-10-28 the defaults track each provider's current
flagship (see design/grok-default-routing.md):
Grok 4.7 for reasoning-heavy stages and Gemini 3.5 Flash-Lite for bulk
writing when both keys are set. The current static plan is about $1.08 for
the base Strategic Overview and $1.26 when the default AI Strategy is
included. The earlier Grok 4.3 recipe measured $0.79 in the v1.24.0 eval.
--grok-tier max routes writing through Grok 4.7 as well as reasoning. It is
a higher-cost opt-in: the current static MAX plan is about $8.65 for the
base report or $10.05 with the default AI Strategy. Grok 4.7 has a 500k
context window. The analysis fallback chain is Grok 4.7, then Claude Sonnet
5.5, GPT-6.1 Terra, and Gemini 3.8 Flash.
XAI-only writing and utility use Grok 4.7.
OpenAI, Anthropic, and local OpenAI-compatible providers remain available for
fallback, utility, evaluation, and backend-freedom routing. Full-report
execution requires xAI, Gemini, or a configured OpenRouter key with the
separate paid-routing opt-in. OpenAI/Anthropic-only dry-runs quote the
XAI/Gemini planning floor and set execution_ready: false (see
Run Modes). The first production stages on the capability
router are fast.scrape_summary, fast.source_relevance, and
fast.hiring_signals behind --inference cloud|hybrid.
Agentic Architecture¶
- Hypothesis tracking with confidence levels across sessions
- Subagents for scraping, analysis, writing, and QA
- Hook system for governance (cost limits, quality gates)
- Research memory that persists and evolves
Acyclic research composition¶
All first-party Python import components are now acyclic, enforced by an empty
cycle baseline. Fast trust and strategy stages receive typed callbacks from the
research composition root; they never resolve helpers by importing that root.
core.legacy_research owns the existing section pipeline, including its scrape
quality gates, explicit working-folder binding, validated external collection,
progress, and section persistence. The orchestrator calls that same owner;
public research-agent functions remain thin compatibility facades that supply
their existing callback seams. The separate typed structured workflow retains
its existing public interface.
The historical research_agent.main and package-level perform_research exports
resolve lazily on explicit attribute access. Importing research stages does not
initialize the operational CLI. These compatibility exports retain the exact
function identities of their owners, with fresh-interpreter checks protecting
the startup boundary.