Primr Concurrency Model¶
This document describes the live concurrency model, its safety limits, and the
difference between coroutine cancellation and actually stopping a research
job. The standing language and extraction policy is in
design/runtime-language-boundaries.md.
Scope¶
Primr has two distinct concurrency levels:
- Inside one research job: bounded asyncio tasks and thread pools overlap independent external waits and selected blocking operations.
- Across research jobs: the core remains single-job. A consumer or hosted control plane may queue several jobs, but each long job should be owned by a separate supervised process or one-job container when cancellation and isolation matter.
These levels must not share budgets, browser environment mutation, mutable scrape state, or terminal status by accident.
Design rules¶
- Classify the operation before choosing a primitive.
- Await natively asynchronous I/O.
- Put bounded blocking I/O in the existing async bridge or an explicitly owned small thread pool.
- Use a child process when work must be terminable, crash-contained, or resource-limited.
- Bound every pool, semaphore, queue, input, timeout, and retry policy.
- Preserve target-host courtesy limits even when more local workers are idle.
- Do not infer that a cancelled asyncio task stopped a running thread or remote provider operation.
- Measure block rate, p95 latency, memory, queueing, and end-to-end time before changing worker counts.
Live concurrency inside one job¶
Website corpus collection¶
The canonical production path is fetch_web_content() in
src/primr/data/scrape.py. data/parallel_scraper.py is a standalone helper;
its default worker count is not the production site-corpus policy.
The live path is deliberately staged:
- The homepage is fetched and classified first.
- Up to
SCRAPE_PILOT_COUNTpages, default 10, run sequentially. This pilot learns access quality, duplicate behavior, effective tiers, and whether the crawl should stop defensively. - Remaining selected pages run through
ThreadPoolExecutor(max_workers=3). - Every tier request still passes through the shared per-host limiter.
The default RateLimitConfig is three concurrent requests per host and 20
requests per minute. The live corpus orchestrator keeps the three-request cap
but currently uses a 30-request-per-minute override and 0.5 second token-wait
jitter. The token bucket starts full, so the rate is not an evenly spaced
delay. Live 429 handling records a persistent host cooldown; the lower-level
limiter's exponential-backoff methods are not currently the production 429
path.
The worker pool is therefore an upper bound, not permission to send three requests continuously. Tokens, backoff, slow browser tiers, sticky-tier state, and circuit breakers may reduce effective concurrency.
Other bounded pools¶
The codebase uses separate limits for separate resources:
| Capability | Live pattern | Reason |
|---|---|---|
| Post-pilot site pages | 3 worker threads | Conservative same-host collection |
| Raw scrape persistence | 1 writer thread | Keep disk writes out of the page path |
| Standard section writing | Up to 4 worker threads | Bounded writes within the rolling-context and whole-document coherence pipeline |
| Premium accordion writing | One section at a time | Preserve the report argument arc and prior-section continuity |
| Gap-research queries | 3 worker threads | Bounded external search/model work |
| Strategy platforms | Up to 3 worker threads | Independent platform artifacts |
| URL existence verification | Configurable, default 10 threads | Conditional guessed-link verification fanout |
| Browser hard-timeout wrapper | 1 worker in the specific Drission path | Isolate one blocking browser call |
These limits are local to their capability. They must not be copied into another path without a production-shaped load and block-rate measurement.
ResearchExecutor still contains a compatibility chapter fan-out with an
async semaphore of two. It is not part of the active Standard or Premium
production topology and must not be described as the Premium concurrency
policy. Premium uses one Deep Research dossier followed by sequential section
writes.
Conditional guessed-link verification currently receives no shared limiter from the live discovery callers, so it falls back to a no-op limiter. That 10-worker same-host fanout is a documented hardening gap, not a concurrency budget to emulate. It should reuse the corpus limiter or an explicit verification budget before wider crawl concurrency is considered.
The browser timeout wrapper deserves a specific warning: timing out a future does not terminate a browser call already running in its thread. It bounds how long the caller waits, not how long the underlying thread can exist.
Async provider and research work¶
Async paths use semaphores and asyncio.gather() for independent operations.
Some are natively asynchronous, such as citation resolution. Others wrap or
call synchronous provider SDK methods and may still occupy the event loop or a
thread briefly. An async def declaration alone is not proof that the work is
nonblocking.
asyncio.gather() is appropriate only when:
- tasks do not mutate the same unguarded state;
- each task has its own timeout and retry policy;
- provider and host quotas remain bounded;
- exceptions are handled explicitly; and
- downstream stages wait for every required result or record an intentional partial outcome.
Pipeline overlap may start external research after the homepage and initial context exist, but it must not let final synthesis consume an uncommitted partial corpus as if collection had completed.
Async and sync boundaries¶
Use the existing primr.utils.async_utils seam.
Synchronous caller, async implementation¶
Use run_sync():
run_sync() creates the loop when no loop is running and refuses nested-loop
use. Code already inside an async function should await directly.
Do not add new copies of get_event_loop().run_until_complete() or bare
asyncio.run() outside the boundary helper.
Async caller, bounded blocking implementation¶
Use run_async() for ordinary blocking library calls:
from primr.utils.async_utils import run_async
result = await run_async(parse_or_fetch_sync, argument)
The shared bridge has a bounded four-worker executor. A capability may own a smaller explicit pool when it has a different resource budget or ordered shutdown requirement.
Existing MCP pipeline work uses asyncio.to_thread() for long synchronous
research entry points. That keeps the event loop responsive, but it does not
make the work safely cancellable. Local MCP and A2A jobs therefore place the
whole runner behind the supervised process boundary below.
Job ownership and cancellation¶
There are three different cancellation meanings:
| Boundary | What cancellation can guarantee |
|---|---|
| Coroutine | Stops future coroutine progress at cancellation points |
| Running thread | Cannot be forcibly terminated safely by Python |
| Owned child process or container | Can be asked to stop, then terminated after a bounded grace period |
A terminal cancelled job state must not be written merely because a client
requested cancellation. It is terminal only after Primr observes that the
worker it owns has exited or after the job is reconciled to a documented remote
state.
The shipped local MCP/A2A long-job pattern is:
control surface
-> retained job-id to process handle
-> cooperative stop request
-> bounded wait
-> terminate, then kill if required
-> preserve partial artifacts
-> commit terminal status and cancellation method
Provider-side work may be non-interruptible. Primr must record that separately and never imply that stopping a local worker cancelled a remote provider task unless the provider confirmed it.
mcp_server.worker_protocol is the packaged local contract. The parent accepts
only schema-valid, sequenced, job-bound events. It rejects malformed, regressive,
cross-job, and late snapshots, preserves canonical identity, and replaces
worker-reported heartbeat, stage-transition, and completion clocks with its own
observation timestamps. It commits terminal state only after exit and writes a
worker-exit manifest for failed or cancelled exits with a retained supervisor
handle. Spawn failures and restart reconciliation remain journal-only. The
deployment runner predates this contract and should converge on it rather than
being copied into the package.
Workers inherit only an explicit research-provider and runtime environment
allowlist. Control-plane, cloud-identity, telemetry, and CI secrets are removed
and cannot be restored by lazy .env loading. Supervised .env parsing
disables interpolation and rejects interpolation-bearing assignments. Control
commands and lifecycle
events move to private, non-inheritable descriptors before pipeline imports;
ordinary stdin becomes DEVNULL and ordinary stdout becomes worker-log output.
This prevents normal native writes from corrupting JSONL and normal exec-based
descendants from retaining the protocol pipes.
Exactly one controller owns a journal through a non-blocking OS lease. MCP, co-hosted A2A, and standalone A2A share one reference-counted controller lifecycle. After lease acquisition, the controller reloads the journal before restart reconciliation. Final lifecycle exit runs shielded, bounded cooperative, terminate, and kill phases and releases the lease only after all retained workers are reaped and descendant cleanup is confirmed. A cleanup failure retains the worker handle and is retried; lifecycle exit fails and retains the lease so another controller cannot reconcile a possibly live worker.
On Windows, the worker joins a named Job Object before it emits ready; closing
the controller's sole long-lived handle removes the worker tree. On POSIX, the
worker starts in a new session, receives bounded group termination, and uses a
parent-disconnect watchdog. Linux covers the bootstrap transition with
PR_SET_PDEATHSIG, then clears the worker-only signal after the control reader
starts so pipe EOF can kill the complete process group. This POSIX crash path is
best effort if native code holds the GIL long enough to starve that reader or a
descendant deliberately escapes through setsid() or a new process group.
Recovered active journals have no retained handle and reconcile to
failed/server_restart only after the next controller acquires the lease.
Shared state¶
State that is synchronized¶
- Per-host rate-limiter tokens, backoff counters, and semaphore creation are protected by a lock; concurrency slots use per-host semaphores.
- Circuit-breaker state uses guarded per-key state.
- Request-scoped correlation uses
contextvars, not a process-wide mutable request identifier. - MCP job-store methods serialize journal access with a lock for the supported single-job process model. Mutable job objects and cross-thread status notification still require care.
State that is not a multi-job contract¶
ScrapeOrchestratormaintains mutable per-host state.- Adaptive browser execution temporarily changes process environment values.
- Orchestrator singleton construction and scrape trace appends are not explicit cross-thread synchronization contracts.
- Some usage and budget state assumes one research job per process.
- Browser, cache, and provider SDK objects may have thread-affinity or undocumented thread-safety constraints.
Ordinary dictionary operations are not a substitute for a compound-state locking policy. Do not share one orchestrator or process-global budget across concurrent research jobs. Process isolation is the preferred way to keep these ownership assumptions explicit.
Python 3.14 free-threading¶
Standard Python 3.14 is supported. Free-threaded 3.14t remains an informational compatibility and benchmark lane.
Removing the GIL does not supply cancellation, queue durability, browser isolation, or provider parallelism. It can also expose races in mutable state, and an extension that is not free-threading-aware may restore the GIL at import.
Before any promotion:
- run the full suite on every supported platform;
- record which extensions preserve or restore free-threading;
- audit process-wide environment mutation, caches, budgets, and host state;
- compare representative one-job and multi-worker throughput;
- measure CPU, p95, peak memory, failures, and report outcomes; and
- adopt only if the complete workload shows material value.
Overload and backpressure¶
Exceptional behavior under load matters more than maximum worker count:
- Keep queues bounded.
- Reject or defer work before exhausting browser or model capacity.
- Propagate deadlines and budget checkpoints.
- Honor provider
Retry-Afterwhere supported. - Treat a healthy local model server that lacks capacity as
busy, not broken. - Return bounded
retry_after_secondsandretry_atguidance instead of sleeping for hours in the worker. - Never fall through to paid cloud execution during a hard-zero run.
Core Primr does not poll indefinitely or schedule its own later retry. A host or external scheduler may submit one later job after rechecking capacity.
Observability¶
Implemented signals include correlation and request identifiers, run-state events, scrape trace records, provider usage and cost metadata where available, job heartbeats, and control-plane audit projections.
The following are required before concurrency or service-language decisions, but should not be described as fully wired today:
- event-loop lag;
- active and queued work per pool;
- queue depth and admission latency;
- per-phase p50, p95, and p99 duration;
- cancellation request-to-worker-exit latency;
- orphaned worker count;
- peak resident memory; and
- block, 429, fallback, and recovery rates by concurrency profile.
Telemetry remains body-free by default. Do not record page bodies, prompts, reports, credentials, raw endpoint URLs, or installed local model names merely to tune concurrency.
Contributor checklist¶
Before adding or changing concurrency:
- Identify the constrained resource: target host, browser, provider quota, local model, CPU, memory, disk, or control-plane dependency.
- Record the current production-shaped baseline.
- Reuse the existing async, rate-limit, job, and logging seams.
- Define queue and pool bounds, timeout, retry, cancellation, and shutdown.
- Prove shared-state ownership.
- Add deterministic failure and timeout tests.
- Measure p95, failures, block rate, memory, and end-to-end time.
- Update this document if the live policy changed.
Explicitly not¶
- No unbounded
gather(), pool, or queue. - No new event-loop helper.
- No assumption that a timeout killed a thread.
- No global worker-count constant reused across unrelated capabilities.
- No concurrent research jobs sharing one in-process budget or scrape orchestrator.
- No Go rewrite to obtain process isolation already available through the runner protocol.
- No free-threaded default based only on a CPU microbenchmark.
Version history¶
| Version | Changes |
|---|---|
| 2.0 | Reconciled the document with the live pilot plus three-worker corpus path, the shared async seam, truthful cancellation, process isolation, and measured runtime policy |
| 1.0 | Initial documentation |