Run and operate

The repository is production-shaped but not production-ready: durable processing, readiness, structured errors, and rebuildable search exist, while production authentication is intentionally not implemented.

Process topology

Run at least two Node processes from apps/api:

npm start -w @activity-platform/api
npm run worker -w @activity-platform/api

Serve the built apps/web-admin/dist as a static SPA with history fallback and proxy /api plus WebSocket upgrades to the API. MongoDB must support transactions; local development uses a single-node replica set.

DigitalOcean App Platform deployment

The planned DigitalOcean App Platform application includes:

• the API web-service component, using /ready as its readiness check;
• a separately scaled worker component, subject to the preproduction limits below;
• the built apps/web-admin/dist as a static-site component at app.corivum.com; and
• the VitePress documentation as a static-site component at documentation.corivum.com.

Build the documentation component with npm ci && npm run docs:build, set DOCS_BASE=/, and publish docs/.vitepress/dist. Build the admin component with VITE_API_BASE_URL=/api so authenticated browser traffic stays same-origin through app.corivum.com/api, and set VITE_DOCS_BASE_URL=https://documentation.corivum.com/. The API is also exposed directly at api.corivum.com for external API and MCP clients. Configure both static components to redeploy from main when their source or shared package manifests change.

Preproduction async safety envelope

The API does not start background processing in-process. Event ingestion only commits durable outbox records; work runs only when the separate worker command is deployed. For the initial preproduction deployment:

• deploy zero workers if derived feed, search, user, and notification freshness can wait, or exactly one worker instance when those projections are needed;
• disable worker autoscaling and do not run the worker command in the API component;
• do not deploy the demo applications in preproduction unless scheduled public-data ingestion is intentional; each demo starts an importer shortly after boot and repeats it on a timer;
• keep subscription history backfill opt-in and set WORKER_BACKFILL_EVENT_LIMIT to a small environment-specific cap (start with 100, below the application default of 1000);
• keep retry attempts low and observable; the configured exponential delay limits repeated failures, while poisoned records stop automatic retries;
• watch /v1/operations/processing during rollout and stop the worker if pending volume, oldest-item lag, failures, or poison counts move outside the expected test envelope.

One worker awaits one outbox record at a time, so there is no in-process fan-out across jobs. A subscription backfill is also capped, but it can bulk-write up to that cap within its single job. WORKER_POLL_MS only controls empty-queue polling and is not a busy-queue throughput limit. Adding worker replicas increases concurrency, and the current worker has no global jobs-per-second cap; those deployment constraints are therefore part of the preproduction safety boundary.

Restate is the planned job-processing runtime before asynchronous workload or worker scale is materially expanded. See ADR 0005 and milestone 6 for the migration boundary.

Dependency roles

• MongoDB is authoritative for immutable events, resource sequences, outbox work, activities, users, link settings, subscriptions, notifications, and checkpoints.
• OpenSearch is a disposable search projection. The API can serve non-search behavior when it is absent, but search returns SEARCH_UNAVAILABLE.
• The worker is required for feed, search, user, and notification freshness. Event ingestion can succeed while a stopped worker creates projection lag.

Health and observability

Use /health for liveness and /ready for traffic readiness. Use /v1/operations/processing for:

• pending, processing, failed, poisoned, and completed queue counts;
• age and lag of the oldest outstanding work;
• durable checkpoint freshness.

Server logs are structured. Unexpected request failures provide a public logId matching the internal log entry. Alert on sustained queue lag, poisoned records, readiness failures, and repeated SEARCH_UNAVAILABLE responses.

For local hosted-observability evaluation, set OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS in apps/api/.env. Both API and worker then export OTLP logs, traces, and metrics while preserving their normal console output. See the API README's Local OpenTelemetry section for the Better Stack source setup and expected service names.

Traced workflows

Auto-instrumentation covers HTTP, MongoDB, and OpenSearch calls. On top of that, the workflows whose outcome is not visible from a single request carry deliberate spans, each recording what the step read and what it returned:

Trace	Spans	Answers
backfill.run	load_subscription, read_events, apply_filters, record_matches, complete	Where a replay lost the events a subscriber expected
event.accepted	load, project_activity, observe_schema, observe_user, match_subscriptions, index_search	Which derived read model is slow, empty, or failing
outbox.process <topic>	wraps either processor above	Whether a record completed, is retrying, or is poisoned
events.ingest, events.ingest_batch	one child span per item	Which batch items were accepted and which collapsed as replays
ai.chat.reply	one span per assistant run	Model, tool sequence, token usage, and latency of a run
search.index_event, index_user	one index_document child per OpenSearch write	Which projection kind or target failed to index
search.reindex	event replay phases, user replay, and every document write	Rebuild counts, catch-up work, cutover target, and failures

Background work starts its own trace rather than extending the request that queued it, because a job runs long after that request responded. The queueing trace is stored on the outbox record and attached to the job span as a link, so a backfill trace opens from the subscription request that asked for it and vice versa. Records queued before this change simply have no link.

Reading a backfill: corivum.backfill.events_read is what the target query returned, events_filtered_out is what the subscription's conditions rejected, and matches_created versus matches_existing separates new notifications from an idempotent replay. When corivum.backfill.limit_reached is true the run hit WORKER_BACKFILL_EVENT_LIMIT and older matching events were not replayed.

Spans carry identifiers, counts, and outcomes — never event metadata, message content, or AI tool arguments and results.

Search rebuilds

npm run search:reindex

The rebuild writes a fresh versioned index, replays events in resource-sequence order, performs catch-up passes around cutover, moves the stable alias atomically, and retains the previous physical index for rollback. Verify search and index counts before removing an older index.

DigitalOcean production deployment

Production search requires a managed OpenSearch cluster; an App Platform dev database cannot provide OpenSearch. Provision a cluster in the app's region, add the corivum App Platform app as a trusted source, and bind it into .do/app.yaml:

envs:
  - key: OPENSEARCH_URL
    value: ${corivum-opensearch.DATABASE_URL}
    scope: RUN_TIME
    type: SECRET
  - key: OPENSEARCH_CA_CERT
    value: ${corivum-opensearch.CA_CERT}
    scope: RUN_TIME
    type: SECRET

databases:
  - name: corivum-opensearch
    engine: OPENSEARCH
    version: '3.6'
    production: true
    cluster_name: corivum-opensearch

After the API and worker deploy successfully, run npm run search:reindex with the production MongoDB and OpenSearch bindings. Confirm the alias exists, document counts are nonzero, worker lag is draining, and /v1/search no longer returns SEARCH_UNAVAILABLE.

Set OTEL_EXPORTER_OTLP_ENDPOINT and, when required by the collector, OTEL_EXPORTER_OTLP_HEADERS as production secrets on the app as well. Indexing and rebuild spans are no-ops until an exporter endpoint is configured.

Backups and recovery

Back up MongoDB as the system of record. OpenSearch can be rebuilt from MongoDB. A recovery plan must include events, outbox records, projections that cannot be recreated from search, and tenant configuration.

If the worker is interrupted, leases expire and work can be claimed again. Processing is designed to be idempotent. Investigate poisoned records before manually changing queue state.

Security boundary

Do not expose the current development authentication to the internet. Before production:

1. implement a production auth mode with trusted organization/environment claims;
2. replace WebSocket query-string API keys with short-lived credentials over wss://;
3. define key rotation and secret storage;
4. terminate TLS and restrict MongoDB/OpenSearch network access;
5. review visibility policies and principal-scoped subscription behavior;
6. add deployment-specific rate limits, CORS policy, retention, and audit requirements.