Technical enablement · 04

Operations & administration

What the team that runs {{product.name}} needs: managing tenants and access, the scheduled jobs, reading system state, handling the common failure modes, migrations, secrets and cost. Most of this page distils RUNBOOK.md in the product repository; that file remains the detailed reference.

Tenancy & access management

Tenants carry a tier (diagnostic, tier_1, tier_2, tier_3) and a data_region. The role list is on the feature reference.

Monitoring & health

Reading system state

• Zones (Agentic). GREEN is normal; AMBER needs a human decision soon; RED is already paused, stopped or refused at the gateway.
• CPVO states (Attribute). OK, LOW_CONFIDENCE, COVERAGE_CAVEAT, RECONCILIATION_PENDING, NOT_ATTRIBUTABLE and NO_OUTCOMES. Calculation runs are immutable; a recompute always creates a new row.
• Coverage badge (Govern). Percentage of AI spend the platform can see and reconcile against a named source.
• The inferred marker. A figure carrying it is inferred rather than measured. It is deliberately never a dismissible modal.
• Demo badge. Tenants flagged as demo show a DEMO pill in the top bar.

Scheduled jobs & data operations

A Container Apps Job, ca-ai-yield-cron, runs the app's own image every four hours (UTC 0 */4 * * *). It calls three routes in order over HTTP so they execute inside the web app's process.

• Cadence is set by the billing APIs. Azure Cost Management refreshes actual cost only every few hours and each run spends two rate-limited cost queries per Azure connection. The 72-hour lookback covers restatement, so correctness does not depend on frequency. Override with CRON_SCHEDULE.
• Authentication. The routes need the x-cron-secret header matching CRON_SECRET. A 401 means the secret does not match; a 503 means CRON_SECRET is not set on the deployment at all.
• Session check exemption. Job routes are exempt from the session check because a scheduler cannot present a cookie. If they were moved back under it, every scheduled run would be redirected to sign-in. This is invisible locally, where AUTH_BYPASS skips the check, and is covered by tests/domain/proxy-auth.test.ts.
• Request ceiling. The sync runs inside an HTTP request that ingress ends at 240 seconds. Retry budgets are capped (background retries at 90 seconds, best-effort token metrics at 15 seconds) and connections are processed one at a time with a deadline. Remaining connections are deferred to the next run. If sync-billing starts returning 504, suspect a retry budget before the provider.
• Rate limits are normal. A 429 from a billing API is expected. If a scheduled sync still reports 429 after retrying, wait for the next run rather than triggering more.
• Demo tenants are excluded from sync, because their connections are seeded props.
• Errored connections are retried; disconnected ones are not.
• Same secret on both. The app and the job must share CRON_SECRET. After rotating, run both make deploy and make cron-deploy.

Common failure modes

Gateway traffic arriving with no workload

Events from keys that name no workload, or a workload the tenant does not hold, are stored rather than lost, with the key's project identifier retained. Attribute → Workloads lists them, biggest estimated spend first; naming one writes a mapping and backfills the stored events. Resolution order: the key's own workload id (if valid and held by the tenant), then the stored map, then unattributed. A key with no project identifier cannot be adopted; the fix is to set one on the key in the gateway. An event with no tenant is dropped at the gateway, so look there, not in the app.

Provider connections not syncing

• A connection is created disconnected. Verification on save, or the Test connection button, promotes it to active.
• Check the registered adapters in lib/tdal/adapters/registry.ts; a provider absent there stays disconnected by design.
• Test connection reports which Azure strategy was used (auth:managed-identity, auth:service-principal or auth:static-token). Check that first when a connection breaks after a redeploy.
• Rotating CREDENTIAL_ENCRYPTION_KEY makes every stored credential unreadable; they must be re-entered.

Estimated cost (Vertex)

Google publishes no actual-spend API; it exists only in a billing export configured per billing account. Where none is available the adapter prices usage from the public catalogue and marks rows cost_basis = 'estimated'. An estimate is an upper bound: it sees no committed-use discount or credit and cannot be reconciled to an invoice. A model is priced only where exactly one catalogue entry matches, so ambiguous models report usage without cost. If an export becomes available, add it to the connection scope and billed rows supersede estimated ones.

Sign-in failures

A bad tenant id shows as an issuer that will not resolve; a wrong AUTH_URL or missing AUTH_TRUST_HOST shows as a redirect-URI mismatch from Entra. If every page 404s or 500s locally with AUTH_BYPASS, check the development session still has a valid identifier.

Database migrations

• The schema lives in lib/db/schema.ts; migrations in lib/db/migrations/. make db-push applies the schema directly and is fine for early-stage or development use; for data that matters, generate and review a migration (npm run db:generate then db:migrate).
• token_cost_period is partitioned by month. New partitions are added by migration, not db:push.
• Changes that db:push cannot apply to a populated table (for example a unique index on a column whose rows all hold the default) ship as scripts in scripts/migrate-*.ts. They are idempotent and run once per environment: DATABASE_URL=... npx tsx scripts/<name>.ts.
• Ordering rule. An additive migration leads the deploy: apply it first, then release, because the new code reads the new columns and the pages that do return 500 without them. The dangerous order is the opposite one, where a constraint is relaxed ahead of the code that handles the relaxed case.

Releasing and secret rotation

Release flow is on the deployment page. For secrets:

1. Update the value in your local .env

   Never in the repository.

2. Run make deploy

   It re-syncs the app's secrets and forces a new revision. make release also rebuilds and pushes.

3. If it is CRON_SECRET, run make cron-deploy too

   Updating only one side leaves the scheduler receiving 401s.

Cost & usage

The platform reads provider billing data, and some reads are themselves chargeable. Agree reporting with Finance; this section covers the technical mechanism only.

Support & escalation

• Data privacy, GDPR and regulated-data questions go to the Data Privacy Officer.
• Legal, compliance and financial questions go to those functions. The product and its assistants do not give that advice.
• Infrastructure and deploy issues go to whoever holds admin access to the resource group.
• Support channel: {{links.support}} · Contact: {{product.contactEmail}}
• Severity and response are not yet agreed. Proposed model, to be confirmed by the service owner: Severity 1, the service is down or client-mode data may have leaked: respond within one working hour, update every two hours. Severity 2, a feature or a provider sync is failing for one or more engagements: respond the same working day, update daily. Severity 3, a defect with a workaround or a question: respond within three working days.

Runbook index

The repository has one living runbook, RUNBOOK.md, rather than separate numbered runbooks. Its sections: