Operations reference

Draft for review

Everything about running an OrchKernel server, for looking things up: how big it can get, what data goes where, every setting, the command line, tokens and webhooks, backups and restore, upgrades, what survives a crash, watching it run and troubleshooting. To set a server up, start with Install in five minutes, then Production hardening.

OrchKernel is one program, orchkernel, that is both the command line and the server. The guide calls it ok: the install script and the Docker image make an ok link beside it, and alias ok=orchkernel does the same anywhere else. The server keeps everything for one company in one SQLite file, the state file.

       people (browser), the ok CLI, outside agents, webhook senders
                               |
                               | HTTPS (--domain, your certificate, or a proxy)
                               v
              ok serve
                 /api/*   the API and the event stream
                 /*       the web workspace, built into the program
                               |
                               | one writer
                               v
               state.db  (SQLite)  + secrets.key, backups/, tls/
                               |
          +--------------------+---------------------+
          v                    v                     v
   model providers     tool servers and      off-site backups
                       connections           (encrypted first)

How big it can get#

One server, one state file, one company. There are no replicas and no failover: availability comes from fast restarts (a service manager, or the Recreate Deployment with a health probe), and runs that wait for an approval survive a restart. If you need zero downtime, OrchKernel is not ready for you yet.

One company is comfortable at a million items on one server, as measured: saving, lists, dashboards and search, by words and by meaning, answer in a fraction of a second for 16 people at once (see At a million items, with its two caveats). Keep larger systems of record where they are: a CRM's 500,000 leads or an ERP's orders stay in that system as federated collections, which add nothing to this size (see Large CRM and ERP data: federate first). A federated list asks the vendor for the person's rows (their filter, the owner match of their access, the page they need) where the adapter declares those filters, so a rep's list over 500,000 vendor leads is one request; a filter the adapter does not declare is applied to at most 5,000 rows read.

The team's measurements on one Apple M2 laptop (8 cores), with 16 people working at once, on a 324,000-record file after the scale fixes:

Those searches matched words only. With search by meaning on (the local model), a rep with 4,000 notes of five windows each, among 24,000 notes and 38,000 knowledge entries, measured in process: 50 ms p50 for a common word and 32 ms for a rare one, one person at a time, and 630 ms and 340 ms with 16 people at once. Each such search scores every one of the rep's own vectors exactly; a rep with more chunks than OK_SEARCH_CANDIDATES goes through the vector index instead (31 and 18 ms, 400 and 240 ms with 16 at once). The index holds every vector there is: at a million vectors it answers in about a millisecond, from 533 MB of files it maps rather than reads into memory.

Captured mail is cheap to keep: the rules, archived history and the local embedding model cost nothing, and the local model embeds 57 to 63 chunks a second on three cores of such a server, about five hours for a million emails. ok serve runs it on a worker of its own (86 chunks a second on a million-item file, against 93 a second run directly), with requests as fast as before. What curating costs depends on how much mail reaches the model (see What capture costs).

At a million items#

The team measured one 100-person company at about a million items on the same laptop (500,000 leads, 250,000 interactions, 100,000 captured emails, 10,000 knowledge entries: a 3.3 GB file), with the machine busy and the file too large for its file cache, after the round 2 scale fixes:

Two soft spots, said plainly:

• Words and meaning together have a p95 of 3.2 s, from each person's first (cold) search after a restart; later searches take the times above.
• The first ticks after an upgrade that adds indexes hold writes for 8 to 12 s each, once per new index, about seven times.

A person's first search after a restart, with the file larger than the machine's file cache, took 1.4 to 2.5 s while their rows were read from disk; later ones took the times above. Search by meaning with 16 people at once is close to a second, and is the part to watch past this size.

Past these sizes the limits are known: the vector index does not merge its segments yet (measured to a million vectors), building a new dashboard total pauses writes for a few seconds on a collection of 500,000 rows, and 5 and 11 million items have not been measured. Postgres is not needed for any of the sizes above.

Disk. Keep free at least three times the state file: an encrypted backup is made from a copy of the state (about its size) into a bundle (smaller, compressed), plus the backups you keep in backups/. Admins are warned under OK_AUDIT_MIN_FREE_MB (1 GB) free.

Your data#

Purged and deleted content stays in backups until they age out: keep backups only as long as your retention promises allow (Context capture).

Configuration#

Settings come from the environment, and many also from ok serve flags.

The install script reads OK_INSTALL_DIR, OK_DATA_DIR, OK_VERSION, OK_REQUIRE_SIGNATURE, OK_INSTALL_FILE, OK_INSTALL_URL and OK_INSTALL_SHA256 (see Install). The image sets OK_STATE=/data/state.db and OK_BIND=0.0.0.0:8080, and its health check uses OK_HEALTH_PORT (8080; 80 with deploy/compose.https.yml). Tuning settings (OK_DB_READERS, OK_SEARCH_CANDIDATES, OK_ARCHIVE_TICK_MS, the audit checkpoint settings) are in docs/deployment.md in the repository.

Keep provider keys, webhook secrets, OK_SECRETS_KEY, bucket secrets and tokens in a secret store (a Kubernetes Secret, Docker secrets, a systemd EnvironmentFile of mode 0600, Vault). Never commit llm.yaml, .env, secrets.key or a recovery key.

ok doctor#

ok doctor checks the install in one run and never writes the state; it is safe beside a running server:

Each line is ok, warning or error, and the exit status is 1 when any is an error.

The command line#

The commands an operator uses most; CLI and API reference has them all.

People, tokens and access#

People sign in to the workspace with their email address and a password (see Sign-in settings). Tokens are for the API, the CLI's --server mode, outside agents and CI, never for signing in to the workspace.

A token belongs to one person or agent. The server keeps only its hash, so a lost token cannot be shown again: issue a new one.

• A token never issues a token, and never changes how anyone signs in: POST /api/tokens, passwords, two-step, email and sessions answer 403 session_required "Sign in to the workspace to do this; a token cannot." to a bearer.
• Nobody issues a person's token for them. An admin asking for someone else's gets 403 own_tokens_only; each person issues their own.
• Lifetimes. A person's token lives at most 90 days (an admin changes the ceiling in Governance, Sign-in); 30 days unless chosen. A kernel agent's lives at most 90 days and must carry the policy or changes scope.
• Old sign-in tokens. Tokens people pasted into the old sign-in page keep working on the API until they expire; the people panel counts them and revokes them all with one action.

Send a token as Authorization: Bearer <token>. The one exception is the event stream, GET /api/events/stream?token=..., because a browser cannot set headers there (the workspace itself uses its cookie).

Someone leaves. An admin offboards them from the people panel (or POST /api/actors/<id>/offboard): their sessions end, their tokens are deleted, their running work and delegations are cancelled. Deactivating instead is reversible: sign-in is refused, sessions end, and tokens stop working until the person is reactivated.

Webhooks#

POST /api/webhooks/<source> turns an event from another system into tasks, through the triggers (rules in a module that start work) that listen to that source. The support module's helpdesk source, for example, creates a "Triage incoming ticket" task for its triage agent. A source with no secret is refused. The sender signs each delivery:

• x-ok-timestamp: the time of sending in Unix seconds, within five minutes of the server's clock.
• x-ok-signature: sha256= and the hex HMAC-SHA256, keyed with the source's secret, of the timestamp, a dot and the body exactly as sent.
• x-ok-delivery (optional): a unique id. A repeat within 24 hours creates nothing and answers with the first delivery's tasks, so the sender can retry safely.

With the server started with OK_WEBHOOK_SECRETS=helpdesk=... and the same secret in HELPDESK_SECRET:

body='{"external_id":"Z-100","subject":"Charged twice"}'
ts=$(date +%s)
sig=$(printf '%s.%s' "$ts" "$body" | openssl dgst -sha256 -hmac "$HELPDESK_SECRET" | sed 's/^.* //')
curl -sS -X POST https://work.acme.example/api/webhooks/helpdesk \
  -H 'content-type: application/json' \
  -H "x-ok-timestamp: $ts" -H "x-ok-signature: sha256=$sig" \
  -H 'x-ok-delivery: evt-0001' --data-raw "$body"

Senders that cannot sign may send the secret itself in x-ok-webhook-secret. It is accepted (202, with "delivery":null) but offers no protection against a replayed request; prefer signatures.

Backups and restore#

Setting backups up (the recovery key, an off-site bucket, checking Verified, testing a restore) is in Production hardening. This section is how they work.

What a backup holds#

A file removed after a backup stays in that backup until retention prunes it. A restore puts back the state and files/ together, moving the folder as it was aside as pre-restore-<time>.files/ (removed with the pre-restore-<time>.db copy).

Left out, and rebuilt or fetched at start: the search index, the local model, certificates, the lock, -wal and -shm.

The format and the keys#

A backup is orchkernel-<id>.tar.gz.age, the id being its UTC time and six random hex digits (2026-10-07T020000Z-a1b2c3): a tar archive, gzipped, encrypted in the age format to X25519 keys. Beside it, orchkernel-<id>.json lists it without opening it (kind, time, versions, size, checksum, key fingerprints, when and how it was verified); it names no company and holds nothing secret, and a restore never trusts it.

Every backup is encrypted to:

A backup made before a key rotation opens with the kept secrets.key.<kid> or the recovery key. A new recovery key applies to backups from then on.

Opening a backup without OrchKernel, with the age tool and the recovery key file:

age -d -i orchkernel-recovery-key-9f3e1a2b.txt orchkernel-2026-10-07T020000Z-a1b2c3.tar.gz.age | tar xz

or ok backups decrypt <file> --identity <key file> --out <folder>. Either gives the files above; state.db opens with any SQLite tool.

How a run goes#

1. When: at the scheduled time, or five minutes after a start when the last good backup is older than one interval. A run whose state has not changed since the last good one records "unchanged" and writes nothing.
2. Make: copy the state, pack, compress and encrypt into backups/.partial/.
3. Verify here: open it with the server's key, check every file's checksum and the manifest's signature, PRAGMA quick_check and the audit chain of the state inside. Only a verified backup is kept.
4. Upload to the off-site bucket (one upload up to 256 MiB, in 64 MiB parts above), then verify there: download and compare (the default). A copy that differs is removed from the bucket and the run fails.
5. Record: BackupTaken and BackupVerified events, and the run in the backup log the page shows.
6. Prune each destination by retention.
7. Report: on a failure, an inbox item for every admin (once per failing streak per destination) and a BackupFailed event.

One run at a time: Back up now during a run answers 409 busy.

Retention#

Per destination, over verified backups only: the newest backup of each of the last 7 days that have one, of each of the last 4 ISO weeks and of each of the last 6 months (the union is kept). Always kept as well: the newest verified backup, whatever its age, and backups made before a restore for 7 days. Unverified or failed files older than a day are removed only when a newer verified backup exists there. Nothing is pruned after a failed run, and the last verified backup is never removed, from the page either.

Restore from the Backups page#

For a server that runs. On the backup's row, choose Restore:

1. Fetching and checking. The backup is fetched (downloaded from the bucket if needed) and checked as in a run; nothing changes yet. A backup this server's keys cannot open asks for the recovery key it was made with (paste the key file's text).
2. Where it came from. "Made by this workspace" (its signature checks with a key this server holds), "An earlier state of this workspace" (its audit log is an earlier point of this one), or "Not checked". A backup that is not checked is restored only from the command line.
3. What will change: the time it goes back to, how many changes since are undone ("not counted" when the backup is not an earlier point of the workspace as it is now, such as a copy made before an earlier restore), people, records, tasks and threads now and in the backup, tokens issued since (revoked), the secrets key (kept; sealed values re-sealed to it), and configuration files that differ (kept; the backup's written beside them as llm.yaml.from-backup). A red line says what is lost: "Everything done after 7 Oct 2026, 02:00 is lost (42 changes). Everyone is signed out."
4. Type the company's name and choose Restore. A sign-in within the last ten minutes is needed; otherwise the page asks for your password.

Then the server makes a backup of the state as it is now (kind "Before a restore", kept 7 days), answers every request but the health check with 503 restoring, stops running agents, and restarts itself with the same command line (HTTPS included), never letting go of its lock. The new process swaps the backup in before it serves, and its log says "restored from the backup of 2026-10-07 02:00 UTC; everyone signs in again". The page then sends you to sign in. Passwords and two-step stay as they are now (nobody gets back a password they changed); sessions and sign-in links end.

The server's backup setup is not rolled back: the schedule, the off-site destination, the recovery key and the record of past backups stay as they were just before the restore.

If the server does not come back (the restart failed, the process exits with status 3), the page says so after two minutes. Start the server again as you normally do (ok serve, or your service): it finishes the restore before it serves. A crash at any step leaves either the state as it was or the restored one, never a mix; the plain copy of the state as it was, pre-restore-<time>.db beside it, is removed after 7 days.

Undoing a restore. Restore the backup of kind Before a restore the same way; the sheet says "This undoes the restore of ...".

ok restore, with the server stopped#

For a server that will not start, a backup that is not checked, or a new machine:

ok --state /var/lib/orchkernel/state.db restore 2026-10-07T020000Z-a1b2c3
ok --state /var/lib/orchkernel/state.db restore /path/to/orchkernel-2026-10-07T020000Z-a1b2c3.tar.gz.age

A backup id is looked for in the local folder, then in the off-site bucket. It runs the same checks, prints the same summary, asks for the company's name (--confirm NAME gives it; --yes skips the question for a checked backup), and does the same swap. It is refused with exit status 75 while the server runs. A backup that is not checked needs --yes and the company's name, and says why first.

On a new machine (the server and its key are lost): put the backup file and the recovery key file on it, and restore into an empty folder, giving the full state path:

mkdir -m 700 /var/lib/orchkernel
ok --state /var/lib/orchkernel/state.db restore orchkernel-2026-10-07T020000Z-a1b2c3.tar.gz.age \
  --identity orchkernel-recovery-key-9f3e1a2b.txt

It writes state.db, secrets.key (and kept keys) from the backup, and llm.yaml, mcp.yaml and pack folders where none exist (--replace-config replaces existing ones), then prints the exact command to start the server on it:

restored 2026-10-07T020000Z-a1b2c3 into /var/lib/orchkernel/state.db; start the server on it with:
  orchkernel --state /var/lib/orchkernel/state.db serve

Use that command (or set OK_STATE in your service): a plain ok serve elsewhere uses ./.orchkernel/state.db and starts an empty workspace. People sign in with the passwords they had at the backup.

Versions#

A plain copy for scripts: ok backup#

ok backup /backups/state-$(date +%F).db copies the state file alone with SQLite's online backup API, 0600, safe while the server runs. It is not encrypted and holds no key or configuration; restoring it means stopping the server, moving state.db (and any -wal, -shm) aside and copying the file in, with the secrets key it was sealed under. Prefer the encrypted backups above.

Tenants#

Under --tenants, each tenant's admins have their own Backups page: off until they turn it on, kept under the operator's root, encrypted to the tenant's key, its recovery key and OK_BACKUP_RECIPIENTS. A restore from it goes through the tenancy restore, without restarting the server. The operator's nightly copies keep running beside them. See Hosting several organisations.

Upgrades and rollback#

1. Back up (Back up now, or ok backups run with the server stopped), and stop every scheduled job that opens the state file with the CLI.
2. Replace the program (run the install script again) or the image tag.
3. Start the server and check /api/health.

Some releases change how the file is stored, and the first open by the new program converts it:

• It checks there is room first. Without it, the open fails with "upgrading the state file needs N bytes free beside it, and M are" and nothing changes.
• It writes a copy beside the file, state.db.pre-upgrade-v<n>.db (0600), and keeps it. Delete it yourself once the new release has run well.
• It converts in one transaction, so a stop half-way leaves the old file untouched.

An older program refuses a converted file: "the state file is newer than this binary". Rolling back after a storage change means restoring the pre-upgrade copy or a backup, and losing what changed since the upgrade. Releases without a storage change can be rolled back by swapping the program again.

If an old server kept writing after the conversion, the next open may refuse with "the state file was written by two versions of orchkernel". Stop everything and restore the pre-upgrade copy or a later backup.

The first open by a release that adds record indexes builds them before the server answers: a few seconds per index on a large collection, once.

Running commands while the server runs#

The server holds a lock on state.db.lock, beside the state file, from before it opens the file until it exits (and across its own restart after a restore). The operating system lets go of it when the process ends, however it ends, so there is no stale lock to clean up on a local disk.

• Commands that only read open a snapshot of the file and never write it, so they work beside the server and see what it has saved: ok actor list, ok runs, ok tasks, ok inbox, ok log, ok replay, ok threads, ok search, ok brain query, ok module list, ok token list, ok user list, ok mail show, ok connection list, ok policy export, ok backups list and the other listings.

• Commands that change the state (ok init, ok actor add-human, ok token issue, ok pack install, ok kill, ok tick, ok ask, ok secrets rotate, ok backups run, ok restore, every other one) are refused while the server runs, with exit status 75 and nothing written:

  textCopy

  The server is running on /data/state.db (pid 4121, since 09:30 UTC).
  Use the workspace or the API (add --server https://work.acme.example), or stop the server, then run this again.
  

  Two such commands at once are refused the same way, and so is a second server.

• ok backup, ok audit, ok doctor, ok backups decrypt and ok embed fetch load no kernel of their own and work beside the server.

• With --server, the people, mail and policy commands go to the running server's API instead (see CLI reference).

• Under tenancy the same holds per tenant.

In Docker, docker exec orchkernel orchkernel --as ada actor list works beside the server; a command that changes the state needs the container stopped and a one-off container on the same volume.

The state on a network disk (NFS, SMB, AFP, WebDAV, FUSE) is not supported, as SQLite's write-ahead log is not: the start and ok doctor say so.

Break glass#

Where the file system refuses locks altogether (ENOLCK, EOPNOTSUPP), a command that changes the state is refused with "the file system holding /data/state.db does not support locks". If you are certain no server runs on the file, --no-lock lets it through, with "warning: writing without the lock; make sure no server runs on this file". --no-lock is refused while a live process holds the lock: there is no override for a running server, because a write beside it is exactly what the lock prevents.

What survives a crash#

The tests behind this table kill the server with SIGKILL in the middle of 16 people's writes, in the middle of a run, while a run waits for an approval, right after a password change, in the middle of a backup of a 200 MB file, at both steps of a key rotation and at three points of a restore, then check with a fresh process that PRAGMA integrity_check answers ok, ok audit verify passes, every answered write is there and a new server starts (crates/ok-cli/tests/it/crash.rs, restore_crash.rs).

Watching it run#

The server advances its own clock every 30 seconds: scheduled triggers, overdue work, stalled threads, archiving. This is the tick. To drive it from your own scheduler, call POST /api/tick with an admin token (deploy/k8s/tick-cronjob.yaml does this every minute).

When the audit log cannot record a decision (a full disk, say), the server refuses the action with 503 audit_unavailable rather than act unrecorded.

Troubleshooting#

Not possible yet#

• Publishing: no release, image or install script has been published yet; build them yourself (Install).
• More than one server or replica for a company, or failover. There is no Postgres backend.
• Single sign-on (OIDC, SAML) and passkeys.
• Certificates by DNS or TLS-ALPN challenge, and wildcard certificates from Let's Encrypt (give your own with --tls-cert).
• More than one off-site destination, or destinations other than S3-compatible storage (SFTP, for example).
• Restoring to a point between two backups.
• Changing state with the CLI while the server runs.
• Keeping the state on a network disk.
• Exporting metrics or traces to OpenTelemetry; read standard output and the event log.
• Apple-notarized macOS programs (the install script does not quarantine them); Windows.

License#

OrchKernel is source-available under the Business Source License 1.1. You may use, modify and run it in production for your own organization, or run a deployment for one client organization; separate deployments for separate clients are fine. Offering it as a hosted or managed service in which one deployment serves more than one organization needs a commercial license. Each version converts to Apache-2.0 four years after it is first published, and the client libraries under sdk/ are Apache-2.0 already. The LICENSE file in the repository is the authoritative text.

For developers#

• ok serve and the tenancy host start every kernel through one path, crates/ok-server/src/startup.rs (prepare_kernel): a pending restore is finished first (ok_server::restart::finish_pending_restore), then the secrets key check, the context hooks, the process configuration, the module reconcile, the gateway's observer, the audit verifier, the gateway start-up sweep, the statistics refresh, the local indexer worker, the backup job (backup_jobs.rs), then the request state. Each 30-second tick runs tick_one.
• HTTPS is crates/ok-tls (the ACME manager on instant-acme, the certificate store, the SNI resolver, the port 80 router) served by crates/ok-server/src/https.rs; the flags are crates/ok-cli/src/serve_tls.rs.
• Backups are crates/ok-backup (bundle, age encryption, the local and S3 destinations, retention, the run, restore); the routes are crates/ok-server/src/backups_api.rs, the settings and log rows crates/ok-kernel/src/backups.rs. A restore's swap is shared with the tenancy restore (crates/ok-tenancy/src/swap.rs); the restart hands the lock over across exec (crates/ok-server/src/restart.rs, ok_db::lock).
• ok backup loads no kernel: it copies through SQLite's online backup API (Kernel::backup_file) into a .partial file made 0600, renamed into place once complete.
• The state lock is ok_db::lock (flock); crates/ok-cli/src/access.rs classes every command (no kernel, snapshot, write, serve).
• The key file is ok_secrets::keyfile; its resolution, generation and rotation are crates/ok-cli/src/secrets_file.rs. Private files are ok_db::private.
• deploy/install.sh is tested by crates/ok-cli/tests/it/install_script.rs; the release workflow is .github/workflows/release.yml.
• docs/deployment.md is the full operator reference: the Kubernetes bootstrap pod, MCP servers, connection key rotation, archiving and indexes.