Hosting several organisations

Draft for review

One OrchKernel server can serve several separate companies, called tenants. Each tenant is a whole OrchKernel of its own: its own state file, its own event log, its own people and tokens, and its own key for the secrets it stores. Nothing inside a tenant knows the others exist. A request is matched to exactly one tenant before anyone is signed in, and that tenant has never heard of another tenant's tokens, records or secrets.

This page is for the person who runs the server: the operator. It covers setting up a tenancy root, creating tenants, reaching them, backing them up, and suspending and deleting them. A plain single-company server is described in Self-hosting, backups and upgrades; nothing here applies to it.

Is this for you#

License fit#

OrchKernel is source-available under the Business Source License 1.1. Read this before you plan a deployment with several tenants.

• By default, one deployment serves one organization: your own, or one client organization you run it for. Separate deployments for separate clients are allowed.
• Serving more than one organization from one deployment is hosting in the license's terms and needs a commercial license. Every tenant that shares a root, its operator key, its configuration or its operator counts as one deployment. Contact the licensor through https://github.com/arsp2020/orch-kernel.
• Without a commercial license, use tenancy inside one organization: two subsidiaries under common control, or staging beside production.

The words on this page#

Try it on one machine#

These steps create two tenants, lumen and nova, and reach each from a browser. They use ok.localhost as the base host, because browsers send every *.localhost name to your own machine. No model is needed: nothing here asks an agent to do anything. Run them in an empty directory, with ok on your PATH. If ports 8080 or 8081 are taken, pick two others and use them throughout.

1. Make an operator key and pick a root.

   shCopy

   export OK_TENANTS_DIR=$PWD/root
   export OK_OPERATOR_KEY=$(head -c 32 /dev/urandom | base64)
   

   Keep the key: without it no tenant opens. For this test, save it next to the root, not inside it (echo "$OK_OPERATOR_KEY" > operator.key), so a second terminal can set the same value. In production it comes from your secret manager (see The operator key).

2. Create two tenants.

   shCopy

   ok tenant create lumen --blueprint saas-startup --contact "owner@lumen.example" > lumen.token
   ok tenant create nova --admin ada --contact "ops@nova.example" > nova.token
   

   Each prints its first admin's token once, alone on standard output, and says on the terminal "created tenant lumen with admin root; the token above is shown once". lumen gets the saas-startup blueprint (see ok blueprint list); nova starts empty with one admin, ada. The tokens are valid for 30 days.

3. Make an operator token.

   shCopy

   export OK_OPERATOR_TOKEN=$(ok operator token issue --label "on-call" --ttl 90d)
   

   The terminal says "operator token f3c528b5 (on-call) expires 2027-01-03...; shown once". The token itself, starting orchk_op_, goes to standard output, so the line above saves it as OK_OPERATOR_TOKEN. Issue operator tokens before you start the server: the command is refused while one runs.

4. Start the server.

   shCopy

   ok serve --tenants $OK_TENANTS_DIR --bind 127.0.0.1:8080 \
     --base-host ok.localhost --operator-bind 127.0.0.1:8081
   

   It says "OrchKernel serving the tenants of .../root on http://127.0.0.1:8080; operator API on http://127.0.0.1:8081". With no model key set, it also warns that no model provider is configured; that is fine for these steps. Leave it running and use a second terminal, with the same OK_TENANTS_DIR, OK_OPERATOR_KEY and OK_OPERATOR_TOKEN set.

5. Reach each tenant. Open http://lumen.ok.localhost:8080, paste the token from lumen.token and choose Sign in. You land on Threads as root, with lumen's agents under Talk to an agent (Analyst Agent, Product Agent, Sales Assistant, Support Agent and the rest). Open http://nova.ok.localhost:8080 in another tab and paste nova.token: you are ada, and because nova has no blueprint you land on Setup ("Set up your company"). Paste lumen's token at nova and it is refused: "That token wasn't recognised or has expired."

6. Open the Tenants page. Open http://127.0.0.1:8081 (it goes to /operator, titled "OrchKernel operator"), paste the operator token and choose Sign in. The Tenants page lists lumen and nova, both Active. Needs attention shows "No backup in 48 h" for both, because neither has been backed up yet. Records, Events, Size and Version show a dash until the tenant is first checked (Check now) or closed.

From here, back one up, and suspend and delete one.

The tenancy root#

<root>/
  operator.db            the list of tenants, operator tokens, the operator log
  server.lock            held by the one server serving this root
  tenants/<name>/
    state.db             the tenant's data
    key.sealed           the tenant's key, locked under the operator key
    keys/<kid>.sealed    older tenant keys a kept backup still needs
    tenant.json          a copy of the tenant's row, for recovery
    owner.lock           held by whichever process has the tenant open
    pre-restore-<time>.db  the data as it was before a restore
  backups/<name>/<time>.db and .json   backups and their manifests
  backups/operator/<time>.db           nightly copies of operator.db

• Directories are created readable by the owner only.
• operator.db decides which tenants exist. A directory with no row is never served.
• If operator.db is lost, ok tenant registry rebuild brings the tenants back from each tenant.json ("restored 2 tenant(s): acme, beta"). Operator tokens and the operator log are not in those files: issue new tokens, or copy back a file from backups/operator/ instead. Run the rebuild with the server stopped.
• One owner per file. A second server on the same root is refused ("another server runs on .../root (its server.lock is held)"). A command that opens a tenant the server has open is refused too ("tenant lumen is open in a running server; use ok --server or close it").

Tenant names#

A name is 1 to 63 characters of a-z, 0-9 and -, does not start or end with -, and has no -- in positions 3 and 4. It is the tenant's web address, its directory and part of its key, so it can never change.

Reserved names: operator, www, api, admin, mcp, static, assets, status, health, login, auth, mail.

Routing and DNS#

The front picks the tenant from either:

• the host name, <name>.<base host> (set --base-host, or OK_TENANT_BASE_HOST). Port, letter case and a trailing dot do not matter; or
• the x-ok-tenant: <name> header.

Browsers and vendor webhooks use the host name. Agents, the SDKs and the CLI can use either.

In production, point a wildcard DNS record such as *.ok.example.com at your TLS proxy, give the proxy a wildcard certificate, and have it pass the Host header through unchanged. Each tenant is then at https://<name>.ok.example.com. Set --trusted-proxy to the proxy's address so failed sign-ins are counted per client, not per proxy. Self-hosting has the proxy setup.

What callers see:

A request without a token gets 401 before any tenant is opened, so anonymous traffic cannot wake tenants.

Inside a tenant everything works as in a single-company server: the workspace, the API, webhooks, the gateway and MCP.

Running commands against a root#

Against a running server, the usual remote commands reach one tenant with OK_TENANT (it sends x-ok-tenant) and that tenant's token in OK_TOKEN:

export OK_TOKEN=$(cat lumen.token)
OK_TENANT=lumen ok policy export ./lumen-policy --server http://127.0.0.1:8080

You see "exported 11 files to ./lumen-policy". In production you can name the tenant by its host instead: --server https://lumen.ok.example.com. Plain http is accepted only to a loopback address such as 127.0.0.1; the CLI does not count lumen.ok.localhost as one, so on one machine use 127.0.0.1 with OK_TENANT.

ok tenant list --health adds each tenant's health; --json prints the rows; the global --state suspended keeps only suspended tenants.

Creating and adopting tenants#

ok tenant create <name> takes:

A failed create leaves nothing behind. The first admin's token is valid for 30 days; to give someone else in the tenant a token, that admin signs in and issues one from the People page.

Adopting a single-company server. ok tenant adopt copies an existing state file into the root as a new tenant and moves its stored secrets from its old key to a new tenant key. The original file is not changed.

export OLD_KEY=<the server's OK_SECRETS_KEY>
ok tenant adopt acme --state /srv/ok/state.db --secrets-key-env OLD_KEY

You see "adopted /srv/ok/state.db as tenant acme". Run it with the tenancy server stopped. The single-company server that owns the file can keep running: the adopt reads a copy.

To try it with the demo company, run ok demo in the same directory, then:

export OLD_KEY=$(head -c 32 /dev/urandom | base64)
ok tenant adopt acme --state orchkernel-demo/state.db --secrets-key-env OLD_KEY
ok tenant admin-token acme --actor ada --reason "adopted the demo"

The demo stores no secrets, so any key works in OLD_KEY. ok --tenant acme --as ada actor list then shows ada, alice, maya, erin and the rest of the demo company, with its agents. The last command gives ada a sign-in token for acme (break-glass); it says "break-glass token for ada in tenant acme; recorded in the operator log and the tenant's log".

For a file that does store secrets, the code refuses the adopt when none of them opens with the key you named ("no sealed secret opens with the old key; check --secrets-key-env"). That case was not tried for this guide.

The operator key and its custody#

operator key (OK_OPERATOR_KEY)       outside the root, from your secret manager
  locks: one key per tenant          tenants/<name>/key.sealed
           locks: that tenant's connection, webhook and callback secrets

• Never store the operator key under the root. Inject it into the server's environment from your secret manager or KMS. OrchKernel has no built-in KMS client.
• Back it up off the host, apart from the root's backups. Without it no tenant opens. With no key set, the server refuses to start ("OK_OPERATOR_KEY is not set; it must hold the operator key"). With a wrong key it starts, but every tenant answers 503 tenant_unavailable, and the server's log says "key.sealed does not open with the operator key (or the previous one)".
• A key.sealed copied into another tenant's directory does not open there.
• Someone who steals the files without the operator key can read records (the data files are not encrypted; use disk encryption) but not secrets. The key without the files opens nothing.
• OK_SECRETS_KEY is refused under --tenants: each tenant uses its own key.

Rotations#

One tenant's key.

ok tenant rotate-key lumen --server http://127.0.0.1:8081

The tenant pauses briefly (503 tenant_draining), its secrets are re-locked under a new key, and the answer shows old_kid and new_kid (key ids, never the keys). The old key is kept as tenants/lumen/keys/<old_kid>.sealed while a backup still needs it. Other tenants are not touched, and lumen's tokens keep working.

The operator key. With the server stopped:

1. Make a new key: export NEW_OPERATOR_KEY=$(head -c 32 /dev/urandom | base64).
2. Run ok operator rotate-key --new-key-env NEW_OPERATOR_KEY. You see "operator key bd1c0920 -> ac6a7163: re-sealed 2 file(s), 0 already under the new key" and "now set OK_OPERATOR_KEY to the new key before starting the server". Running it again is safe: it skips files already done ("re-sealed 0 file(s), 2 already under the new key").
3. Set OK_OPERATOR_KEY to the new key. If the command listed files still under the old key, also set OK_OPERATOR_KEY_PREVIOUS to the old one until a rerun reports none left.
4. Start the server, and store the new key off the host.

Backups#

A backup is a copy of one tenant's data, with a manifest (a .json file that describes the copy) beside it. Over --server it is safe while people use the tenant. A local ok tenant backup is refused while the server runs.

1. Back up nova now.

   shCopy

   ok tenant backup nova --server http://127.0.0.1:8081
   

   The answer shows the copy's time, size, SHA-256, the tenant key's id (kid, never the key) and the head of its event log (head, the last event's number and hash). The files appear under root/backups/nova/, as <time>.db and <time>.json. On the Tenants page, open nova and choose Back up now to do the same; the backup appears in the tenant's Backups table.

2. See what is kept. ok tenant backups nova lists each backup with its path, such as backups/nova/2026-10-05T120912.465Z.db. It works whether or not the server runs.

With the server stopped, ok tenant backup --all backs up every active and suspended tenant and prints one line each ("lumen: 552960 bytes, sha256 ..., head #53"); ok tenant backup nova --to <dir> writes the copy and its manifest to another directory instead.

While the server runs it also:

• backs up every changed tenant, and operator.db, nightly at OK_TENANT_BACKUP_AT (default 02:00 UTC);
• deletes backups older than OK_TENANT_BACKUP_KEEP_DAYS (default 30).

Offsite copies are your job. Copy root/backups/ to another place on a schedule, check each .db against the sha256 in its .json, and keep the operator key out of that place.

Restore#

A restore replaces a tenant's data with a backup. Everything after the backup is lost.

ok tenant backups lumen
ok tenant restore lumen 2026-10-05T112024.620Z.db --server http://127.0.0.1:8081

What happens:

1. The backup is checked first (it belongs to this tenant, its SHA-256 matches, the file and its event log are sound). If not, nothing changes.
2. The tenant pauses, the current data is kept as tenants/lumen/pre-restore-<time>.db for 7 days, and the backup takes its place. The answer names that file.
3. A state_restored event is added to the tenant's log, so its history shows the rewind. Tokens revoked since the backup stay revoked.

To see it: back up lumen, then add a feature request (for example as root, through POST /api/brain/collections/feature_requests/records with {"id": "fr-dark-mode", "fields": {"title": "Dark mode"}}), then restore the backup. The feature request is gone (404), lumen's admin token still works, and lumen's Events end with state_restored by operator.

Over --server a restore names a file under the tenant's own backups/ folder. A path is refused ("name a backup file under the tenant's backups, not a path").

Copying one tenant into another. To give a new tenant beta a copy of lumen's data:

ok tenant create beta --admin ada --server http://127.0.0.1:8081 > beta.token
ok tenant restore beta 2026-10-05T121033.075Z.db --from-tenant lumen --server http://127.0.0.1:8081

Without --from-tenant, the file is looked for under beta's own backups and is not found ("no backup ... of tenant beta"). With it, the answer shows "from_tenant": "lumen", "resealed": true (lumen's secrets now locked under beta's key) and tokens_revoked: every token in the copy, and beta's own first token, stop working. Give an admin a way back in with a break-glass token, for example ok tenant admin-token beta --actor root --reason "after clone" --server http://127.0.0.1:8081 (root is lumen's admin, and now beta's).

Upgrades#

Each data file carries a storage version. A newer binary moves each tenant up when it first opens it, and at start upgrades closed tenants in the background with a pre-upgrade-v<old> backup each (turn this off with OK_TENANT_UPGRADE_AT_START=0). An older binary refuses a newer file.

1. ok tenant backup --all, with the server stopped.
2. Replace the binary and start the server.
3. Watch the Tenants page's "storage version" line, or by_version in GET /operator/health, until every tenant is on the new version. ok tenant upgrade --all runs it on demand and skips tenants already done.

To roll back, restore each tenant's pre-upgrade-v<N> backup under the older binary. Changes since the upgrade are lost.

Health and the Tenants page#

The Tenants page is on the operator listener only: a tenant's own address answers 404 for /operator, so an operator token is never asked for there. Reach it through an SSH tunnel (ssh -L 8081:127.0.0.1:8081 host). The server warns if --operator-bind is not a loopback address.

It shows counts, sizes and times, never a tenant's content.

On a tenant, Suspend asks "Why suspend nova? Its users get 403 at once; its data is kept." and needs a reason. Resume (shown greyed out unless the tenant is suspended), Back up now and Check now act at once; after a check the panel says "Checks passed". Refresh reloads the figures. Create, delete, restore and key rotation stay on the command line, as the panel says. The operator token is kept when you reload the page, until you close the tab or choose Sign out. On a phone the cards stack and the list scrolls sideways inside its card.

What the page does not show yet, in this version:

• Records, Events, Size, Version and the tenant key id show a dash until the tenant has been checked or closed once.
• "last opened" and "last tick" always show a dash.
• The list says Health "Ok" where the tenant's panel says "Healthy", and a purged tenant still shows "Ok".
• The list and the panel give the same file in different units: 344.1 KB in the list is 336.0 KB in the panel.

Checks. Nightly at OK_TENANT_CHECK_AT (default 03:00 UTC), and on demand with ok tenant check lumen or Check now, the server checks each changed tenant's file, its search index and its event log. A tenant that is open gets the file and log checks only. A broken event log in an open tenant is raised to that tenant's admins.

The operator log. GET /operator/log lists, newest first, every tenant_created, tenant_state_changed, tenant_admin_token_issued, tenant_key_rotated, operator_key_rotated, tenant_restored, tenant_upgraded, tenant_purged, operator_token_issued and operator_token_revoked, with reasons. It is chained like a tenant's log: ok audit verify --state root/operator.db checks it.

Break-glass admin tokens#

When a tenant's admin loses their token, or asks for support:

ok tenant admin-token nova --actor ada --reason "lost first token" --server http://127.0.0.1:8081

It prints a token for an existing admin, valid 30 days. Only an active human admin can get one: --actor maya at a tenant where maya is not an admin is refused ("maya is not an active human admin of this kernel"). It is recorded twice: in the operator log, and in nova's own Events as a token issued by operator with the label "break-glass: lost first token", where nova's admins see it.

The clock for closed tenants#

Tenants that nobody is using are closed to save memory, but their scheduled work (triggers, deadlines, reminders, retries) still runs:

• Every 30 seconds the server ticks open tenants that have work due, and every open tenant at least every 5 minutes.
• A closed tenant whose work is due is opened, ticked and closed again. Work that may not make progress (waiting agent work, a paused module's retry) wakes a closed tenant at most every OK_TENANT_RETRY_WAKE (default 5m).
• Every active tenant is woken at least once a day.
• Suspended and deleting tenants are never woken. A resumed tenant catches up once.

A webhook, an agent action or an MCP call to a closed tenant opens it and is answered as usual. The Tenants page shows each tenant's next wake.

Suspending and deleting a tenant#

Add --server http://127.0.0.1:8081 while the server runs. Every command needs a reason except resume and undelete.

1. Suspend. The answer shows "state": "suspended". Signed-in users are cut off at once, including open event streams: ada's next request at nova gets 403, and her browser shows "Suspended". ok tenant list --state suspended lists nova.
2. Resume. The answer shows "state": "active", and ada's token works again without signing in anew.
3. Delete. The answer shows "state": "deleting" and purge_after. ok tenant list shows "nova deleting created ... purge_after ...". Within the grace period, ok tenant undelete nova puts it back to suspended, and resume serves it again.
4. Purge. Once purge_after passes, the running server purges the tenant within a minute. With --now it happens immediately; the answer counts the files, bytes and backups removed and says "key_destroyed": true. ok tenant list then shows "nova purged", and a new ok tenant create nova is refused: the name is taken.

A purge removes, in order: the tenant's key (which makes every stored secret unreadable in every copy, offsite ones included), its data file and directory, and its local backups. The row stays as a purged tombstone and the name is taken for 400 days. The operator log keeps the name, dates, counts and reason, never content. Copies outside the root, such as your offsite backups, are not reached; they age out under your own retention.

ok tenant set nova --final-backup asks for a final backup before a purge. In this version that backup is written under backups/nova/ and then deleted by the same purge, so nothing of it is left. To keep a last copy, run ok tenant backup nova --to <dir> (server stopped) before you delete, or copy the newest file from backups/nova/ yourself.

Other settings with ok tenant set (add --server while the server runs):

The answer shows the tenant's whole row, with "applied": true once an open tenant has taken the change.

Limits#

• One server per root, on one host. No high availability: a host failure is recovered from backups. To move a tenant to another host, stop, copy its directory, run ok tenant registry rebuild there and start.
• Capacity. Up to OK_TENANTS_OPEN (200) tenants stay open; the least recently used idle ones are closed beyond that. At OK_TENANTS_MAX_OPEN (400) open and none idle, another tenant gets 503 tenant_capacity. One tenant takes up to 64 requests at once, 256 event streams, and 8 streams per token.
• As measured on an 8-core laptop: about 2.5 MB of disk per tenant with 1,000 records, about 2 MB of memory per open tenant, and a few milliseconds to open a closed tenant. Tenants share CPU and the model providers' rate limits; there is no per-tenant CPU quota.
• Shared by every tenant: the model providers and their keys (spend is counted per tenant), the server-wide connection host list, audit and gateway settings, and the failed sign-in limiter.
• Refused under --tenants, because each would act for every tenant at once: --dev-auth, --dev-echo-tools, OK_SECRETS_KEY, OK_WEBHOOK_SECRET(S), OK_MCP_CONFIG and a ./mcp.yaml. The server stops at start and names the setting.

Not possible yet#

• Custom domains per tenant, or renaming a tenant.
• More than one server per root, or per-tenant model providers.
• A built-in KMS client for the operator key.
• Sign-up by tenants themselves.
• One admin change applied to every tenant at once.
• Creating, deleting, restoring or rotating keys from the Tenants page.
• Issuing or revoking an operator token, adopting a server or rebuilding the tenant list while the server runs.
• A final backup that survives the purge it was made for.
• Per-tenant model spend on the Tenants page.

For developers#

Operator API#

On --operator-bind (default 127.0.0.1:8081), with Authorization: Bearer orchk_op_.... A tenant token gets 401 here, and an operator token gets 401 at every tenant.

Errors include 409 name_taken on create and 409 tenant_busy when a restore cannot drain the tenant within 30 seconds.

Environment variables#

The full operator reference, with the backup manifest, the restore journal and the bench figures, is docs/tenancy.md in the repository. Commands are listed in the CLI and API reference.