Events and the audit log

Draft for review

Everything that happens in OrchKernel is written to one append-only event log: every policy decision, every write, every approval, every step of an agent's run and every tool it calls. The log is what the Events page shows, what a run's replay is built from, and the company's audit record.

Each event is also chained to the one before it by a hash (a fingerprint of the event's content), so a row that is later edited, deleted or slipped in by hand can be found. This page explains what is recorded, how to find and replay the events of one run, and how to check the chain. By the end you can find every event of the demo refund, replay it, verify the log and see a checkpoint.

All of it works in the demo without a model key: with no model configured, the demo's stub planner answers instead of a model, and it knows the refund ask from the Quickstart. The examples below come from a fresh demo company after two steps of the Quickstart: maya's refund ask approved by ada ("Approve the parked refund as Ada"), and, where noted, alice's follow-up email ("Watch a governed write wait"). Event numbers in your demo match if you do the same steps in the same order; hashes, ids and times differ.

This page describes the event log as it works today on the golive branch. What is not possible yet, and known problems, are listed at the end.

What is recorded#

Every event has a number (#403), a time, the person or agent responsible (or kernel when the system itself did it), and, where they apply, the run, task and thread it belongs to. The product never changes or removes an event; it adds a new one instead.

Two things worth knowing:

• The gate's decision is written first. The gate is the check every action passes before it runs (see How the gate decides). Every step gets a Policy decision event (allowed, needs approval or denied) before it runs, even if the step then fails. If the gate cannot write its decision (a full disk, a failing file), it refuses the action rather than act unrecorded.
• Provenance is on both sides. The event says who did what, in which run, task and thread; each version of a record in the brain also names who wrote it, who it acted for, the run, the skill and the source (see The brain).

The Events page#

Click Events in the sidebar, under Company. The page lists events newest first, one line each, and adds new events at the top as they happen, without a reload.

What you see depends on who you are:

In the demo, after the refund, maya sees 37 events: the 6 ticket writes she made when the demo was set up, and the 31 events of her conversation, her task and the watcher's run. alice sees only the 10 events of the records she wrote at setup.

Each line has three parts: the time, the kind of event, and a sentence saying what happened. Some examples from the demo:

Some kinds have no sentence yet and repeat their name instead: Run step, Run finished, Model called, Thread created, Message posted and Audit checkpoint. Open the line to see the detail.

Click a line to open it. It shows the event number, the full date and time with your time zone, who did it, the event's type as stored (tool_called) and every field it carries:

#398 · Monday, October 5, 2026 at 5:27:27 PM GMT+5:30 · by Churn-risk watcher · tool_called
{ "by": "churn-watcher", "cost_cents": 0, "ok": true, "tool": "support-ops.billing.refund" }

Filtering by type#

The page has one filter, Type. It lists every kind of event, grouped by the areas in the table above. A kind the list does not know yet shows under Other.

1. Sign in as ada and click Events.
2. In Type, choose Tool called. After the refund you see two lines: "support-ops.billing.refund called by Churn-risk watcher" and "support-ops.billing.account called by Churn-risk watcher". If you also did alice's email step, a third line reads "gmail.send called by Sales Assistant".
3. Choose Approval requested: "Approval asked of Ada Park for tool.call(support-ops.billing.refund)".
4. Choose Audit checkpoint. Before the first checkpoint the page says "No audit checkpoint events."
5. Choose All types to go back.

The page shows the newest 200 events of the type chosen. It cannot filter by person, run, task, thread or date; to follow one run, use the replay below.

Following one run, task or thread#

In the workspace#

The run panel rebuilds a run from the log.

1. Sign in as ada and click Work, then the Runs tab.
2. Click the Churn-risk watcher run. A panel opens with "Churn-risk watcher acting for Maya Patel", the run's Trace and its Output. The address bar now ends in ?run=<run id>; keep that id for the command line.
3. Open Replay (raw events) at the bottom: one line per event, with number, who, type and fields. For the refund you see 23 lines, from run_started (#378) to run_finished (#402). The refund's policy_evaluated (#391) says require_approval with the reason rule support-ops-refunds-need-admin: ..., followed by approval_requested, run_parked, and approval_resolved by ada.

Replay (raw events) is shown to admins only. maya can open the same run from her Runs tab and see its Trace and Output, but not the replay. To see the events, she opens the task instead:

1. Sign in as maya and click Work. The task is done, so the list of open tasks is empty ("Open tasks (0) · 1 done").
2. Choose All, then Details on "Northwind wants a refund for the double charge".
3. Its Events section lists the task's events, each with its number, time, who, type and fields, starting with task_created (#377).

From the command line#

ok replay prints the events of one run, task or thread, oldest first. It reads the log; it does not run anything again.

ok --state <path/to/state.db> replay --run <run id>
ok --state <path/to/state.db> replay --task <task id>
ok --state <path/to/state.db> replay --thread <thread id>

ok replay and ok log open the state file as a full kernel and write it back when they finish. Do not run them on the file a running server is using: stop the server first, or work on a copy. ok backup is safe while the server runs. In the directory where you ran ok demo:

1. Make a copy: ok --state orchkernel-demo/state.db backup copy.db. You see backed up orchkernel-demo/state.db to copy.db.

2. Replay the run, with the id from the run panel's address: ok --state copy.db replay --run 8c46321e-7084-49b3-b32a-14c7510e655c (your id differs). You see the same 23 lines as Replay (raw events), for example:

   Copy

   #391   churn-watcher policy_evaluated       {"action":"tool.call(support-ops.billing.refund)", ... "decision":"require_approval", ...}
   #392   churn-watcher approval_requested     {"action":"tool.call(support-ops.billing.refund)", ... "approver":"ada"}
   #393   churn-watcher run_parked             {...}
   #395   ada approval_resolved      {... "approved":true,"by":"ada"}
   #396   kernel run_resumed            {...}
   

3. Replay the task. Its id is the task field on the run_started line (or the end of the address, ?task=<task id>, after Details). You see 23 lines again, now with task_created and the three task_state_changed lines (running, parked, done), and without the model calls and run_resumed.

4. Replay the thread. Its id is the end of the address when maya has the conversation open (/threads/<thread id>). You see 11 lines: the thread's creation, maya's message and the policy decisions and approval request of the run (see Known problems).

An id that matches nothing prints nothing.

Run, task and thread do not cover the same events:

To see everything an agent did, replay by run; for the conversation, open the thread (see Threads).

ok log#

ok log prints the newest events of the whole log, 50 by default, oldest of them first:

ok --state copy.db log                         # the newest 50 events
ok --state copy.db log --type tool_called      # only one type
ok --state copy.db log --limit 200             # the newest 200

Each line is the number, who, the type and the event as JSON. --type takes one type name as stored (approval_requested, not Approval requested). The command line reads the file directly: whoever can read the file sees every event, whatever --as says.

The tamper-evident chain#

Each event row stores the hash of the row before it and a hash of itself. A row that is changed, removed, renumbered or inserted by hand no longer links, and ok audit verify names the first place where it does not. The product itself never changes a row: the database refuses updates, deletes and unchained inserts. Those guards stop mistakes and old software, not someone who can edit the file directly; the chain is what catches that.

What the chain proves, and what it does not#

Any other event that cannot be written is counted, and an Events lost event (how many, since when) is written before the next one, so the gap is in the chain. When the disk runs low, the server writes Disk low and tells every admin in their inbox, before the gate has to start refusing actions.

Checkpoints#

While the server runs, it checks the newest part of the log on its own and records a checkpoint: an Audit checkpoint event saying "every event up to #N was verified, and the hash at #N is this". It writes one once 1,000 events have been added since the last checkpoint, or 60 minutes after it, whichever comes first, so even a quiet hour gets one. Before the first checkpoint, the count starts at event #1 and the hour starts when the server starts. Separately, every 24 hours it verifies the whole log and records the result as the last full verify.

To see a checkpoint in the demo without waiting an hour:

1. Bring the log past 1,000 events. Each save of a record is two events (the policy decision and the write), so save one of pat's roadmap items 500 times, with pat's token from the ok demo serve output and the port it printed under Workspace (8080 unless you chose another):

   shCopy

   TOKEN=<pat's token>
   URL=http://127.0.0.1:8080/api/brain/collections/roadmap_items/records
   for i in $(seq 500); do
     curl -s -o /dev/null -X POST -H "Authorization: Bearer $TOKEN" \
       -H 'content-type: application/json' \
       -d '{"id":"checkpoint-test","fields":{"owner":"pat","quarter":"Q3","size":"s","status":"proposed","title":"Checkpoint test"}}' \
       "$URL"
   done
   

   This creates the roadmap item "Checkpoint test" and saves it 499 more times.

2. A checkpoint is written within half a minute of the log passing #1,000, often while the loop is still running. As ada, the line next to the Type filter on Events changes from "Audit chain: 403 events, not checkpointed yet" to "Audit chain verified through event #1,189 · 10/5/2026, 5:32:05 PM". Your number depends on when the server's clock ticked.

3. In Type, choose Audit checkpoint and open the line: by kernel, with "events": 1189, "head": "sha256:5306...", "through": 1189 and "verified_from": 1.

4. On the command line, ok --state orchkernel-demo/state.db audit checkpoints lists it: #1190 through #1189 sha256:5306... events 1189 verified from #1 at 2026-10-05T12:02:05.... Before the first checkpoint the command prints nothing.

The ok audit commands are safe beside a running server: they open the file read-only and never write it.

The intervals are server settings, read when the server starts:

Anchors: a hash kept elsewhere#

Because the checkpoints live in the same file, someone who rewrites the log can rewrite them too. An anchor is an event number and its hash, written down somewhere the file's editor cannot reach: a ticket, a password manager, another system. Verifying against an anchor catches a rewrite that recomputed every hash.

ok --state orchkernel-demo/state.db audit head                       # the current head: an anchor
ok --state orchkernel-demo/state.db audit checkpoints --anchors > anchors.txt
ok --state orchkernel-demo/state.db audit verify --anchors anchors.txt
ok --state orchkernel-demo/state.db audit verify --anchor 403=sha256:8901...

ok audit head prints the log id, how the chain started, the head (head #403 sha256:8901...), the last checkpoint and the last full verify (none until there is one). checkpoints --anchors prints one <number> sha256:<hash> line per checkpoint, the format verify --anchors reads. An anchor on the command line is written <number>=sha256:<hash>, with the full 64-character hash.

We checked this on a copy of the demo: after editing event #398 and recomputing every hash after it, ok audit verify reported "no break", and the same command with the head written down before the edit (--anchor 403=sha256:8901...) reported BREAK at #403: anchor_mismatch: the stored hash differs from the anchor. An anchor tells you the log changed at or before its number, not where: the "last good" line it prints (#402) is just the event before the anchor.

When the chain breaks#

When a checkpoint finds a break instead, the server:

• writes Audit chain broken with the event number and the kind of break;
• puts "The audit chain is broken at #1030: hash_mismatch" in every admin's inbox;
• shows admins a red banner on the Events page: "The audit chain is broken at #1030: hash_mismatch. Investigate before anything is pruned or exported."

The banner stays after later checkpoints, which only check the events after the break. It goes away only when the server's next full verify passes, which can be up to 24 hours later. A passing Verify now does not clear it.

The Events page shows what is stored, edited row included; it does not mark the edited row. Use ok audit verify to find it.

Restores and backups#

A backup is a state file like any other: verify it before you trust it (ok --state /backups/state-2026-10-12.db audit verify). A copy made with ok backup keeps the log id of the file it came from, so anchors taken from the live file apply to it. See Self-hosting, backups and upgrades.

Verifying the chain#

With ok audit verify#

1. Run ok --state orchkernel-demo/state.db audit verify. On an untouched demo after the refund you see:

   Copy

   log 98e5d089-...  chain audit-chain/1  genesis #0 (started chained)
   checked #1..#403 (403 events, 0 anchors) in 0.0 s; trust: genesis
   no break; head #403 sha256:8901...
   

   and the exit code is 0. "genesis" is the start of the chain: the check began there, so it trusts nothing it did not check itself.

2. Check only part of the log with --from and --to: audit verify --from 350 --to 403. The line now says trust: stored hash of #349: the range is consistent with itself and with the hash stored at #349, nothing more. Add an anchor at or before the start for a stronger answer.

3. Add --json for the same result as JSON, or --all-breaks to list up to 100 breaks instead of stopping at the first.

When something was changed, the output names the first break and the last good event. We deleted #300 and edited #398 on a copy. Plain verify stops at the first break:

checked #1..#403 (300 events, 0 anchors) in 0.0 s; trust: genesis
BREAK at #300: missing: row #300 is absent
  last good  #299 sha256:49f8...

--all-breaks adds BREAK at #398: hash_mismatch: the stored body does not hash to the stored hash. With --from 350 only the edit shows, and a third line says whether the next event still links to the broken one (one row edited in place) or not (a rewrite from there on):

BREAK at #398: hash_mismatch: the stored body does not hash to the stored hash
  last good  #397 sha256:eebb...
  next row   #399 links to the stored hash of #398 (one row edited in place)

In Governance, Audit#

Admins have an Audit tab under Governance. Others do not see the tab; opening its address (/governance/audit) directly says "The audit log is for admins."

1. Sign in as ada, click Governance, then Audit.
2. The log shows the log id, how the chain started ("The chain started with the log: every event is chained."), and the head, shortened (head #403 sha256:8901d2bb3c2daade…), with a reminder to write the head down.
3. Checks shows "last checkpoint: none yet" (or "through #1189 at 10/5/2026, 5:32:05 PM" once there is one) and "last full verify: not yet run" until the server's first daily verify.
4. Press Verify now. The whole log is checked and the result appears below: "Verify: no break", "checked #1..#403 (403 events) · trust: genesis". On a broken log it lists each break with its kind, number and detail, for example "hash_mismatch at #1030: the stored body does not hash to the stored hash (one row edited in place)", and the last good event.

Verify now writes nothing: it does not count as the last full verify, add an event or clear the Events banner.

A log from before the chain#

A state file written before the chain existed is sealed the first time a version with the chain opens it, and an Audit chain started event records how many events were sealed. The Audit tab then reads, for example, "41,207 earlier events sealed at upgrade on 12 October; the chain is strict from #41,208." Sealing proves only that those events have not changed since the upgrade. Every program that opens the file must run the new version, or its events are lost; see Self-hosting, backups and upgrades. A new company, like the demo, starts chained.

Not possible yet#

• Filtering the Events page by person, run, task, thread or date, or paging past the newest 200 events. Use ok replay, or the API below.
• Recording a full verify from the screen, or clearing the break banner after investigating, other than waiting for the server's daily verify.
• Streaming events to a SIEM (a security monitoring system), redacting them, or pruning old events.
• Changing checkpoint intervals from the screen (server settings only).

Known problems on this page#

• A thread's replay misses its runs. Runs in a thread write their start, steps, tool calls, approval decision and end without the thread's id, so ok replay --thread (and the API's thread filter) shows 11 of the 31 events of maya's refund conversation. Replay the run instead.
• ok log and ok replay write the state file. They only read, but they save the file when they finish (a copy's checksum changes after ok log --limit 1), so beside a running server they can write over what the server saved. Use a copy made with ok backup.
• One unreadable event empties the Events page. If a row was edited so that it no longer reads as an event, the page shows "Couldn't load events" for everyone, just when the chain banner asks you to investigate. Filtering by another type still works.
• Thin sentences. Run finished never says whether the run failed, Thread created never shows the goal, and Run step, Model called, Message posted, Audit checkpoint and Audit chain broken only repeat their name.
• Quiet commands. ok log --type a,b prints nothing (it takes one type), and ok replay with an unknown id prints nothing and succeeds.
• "Refusals summarised". The Type filter spells this one label the British way.

For developers#

API#

All under /api, with Authorization: Bearer <token>.

The /audit routes are for human admins only. Anyone else, an agent acting for an admin included, gets 403 denied ("policy denied audit.read: the audit log is for human admins only"), and no token scope reaches them.

Audit events#

Break kinds#

The hash construction and the upgrade rules are in docs/audit.md.