Audit Trail¶
Learning objectives
After reading this page you will understand:
- Why a full event log matters for an exchange and what
pm-auditrecords - The JSONL log format, how
pm-auditcloses each day and how days are archived - All
pm-auditstartup options including write buffering - How to query audit logs from the command line using every
pm-audit-cliverb - When to use the optional SQLite index for faster queries
- A practical cookbook for common audit investigation workflows
Overview¶
EduMatcher maintains a complete, append-only, tamper-evident record of every message that passes through the ZeroMQ bus. This is the audit trail — a chronological log of every order, fill, trade, cancellation, session transition, and system event that occurred during a trading session.
The audit system has two independent components:
| Component | Role | Type |
|---|---|---|
pm-audit |
Subscriber — records every event to a JSONL log file, closed day by day | Long-running process |
pm-audit-cli |
Query tool — reads audit logs and prints structured output | One-shot CLI |
The recorder and the query tool are completely independent. pm-audit-cli
reads log files directly from disk and works even when pm-audit is not
running.
flowchart TB
subgraph Runtime
ENG["pm-engine\n(ZMQ PUB :5556)"]
AUD["pm-audit\nSUB :5556"]
LOG[("data/audit.log\n+ audit/ day parts\n+ audit/*.log.zst")]
ENG -- "all topics" --> AUD
AUD -- "buffered JSONL writes" --> LOG
end
subgraph Operator
CLI["pm-audit-cli\n(read-only)"]
LOG -- "stream parse" --> CLI
CLI -. "optional fast path" .-> IDX[("audit_index.db")]
end
Why a full event log matters¶
A financial exchange must be able to answer questions after the fact:
- Was a specific order ever acknowledged by the engine?
- What sequence of events led to a particular fill?
- Did a circuit breaker fire before or after an order was submitted?
- Which gateways were active at a given time?
The audit trail answers all of these by preserving every message exactly as it was broadcast, with a millisecond-precision UTC timestamp.
Data Directory¶
Audit logs are stored in the same data directory used by all other EduMatcher persistent files:
| Running mode | Default location |
|---|---|
Source checkout (poetry run pm-audit) |
<repo>/src/data/audit.log |
Installed (pm-audit on PATH) |
~/.local/share/edumatcher/audit.log |
Override with EDUMATCHER_DATA_DIR or --audit-log-file:
export EDUMATCHER_DATA_DIR="$HOME/sessions/morning"
pm-audit # writes to $HOME/sessions/morning/audit.log
pm-audit — Event Recorder¶
pm-audit subscribes to all topics on the engine's PUB socket and appends
every message as a single JSON line to the configured log file. It also binds
a small health socket (port 5562) that answers
pm-audit-cli health pings.
ZeroMQ PUB/SUB subscriptions join asynchronously. A publication made before
pm-audit's subscription has reached the engine is not queued or replayed;
the recorder cannot write an event it never received. Start pm-audit before
starting pm-engine, wait for pm-audit-cli health to report the recorder
healthy, and verify that the engine's system.startup event appears in the
audit log before trading. The health check confirms the recorder's receive
loop is running; it does not itself acknowledge the upstream subscription.
If attaching to an already-running engine, allow the subscription to join
before relying on the trail. The write buffer affects when received events
appear on disk, not whether the subscription receives them.
Startup options¶
| Flag | Default | Description |
|---|---|---|
--audit-log-file PATH |
<DATA_DIR>/audit.log |
Audit-trail output log file path |
--terminal / -t |
off | Also print each entry to stdout (useful during demos) |
--buffer-size N |
100 |
Number of messages to buffer in memory before flushing to disk |
--flush-interval SECONDS |
10.0 |
Maximum seconds to wait before flushing buffer regardless of size |
--max-file-mb MB |
10 |
Close the audit log as a day part when it reaches this size — see Day parts and archives |
--keep-days N |
0 |
Delete daily archives older than N days; 0 keeps them all |
--fsync-interval SECONDS |
1.0 |
Sync the active file to disk at most this often; 0 syncs every flush — see What can be lost |
--log-level |
WARNING |
Explicit level: CRITICAL, ERROR, WARNING, INFO, DEBUG |
-v / --verbose |
off | Increase verbosity (-v → INFO, -vv → DEBUG) |
-q / --quiet |
off | No-op — the default level is already WARNING, so --quiet currently sets the same level and has no observable effect |
--log-target |
server |
Where this process's own operational log records go (not the audit trail): server (auto-detected pm-log-srv), stdout, or file |
--log-file PATH |
— | Operational log file path — required when --log-target file. Distinct from --audit-log-file, which is the audit-trail output |
--log-failover-timeout SECONDS |
30 |
Grace window before falling back to a local log file once pm-log-srv becomes unreachable |
--log-level takes priority over -v/-q when both are given.
Write buffering¶
Writing every message to disk individually would generate enormous I/O on
active sessions. pm-audit batches messages in memory and flushes as a
group when either limit is reached:
- Buffer full: buffer contains
--buffer-sizemessages (default 100) - Timer expired:
--flush-intervalseconds have elapsed since the last flush (default 10 s) - Shutdown: Ctrl-C, SIGINT, or SIGTERM always flushes remaining messages before exit
No messages are lost on clean shutdown. A hard kill (kill -9) may lose up
to --buffer-size messages that had not yet been flushed. See
What can be lost for the whole picture.
Tuning examples:
# Default: good balance of throughput and latency
pm-audit
# High-throughput session: larger buffer, less frequent disk I/O
pm-audit --buffer-size 500 --flush-interval 30
# Low-latency / compliance: flush aggressively
pm-audit --buffer-size 1 --flush-interval 1
# Demo mode: see every event on the terminal as well
pm-audit --terminal
What can be lost¶
pm-audit is a subscriber, so the trail holds what it received. Four things
sit between an engine publication and the disk, and each has a bound:
| Where | What is lost | Bound |
|---|---|---|
pm-audit is not running |
Everything the engine publishes meanwhile | Unbounded. The next line shows a jump in eseq (replay: ESEQ_GAP), and a # recorder-start marker before it (replay: RECORDER_RESTART) says why |
| The receive queue overflows (a burst faster than the writer) | The messages past the queue's 200 000 | Counted by audit.missed_messages when the next message shows the jump |
pm-audit is killed (kill -9) |
Lines buffered but not yet written | At most --buffer-size messages or --flush-interval seconds, whichever comes first |
| The host loses power or the kernel crashes | Lines written but not yet synced | About --fsync-interval seconds (default 1) plus the line above. A write syncs when one is due, and while no line arrives the receive loop does (it wakes at least every 0.3 s), so a quiet tail is not left in the page cache |
A stopped-and-started pm-audit writes one comment line, # edumatcher-audit
recorder-start pm-audit=<version> started=<time>, before its first line, so the
restart is visible in the file itself and not only as a hole in the numbering.
pm-audit-replay reports it as RECORDER_RESTART (info) and does not count it
as an unreadable line.
The health beacon carries audit.missed_messages: the number of engine messages
pm-audit can prove it missed from forward sequence gaps. It stays 0 on a
healthy exchange. An engine restart is not a miss (the sequence begins again at
system.startup). The counter sees only a jump it observes, so messages lost
after the last one received, because pm-audit was stopped, appear in the
next run's trail as the gap, not in this counter.
The separate audit.sequence_discontinuities counter records a repeated or
decreasing eseq without an observed system.startup. Such a discontinuity
can indicate a missed restart marker or duplicate/out-of-order publication; it
does not reveal an exact number of missing messages. Check the trail and
recorder health before treating the run as complete.
Log format¶
Every line is an independent record in this format:
[2026-07-08T09:30:00.123+00:00] [trade.executed] [seq=42 msg=01J… cause=01J… chain=01J…] {"id": "TRD-001", ...}
| Part | Example | Description |
|---|---|---|
[timestamp] |
2026-07-08T09:30:00.123+00:00 |
UTC ISO-8601 with millisecond precision. This is pm-audit's receipt clock, not the publisher's |
[topic] |
trade.executed |
ZeroMQ topic exactly as broadcast by the engine |
[metadata] |
seq=42 msg=01J… cause=01J… chain=01J… |
Envelope frames, recorded as key=value pairs. Optional — see below |
{...} |
{"symbol": "AAPL", ...} |
Full JSON payload of the message, byte-identical to what was published |
The metadata section
| Key | Meaning |
|---|---|
seq |
The publisher's per-topic sequence number. Dense within a run, so a gap means messages were dropped — PUB/SUB discards silently once a subscriber falls behind, and this is how you find out |
eseq |
The publisher's count over all its messages: one total order of everything the engine published since it started. A gap is a lost message even where seq cannot show one — the last message of a topic |
msg |
ULID of this message |
cause |
ULID of the message that caused this one. Absent means nothing caused it — a scheduler tick, a circuit-breaker trip — not that the cause is unknown |
chain |
ULID shared by every message descending from one original request |
The section is omitted entirely for a message whose publisher stamps no envelope, and lines archived before the section existed are still read correctly, so a mixed archive needs no flag.
The header line and the engine's start
Every file — audit.log and each day part — begins with one header line
that names the format and the recorder:
Readers skip it (it does not start with [). The first record the engine
publishes on every start is system.startup: its version and source
commit, the run_seq it took, the hashes of the persistence files it is
about to restore from, and the compiled configuration it runs on,
verbatim. An audit trail therefore describes its own run — a replay can
rebuild the engine from that one message, after the data directory's
ref_data/ has been redeployed. Every per-topic seq starts again at 1
after it.
The engine also records every book whole, as system.book_checkpoint.<SYM>:
after startup (what the restore and the seeds made), after each session
transition, and at shutdown — each resting order in priority order with
its hidden iceberg quantity, each armed stop with its current trigger, and
the last prices. A replay can start from any of them, so a day's audit
file, or an audit that runs across an engine restart, can be checked on
its own.
This is what makes a post-mortem tractable. To follow one order submission through everything it caused:
# find the submission
poetry run pm-audit-cli --format json events --topic order.new --limit 1
# then everything in its causal chain, in order
grep 'chain=01ARZ3NDEKTSV4RRFFQ69G5FAV' data/audit.log
Lines are appended in arrival order. Within a single session they are
chronologically monotonic by receipt time — but note that receipt order across
several publishers is not causal order. cause and chain are what establish
causality; the timestamp only establishes when pm-audit saw it.
Day parts and archives¶
audit.log always holds one trading day: the day is midnight to midnight
in the exchange's locale.session_timezone (as in pm-stats), judged by each
line's receive time. At the date change, or when it reaches --max-file-mb,
pm-audit closes it as a numbered part of its day. Parts are never renamed
again and nothing is ever deleted by the recorder:
data/audit.log ← active: today's lines, being written
data/audit/audit-2026-10-25.1.log ← closed parts of a day, in write order
data/audit/audit-2026-10-25.2.log
data/audit/audit-2026-10-24.log.zst ← one archive per past day
A line received at 23:59:59.990 and flushed at 00:00:05 still lands in the
old day's part, so every line in audit-D.* was received on day D. A day that
ends while the exchange is idle, or while pm-audit is down, is closed when
pm-audit next starts.
Archives. Right after closing a day, and once at start-up, pm-audit
starts a low-priority child process, pm-audit-cli archive --pending. It
concatenates the day's parts into one zstd stream, audit-D.log.zst
(≈ 13:1 on this data, ~140 MB/s at the default level 9), checks it by
decompressing and comparing a SHA-256 and the line count, renames it into
place and only then deletes the parts. A crash at any point leaves the parts,
or the archive and the parts, never neither; the next run finishes the job.
zstdcat audit-2026-10-24.log.zst | grep … works, since the archive is the
day's log byte for byte.
A 100-trader AI simulation writes about 1.2 GB an hour, so a day is 10–14 GB
as parts and about 1 GB as an archive. Disk use is governed by --keep-days:
Only archives are ever deleted, and only those older than --keep-days days.
Recovering a failed archive by hand. A failed archive leaves the day's
parts in place, where they are still read as part of the live trail, and logs
an ERROR (pm-log-srv) and counts audit.archive_errors, which pm-monitor
shows as an audit_archive_errors WARN. Free the cause (usually disk space:
the archive needs about a tenth of the parts' size, but the run refuses below
25 %) and run it yourself:
pm-audit-cli archive --date 2026-10-24 # one day, must be before today
pm-audit-cli archive --pending # every closed day that has parts
pm-audit-cli archive --pending --level 12 --keep-days 30
Two runs never overlap; a second one exits at once with a note. A leftover
audit-D.log.zst.part file is deleted and redone. A day that audit.log
still holds (pm-audit was down at midnight) is not archived until
pm-audit has closed it at its next start.
An archive is never deleted by a run. Parts found beside it are either what a
crash left while deleting them (they are the archive's tail, and are deleted)
or lines of that day written after it was archived (they are appended to it).
If the archive itself cannot be decompressed, the run refuses that day and
leaves both: check it with zstd -t audit-D.log.zst, and only if it is
damaged and the parts are known to be the whole day, move it aside and re-run.
Proving a day is unchanged: the evidence manifest¶
Each archive appends one line to data/audit/manifest.jsonl: the day, its
line and byte counts, and the SHA-256 of its uncompressed content. Every line
also carries the SHA-256 of the line before it, so changing or removing any
entry breaks every entry after it. Retention writes a retired line before it
deletes an archive, so the chain survives --keep-days. A day reopened after
it was archived appends a second entry, and the newest is current.
pm-audit-cli verify # the whole chain, and every archive against it
pm-audit-cli verify --date 2026-10-24 # the chain, and that day's archive
verify exits 1 and names the day when an archive's content differs from its
entry, an entry is missing, an archive is missing without being retired, or the
chain is broken. It ends with the head hash, the hash of the newest line.
A crash while a line was being appended leaves it torn (no newline, invalid
JSON). verify reports it on every run from then on, as it should: the record
of that moment is damaged. The next entry starts on a line of its own and
chains to the torn line, so everything after it still verifies; the archive
whose entry was torn is recorded again when the archiver finishes it.
This makes an edit evident; it does not prevent one. Someone who can
rewrite the files can rewrite the whole chain, or cut off its newest entries.
To rule that out, copy the head hash somewhere the recorder host cannot write
(a ticket, an email, a separate system) after each close, and compare it with
the one verify prints when a day is disputed.
Subscribed topics¶
pm-audit subscribes with an empty topic filter, which causes ZeroMQ to
deliver every published message without exception:
| Topic | Source |
|---|---|
order.new |
Gateway order submissions |
order.ack.* |
Engine order acknowledgements |
order.fill.* |
Fill notifications |
order.cancelled.* |
Cancel confirmations |
order.expired.* |
TIF expiry notifications |
trade.executed |
Every matched trade pair |
book.* |
Book snapshots after every change (each carries ts_ns, so a snapshot can be ordered against the trades around it) |
session.state |
Session phase transitions (command_id present when an operator, not the scheduler, requested it) |
system.* |
Gateway auth, symbols, EOD, GTC-restore summary and per-entity restore outcomes, absorbed internal failures |
| (all others) | Admin events, combos, quotes, circuit breaker |
Audit completeness — every state-changing decision, not just its outcome¶
pm-audit can only ever see what the engine publishes on :5556 — it is a
bare subscriber with no other visibility into the process. Reconstructing
"what happened and why" from the log is therefore only as complete as the
publish call sites are, and a few decision points used to publish an outcome
without the reason behind it, or nothing at all. The fields and events below
close those gaps:
| Gap closed | Where it shows up |
|---|---|
| Every engine-initiated cancel now states why | order.cancelled.{GW_ID}'s cancel_reason covers a kill switch (self or admin), a circuit-breaker halt, a gateway disconnect, an admin symbol-level mass cancel, a quote being replaced, and a quote leg cancelled because its sibling filled — in addition to the original SELF_MATCH_PREVENTED / INSUFFICIENT_LIQUIDITY. Still null for a client-requested cancel, which is not the exchange's decision to explain. |
| Cancels and quote removals join back to the command that caused them | order.cancelled.{GW_ID} and quote.status.{GW_ID} both carry command_id when an admin/kill-switch command caused the event — the same command_id echoed on that command's ack and on admin.action. null when there is no such command (self-match prevention, a participant's own quote cancel, and so on). |
| A resting order can be told apart from a config-seeded one | order.ack.{GW_ID} and order.fill.{GW_ID} carry is_seed — true for an order or quote leg created by market_maker_quotes/market_maker_combos at startup rather than a live gateway submission (an order's acknowledgement carries it too, and order.new itself is in the trail, since the engine echoes every command it is asked to perform before acting on it). The on-demand order.orders/order.price_level_orders snapshot rows carry it too. Purely observational: nothing in the engine branches on it. |
| Every combo leg gets an acknowledgement | order.ack.{GW_ID} now fires once per child leg when a combo is accepted — live or config-seeded. Previously no combo leg published an ack at all; a subscriber that started after acceptance had no way to learn a leg's symbol, side, price, or qty until it was cancelled, and order.cancelled's minimal shape cannot supply them either. |
| Startup recovery is on the record | system.startup_recovery, published once right after _restore_gtc(), reports how many orders were restored, discarded as stale TIF=DAY, or failed to restore; how many single-leg quote remnants and rebuilt QuoteIndex entries resulted; and how many GTC combos came back. Previously these counts reached only log.info/log.error. system.recovery_item now accompanies it with one line per entity — entity_id, kind (ORDER/COMBO) and outcome (RESTORED/DISCARDED_STALE_DAY/FAILED/QUOTE_REMNANT), plus detail carrying the exception text for a FAILED order — so a single bad GTC record is a wire fact with a reason attached, not just a decremented count. |
| Kill-switch and admin sweeps say which ids they hit, not just how many | risk.kill_switch_ack, kill_switch_gateway_ack, kill_switch_global_ack, symbol_halt_ack, cancel_symbol_ack, circuit_breaker_halt_all_ack, circuit_breaker_resume_all_ack, and admin.action's scope all carry an id list beside every count they already reported — cancelled_order_ids, cancelled_quote_order_ids, halted_symbol_ids, resumed_symbol_ids, affected_gateway_ids. Previously undoing or auditing the effect of a mass cancel meant re-deriving the affected set from order.cancelled lines by command_id; the ack now says it directly, and a length mismatch against the count is itself a detectable defect. |
| Cancellations, expiries and amendments can be filtered by symbol | order.cancelled, order.expired and order.amended all carry symbol now. pm-audit-cli events --topic order.cancelled --symbol AAPL previously returned nothing for any symbol — the engine had symbol at every publish site but never put it on the wire. |
| Book and depth snapshots carry a clock | book.{symbol} and depth.{symbol} both gain ts_ns, stamped from the same strictly-monotonic clock trade.executed uses. Previously a snapshot carried no time field at all, so there was no way to tell from the log alone whether a given snapshot reflects a particular trade. |
| An operator-driven session transition is distinguishable from a scheduled one | session.state gains an optional command_id, echoing the command_id on the session.transition that caused it — present only when an operator (not pm-scheduler) requested the transition. Previously the broadcast looked identical either way; telling them apart meant inferring it from what else was on the bus at the time. |
| Absorbed internal failures are on the record | system.diagnostic fires for a maintenance-flush exception, a dispatch-handler crash, an undecodable inbound message, or a message on a topic with no handler — cases the engine deliberately survives so trading is not interrupted, but which previously reached only the process log. Carries component, detail (the flush name or topic, when there is one), error (the exception text, when there is one), and count (that component's running occurrence count). The client-facing reject for a crashed order handler still stays generic — see order.ack's INTERNAL_ERROR — so system.diagnostic is the only wire-visible place the real exception text appears. |
| Amendments show the prior price/qty | order.amended.{GW_ID} carries old_price/old_qty alongside the new values, so a post-mortem does not need the preceding order.new/order.amended to know what changed. |
| Index corporate actions show the prior divisor | index.corp_action_ack, index.constituent_change_ack, and index.rebalance_ack all carry old_divisor alongside divisor, mirroring HistoryRecord.old_divisor in pm-index's own local archive — a post-mortem no longer needs that archive to see what changed. |
Refusals made at a gateway¶
The ALF, BALF and REST gateways refuse some requests themselves, before the
engine sees them: a command before login, a failed authentication, a request
the key's role does not permit, and a client over its rate limit. Each is sent
to the engine as system.gateway_reject and echoed into the trail, so "I sent
it and you ignored it" can be answered either way. A record carries the
gateway (source), the participant it believed it was talking to
(gateway_id, claimed and empty before login), the remote address, the reject
code, the reason and the gateway's clock. It never carries an API key.
A gateway sends at most one record per peer host and code per second and puts
the number it folded in as count, so a flood of refused commands is one line a
second, not a flood. The host, not the host and port: a client that opens a new
connection per request comes from a new port each time. The record names the
full peer of the refusal that sent it. The count of the last burst is lost if the peer then goes
quiet, and the first record of the burst is not.
Not recorded, on purpose: malformed frames and field-validation refusals
(missing field, bad value), which the gateways count, and a read-only REST key
attempting a write. pm-audit-replay story narrates a record as "A request
from GW01 (10.0.0.5:4411) was refused at the ALF gateway — rate limiting".
Which clock a line is stamped with¶
Every audit line starts with the time pm-audit received it, which is the
only clock present on every line. Three messages also carry the engine's
clock, in Unix epoch nanoseconds: trade.executed.ts_ns (the match),
order.fill.ts_ns (the same value as the trade it reports, so a fill and its
print share one reading) and order.ack.ts_ns (when the engine accepted or
refused the order, from the same strictly increasing source). For "when did the
exchange accept this order", use the ack's ts_ns; the receipt time can lag it
under load. Ordering between messages is by the engine's eseq, not by either
clock. Lines recorded before this field existed carry the receipt time only.
Commands the trail may not hold¶
The engine echoes every inbound command into the trail before it acts on it,
which is what lets a command that had no effect still be found. A command that
reaches the engine without an envelope frame (a client that built its own
PUSH socket instead of using the project's client library) cannot be echoed, and
a failed echo is the other way one goes missing. Both are counted in
engine.diagnostics{component="UNRECORDED_COMMAND"} in the engine's health
beacon, and the first of each thousand is logged at WARNING. On a healthy
exchange the counter stays at 0; any other value means the trail is not a
complete record of what the exchange was asked to do.
An audited run must also be started from a deployed config artifact
(pm-config-deploy). With none, system.startup carries no config and
pm-audit-replay anomalies reports CONFIG_NOT_RECORDED.
Signal handling and graceful shutdown¶
The buffer flush happens synchronously before the process exits so the last batch of messages is always written to disk.
pm-audit-cli — Query Tool¶
pm-audit-cli is a read-only command-line tool for querying audit logs
without shell pipelines or SQL knowledge. It reads JSONL files directly from
disk and produces structured output. The one command that talks to the running
recorder is health, which pings
pm-audit to confirm it is alive.
Why a separate query tool¶
pm-audit is a long-running writer. Querying is a separate one-shot
operation. Keeping them separate means:
- You can query logs while
pm-auditis still writing - The tool works on archived logs from previous sessions
- Queries never affect the running recorder
This follows the same pattern as pm-stats-cli and pm-clearing-cli.
Global options¶
These flags are parsed once, before the subcommand:
| Flag | Default | Description |
|---|---|---|
--log-file PATH |
data/audit.log |
Primary audit log file to read |
--log-dir PATH |
(parent of --log-file) |
Directory holding the audit/ day parts and archives |
--format table\|json\|csv |
table |
Output format |
--no-header |
off | Suppress CSV header row |
--use-index PATH |
(auto) | Path to SQLite index file; auto-detected as <the directory containing --log-file>/audit_index.db when present |
Not every flag applies to every command
--use-index only affects the events command — orders, trades,
topics, gateways, and timeline always stream the JSONL log files
directly and never consult the SQLite index, even when one exists and
--use-index is given. --format/--no-header are also ignored by
stats (its output is a fixed text summary, not a table) and by index
(which prints build progress, not rows). health honours --format json
(the report as a JSON object); csv prints the same text as table.
The auto-detected index path is always <the directory containing
--log-file>/audit_index.db — it does not use --log-dir. If
--log-dir points somewhere other than --log-file's own parent
directory, the index lookup still only checks the log file's parent.
Output formats:
| Format | When to use |
|---|---|
table |
Interactive terminal — aligned columns, human-readable |
json |
Automation, scripts, downstream processing |
csv |
Export to spreadsheets or pipelines |
No-result behaviour:
table: printsNo matching events found.json: prints[]csv: prints header only (or nothing with--no-header)
Exit code is always 0 when a query succeeds with no results. Non-zero only
on argument errors (exit 2) or I/O failures (exit 1).
Which files are read¶
By default pm-audit-cli reads the live trail: every closed day part that
is not yet archived, oldest first, then audit.log. That is today, plus any
day that has not been archived yet. Archives are not read unless you ask:
| You give | It reads |
|---|---|
| nothing | the live trail |
--date 2026-10-24 |
that trading day (session timezone): its archive, or its parts if not archived yet, plus audit.log if it still holds that day |
--log-file data/audit/audit-2026-10-24.log.zst |
exactly that archive (a day part path works too) |
--date selects a day's files, not a time window, and cannot be combined
with an archive --log-file or with --follow. Results are chronological
across all files.
events — General event search¶
Search log entries by topic prefix, gateway, symbol, and time range. This is the most flexible command and the right starting point for any investigation.
Options:
| Flag | Default | Description |
|---|---|---|
--topic PREFIX |
(all) | Filter by topic prefix, e.g. order.fill or trade. |
--gateway GW_ID |
(all) | Filter by gateway ID appearing in the payload |
--symbol SYMBOL |
(all) | Filter by instrument symbol |
--date YYYY-MM-DD |
(live trail) | Read this trading day's archive or parts — see Which files are read |
--from ISO_TS |
(none) | Start of time range (inclusive) |
--to ISO_TS |
(none) | End of time range (inclusive) |
--limit N |
100 |
Maximum rows returned |
--reverse |
off | Show newest events first |
-f, --follow |
off | Keep watching the log and print new events as they arrive; see Follow mode below |
--interval SEC |
1.0 |
Polling interval for --follow (ignored otherwise) |
Output columns:
| Column | Description |
|---|---|
timestamp |
UTC timestamp of the event |
topic |
ZeroMQ topic |
gateway |
Gateway ID extracted from payload (if present) |
symbol |
Instrument symbol extracted from payload (if present) |
order_id |
Order ID extracted from payload (if present) |
summary |
Brief human-readable description of the event |
summary format differs when the SQLite index is used
When events is served from the JSONL log files, summary is a
structured, topic-aware description (e.g. FILL 60@150.00). When a
usable index exists at the resolved --use-index path, events is
served from SQLite instead, and summary is just the payload dict's
str() representation truncated to 80 characters — a different, less
readable format. Rebuild the index after querying if you need the
latest events; the index is never queried automatically for freshness.
--follow always reads the JSONL files directly and never uses the
index, even when one is available — see below.
Examples:
# All events in the log (last 100)
pm-audit-cli events
# All fills for gateway GW01 on a specific date
pm-audit-cli events --topic order.fill --gateway GW01 --date 2026-07-08
# All trade executions for AAPL in a two-hour window
pm-audit-cli events --topic trade.executed --symbol AAPL \
--from 2026-07-08T09:30:00+00:00 --to 2026-07-08T11:30:00+00:00
# Most recent 50 events across all topics
pm-audit-cli events --limit 50 --reverse
# All session phase transitions
pm-audit-cli events --topic session.state
# All order rejections
pm-audit-cli events --topic order.ack --symbol AAPL
# Export all trade events as JSON for analysis
pm-audit-cli --format json events --topic trade.executed --limit 5000 > trades.json
# Order submissions from a specific gateway in CSV format
pm-audit-cli --format csv events --topic order.new --gateway GW01 > gw01_orders.csv
# Watch fills arrive live during a training session
pm-audit-cli events --topic order.fill --follow
# Watch one symbol, polling every 2 seconds instead of the 1s default
pm-audit-cli events --symbol AAPL --follow --interval 2
Follow mode¶
events and timeline both accept -f/--follow to keep the query
running: after printing the normal (filtered, --limit-capped) result, the
command keeps polling the log files every --interval seconds and prints
each newly-appended, matching entry as it arrives, until interrupted with
Ctrl-C. This is the audit-log equivalent of pm-log-cli tail.
# Watch the whole log live, table format
pm-audit-cli events --follow
# Watch order lifecycle events for one gateway, as JSON
pm-audit-cli --format json events --topic order. --gateway GW01 --follow
A few things behave differently in follow mode:
--limitonly bounds the initial backfill. The first batch printed respects--limit(100by default forevents,500fortimeline) exactly as it would without--follow. Once the poll loop starts, every subsequent batch prints however many new matching entries actually arrived since the last poll — there is no cap. This mirrorspm-log-cli tail --before N.events --reverse --followtogether is rejected (exit code2):timelinehas no--reverseflag, and forevents, newest-first ordering is meaningless once the log is still growing.events --followalways reads the JSONL log files, even when a usable SQLite index exists at the resolved--use-indexpath. The index is a point-in-time snapshot and is never refreshed by a running query, so follow mode bypasses it rather than showing stale data.- Closing a day is handled automatically. Each poll re-discovers the
live trail, so the midnight close —
audit.logbecoming a day part and a freshaudit.logstarting — is picked up on the next poll without restarting the command, and no line is printed twice. - No
seqrequirement. Resuming after each poll is based on the entries' timestamps, not the optionalseq=field in the log line's metadata section — so--followworks identically whether or notpm-audithappens to be stamping envelope metadata.
--format csv/--format json both work under --follow. Each poll batch
is printed as its own block, one after another, rather than as a single
combined document: for csv, only the very first batch gets a header row
(later batches never repeat it, regardless of --no-header); for json,
each batch is its own JSON array printed back to back with no separator
between them. A consumer streaming the output should parse it incrementally
(read one CSV block or JSON array at a time) rather than treating the whole
stream as one JSON value or one CSV table.
orders — Order lifecycle investigation¶
Find all audit events related to specific order IDs or filter by gateway and symbol to reconstruct an order's full journey through the engine.
Options:
| Flag | Default | Description |
|---|---|---|
--id ORDER_ID |
(none) | Order ID to find (repeatable; searches order_id and id fields in payload) |
--gateway GW_ID |
(all) | Filter by gateway ID |
--symbol SYMBOL |
(all) | Filter by symbol |
--date YYYY-MM-DD |
(live trail) | Read this trading day — see Which files are read |
--from ISO_TS |
(none) | Start of time range |
--to ISO_TS |
(none) | End of time range |
--limit N |
100 |
Maximum rows |
Output columns:
| Column | Description |
|---|---|
timestamp |
Event timestamp |
order_id |
Order identifier |
event |
Last dot-separated segment of the topic. For two-part topics like order.new this is the verb (new). For per-gateway topics like order.ack.GW01, order.fill.GW01, or order.cancelled.GW01 it is the gateway ID, not the verb — the topic's own verb segment (ack, fill, cancelled) is dropped. |
gateway |
Gateway ID |
symbol |
Instrument symbol |
side |
BUY or SELL |
qty |
quantity or fill_qty or remaining_qty — the first truthy value in that order (not merely the first present key; a legitimate 0 or null falls through to the next) |
price |
Price (limit price or fill price depending on event type) |
status |
Order status after the event |
reason |
reject_code (from an order.ack rejection), cancel_reason (from order.cancelled), or a generic reason string — whichever the event carries |
Examples:
# Full lifecycle of one specific order
pm-audit-cli orders --id ORD-GW01-00142
# Two orders side by side
pm-audit-cli orders --id ORD-GW01-00142 --id ORD-GW01-00143
# All order events for gateway GW01 today
pm-audit-cli orders --gateway GW01 --date 2026-07-08 --limit 50
# Order events for AAPL in a session
pm-audit-cli orders --symbol AAPL --from 2026-07-08T09:30:00+00:00
# Export one order's lifecycle as JSON for a support ticket
pm-audit-cli --format json orders --id ORD-GW01-00142
# Order activity for market-maker in CSV
pm-audit-cli --format csv orders --gateway MM_AAPL_01 > mm_orders.csv
trades — Trade execution search¶
Find trade.executed events with precise filtering on both sides of the
trade, price ranges, and minimum quantities.
Options:
| Flag | Default | Description |
|---|---|---|
--symbol SYMBOL |
(all) | Filter by instrument symbol |
--gateway GW_ID |
(all) | Match trades where this gateway appears on either side |
--buy-gateway GW_ID |
(all) | Filter by buyer gateway specifically |
--sell-gateway GW_ID |
(all) | Filter by seller gateway specifically |
--min-price PRICE |
(none) | Minimum trade price (inclusive) |
--max-price PRICE |
(none) | Maximum trade price (inclusive) |
--min-qty QTY |
(none) | Minimum trade quantity (inclusive) |
--date YYYY-MM-DD |
(live trail) | Read this trading day — see Which files are read |
--from ISO_TS |
(none) | Start of time range |
--to ISO_TS |
(none) | End of time range |
--limit N |
100 |
Maximum rows |
--reverse |
off | Show newest first |
Output columns:
| Column | Description |
|---|---|
timestamp |
Trade execution timestamp |
trade_id |
Durable trade identifier, formatted as run_seq-counter (for example, 000042-000000001). The prefix identifies the engine run; the suffix is monotonic within that run. |
symbol |
Instrument symbol |
price |
Execution price |
quantity |
Matched quantity |
buy_gateway |
Buyer gateway ID |
sell_gateway |
Seller gateway ID |
aggressor |
Which side crossed the spread: BUY, SELL, or AUCTION for trades produced by an auction match |
Examples:
# All trades for AAPL today
pm-audit-cli trades --symbol AAPL --date 2026-07-08
# Block trades (>= 1,000 shares) across all symbols
pm-audit-cli trades --min-qty 1000
# Trades between two specific gateways
pm-audit-cli trades --buy-gateway GW01 --sell-gateway MM_AAPL_01
# MSFT trades in a specific price band
pm-audit-cli trades --symbol MSFT --min-price 410.0 --max-price 420.0
# Most recent 20 trades
pm-audit-cli trades --limit 20 --reverse
# Export full day as CSV for post-trade analysis
pm-audit-cli --format csv trades --date 2026-07-08 --limit 10000 > day_trades.csv
# All trades involving market maker
pm-audit-cli --format json trades --gateway MM_AAPL_01
topics — Topic discovery and statistics¶
List all topics present in the audit logs with event counts and first/last occurrence timestamps. Useful for understanding what happened in a session and for checking whether expected event types were produced.
Options:
| Flag | Default | Description |
|---|---|---|
--date YYYY-MM-DD |
(live trail) | Read this trading day — see Which files are read |
--from ISO_TS |
(none) | Start of time range |
--to ISO_TS |
(none) | End of time range |
--prefix PREFIX |
(all) | Filter topics by prefix, e.g. order. |
--sort count\|alpha |
count |
Sort by event count descending (default) or alphabetically |
Output columns:
| Column | Description |
|---|---|
topic |
ZeroMQ topic name |
count |
Number of events with this topic |
first_seen |
Timestamp of the first occurrence |
last_seen |
Timestamp of the last occurrence |
Examples:
# All topics in the log, most frequent first
pm-audit-cli topics
# Topics on a specific trading date
pm-audit-cli topics --date 2026-07-08
# Order-related topics only, sorted alphabetically
pm-audit-cli topics --prefix order. --sort alpha
# What happened during the opening auction?
pm-audit-cli topics --from 2026-07-08T09:00:00+00:00 --to 2026-07-08T09:30:00+00:00
# Export all topic stats as JSON
pm-audit-cli --format json topics
# Check that trades were produced today
pm-audit-cli topics --prefix trade. --date 2026-07-08
gateways — Gateway activity summary¶
List all gateways seen in the audit logs and show a summary of their activity: order submissions, fills, and trades participated in.
Options:
| Flag | Default | Description |
|---|---|---|
--date YYYY-MM-DD |
(live trail) | Read this trading day — see Which files are read |
--from ISO_TS |
(none) | Start of time range |
--to ISO_TS |
(none) | End of time range |
--min-events N |
(none) | Only show gateways with at least N events |
Output columns:
| Column | Description |
|---|---|
gateway_id |
Gateway identifier |
events |
Total event count (all topics) |
orders |
order.new events submitted |
fills |
order.fill.* events received |
trades |
trade.executed events participated in |
first_seen |
First event timestamp |
last_seen |
Last event timestamp |
Results are sorted by total event count descending — the most active gateway appears first.
Examples:
# All gateways in the logs
pm-audit-cli gateways
# Active gateways on a specific date
pm-audit-cli gateways --date 2026-07-08
# Only gateways with meaningful activity
pm-audit-cli gateways --min-events 10
# Gateway activity in the first hour of trading
pm-audit-cli gateways --from 2026-07-08T09:30:00+00:00 --to 2026-07-08T10:30:00+00:00
# Export as JSON for automation
pm-audit-cli --format json gateways
# Check which gateways connected during the closing auction
pm-audit-cli gateways \
--from 2026-07-08T15:30:00+00:00 --to 2026-07-08T16:00:00+00:00
timeline — Chronological event stream¶
Show a raw chronological stream of events for session replay and forensic
investigation. Unlike events, the timeline command exposes the full JSON
payload as a single column, making it suitable for detailed reconstruction of
what happened over a time window.
Options:
| Flag | Default | Description |
|---|---|---|
--from ISO_TS |
(none) | Start time |
--to ISO_TS |
(none) | End time |
--topic PREFIX |
(all) | Filter by topic prefix |
--gateway GW_ID |
(all) | Filter by gateway |
--symbol SYMBOL |
(all) | Filter by symbol |
--limit N |
500 |
Maximum events |
-f, --follow |
off | Keep watching the log and print new events as they arrive; see Follow mode above |
--interval SEC |
1.0 |
Polling interval for --follow (ignored otherwise) |
Output columns:
| Column | Description |
|---|---|
timestamp |
Event timestamp |
topic |
ZeroMQ topic |
gateway |
Gateway ID (if present in payload) |
symbol |
Symbol (if present in payload) |
payload |
Full JSON payload as a string |
Examples:
# Opening 5 minutes of trading
pm-audit-cli timeline \
--from 2026-07-08T09:30:00+00:00 --to 2026-07-08T09:35:00+00:00
# Gateway activity in a 10-minute window
pm-audit-cli timeline --gateway GW01 \
--from 2026-07-08T14:00:00+00:00 --to 2026-07-08T14:10:00+00:00
# Symbol-focused replay with higher limit
pm-audit-cli timeline --symbol AAPL \
--from 2026-07-08T09:30:00+00:00 --to 2026-07-08T16:00:00+00:00 \
--limit 2000
# All session state changes across the day (timeline has no --date flag;
# use --from/--to to bound a trading day instead)
pm-audit-cli timeline --topic session.state \
--from 2026-07-08T00:00:00+00:00 --to 2026-07-08T23:59:59+00:00
# Only trades and fills in the closing auction
pm-audit-cli timeline --topic trade. \
--from 2026-07-08T15:30:00+00:00 --to 2026-07-08T16:00:00+00:00
# Export for external replay tooling
pm-audit-cli --format json timeline \
--from 2026-07-08T09:30:00+00:00 --to 2026-07-08T16:00:00+00:00 \
--limit 100000 > session_replay.json
# Watch the full raw event stream live (see Follow mode above)
pm-audit-cli timeline --follow
stats — Log file statistics¶
Show a health-check summary of the audit log files: total events, file sizes,
date range, and counts of unique topics and gateways. Run this to confirm
that pm-audit has been recording as expected.
Options:
| Flag | Default | Description |
|---|---|---|
--verbose |
off | Show a per-file breakdown in addition to the summary |
Example output (default):
Audit Log Statistics
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Total events: 45,678
Total size: 12.4 MB
Log files: 3
Oldest event: 2026-07-01T09:30:00.123+00:00
Newest event: 2026-07-08T15:59:59.987+00:00
Topics seen: 24
Gateways seen: 5
Example output (--verbose):
Audit Log Statistics
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Total events: 45,678
Total size: 12.4 MB
Log files: 3
Oldest event: 2026-07-01T09:30:00.123+00:00
Newest event: 2026-07-08T15:59:59.987+00:00
Topics seen: 24
Gateways seen: 5
Per-file breakdown:
data/audit/audit-2026-07-07.1.log 12,000 events 4.1 MB
data/audit/audit-2026-07-07.2.log 18,234 events 6.2 MB
data/audit.log 15,444 events 2.1 MB
Examples:
# Quick health check
pm-audit-cli stats
# Full breakdown including per-file stats
pm-audit-cli stats --verbose
# An archived day, by path
pm-audit-cli --log-file data/audit/audit-2026-07-01.log.zst stats
index — Build or update the SQLite index¶
For large audit histories or frequent complex queries, pm-audit-cli can
build an optional SQLite index from the JSONL log files. Once built, the
events command uses it automatically for significantly faster response.
Options:
| Flag | Default | Description |
|---|---|---|
--output PATH |
<the directory containing --log-file>/audit_index.db |
Destination SQLite database |
--days N |
(none) | Index only the last N days of logs |
--from ISO_TS |
(none) | Start of index range |
--to ISO_TS |
(none) | End of index range |
--rebuild |
off | Delete all existing index data and rebuild from scratch |
--incremental |
off | Add only entries newer than the last indexed timestamp |
--rebuild and --incremental are mutually exclusive.
Re-running index without --rebuild/--incremental duplicates rows
Without either flag, index re-inserts every entry in the requested
range on each run with no deduplication — running the same --days 7
command twice doubles the row count for that window. Use --incremental
for routine re-indexing, or --rebuild to start clean.
Index schema highlights:
The index stores one row per log line in a audit_events table with
dedicated columns for timestamp, topic, gateway_id, symbol,
order_id, and trade_id, each with a covering index. The full JSON payload
is kept in a payload column so no information is lost.
Performance comparison (1 M events):
| Query type | JSONL streaming | SQLite index |
|---|---|---|
| Recent 100 events | < 1 s | < 0.1 s |
| Single topic filter | 4–6 s | 0.2 s |
| Multi-filter (topic + gateway + date) | 10–15 s | 0.5 s |
| Order ID lookup | 8 s | 0.05 s |
Index auto-detection:
When audit_index.db exists in the log directory, the events command uses
it automatically. Override with --use-index PATH or point at a different
index location.
Examples:
# Build index for the last 7 days
pm-audit-cli index --days 7
# Build complete index from all log files
pm-audit-cli index --rebuild
# Incremental update — only adds entries not yet in the index
pm-audit-cli index --incremental
# Index a specific date range
pm-audit-cli index \
--from 2026-07-01T00:00:00+00:00 --to 2026-07-08T23:59:59+00:00
# Build index to a custom location
pm-audit-cli index --output /tmp/session_idx.db --rebuild
# Use a non-default index when querying
pm-audit-cli --use-index /tmp/session_idx.db \
events --topic trade.executed --limit 100
When to build an index
Build an index when:
- The audit log contains more than ~500,000 events
- You run more than a few queries per session
- You need to correlate events across multiple fields (topic + gateway + symbol + date)
For ad-hoc queries on small logs, streaming JSONL is fast enough without the extra disk space.
health — Recorder and log file health¶
Check that pm-audit is alive and actually recording. By default two fast
checks run:
- Process — a
pm-auditprocess is running (found withpgrep). - ZMQ ping —
pm-auditanswers a ping on its health socket (port5562) with its current UTC time. The reply comes from the recorder's own receive loop, so it proves the loop is running, not just that the process exists.
The log file check is opt-in with --extended. It checks that --log-file
exists and is a regular file, that its last entry is a well-formed audit line
(an empty log is fine — a freshly started pm-audit has written nothing yet),
and that the file can be opened for append. Nothing is ever written to the
log. Only the tail of the file is read, so the check stays fast however large
the log grows; it is still bounded at 15 seconds so a stalled or
network-mounted volume cannot hang the command.
Unlike the query commands, health runs even when the log file does not
exist; a missing file is only reported as a failure under --extended.
pm-audit-cli health
pm-audit-cli health --extended
pm-audit-cli health --extended --verbose
pm-audit-cli --format json health
pm-audit-cli health --quiet
pm-audit process: OK - pm-audit is running
audit log file: OK - skipped (pass --extended to check read/write access)
audit ZMQ ping: OK - replied at 2026-10-17T14:23:37.797827+00:00
health: OK
Options:
| Option | Description |
|---|---|
--extended |
Also check the audit log file is readable, writable and ends in a well-formed entry (up to 15 s) |
-v, --verbose |
List exactly which checks ran; with --extended, also print the log size, day parts, archives (with the compression ratio) and last entry time |
-q, --quiet |
Print nothing; report only through the exit code |
With --extended --verbose the report ends with the checks performed and the
log file figures. Day parts and archives are found under <dir>/audit/, so
--log-dir applies:
Checks performed:
pm-audit process (pgrep): yes
ZMQ health ping: yes
Log file read/write/entry: yes
Log file stats:
path: data/audit.log
size: 2.1MB
day parts: 2 (10.3MB)
archives: 3 (2.1MB, 13.4:1)
total size: 14.5MB
last entry: 2026-07-08T16:29:59.874+00:00
The exit code is 0 when healthy and 1 otherwise. With --quiet nothing is
printed and the exit code is 0 when healthy or -1 (255 in a shell)
otherwise.
The health socket binds to 127.0.0.1:5562 by default; set
EDUMATCHER_AUDIT_BIND_HOST / EDUMATCHER_AUDIT_HEALTH_PORT to change where
pm-audit binds it, and EDUMATCHER_ENGINE_HOST /
EDUMATCHER_AUDIT_HEALTH_PORT to change where pm-audit-cli connects.
Cookbook — Common Audit Workflows¶
Investigate a specific order¶
# Find all events for order ORD-GW01-00142
pm-audit-cli orders --id ORD-GW01-00142
# Same as JSON (attach to support ticket)
pm-audit-cli --format json orders --id ORD-GW01-00142
# Find by gateway and time window (when order ID is not known)
pm-audit-cli orders --gateway GW01 \
--from 2026-07-08T09:30:00+00:00 --to 2026-07-08T09:35:00+00:00
Reconstruct a trade¶
# Find trade details by symbol and time
pm-audit-cli trades --symbol AAPL \
--from 2026-07-08T10:15:00+00:00 --to 2026-07-08T10:15:01+00:00
# Find all large trades above a price threshold
pm-audit-cli trades --min-qty 500 --min-price 150.0 --date 2026-07-08
Verify session phases fired correctly¶
# See all session transitions in order
pm-audit-cli events --topic session.state --date 2026-07-08
# Timeline around market open
pm-audit-cli timeline --topic session. \
--from 2026-07-08T09:25:00+00:00 --to 2026-07-08T09:35:00+00:00
Identify the most active gateway¶
Check log health before end of day¶
pm-audit-cli health --extended
pm-audit-cli stats --verbose
pm-audit-cli topics --date 2026-07-08 --sort count | head -10
Export a full session for offline analysis¶
# Build index first for speed
pm-audit-cli index --rebuild
# Export all trades
pm-audit-cli --format csv trades --date 2026-07-08 --limit 50000 > trades.csv
# Export all order events for gateway GW01
pm-audit-cli --format json events --topic order. --gateway GW01 \
--date 2026-07-08 --limit 50000 > gw01_orders.json
Look at an archived day¶
# The live trail (unarchived parts + audit.log) is the default
pm-audit-cli trades --symbol AAPL --limit 10000
# One past day, read from its archive (or its parts, if not archived yet)
pm-audit-cli trades --symbol AAPL --date 2026-07-07 --limit 10000
# A single archive by path, or straight from the shell
pm-audit-cli --log-file data/audit/audit-2026-07-07.log.zst trades --symbol AAPL
zstdcat data/audit/audit-2026-07-07.log.zst | grep '\[trade\.executed\]'
Watch a live training session¶
# Everything, as it happens
pm-audit-cli events --follow
# Just order lifecycle events for one gateway, while proctoring a session
pm-audit-cli events --gateway GW01 --topic order. --follow
# Full raw payloads for the whole floor, piped to a file for later replay
# (each poll batch is its own JSON array back to back -- see Follow mode
# above -- not JSON Lines, so replay tooling should parse incrementally)
pm-audit-cli --format json timeline --follow > live_session.json
Error Handling¶
| Exit code | Meaning |
|---|---|
0 |
Success (including queries that returned zero rows) |
1 |
I/O error (log file not found, unreadable database) |
2 |
Argument error (invalid date format, bad --limit, conflicting flags) |
Invalid log lines (malformed JSON, truncated writes) are always silently
skipped — no warning is printed, and there is no global --verbose flag on
pm-audit-cli to enable one. (stats --verbose is a different, per-command
flag that only adds a per-file size/count breakdown to the stats output.)
See Also¶
In this book:
- Persistence — which files each process writes and where they live
- Statistics and Reporting —
pm-statsandpm-stats-clifor structured market data queries
Participant Guide:
- P&L & Clearing —
pm-clearingandpm-clearing-clifor trade settlement
Reference Manual:
- Processes — pm-audit — startup reference table in the process overview