Skip to content

Audit Trail

Learning objectives

After reading this page you will understand:

  • Why a full event log matters for an exchange and what pm-audit records
  • The JSONL log format, how pm-audit closes each day and how days are archived
  • All pm-audit startup options including write buffering
  • How to query audit logs from the command line using every pm-audit-cli verb
  • When to use the optional SQLite index for faster queries
  • A practical cookbook for common audit investigation workflows

Overview

EduMatcher maintains a complete, append-only, tamper-evident record of every message that passes through the ZeroMQ bus. This is the audit trail — a chronological log of every order, fill, trade, cancellation, session transition, and system event that occurred during a trading session.

The audit system has two independent components:

Component Role Type
pm-audit Subscriber — records every event to a JSONL log file, closed day by day Long-running process
pm-audit-cli Query tool — reads audit logs and prints structured output One-shot CLI

The recorder and the query tool are completely independent. pm-audit-cli reads log files directly from disk and works even when pm-audit is not running.

flowchart TB
    subgraph Runtime
        ENG["pm-engine\n(ZMQ PUB :5556)"]
        AUD["pm-audit\nSUB :5556"]
        LOG[("data/audit.log\n+ audit/ day parts\n+ audit/*.log.zst")]
        ENG -- "all topics" --> AUD
        AUD -- "buffered JSONL writes" --> LOG
    end
    subgraph Operator
        CLI["pm-audit-cli\n(read-only)"]
        LOG -- "stream parse" --> CLI
        CLI -. "optional fast path" .-> IDX[("audit_index.db")]
    end

Why a full event log matters

A financial exchange must be able to answer questions after the fact:

  • Was a specific order ever acknowledged by the engine?
  • What sequence of events led to a particular fill?
  • Did a circuit breaker fire before or after an order was submitted?
  • Which gateways were active at a given time?

The audit trail answers all of these by preserving every message exactly as it was broadcast, with a millisecond-precision UTC timestamp.

Data Directory

Audit logs are stored in the same data directory used by all other EduMatcher persistent files:

Running mode Default location
Source checkout (poetry run pm-audit) <repo>/src/data/audit.log
Installed (pm-audit on PATH) ~/.local/share/edumatcher/audit.log

Override with EDUMATCHER_DATA_DIR or --audit-log-file:

export EDUMATCHER_DATA_DIR="$HOME/sessions/morning"
pm-audit  # writes to $HOME/sessions/morning/audit.log

pm-audit — Event Recorder

pm-audit subscribes to all topics on the engine's PUB socket and appends every message as a single JSON line to the configured log file. It also binds a small health socket (port 5562) that answers pm-audit-cli health pings.

ZeroMQ PUB/SUB subscriptions join asynchronously. A publication made before pm-audit's subscription has reached the engine is not queued or replayed; the recorder cannot write an event it never received. Start pm-audit before starting pm-engine, wait for pm-audit-cli health to report the recorder healthy, and verify that the engine's system.startup event appears in the audit log before trading. The health check confirms the recorder's receive loop is running; it does not itself acknowledge the upstream subscription. If attaching to an already-running engine, allow the subscription to join before relying on the trail. The write buffer affects when received events appear on disk, not whether the subscription receives them.

pm-audit [options]

Startup options

Flag Default Description
--audit-log-file PATH <DATA_DIR>/audit.log Audit-trail output log file path
--terminal / -t off Also print each entry to stdout (useful during demos)
--buffer-size N 100 Number of messages to buffer in memory before flushing to disk
--flush-interval SECONDS 10.0 Maximum seconds to wait before flushing buffer regardless of size
--max-file-mb MB 10 Close the audit log as a day part when it reaches this size — see Day parts and archives
--keep-days N 0 Delete daily archives older than N days; 0 keeps them all
--fsync-interval SECONDS 1.0 Sync the active file to disk at most this often; 0 syncs every flush — see What can be lost
--log-level WARNING Explicit level: CRITICAL, ERROR, WARNING, INFO, DEBUG
-v / --verbose off Increase verbosity (-v → INFO, -vv → DEBUG)
-q / --quiet off No-op — the default level is already WARNING, so --quiet currently sets the same level and has no observable effect
--log-target server Where this process's own operational log records go (not the audit trail): server (auto-detected pm-log-srv), stdout, or file
--log-file PATH — Operational log file path — required when --log-target file. Distinct from --audit-log-file, which is the audit-trail output
--log-failover-timeout SECONDS 30 Grace window before falling back to a local log file once pm-log-srv becomes unreachable

--log-level takes priority over -v/-q when both are given.

Write buffering

Writing every message to disk individually would generate enormous I/O on active sessions. pm-audit batches messages in memory and flushes as a group when either limit is reached:

  • Buffer full: buffer contains --buffer-size messages (default 100)
  • Timer expired: --flush-interval seconds have elapsed since the last flush (default 10 s)
  • Shutdown: Ctrl-C, SIGINT, or SIGTERM always flushes remaining messages before exit

No messages are lost on clean shutdown. A hard kill (kill -9) may lose up to --buffer-size messages that had not yet been flushed. See What can be lost for the whole picture.

Tuning examples:

# Default: good balance of throughput and latency
pm-audit

# High-throughput session: larger buffer, less frequent disk I/O
pm-audit --buffer-size 500 --flush-interval 30

# Low-latency / compliance: flush aggressively
pm-audit --buffer-size 1 --flush-interval 1

# Demo mode: see every event on the terminal as well
pm-audit --terminal

What can be lost

pm-audit is a subscriber, so the trail holds what it received. Four things sit between an engine publication and the disk, and each has a bound:

Where What is lost Bound
pm-audit is not running Everything the engine publishes meanwhile Unbounded. The next line shows a jump in eseq (replay: ESEQ_GAP), and a # recorder-start marker before it (replay: RECORDER_RESTART) says why
The receive queue overflows (a burst faster than the writer) The messages past the queue's 200 000 Counted by audit.missed_messages when the next message shows the jump
pm-audit is killed (kill -9) Lines buffered but not yet written At most --buffer-size messages or --flush-interval seconds, whichever comes first
The host loses power or the kernel crashes Lines written but not yet synced About --fsync-interval seconds (default 1) plus the line above. A write syncs when one is due, and while no line arrives the receive loop does (it wakes at least every 0.3 s), so a quiet tail is not left in the page cache

A stopped-and-started pm-audit writes one comment line, # edumatcher-audit recorder-start pm-audit=<version> started=<time>, before its first line, so the restart is visible in the file itself and not only as a hole in the numbering. pm-audit-replay reports it as RECORDER_RESTART (info) and does not count it as an unreadable line.

The health beacon carries audit.missed_messages: the number of engine messages pm-audit can prove it missed from forward sequence gaps. It stays 0 on a healthy exchange. An engine restart is not a miss (the sequence begins again at system.startup). The counter sees only a jump it observes, so messages lost after the last one received, because pm-audit was stopped, appear in the next run's trail as the gap, not in this counter.

The separate audit.sequence_discontinuities counter records a repeated or decreasing eseq without an observed system.startup. Such a discontinuity can indicate a missed restart marker or duplicate/out-of-order publication; it does not reveal an exact number of missing messages. Check the trail and recorder health before treating the run as complete.

Log format

Every line is an independent record in this format:

[2026-07-08T09:30:00.123+00:00] [trade.executed] [seq=42 msg=01J… cause=01J… chain=01J…] {"id": "TRD-001", ...}
Part Example Description
[timestamp] 2026-07-08T09:30:00.123+00:00 UTC ISO-8601 with millisecond precision. This is pm-audit's receipt clock, not the publisher's
[topic] trade.executed ZeroMQ topic exactly as broadcast by the engine
[metadata] seq=42 msg=01J… cause=01J… chain=01J… Envelope frames, recorded as key=value pairs. Optional — see below
{...} {"symbol": "AAPL", ...} Full JSON payload of the message, byte-identical to what was published

The metadata section

Key Meaning
seq The publisher's per-topic sequence number. Dense within a run, so a gap means messages were dropped — PUB/SUB discards silently once a subscriber falls behind, and this is how you find out
eseq The publisher's count over all its messages: one total order of everything the engine published since it started. A gap is a lost message even where seq cannot show one — the last message of a topic
msg ULID of this message
cause ULID of the message that caused this one. Absent means nothing caused it — a scheduler tick, a circuit-breaker trip — not that the cause is unknown
chain ULID shared by every message descending from one original request

The section is omitted entirely for a message whose publisher stamps no envelope, and lines archived before the section existed are still read correctly, so a mixed archive needs no flag.

The header line and the engine's start

Every file — audit.log and each day part — begins with one header line that names the format and the recorder:

# edumatcher-audit format=1 pm-audit=1.0.0 started=2026-07-08T00:00:00.002+00:00

Readers skip it (it does not start with [). The first record the engine publishes on every start is system.startup: its version and source commit, the run_seq it took, the hashes of the persistence files it is about to restore from, and the compiled configuration it runs on, verbatim. An audit trail therefore describes its own run — a replay can rebuild the engine from that one message, after the data directory's ref_data/ has been redeployed. Every per-topic seq starts again at 1 after it.

The engine also records every book whole, as system.book_checkpoint.<SYM>: after startup (what the restore and the seeds made), after each session transition, and at shutdown — each resting order in priority order with its hidden iceberg quantity, each armed stop with its current trigger, and the last prices. A replay can start from any of them, so a day's audit file, or an audit that runs across an engine restart, can be checked on its own.

This is what makes a post-mortem tractable. To follow one order submission through everything it caused:

# find the submission
poetry run pm-audit-cli --format json events --topic order.new --limit 1 

# then everything in its causal chain, in order
grep 'chain=01ARZ3NDEKTSV4RRFFQ69G5FAV' data/audit.log

Lines are appended in arrival order. Within a single session they are chronologically monotonic by receipt time — but note that receipt order across several publishers is not causal order. cause and chain are what establish causality; the timestamp only establishes when pm-audit saw it.

Day parts and archives

audit.log always holds one trading day: the day is midnight to midnight in the exchange's locale.session_timezone (as in pm-stats), judged by each line's receive time. At the date change, or when it reaches --max-file-mb, pm-audit closes it as a numbered part of its day. Parts are never renamed again and nothing is ever deleted by the recorder:

data/audit.log                          ← active: today's lines, being written
data/audit/audit-2026-10-25.1.log       ← closed parts of a day, in write order
data/audit/audit-2026-10-25.2.log
data/audit/audit-2026-10-24.log.zst     ← one archive per past day

A line received at 23:59:59.990 and flushed at 00:00:05 still lands in the old day's part, so every line in audit-D.* was received on day D. A day that ends while the exchange is idle, or while pm-audit is down, is closed when pm-audit next starts.

Archives. Right after closing a day, and once at start-up, pm-audit starts a low-priority child process, pm-audit-cli archive --pending. It concatenates the day's parts into one zstd stream, audit-D.log.zst (≈ 13:1 on this data, ~140 MB/s at the default level 9), checks it by decompressing and comparing a SHA-256 and the line count, renames it into place and only then deletes the parts. A crash at any point leaves the parts, or the archive and the parts, never neither; the next run finishes the job. zstdcat audit-2026-10-24.log.zst | grep … works, since the archive is the day's log byte for byte.

A 100-trader AI simulation writes about 1.2 GB an hour, so a day is 10–14 GB as parts and about 1 GB as an archive. Disk use is governed by --keep-days:

pm-audit --max-file-mb 256 --keep-days 7    # what the ai-sim emo profile runs

Only archives are ever deleted, and only those older than --keep-days days.

Recovering a failed archive by hand. A failed archive leaves the day's parts in place, where they are still read as part of the live trail, and logs an ERROR (pm-log-srv) and counts audit.archive_errors, which pm-monitor shows as an audit_archive_errors WARN. Free the cause (usually disk space: the archive needs about a tenth of the parts' size, but the run refuses below 25 %) and run it yourself:

pm-audit-cli archive --date 2026-10-24      # one day, must be before today
pm-audit-cli archive --pending              # every closed day that has parts
pm-audit-cli archive --pending --level 12 --keep-days 30

Two runs never overlap; a second one exits at once with a note. A leftover audit-D.log.zst.part file is deleted and redone. A day that audit.log still holds (pm-audit was down at midnight) is not archived until pm-audit has closed it at its next start.

An archive is never deleted by a run. Parts found beside it are either what a crash left while deleting them (they are the archive's tail, and are deleted) or lines of that day written after it was archived (they are appended to it). If the archive itself cannot be decompressed, the run refuses that day and leaves both: check it with zstd -t audit-D.log.zst, and only if it is damaged and the parts are known to be the whole day, move it aside and re-run.

Proving a day is unchanged: the evidence manifest

Each archive appends one line to data/audit/manifest.jsonl: the day, its line and byte counts, and the SHA-256 of its uncompressed content. Every line also carries the SHA-256 of the line before it, so changing or removing any entry breaks every entry after it. Retention writes a retired line before it deletes an archive, so the chain survives --keep-days. A day reopened after it was archived appends a second entry, and the newest is current.

pm-audit-cli verify                     # the whole chain, and every archive against it
pm-audit-cli verify --date 2026-10-24   # the chain, and that day's archive

verify exits 1 and names the day when an archive's content differs from its entry, an entry is missing, an archive is missing without being retired, or the chain is broken. It ends with the head hash, the hash of the newest line.

A crash while a line was being appended leaves it torn (no newline, invalid JSON). verify reports it on every run from then on, as it should: the record of that moment is damaged. The next entry starts on a line of its own and chains to the torn line, so everything after it still verifies; the archive whose entry was torn is recorded again when the archiver finishes it.

This makes an edit evident; it does not prevent one. Someone who can rewrite the files can rewrite the whole chain, or cut off its newest entries. To rule that out, copy the head hash somewhere the recorder host cannot write (a ticket, an email, a separate system) after each close, and compare it with the one verify prints when a day is disputed.

Subscribed topics

pm-audit subscribes with an empty topic filter, which causes ZeroMQ to deliver every published message without exception:

Topic Source
order.new Gateway order submissions
order.ack.* Engine order acknowledgements
order.fill.* Fill notifications
order.cancelled.* Cancel confirmations
order.expired.* TIF expiry notifications
trade.executed Every matched trade pair
book.* Book snapshots after every change (each carries ts_ns, so a snapshot can be ordered against the trades around it)
session.state Session phase transitions (command_id present when an operator, not the scheduler, requested it)
system.* Gateway auth, symbols, EOD, GTC-restore summary and per-entity restore outcomes, absorbed internal failures
(all others) Admin events, combos, quotes, circuit breaker

Audit completeness — every state-changing decision, not just its outcome

pm-audit can only ever see what the engine publishes on :5556 — it is a bare subscriber with no other visibility into the process. Reconstructing "what happened and why" from the log is therefore only as complete as the publish call sites are, and a few decision points used to publish an outcome without the reason behind it, or nothing at all. The fields and events below close those gaps:

Gap closed Where it shows up
Every engine-initiated cancel now states why order.cancelled.{GW_ID}'s cancel_reason covers a kill switch (self or admin), a circuit-breaker halt, a gateway disconnect, an admin symbol-level mass cancel, a quote being replaced, and a quote leg cancelled because its sibling filled — in addition to the original SELF_MATCH_PREVENTED / INSUFFICIENT_LIQUIDITY. Still null for a client-requested cancel, which is not the exchange's decision to explain.
Cancels and quote removals join back to the command that caused them order.cancelled.{GW_ID} and quote.status.{GW_ID} both carry command_id when an admin/kill-switch command caused the event — the same command_id echoed on that command's ack and on admin.action. null when there is no such command (self-match prevention, a participant's own quote cancel, and so on).
A resting order can be told apart from a config-seeded one order.ack.{GW_ID} and order.fill.{GW_ID} carry is_seed — true for an order or quote leg created by market_maker_quotes/market_maker_combos at startup rather than a live gateway submission (an order's acknowledgement carries it too, and order.new itself is in the trail, since the engine echoes every command it is asked to perform before acting on it). The on-demand order.orders/order.price_level_orders snapshot rows carry it too. Purely observational: nothing in the engine branches on it.
Every combo leg gets an acknowledgement order.ack.{GW_ID} now fires once per child leg when a combo is accepted — live or config-seeded. Previously no combo leg published an ack at all; a subscriber that started after acceptance had no way to learn a leg's symbol, side, price, or qty until it was cancelled, and order.cancelled's minimal shape cannot supply them either.
Startup recovery is on the record system.startup_recovery, published once right after _restore_gtc(), reports how many orders were restored, discarded as stale TIF=DAY, or failed to restore; how many single-leg quote remnants and rebuilt QuoteIndex entries resulted; and how many GTC combos came back. Previously these counts reached only log.info/log.error. system.recovery_item now accompanies it with one line per entity — entity_id, kind (ORDER/COMBO) and outcome (RESTORED/DISCARDED_STALE_DAY/FAILED/QUOTE_REMNANT), plus detail carrying the exception text for a FAILED order — so a single bad GTC record is a wire fact with a reason attached, not just a decremented count.
Kill-switch and admin sweeps say which ids they hit, not just how many risk.kill_switch_ack, kill_switch_gateway_ack, kill_switch_global_ack, symbol_halt_ack, cancel_symbol_ack, circuit_breaker_halt_all_ack, circuit_breaker_resume_all_ack, and admin.action's scope all carry an id list beside every count they already reported — cancelled_order_ids, cancelled_quote_order_ids, halted_symbol_ids, resumed_symbol_ids, affected_gateway_ids. Previously undoing or auditing the effect of a mass cancel meant re-deriving the affected set from order.cancelled lines by command_id; the ack now says it directly, and a length mismatch against the count is itself a detectable defect.
Cancellations, expiries and amendments can be filtered by symbol order.cancelled, order.expired and order.amended all carry symbol now. pm-audit-cli events --topic order.cancelled --symbol AAPL previously returned nothing for any symbol — the engine had symbol at every publish site but never put it on the wire.
Book and depth snapshots carry a clock book.{symbol} and depth.{symbol} both gain ts_ns, stamped from the same strictly-monotonic clock trade.executed uses. Previously a snapshot carried no time field at all, so there was no way to tell from the log alone whether a given snapshot reflects a particular trade.
An operator-driven session transition is distinguishable from a scheduled one session.state gains an optional command_id, echoing the command_id on the session.transition that caused it — present only when an operator (not pm-scheduler) requested the transition. Previously the broadcast looked identical either way; telling them apart meant inferring it from what else was on the bus at the time.
Absorbed internal failures are on the record system.diagnostic fires for a maintenance-flush exception, a dispatch-handler crash, an undecodable inbound message, or a message on a topic with no handler — cases the engine deliberately survives so trading is not interrupted, but which previously reached only the process log. Carries component, detail (the flush name or topic, when there is one), error (the exception text, when there is one), and count (that component's running occurrence count). The client-facing reject for a crashed order handler still stays generic — see order.ack's INTERNAL_ERROR — so system.diagnostic is the only wire-visible place the real exception text appears.
Amendments show the prior price/qty order.amended.{GW_ID} carries old_price/old_qty alongside the new values, so a post-mortem does not need the preceding order.new/order.amended to know what changed.
Index corporate actions show the prior divisor index.corp_action_ack, index.constituent_change_ack, and index.rebalance_ack all carry old_divisor alongside divisor, mirroring HistoryRecord.old_divisor in pm-index's own local archive — a post-mortem no longer needs that archive to see what changed.

Refusals made at a gateway

The ALF, BALF and REST gateways refuse some requests themselves, before the engine sees them: a command before login, a failed authentication, a request the key's role does not permit, and a client over its rate limit. Each is sent to the engine as system.gateway_reject and echoed into the trail, so "I sent it and you ignored it" can be answered either way. A record carries the gateway (source), the participant it believed it was talking to (gateway_id, claimed and empty before login), the remote address, the reject code, the reason and the gateway's clock. It never carries an API key.

A gateway sends at most one record per peer host and code per second and puts the number it folded in as count, so a flood of refused commands is one line a second, not a flood. The host, not the host and port: a client that opens a new connection per request comes from a new port each time. The record names the full peer of the refusal that sent it. The count of the last burst is lost if the peer then goes quiet, and the first record of the burst is not.

Not recorded, on purpose: malformed frames and field-validation refusals (missing field, bad value), which the gateways count, and a read-only REST key attempting a write. pm-audit-replay story narrates a record as "A request from GW01 (10.0.0.5:4411) was refused at the ALF gateway — rate limiting".

Which clock a line is stamped with

Every audit line starts with the time pm-audit received it, which is the only clock present on every line. Three messages also carry the engine's clock, in Unix epoch nanoseconds: trade.executed.ts_ns (the match), order.fill.ts_ns (the same value as the trade it reports, so a fill and its print share one reading) and order.ack.ts_ns (when the engine accepted or refused the order, from the same strictly increasing source). For "when did the exchange accept this order", use the ack's ts_ns; the receipt time can lag it under load. Ordering between messages is by the engine's eseq, not by either clock. Lines recorded before this field existed carry the receipt time only.

Commands the trail may not hold

The engine echoes every inbound command into the trail before it acts on it, which is what lets a command that had no effect still be found. A command that reaches the engine without an envelope frame (a client that built its own PUSH socket instead of using the project's client library) cannot be echoed, and a failed echo is the other way one goes missing. Both are counted in engine.diagnostics{component="UNRECORDED_COMMAND"} in the engine's health beacon, and the first of each thousand is logged at WARNING. On a healthy exchange the counter stays at 0; any other value means the trail is not a complete record of what the exchange was asked to do.

An audited run must also be started from a deployed config artifact (pm-config-deploy). With none, system.startup carries no config and pm-audit-replay anomalies reports CONFIG_NOT_RECORDED.

Signal handling and graceful shutdown

Ctrl-C / SIGINT / SIGTERM → flush buffer → close socket → exit

The buffer flush happens synchronously before the process exits so the last batch of messages is always written to disk.

pm-audit-cli — Query Tool

pm-audit-cli is a read-only command-line tool for querying audit logs without shell pipelines or SQL knowledge. It reads JSONL files directly from disk and produces structured output. The one command that talks to the running recorder is health, which pings pm-audit to confirm it is alive.

pm-audit-cli [global-options] COMMAND [command-options]

Why a separate query tool

pm-audit is a long-running writer. Querying is a separate one-shot operation. Keeping them separate means:

  • You can query logs while pm-audit is still writing
  • The tool works on archived logs from previous sessions
  • Queries never affect the running recorder

This follows the same pattern as pm-stats-cli and pm-clearing-cli.

Global options

These flags are parsed once, before the subcommand:

Flag Default Description
--log-file PATH data/audit.log Primary audit log file to read
--log-dir PATH (parent of --log-file) Directory holding the audit/ day parts and archives
--format table\|json\|csv table Output format
--no-header off Suppress CSV header row
--use-index PATH (auto) Path to SQLite index file; auto-detected as <the directory containing --log-file>/audit_index.db when present

Not every flag applies to every command

--use-index only affects the events command — orders, trades, topics, gateways, and timeline always stream the JSONL log files directly and never consult the SQLite index, even when one exists and --use-index is given. --format/--no-header are also ignored by stats (its output is a fixed text summary, not a table) and by index (which prints build progress, not rows). health honours --format json (the report as a JSON object); csv prints the same text as table.

The auto-detected index path is always <the directory containing --log-file>/audit_index.db — it does not use --log-dir. If --log-dir points somewhere other than --log-file's own parent directory, the index lookup still only checks the log file's parent.

Output formats:

Format When to use
table Interactive terminal — aligned columns, human-readable
json Automation, scripts, downstream processing
csv Export to spreadsheets or pipelines

No-result behaviour:

  • table: prints No matching events found.
  • json: prints []
  • csv: prints header only (or nothing with --no-header)

Exit code is always 0 when a query succeeds with no results. Non-zero only on argument errors (exit 2) or I/O failures (exit 1).

Which files are read

By default pm-audit-cli reads the live trail: every closed day part that is not yet archived, oldest first, then audit.log. That is today, plus any day that has not been archived yet. Archives are not read unless you ask:

You give It reads
nothing the live trail
--date 2026-10-24 that trading day (session timezone): its archive, or its parts if not archived yet, plus audit.log if it still holds that day
--log-file data/audit/audit-2026-10-24.log.zst exactly that archive (a day part path works too)

--date selects a day's files, not a time window, and cannot be combined with an archive --log-file or with --follow. Results are chronological across all files.

Search log entries by topic prefix, gateway, symbol, and time range. This is the most flexible command and the right starting point for any investigation.

pm-audit-cli events [options]

Options:

Flag Default Description
--topic PREFIX (all) Filter by topic prefix, e.g. order.fill or trade.
--gateway GW_ID (all) Filter by gateway ID appearing in the payload
--symbol SYMBOL (all) Filter by instrument symbol
--date YYYY-MM-DD (live trail) Read this trading day's archive or parts — see Which files are read
--from ISO_TS (none) Start of time range (inclusive)
--to ISO_TS (none) End of time range (inclusive)
--limit N 100 Maximum rows returned
--reverse off Show newest events first
-f, --follow off Keep watching the log and print new events as they arrive; see Follow mode below
--interval SEC 1.0 Polling interval for --follow (ignored otherwise)

Output columns:

Column Description
timestamp UTC timestamp of the event
topic ZeroMQ topic
gateway Gateway ID extracted from payload (if present)
symbol Instrument symbol extracted from payload (if present)
order_id Order ID extracted from payload (if present)
summary Brief human-readable description of the event

summary format differs when the SQLite index is used

When events is served from the JSONL log files, summary is a structured, topic-aware description (e.g. FILL 60@150.00). When a usable index exists at the resolved --use-index path, events is served from SQLite instead, and summary is just the payload dict's str() representation truncated to 80 characters — a different, less readable format. Rebuild the index after querying if you need the latest events; the index is never queried automatically for freshness. --follow always reads the JSONL files directly and never uses the index, even when one is available — see below.

Examples:

# All events in the log (last 100)
pm-audit-cli events

# All fills for gateway GW01 on a specific date
pm-audit-cli events --topic order.fill --gateway GW01 --date 2026-07-08

# All trade executions for AAPL in a two-hour window
pm-audit-cli events --topic trade.executed --symbol AAPL \
  --from 2026-07-08T09:30:00+00:00 --to 2026-07-08T11:30:00+00:00

# Most recent 50 events across all topics
pm-audit-cli events --limit 50 --reverse

# All session phase transitions
pm-audit-cli events --topic session.state

# All order rejections
pm-audit-cli events --topic order.ack --symbol AAPL

# Export all trade events as JSON for analysis
pm-audit-cli --format json events --topic trade.executed --limit 5000 > trades.json

# Order submissions from a specific gateway in CSV format
pm-audit-cli --format csv events --topic order.new --gateway GW01 > gw01_orders.csv

# Watch fills arrive live during a training session
pm-audit-cli events --topic order.fill --follow

# Watch one symbol, polling every 2 seconds instead of the 1s default
pm-audit-cli events --symbol AAPL --follow --interval 2

Follow mode

events and timeline both accept -f/--follow to keep the query running: after printing the normal (filtered, --limit-capped) result, the command keeps polling the log files every --interval seconds and prints each newly-appended, matching entry as it arrives, until interrupted with Ctrl-C. This is the audit-log equivalent of pm-log-cli tail.

# Watch the whole log live, table format
pm-audit-cli events --follow

# Watch order lifecycle events for one gateway, as JSON
pm-audit-cli --format json events --topic order. --gateway GW01 --follow

A few things behave differently in follow mode:

  • --limit only bounds the initial backfill. The first batch printed respects --limit (100 by default for events, 500 for timeline) exactly as it would without --follow. Once the poll loop starts, every subsequent batch prints however many new matching entries actually arrived since the last poll — there is no cap. This mirrors pm-log-cli tail --before N.
  • events --reverse --follow together is rejected (exit code 2): timeline has no --reverse flag, and for events, newest-first ordering is meaningless once the log is still growing.
  • events --follow always reads the JSONL log files, even when a usable SQLite index exists at the resolved --use-index path. The index is a point-in-time snapshot and is never refreshed by a running query, so follow mode bypasses it rather than showing stale data.
  • Closing a day is handled automatically. Each poll re-discovers the live trail, so the midnight close — audit.log becoming a day part and a fresh audit.log starting — is picked up on the next poll without restarting the command, and no line is printed twice.
  • No seq requirement. Resuming after each poll is based on the entries' timestamps, not the optional seq= field in the log line's metadata section — so --follow works identically whether or not pm-audit happens to be stamping envelope metadata.

--format csv/--format json both work under --follow. Each poll batch is printed as its own block, one after another, rather than as a single combined document: for csv, only the very first batch gets a header row (later batches never repeat it, regardless of --no-header); for json, each batch is its own JSON array printed back to back with no separator between them. A consumer streaming the output should parse it incrementally (read one CSV block or JSON array at a time) rather than treating the whole stream as one JSON value or one CSV table.

orders — Order lifecycle investigation

Find all audit events related to specific order IDs or filter by gateway and symbol to reconstruct an order's full journey through the engine.

pm-audit-cli orders [options]

Options:

Flag Default Description
--id ORDER_ID (none) Order ID to find (repeatable; searches order_id and id fields in payload)
--gateway GW_ID (all) Filter by gateway ID
--symbol SYMBOL (all) Filter by symbol
--date YYYY-MM-DD (live trail) Read this trading day — see Which files are read
--from ISO_TS (none) Start of time range
--to ISO_TS (none) End of time range
--limit N 100 Maximum rows

Output columns:

Column Description
timestamp Event timestamp
order_id Order identifier
event Last dot-separated segment of the topic. For two-part topics like order.new this is the verb (new). For per-gateway topics like order.ack.GW01, order.fill.GW01, or order.cancelled.GW01 it is the gateway ID, not the verb — the topic's own verb segment (ack, fill, cancelled) is dropped.
gateway Gateway ID
symbol Instrument symbol
side BUY or SELL
qty quantity or fill_qty or remaining_qty — the first truthy value in that order (not merely the first present key; a legitimate 0 or null falls through to the next)
price Price (limit price or fill price depending on event type)
status Order status after the event
reason reject_code (from an order.ack rejection), cancel_reason (from order.cancelled), or a generic reason string — whichever the event carries

Examples:

# Full lifecycle of one specific order
pm-audit-cli orders --id ORD-GW01-00142

# Two orders side by side
pm-audit-cli orders --id ORD-GW01-00142 --id ORD-GW01-00143

# All order events for gateway GW01 today
pm-audit-cli orders --gateway GW01 --date 2026-07-08 --limit 50

# Order events for AAPL in a session
pm-audit-cli orders --symbol AAPL --from 2026-07-08T09:30:00+00:00

# Export one order's lifecycle as JSON for a support ticket
pm-audit-cli --format json orders --id ORD-GW01-00142

# Order activity for market-maker in CSV
pm-audit-cli --format csv orders --gateway MM_AAPL_01 > mm_orders.csv

Find trade.executed events with precise filtering on both sides of the trade, price ranges, and minimum quantities.

pm-audit-cli trades [options]

Options:

Flag Default Description
--symbol SYMBOL (all) Filter by instrument symbol
--gateway GW_ID (all) Match trades where this gateway appears on either side
--buy-gateway GW_ID (all) Filter by buyer gateway specifically
--sell-gateway GW_ID (all) Filter by seller gateway specifically
--min-price PRICE (none) Minimum trade price (inclusive)
--max-price PRICE (none) Maximum trade price (inclusive)
--min-qty QTY (none) Minimum trade quantity (inclusive)
--date YYYY-MM-DD (live trail) Read this trading day — see Which files are read
--from ISO_TS (none) Start of time range
--to ISO_TS (none) End of time range
--limit N 100 Maximum rows
--reverse off Show newest first

Output columns:

Column Description
timestamp Trade execution timestamp
trade_id Durable trade identifier, formatted as run_seq-counter (for example, 000042-000000001). The prefix identifies the engine run; the suffix is monotonic within that run.
symbol Instrument symbol
price Execution price
quantity Matched quantity
buy_gateway Buyer gateway ID
sell_gateway Seller gateway ID
aggressor Which side crossed the spread: BUY, SELL, or AUCTION for trades produced by an auction match

Examples:

# All trades for AAPL today
pm-audit-cli trades --symbol AAPL --date 2026-07-08

# Block trades (>= 1,000 shares) across all symbols
pm-audit-cli trades --min-qty 1000

# Trades between two specific gateways
pm-audit-cli trades --buy-gateway GW01 --sell-gateway MM_AAPL_01

# MSFT trades in a specific price band
pm-audit-cli trades --symbol MSFT --min-price 410.0 --max-price 420.0

# Most recent 20 trades
pm-audit-cli trades --limit 20 --reverse

# Export full day as CSV for post-trade analysis
pm-audit-cli --format csv trades --date 2026-07-08 --limit 10000 > day_trades.csv

# All trades involving market maker
pm-audit-cli --format json trades --gateway MM_AAPL_01

topics — Topic discovery and statistics

List all topics present in the audit logs with event counts and first/last occurrence timestamps. Useful for understanding what happened in a session and for checking whether expected event types were produced.

pm-audit-cli topics [options]

Options:

Flag Default Description
--date YYYY-MM-DD (live trail) Read this trading day — see Which files are read
--from ISO_TS (none) Start of time range
--to ISO_TS (none) End of time range
--prefix PREFIX (all) Filter topics by prefix, e.g. order.
--sort count\|alpha count Sort by event count descending (default) or alphabetically

Output columns:

Column Description
topic ZeroMQ topic name
count Number of events with this topic
first_seen Timestamp of the first occurrence
last_seen Timestamp of the last occurrence

Examples:

# All topics in the log, most frequent first
pm-audit-cli topics

# Topics on a specific trading date
pm-audit-cli topics --date 2026-07-08

# Order-related topics only, sorted alphabetically
pm-audit-cli topics --prefix order. --sort alpha

# What happened during the opening auction?
pm-audit-cli topics --from 2026-07-08T09:00:00+00:00 --to 2026-07-08T09:30:00+00:00

# Export all topic stats as JSON
pm-audit-cli --format json topics

# Check that trades were produced today
pm-audit-cli topics --prefix trade. --date 2026-07-08

gateways — Gateway activity summary

List all gateways seen in the audit logs and show a summary of their activity: order submissions, fills, and trades participated in.

pm-audit-cli gateways [options]

Options:

Flag Default Description
--date YYYY-MM-DD (live trail) Read this trading day — see Which files are read
--from ISO_TS (none) Start of time range
--to ISO_TS (none) End of time range
--min-events N (none) Only show gateways with at least N events

Output columns:

Column Description
gateway_id Gateway identifier
events Total event count (all topics)
orders order.new events submitted
fills order.fill.* events received
trades trade.executed events participated in
first_seen First event timestamp
last_seen Last event timestamp

Results are sorted by total event count descending — the most active gateway appears first.

Examples:

# All gateways in the logs
pm-audit-cli gateways

# Active gateways on a specific date
pm-audit-cli gateways --date 2026-07-08

# Only gateways with meaningful activity
pm-audit-cli gateways --min-events 10

# Gateway activity in the first hour of trading
pm-audit-cli gateways --from 2026-07-08T09:30:00+00:00 --to 2026-07-08T10:30:00+00:00

# Export as JSON for automation
pm-audit-cli --format json gateways

# Check which gateways connected during the closing auction
pm-audit-cli gateways \
  --from 2026-07-08T15:30:00+00:00 --to 2026-07-08T16:00:00+00:00

timeline — Chronological event stream

Show a raw chronological stream of events for session replay and forensic investigation. Unlike events, the timeline command exposes the full JSON payload as a single column, making it suitable for detailed reconstruction of what happened over a time window.

pm-audit-cli timeline [options]

Options:

Flag Default Description
--from ISO_TS (none) Start time
--to ISO_TS (none) End time
--topic PREFIX (all) Filter by topic prefix
--gateway GW_ID (all) Filter by gateway
--symbol SYMBOL (all) Filter by symbol
--limit N 500 Maximum events
-f, --follow off Keep watching the log and print new events as they arrive; see Follow mode above
--interval SEC 1.0 Polling interval for --follow (ignored otherwise)

Output columns:

Column Description
timestamp Event timestamp
topic ZeroMQ topic
gateway Gateway ID (if present in payload)
symbol Symbol (if present in payload)
payload Full JSON payload as a string

Examples:

# Opening 5 minutes of trading
pm-audit-cli timeline \
  --from 2026-07-08T09:30:00+00:00 --to 2026-07-08T09:35:00+00:00

# Gateway activity in a 10-minute window
pm-audit-cli timeline --gateway GW01 \
  --from 2026-07-08T14:00:00+00:00 --to 2026-07-08T14:10:00+00:00

# Symbol-focused replay with higher limit
pm-audit-cli timeline --symbol AAPL \
  --from 2026-07-08T09:30:00+00:00 --to 2026-07-08T16:00:00+00:00 \
  --limit 2000

# All session state changes across the day (timeline has no --date flag;
# use --from/--to to bound a trading day instead)
pm-audit-cli timeline --topic session.state \
  --from 2026-07-08T00:00:00+00:00 --to 2026-07-08T23:59:59+00:00

# Only trades and fills in the closing auction
pm-audit-cli timeline --topic trade. \
  --from 2026-07-08T15:30:00+00:00 --to 2026-07-08T16:00:00+00:00

# Export for external replay tooling
pm-audit-cli --format json timeline \
  --from 2026-07-08T09:30:00+00:00 --to 2026-07-08T16:00:00+00:00 \
  --limit 100000 > session_replay.json

# Watch the full raw event stream live (see Follow mode above)
pm-audit-cli timeline --follow

stats — Log file statistics

Show a health-check summary of the audit log files: total events, file sizes, date range, and counts of unique topics and gateways. Run this to confirm that pm-audit has been recording as expected.

pm-audit-cli stats [--verbose]

Options:

Flag Default Description
--verbose off Show a per-file breakdown in addition to the summary

Example output (default):

Audit Log Statistics
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Total events:       45,678
  Total size:         12.4 MB
  Log files:          3
  Oldest event:       2026-07-01T09:30:00.123+00:00
  Newest event:       2026-07-08T15:59:59.987+00:00
  Topics seen:        24
  Gateways seen:      5

Example output (--verbose):

Audit Log Statistics
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Total events:       45,678
  Total size:         12.4 MB
  Log files:          3
  Oldest event:       2026-07-01T09:30:00.123+00:00
  Newest event:       2026-07-08T15:59:59.987+00:00
  Topics seen:        24
  Gateways seen:      5

  Per-file breakdown:
    data/audit/audit-2026-07-07.1.log  12,000 events  4.1 MB
    data/audit/audit-2026-07-07.2.log  18,234 events  6.2 MB
    data/audit.log     15,444 events  2.1 MB

Examples:

# Quick health check
pm-audit-cli stats

# Full breakdown including per-file stats
pm-audit-cli stats --verbose

# An archived day, by path
pm-audit-cli --log-file data/audit/audit-2026-07-01.log.zst stats

index — Build or update the SQLite index

For large audit histories or frequent complex queries, pm-audit-cli can build an optional SQLite index from the JSONL log files. Once built, the events command uses it automatically for significantly faster response.

pm-audit-cli index [options]

Options:

Flag Default Description
--output PATH <the directory containing --log-file>/audit_index.db Destination SQLite database
--days N (none) Index only the last N days of logs
--from ISO_TS (none) Start of index range
--to ISO_TS (none) End of index range
--rebuild off Delete all existing index data and rebuild from scratch
--incremental off Add only entries newer than the last indexed timestamp

--rebuild and --incremental are mutually exclusive.

Re-running index without --rebuild/--incremental duplicates rows

Without either flag, index re-inserts every entry in the requested range on each run with no deduplication — running the same --days 7 command twice doubles the row count for that window. Use --incremental for routine re-indexing, or --rebuild to start clean.

Index schema highlights:

The index stores one row per log line in a audit_events table with dedicated columns for timestamp, topic, gateway_id, symbol, order_id, and trade_id, each with a covering index. The full JSON payload is kept in a payload column so no information is lost.

Performance comparison (1 M events):

Query type JSONL streaming SQLite index
Recent 100 events < 1 s < 0.1 s
Single topic filter 4–6 s 0.2 s
Multi-filter (topic + gateway + date) 10–15 s 0.5 s
Order ID lookup 8 s 0.05 s

Index auto-detection:

When audit_index.db exists in the log directory, the events command uses it automatically. Override with --use-index PATH or point at a different index location.

Examples:

# Build index for the last 7 days
pm-audit-cli index --days 7

# Build complete index from all log files
pm-audit-cli index --rebuild

# Incremental update — only adds entries not yet in the index
pm-audit-cli index --incremental

# Index a specific date range
pm-audit-cli index \
  --from 2026-07-01T00:00:00+00:00 --to 2026-07-08T23:59:59+00:00

# Build index to a custom location
pm-audit-cli index --output /tmp/session_idx.db --rebuild

# Use a non-default index when querying
pm-audit-cli --use-index /tmp/session_idx.db \
  events --topic trade.executed --limit 100

When to build an index

Build an index when:

  • The audit log contains more than ~500,000 events
  • You run more than a few queries per session
  • You need to correlate events across multiple fields (topic + gateway + symbol + date)

For ad-hoc queries on small logs, streaming JSONL is fast enough without the extra disk space.

health — Recorder and log file health

Check that pm-audit is alive and actually recording. By default two fast checks run:

  • Process — a pm-audit process is running (found with pgrep).
  • ZMQ ping — pm-audit answers a ping on its health socket (port 5562) with its current UTC time. The reply comes from the recorder's own receive loop, so it proves the loop is running, not just that the process exists.

The log file check is opt-in with --extended. It checks that --log-file exists and is a regular file, that its last entry is a well-formed audit line (an empty log is fine — a freshly started pm-audit has written nothing yet), and that the file can be opened for append. Nothing is ever written to the log. Only the tail of the file is read, so the check stays fast however large the log grows; it is still bounded at 15 seconds so a stalled or network-mounted volume cannot hang the command.

Unlike the query commands, health runs even when the log file does not exist; a missing file is only reported as a failure under --extended.

pm-audit-cli health
pm-audit-cli health --extended
pm-audit-cli health --extended --verbose
pm-audit-cli --format json health
pm-audit-cli health --quiet
pm-audit process: OK - pm-audit is running
audit log file:   OK - skipped (pass --extended to check read/write access)
audit ZMQ ping:   OK - replied at 2026-10-17T14:23:37.797827+00:00
health: OK

Options:

Option Description
--extended Also check the audit log file is readable, writable and ends in a well-formed entry (up to 15 s)
-v, --verbose List exactly which checks ran; with --extended, also print the log size, day parts, archives (with the compression ratio) and last entry time
-q, --quiet Print nothing; report only through the exit code

With --extended --verbose the report ends with the checks performed and the log file figures. Day parts and archives are found under <dir>/audit/, so --log-dir applies:

Checks performed:
  pm-audit process (pgrep):     yes
  ZMQ health ping:               yes
  Log file read/write/entry:     yes

Log file stats:
  path: data/audit.log
  size: 2.1MB
  day parts: 2 (10.3MB)
  archives: 3 (2.1MB, 13.4:1)
  total size: 14.5MB
  last entry: 2026-07-08T16:29:59.874+00:00

The exit code is 0 when healthy and 1 otherwise. With --quiet nothing is printed and the exit code is 0 when healthy or -1 (255 in a shell) otherwise.

The health socket binds to 127.0.0.1:5562 by default; set EDUMATCHER_AUDIT_BIND_HOST / EDUMATCHER_AUDIT_HEALTH_PORT to change where pm-audit binds it, and EDUMATCHER_ENGINE_HOST / EDUMATCHER_AUDIT_HEALTH_PORT to change where pm-audit-cli connects.

Cookbook — Common Audit Workflows

Investigate a specific order

# Find all events for order ORD-GW01-00142
pm-audit-cli orders --id ORD-GW01-00142

# Same as JSON (attach to support ticket)
pm-audit-cli --format json orders --id ORD-GW01-00142

# Find by gateway and time window (when order ID is not known)
pm-audit-cli orders --gateway GW01 \
  --from 2026-07-08T09:30:00+00:00 --to 2026-07-08T09:35:00+00:00

Reconstruct a trade

# Find trade details by symbol and time
pm-audit-cli trades --symbol AAPL \
  --from 2026-07-08T10:15:00+00:00 --to 2026-07-08T10:15:01+00:00

# Find all large trades above a price threshold
pm-audit-cli trades --min-qty 500 --min-price 150.0 --date 2026-07-08

Verify session phases fired correctly

# See all session transitions in order
pm-audit-cli events --topic session.state --date 2026-07-08

# Timeline around market open
pm-audit-cli timeline --topic session. \
  --from 2026-07-08T09:25:00+00:00 --to 2026-07-08T09:35:00+00:00

Identify the most active gateway

pm-audit-cli gateways --date 2026-07-08

Check log health before end of day

pm-audit-cli health --extended
pm-audit-cli stats --verbose
pm-audit-cli topics --date 2026-07-08 --sort count | head -10

Export a full session for offline analysis

# Build index first for speed
pm-audit-cli index --rebuild

# Export all trades
pm-audit-cli --format csv trades --date 2026-07-08 --limit 50000 > trades.csv

# Export all order events for gateway GW01
pm-audit-cli --format json events --topic order. --gateway GW01 \
  --date 2026-07-08 --limit 50000 > gw01_orders.json

Look at an archived day

# The live trail (unarchived parts + audit.log) is the default
pm-audit-cli trades --symbol AAPL --limit 10000

# One past day, read from its archive (or its parts, if not archived yet)
pm-audit-cli trades --symbol AAPL --date 2026-07-07 --limit 10000

# A single archive by path, or straight from the shell
pm-audit-cli --log-file data/audit/audit-2026-07-07.log.zst trades --symbol AAPL
zstdcat data/audit/audit-2026-07-07.log.zst | grep '\[trade\.executed\]'

Watch a live training session

# Everything, as it happens
pm-audit-cli events --follow

# Just order lifecycle events for one gateway, while proctoring a session
pm-audit-cli events --gateway GW01 --topic order. --follow

# Full raw payloads for the whole floor, piped to a file for later replay
# (each poll batch is its own JSON array back to back -- see Follow mode
# above -- not JSON Lines, so replay tooling should parse incrementally)
pm-audit-cli --format json timeline --follow > live_session.json

Error Handling

Exit code Meaning
0 Success (including queries that returned zero rows)
1 I/O error (log file not found, unreadable database)
2 Argument error (invalid date format, bad --limit, conflicting flags)

Invalid log lines (malformed JSON, truncated writes) are always silently skipped — no warning is printed, and there is no global --verbose flag on pm-audit-cli to enable one. (stats --verbose is a different, per-command flag that only adds a per-file size/count breakdown to the stats output.)

See Also

In this book:

Participant Guide:

  • P&L & Clearing — pm-clearing and pm-clearing-cli for trade settlement

Reference Manual: