Skip to content

ALF TCP Gateway Protocol Training

Objective

Train on end-to-end usage of the ALF protocol through pm-alf-gwy using OS CLI tools, the example parser libraries, and the interactive example clients.

You will practise:

  • hand-writing an alf_gateway: config section (pm-config-gen has no --alf-gateway flag)
  • starting pm-alf-gwy and verifying the port is listening using OS tools
  • testing connectivity and the full session lifecycle manually with nc and telnet
  • submitting, amending, and cancelling orders over a raw TCP session
  • using the Python alf_parser.py / alf_client.py examples for scripted and interactive access
  • building and running the C alf_client example
  • operating control commands (PING, ORDERS, QBOOT, KILL)
  • diagnosing error conditions from ERR|CODE=...|REJECT_CODE=...

Background reading

Prerequisites

  • Chapters 01-26 completed.
  • Engine running and configured with at least two gateway IDs.
  • pm-alf-gwy available in the environment (poetry run pm-alf-gwy or installed via pipx).
  • Support libraries and clients available in docs/examples/alf (in a repository checkout; on a container or pipx install they are under $EX/alf, with EX=$(python -c "from edumatcher.config_deploy import examples_root; print(examples_root().parent)"); the C client needs make, a C compiler and the readline headers, so its exercise is optional in the container image):
  • python/alf_parser.py, python/alf_client.py
  • c/alf_parser.h, c/alf_parser.c, c/alf_client.c, c/Makefile

Recommended startup terminals:

  1. Engine: pm-engine --verbose
  2. ALF gateway: pm-alf-gwy
  3. One or more client terminals for connection exercises

Background

pm-alf-gwy is the TCP gateway for the ALF (Almost FIX) protocol. It accepts multiple simultaneous connections from external bots and scripts written in any language, and translates ALF commands into the same engine ZMQ messages used by the interactive pm-alf-console terminal.

In this chapter, the most important operational ideas are:

  • Session establishment: HELLO → auth round-trip → WELCOME with automatic SYMBOLS bootstrap
  • Authenticated order entry and management: NEW, AMEND, CANCEL, QUOTE
  • Gateway kill-switch: KILL (with optional SYM= scope)
  • Bootstrap state recovery: ORDERS and QBOOT
  • Liveness: PING/PONG and unsolicited HB heartbeat lines
  • Error escalation: per-error ERR|CODE=...|REJECT_CODE=... and sliding-window forced disconnect

Exercise 1: Generate a Config That Enables pm-alf-gwy

Unlike post_trade_gateway, market_data_gateway, api_gateways, and balf_gateway, pm-config-gen has no --alf-gateway* flags — there is no CLI generator support for the alf_gateway: section. Generate the base config (--no-mm-seed-quotes because MM01 is a market maker and the generator would otherwise emit price-less seed stubs that pm-config-deploy refuses), then add alf_gateway: by hand:

pm-config-gen \
  --symbols AAPL MSFT \
  --participants TRADER01 TRADER02 OPS01:ADMIN MM01:MARKET_MAKER \
  --no-mm-seed-quotes \
  --output engine_config.yaml --force

Append the alf_gateway: section yourself:

alf_gateway:
  enabled: true
  name: "alf-gwy01"
  bind_address: "0.0.0.0"          # all interfaces; use 127.0.0.1 if you do not want other machines to reach it
  port: 5565
  heartbeat_interval_sec: 5
  handshake_timeout_sec: 10
  idle_timeout_sec: 30

Inspect the section:

grep -A10 '^alf_gateway:' engine_config.yaml

See ALF TCP Gateway → Configuration for the full field reference and defaults (the section is optional — omitting it entirely still works, using every default shown there).

Back up, check, deploy — then restart

This chapter starts from a fresh configuration (its operator is OPS01), and --force replaces your engine_config.yaml: save a copy first if you want to come back to it (cp engine_config.yaml engine_config.before-ch27.yaml). As always, nothing changes until you deploy it and restart the engine: pm-cverifier engine_config.yaml, then pm-config-deploy engine_config.yaml.

Start processes with that config:

pm-engine --verbose
pm-alf-gwy

Orders from earlier chapters are still in the data directory

Resting DAY orders and market-maker quote legs from the previous chapters are restored when the engine starts. Exercise 4's "COUNT=0" answers and Exercise 8's "kill clears everything" assume MM01 and the traders start with nothing resting. For a clean venue stop every process and run pm-opctl-cli clear --state --yes before starting the engine (this also deletes clearing.db and stats.db). See Starting over for the reset matrix and preconditions.

Checkpoint: pm-alf-gwy starts, logs that it is listening on port 5565, and reports a successful engine ZMQ connection.

Exercise 2: Verify the Port Is Listening with CLI Tools

Before writing any client code, confirm the gateway is actually bound to the configured port using OS tools. This is the fastest way to rule out startup failures, bind-address mismatches, and firewall issues.

macOS:

# lsof — shows the process name and PID holding the port
sudo lsof -iTCP:5565 -sTCP:LISTEN

# BSD netstat (ships with macOS)
netstat -an | grep LISTEN | grep 5565

Expected: a line with pm-alf-gwy (or Python if running under Poetry) and LISTEN.

Linux:

# ss — preferred on modern Linux
ss -tlnp 'sport = :5565'

# lsof
sudo lsof -iTCP:5565 -sTCP:LISTEN

# netstat (older distributions)
netstat -tlnp | grep 5565

Both commands should show a process bound to *:5565 or 127.0.0.1:5565.

If no output appears:

  • the gateway is not running or failed to start — check logs
  • alf_gateway.enabled may be false in the config
  • the port may differ from what you expect — re-run grep -A5 'alf_gateway:' engine_config.yaml
  • pm-engine must be running before pm-alf-gwy can complete its ZMQ handshake

Checkpoint: at least one CLI command shows a process listening on port 5565.

Exercise 3: Manual Handshake Probe with nc

Use nc (netcat) to open a raw TCP connection and type ALF lines by hand. This verifies the full session lifecycle — HELLO, auth, WELCOME, SYMBOLS — with no code.

nc 127.0.0.1 5565

Type these lines, pressing Enter after each:

HELLO|CLIENT=test|PROTO=ALF1|ID=TRADER01
SYMBOLS
EXIT

Expected response pattern:

  1. WELCOME|PROTO=ALF1|GW=alf-gwy01|ID=TRADER01|HBINT=5|IDLE=30
  2. Automatic SYMBOLS|COUNT=N block with one SYMBOL|SYM=...|TICK=... per instrument
  3. END|TYPE=SYMBOLS
  4. HB|TS=... every 5 seconds when quiet
  5. Connection closes after EXIT

Also try ORDERS to see the empty bootstrap snapshot:

HELLO|CLIENT=test|PROTO=ALF1|ID=TRADER01
ORDERS
EXIT

Expected: ORDERS|COUNT=0|GW=TRADER01 and END|TYPE=ORDERS.

Checkpoint: nc session reaches WELCOME and receives a valid SYMBOLS block and an END|TYPE=ORDERS line.

Exercise 3B: telnet and Non-Interactive One-Liner

telnet (macOS / Linux):

telnet 127.0.0.1 5565

Type HELLO|CLIENT=test|PROTO=ALF1|ID=TRADER01 and press Enter. telnet echoes characters locally so the line looks doubled in the terminal — the WELCOME response confirms the gateway accepted it. Press Ctrl-], then type quit to close.

Non-interactive probe — useful in scripts or CI pipelines:

printf 'HELLO|CLIENT=ci|PROTO=ALF1|ID=TRADER01\nSYMBOLS\nEXIT\n' \
  | nc 127.0.0.1 5565

Expected: the full session including WELCOME, SYMBOLS block, and END|TYPE=SYMBOLS is printed to stdout and the command exits cleanly.

Checkpoint: the one-liner prints WELCOME and END|TYPE=SYMBOLS without interaction.

Exercise 4: Submit and Cancel an Order over nc

Open a new nc session and type all commands (press Enter after each line):

nc 127.0.0.1 5565

HELLO|CLIENT=manual|PROTO=ALF1|ID=TRADER01
NEW|SYM=AAPL|SIDE=BUY|TYPE=LIMIT|QTY=100|PRICE=150.00|TAG=NC-ORDER-001
ORDERS

Expected behavior:

  • ACK|ORDER_ID=<uuid>|ACCEPTED=TRUE|... arrives immediately after NEW
  • the ACK and later lifecycle events echo TAG=NC-ORDER-001
  • ORDERS shows the resting order with STATUS=NEW and REMAINING=100
  • If MM01 is quoting, the order may fill immediately and ORDERS returns COUNT=0

When an order fills, its FILL event carries TRADE_IDS=<id[,id...]>. Keep those durable public trade IDs with the order record: a swept order can contain several, in match order, and they join the private fill to the public trade tape, CALF, and drop copy without matching on price or time.

FILL also carries LIQUIDITY=<MAKER|TAKER> — whether this order supplied liquidity by resting on the book (MAKER) or removed it by crossing the spread (TAKER). If MM01 filled your order immediately, expect TAKER; if you place a resting LIMIT and a later order crosses it, that later FILL (for the counterparty) says TAKER while yours says MAKER. It is the same attribution the drop-copy DC_FILL line has always reported (see Exercise 8 → DC below, or Market Data & Drop Copy) — you no longer need drop copy just to find out which side of a trade you were on.

Cancel the resting order using the UUID from the ACK and a request tag:

CANCEL|ID=<uuid>|RTAG=NC-CXL-001

Expected: CANCELLED|ORDER_ID=<uuid>|TAG=NC-ORDER-001|RTAG=NC-CXL-001.

Try amending before cancelling to see priority rules in action:

NEW|SYM=AAPL|SIDE=BUY|TYPE=LIMIT|QTY=200|PRICE=149.00|TAG=NC-ORDER-002
AMEND|ID=<uuid>|PRICE=149.50|QTY=150|RTAG=NC-AMD-001

Expected: AMENDED|ORDER_ID=...|PRICE=149.5|QTY=150|REMAINING=150|PRIORITY_RESET=TRUE|TAG=NC-ORDER-002|RTAG=NC-AMD-001

RTAG is optional, but it is the easiest way to prove which amend or cancel a later TCP response belongs to.

Checkpoint: you can submit, confirm, amend, and cancel an order using only raw TCP and a terminal.

Exercise 5: Use the Python Interactive Client

alf_client.py provides a pm-alf-console-style interactive session over TCP, with tab-completion, background event display, and local P&L tracking.

cd docs/examples/alf/python
python3 alf_client.py --id TRADER01

At the prompt:

[TRADER01]> SYMBOLS
[TRADER01]> NEW|SYM=AAPL|SIDE=BUY|TYPE=LIMIT|QTY=100|PRICE=150.00|TAG=PY-ORDER-001
[TRADER01]> ORDERS
[TRADER01]> POS
[TRADER01]> STATUS
[TRADER01]> HELP
[TRADER01]> EXIT

Expected behavior:

  • POS shows positions and realized P&L from fills seen by this connection; POS|GW=TRADER01 queries the engine's position snapshot
  • STATUS shows session info: gateway name, heartbeat interval, idle timeout
  • fills, HB, and broadcast SESSION/TRADE events appear in real time while you type other commands
  • Tab completes command verbs and field names
  • Command history is saved across sessions to ~/.alf_client_history
  • Both interactive clients send keepalive PING automatically; DC_FILL is displayed separately without counting its execution a second time

Connect to a remote gateway:

python3 alf_client.py --host 10.0.0.5 --port 5565 --id TRADER01

Checkpoint: interactive Python client connects, shows WELCOME, and ORDERS and POS respond correctly.

Exercise 6: Use the Python Library Directly

alf_parser.py exposes parse_alf_line, build_alf_line, and the AlfSession high-level class for use in scripts.

cd docs/examples/alf/python
python3 - <<'EOF'
from alf_parser import parse_alf_line, build_alf_line, AlfSession, AlfMessage

# Parse a line received from the gateway
msg: AlfMessage = parse_alf_line("ACK|ORDER_ID=abc|ACCEPTED=FALSE|REJECT_CODE=ORDER_NOT_FOUND|REASON=missing")
print(msg.msg_type)   # ACK
print(msg.fields)     # {"ORDER_ID": "abc", "ACCEPTED": "FALSE", "REJECT_CODE": "ORDER_NOT_FOUND", ...}

# Build a line to send
line = build_alf_line("NEW", {
    "SYM": "AAPL", "SIDE": "BUY",
    "TYPE": "LIMIT", "QTY": "100", "PRICE": "150.00", "TAG": "NC-ORDER-003",
})
print(repr(line))  # 'NEW|SYM=AAPL|SIDE=BUY|TYPE=LIMIT|QTY=100|PRICE=150.00|TAG=NC-ORDER-003\n'

# High-level session: connect, HELLO/WELCOME, consume a complete query
session = AlfSession.connect("127.0.0.1", 5565, "TRADER01")
try:
    print(session.welcome.gw_name)   # alf-gwy01
    session.sock.settimeout(2.0)    # fail visibly if this short exercise stalls
    session.send("ORDERS")
    while True:
        msg = session.recv_msg()
        print(msg.msg_type, msg.fields)
        if msg.msg_type == "END" and msg.fields.get("TYPE") == "ORDERS":
            break
finally:
    session.close()
EOF

Expected output: ACK, the parsed fields dict, the built line string, the gateway name from WELCOME, and the complete ORDERS response, possibly interleaved with heartbeats or broadcasts.

This short exercise closes after one query. AlfSession validates the handshake and buffers TCP lines but does not start keepalives or reconnect. For a persistent application, copy alf_parser.py beside your program and use the application integration walkthrough, which includes periodic PING, a query deadline, error handling, and reconciliation responsibilities.

Checkpoint: AlfSession.connect completes the HELLO/WELCOME handshake and recv_msg returns a parsed message.

Optional Exercise 7: Build and Run the C Client

The container image does not include make, a C compiler, or readline development headers. Skip this exercise there; Exercise 5's Python interactive client provides the same gateway workflow.

macOS — install Homebrew readline for full callback support if not already present:

brew install readline

Linux:

sudo apt install libreadline-dev   # Debian/Ubuntu

Build:

cd docs/examples/alf/c
make

Run:

./alf_client --id TRADER01

At the prompt:

[TRADER01]> SYMBOLS
[TRADER01]> NEW|SYM=AAPL|SIDE=BUY|TYPE=LIMIT|QTY=100|PRICE=150.00|TAG=C-ORDER-001
[TRADER01]> ORDERS
[TRADER01]> EXIT

Expected behavior:

  • the C client uses select() to multiplex the socket and stdin, so gateway events (fills, heartbeats, broadcasts) appear in real time while you type
  • readline provides tab completion and persistent command history
  • --no-color disables ANSI colour output for plain terminals

Try connecting to a remote host:

./alf_client --host 10.0.0.5 --port 5565 --id TRADER01

Optional checkpoint: the C client connects, shows WELCOME, and a NEW order returns an ACK.

Exercise 8: Control Commands and the Kill-Switch

PING / PONG

nc 127.0.0.1 5565

HELLO|CLIENT=ctrl|PROTO=ALF1|ID=TRADER01
PING

Expected: PONG|TS=<ISO-8601-timestamp>.

QBOOT — quote bootstrap state (MARKET_MAKER role)

Open a second terminal and connect with the MM01 identity:

nc 127.0.0.1 5565

HELLO|CLIENT=mm|PROTO=ALF1|ID=MM01
QBOOT

Expected: QBOOT|COUNT=0 and END|TYPE=QBOOT initially.

Submit a quote then re-check:

QUOTE|SYM=AAPL|BID=149.90|ASK=150.10|BID_QTY=500|ASK_QTY=500
QBOOT

Expected: QBOOT|COUNT=1 with one active QUOTE|... line, then END|TYPE=QBOOT. (If MM01 still had a quote on AAPL from an earlier chapter you also see a CANCELLED|…|CANCEL_REASON=QUOTE_REPLACED line for the old one.)

KILL — cancel all resting orders and active quotes

KILL

Expected: KILL_ACK|ACCEPTED=TRUE|ORDERS=N|QUOTES=N.

Confirm everything was cleared:

ORDERS
QBOOT

KILL_ACK reports non-quote orders in ORDERS and quote-origin orders in QUOTES. A partially filled quote can be inactive while its unfilled remainder still rests; KILL removes that remainder and includes it in QUOTES, even after an engine restart.

For example, if 340 shares of a 1,000-share quote leg filled before restart, ORDERS may show its 660-share remainder. After KILL, the acknowledgement reports ORDERS=0 and QUOTES=1 for that remainder. ORDERS then returns ORDERS|COUNT=0|GW=MM01 followed by END|TYPE=ORDERS; QBOOT is empty because the quote was already inactive. If other orders or quotes are resting, their cancellations are included in the corresponding acknowledgement count.

Scope the kill to one symbol:

NEW|SYM=AAPL|SIDE=BUY|TYPE=LIMIT|QTY=100|PRICE=149.00
NEW|SYM=MSFT|SIDE=BUY|TYPE=LIMIT|QTY=50|PRICE=400.00
KILL|SYM=AAPL
ORDERS

Expected: only the MSFT order remains.

Checkpoint: you can use PING, QBOOT, and KILL (with and without SYM=) and interpret the responses.

Exercise 9: Error Conditions and Operational Interpretation

Test typical protocol errors. Start a fresh nc session and send each line before completing the HELLO handshake, or with invalid content:

NEW|SYM=AAPL|SIDE=BUY|TYPE=LIMIT|QTY=100|PRICE=150.00|TAG=BAD-AUTH-001    (before HELLO)
HELLO|CLIENT=test|PROTO=WRONG|ID=TRADER01
HELLO|CLIENT=test|PROTO=ALF1|ID=NONEXISTENT

After a successful HELLO, try:

NEW|SYM=AAPL|SIDE=BUY|TYPE=LIMIT|QTY=100|TAG=BAD-MISSING-001          (missing PRICE for LIMIT)
NEW|SYM=UNKNOWN|SIDE=BUY|TYPE=LIMIT|QTY=100|PRICE=150.00|TAG=BAD-SYM-001
QUOTE|SYM=AAPL|BID=150.00|ASK=150.10|BID_QTY=500|ASK_QTY=500   (TRADER role)

Map each observed ERR|CODE=...|REJECT_CODE=... to its operator remediation:

CODE Typical REJECT_CODE Practical interpretation
AUTH_REQUIRED AUTH_REQUIRED Client sent a non-HELLO first line; fix client code ordering
PROTO_MISMATCH AUTH_FAILED Wrong PROTO= value; must be exactly ALF1
AUTH_FAILED UNKNOWN Gateway ID not in participants; add to config and restart engine
GATEWAY_ALREADY_CONNECTED AUTH_FAILED Same ID connected from another session; disconnect it first
MISSING_FIELD MISSING_FIELD Required field absent for this order type; check command reference
INVALID_VALUE INVALID_VALUE Field value fails validation (e.g. PRICE=NaN, QTY=0); fix client
SYMBOL_NOT_CONFIGURED UNKNOWN_SYMBOL Unknown symbol; run SYMBOLS to see the configured list
ROLE_DENIED ROLE_DENIED Command not allowed for this gateway's role (QUOTE requires MARKET_MAKER)
RATE_LIMITED RATE_LIMITED Commands arriving faster than max_commands_per_second; throttle client

CODE is the ALF gateway's local validation category. REJECT_CODE is the canonical order-rejection class used by ALF, REST, and tests. When the rejected line carried TAG or RTAG, the gateway echoes it as TAG on the ERR so a client can correlate the failure.

Checkpoint: you can map each ERR|CODE and REJECT_CODE to the correct config or client-code fix.

Support Libraries and Example Clients

Reference implementations used in this training chapter:

  • docs/examples/alf/python/alf_parser.py — protocol library: parse_alf_line, build_alf_line, AlfSession
  • docs/examples/alf/python/alf_client.py — interactive client with tab-completion, event display, and P&L tracking
  • docs/examples/alf/c/alf_parser.h / alf_parser.c — C protocol library: alf_parse_line, alf_build_line, alf_get_field
  • docs/examples/alf/c/alf_client.c — interactive C client using select() + readline

Use these to bootstrap ALF clients in any language that can open a TCP socket and read newline-delimited text. The Python parser/session and C parser libraries do not need readline; only the interactive clients do. The C parser is not a session library: adapt the C client's transport and lifecycle code for a headless application, check parser/builder return codes, and review its fixed capacities. See the reusable-client guide for the integration checklist.

Summary

You can now:

  • Hand-write the alf_gateway: config section and start pm-alf-gwy.
  • Verify a TCP port is listening using lsof, ss, and netstat on macOS and Linux.
  • Test the complete ALF session lifecycle manually with nc and telnet before writing any code.
  • Submit, amend, and cancel orders over a raw TCP session and read ACK, AMENDED, and CANCELLED responses.
  • Correlate an ALF FILL to its public executions through TRADE_IDS.
  • Use python3 alf_client.py for interactive sessions with tab-completion and live event display.
  • Script against the gateway using AlfSession from alf_parser.py.
  • Build and run the C alf_client for a readline-based interactive session with select() multiplexing.
  • Operate PING, ORDERS, QBOOT, and KILL and interpret their responses.
  • Map ERR|CODE=...|REJECT_CODE=... values to the correct config or client-side fix.

Reflection

pm-alf-gwy (this chapter) and pm-api-gwy (Chapter 25) both allow external clients to submit orders to the same matching engine. Answer the following:

  1. A Python bot connecting to pm-alf-gwy receives a FILL event pushed directly over its persistent TCP connection. A Python bot connecting to pm-api-gwy learns about the same fill through a WebSocket events stream. Both need to act on fills within milliseconds. Which transport has lower latency overhead and why?

  2. A compliance team needs read-only access to every fill for all gateway IDs — not just their own. Can they use pm-alf-gwy for this? If not, which gateway process should they consume instead and with what role?

  3. After a network interruption, an ALF bot reconnects and sends ORDERS to recover resting state. The API gateway equivalent is GET /api/v1/orders. At the protocol level, which reconnect handler is simpler to implement and why?

Handoff for Chapter 28

This completes the external gateway protocol series. pm-engine, pm-alf-gwy, and any other running processes can remain up if you want to continue a live session, or you can stop them and start fresh for the next chapter.

Further Reading