Observability: daemon + viewer

The big picture

Three pieces work together:

• Flow records on disk. Every dispatch, every phase transition, every QA verdict appends a JSON line to ~/.darkmux/flows/<date>.jsonl via the LocalFileSink. Each record carries provenance fields (machine_id) auto-populated at write time. The CLI writes these; no daemon required. Two opt-in sinks compose alongside via TeeSink: AuditFileSink (a BLAKE3 hash-chained log; darkmux flow integrity-check walks the chain and surfaces edits. The chain is un-anchored: it detects edits, absent a full re-chain, rather than giving an OS append-only guarantee. Enable via DARKMUX_AUDIT_DIR) and RedisSink (the cross-machine coordination stream; enable via DARKMUX_REDIS_URL or config.redis).
• The darkmux serve daemon. A small HTTP server (default 127.0.0.1:8765) that reads those JSONL files and exposes them via REST endpoints plus an SSE tail. Sink composition, Redis health, and schema drift are reported by darkmux flow status, not by an HTTP endpoint.
• The web viewer. The daemon serves a single drillable viewer at its own root, http://127.0.0.1:8765/: live timeline, filters, run drill-down. It is a static page whose JavaScript fetches from your local daemon. Nothing is uploaded. (darkmux.com/demo is the same viewer in playback mode over a recorded session, which is why it works with no daemon.)

The daemon and viewer are decoupled from the records: records keep landing on disk whether or not the daemon is running. The daemon being up is only required for live viewing.

Open the viewer

Point any browser at the daemon's own address:

http://127.0.0.1:8765/

Because the page is served by your local daemon, it is same-origin with the data it reads: it loads your records directly, with no "connect" step, no account, and nothing uploaded. (The darkmux.com/flow URL from older docs now redirects to the playback demo; for your live data, use the daemon's own address above. Loading a remote page like darkmux.com against your local daemon would be blocked by the daemon's default CORS policy anyway, which is the safe default.)

The captures here come from a demo fleet, not a real machine. Each card shows:

• Tier stripe: the colored bar on the left. Cyan = operator (manual mission/phase actions), violet = frontier (orchestrator narrative), green = local (model work).
• Action + handle: what happened ("dispatch.start", "dispatch.complete", "phase.review.verdict") and which entity it acted on (the role id, phase id, or mission id).
• Meta pills: category, tier, stage, source, model (if known), phase, session id. The model pill on dispatch cards shows which LMStudio model handled the work.
• Wall-clock graphic: for dispatch records, an SVG ring shows the duration, sweeping while the dispatch is mid-flight and frozen at the final duration once complete.

A source pill in the toolbar tells you the truth about the data path: live (reading your daemon, Redis-aggregated if configured), no daemon (can't reach the daemon), or replay (a static fixture, as on the demo).

Reading a live dispatch

Run something that emits records and watch them land:

darkmux mission launch <your-config>         # cyan: mission launch record (mints + starts the mission)
darkmux dispatch coder "..."                # green: dispatch.start + (later) dispatch.complete
darkmux mission finalize <your-mission-id>   # cyan: mission finalize + duration arc

With the viewer open as you run those commands, the records appear in order: a cyan operator-tier launch record, then a green local-tier dispatch pair (with the model pill visible on each green card), followed by the cyan mission-finalize record with its duration arc.

The records pair via session_id: dispatch.start and dispatch.complete share the same session id, which is how the wall-clock graphic computes duration. The viewer joins them automatically.

Managing the daemon

You started the daemon in getting started. How you operate it depends on how you installed darkmux.

Homebrew install (managed service)

The daemon runs under brew services: it starts at login and respawns on crash. You rarely touch it.

brew services start darkmux       # start now + at every login
brew services restart darkmux     # after `brew upgrade darkmux`
brew services stop darkmux        # stop + unregister from login
brew services info darkmux        # is it running? where are the logs?

The brew service binds 127.0.0.1:8765 by default and resolves its Redis connection (if any) from config.redis plus the Keychain. For the production-grade always-on setup (Redis hardening, audit substrate, log rotation, daily integrity checks), see the always-on hub guide.

From source (manual)

A cargo install build has no brew service, so you run the daemon yourself. It is a foreground process that blocks until you Ctrl-C, so give it its own terminal tab:

darkmux serve                     # foreground; Ctrl-C to stop
darkmux serve --port 9000 --bind 127.0.0.1   # override the defaults

To restart after upgrading: Ctrl-C, cargo install --path . --force, then re-run darkmux serve. The viewer auto-reconnects on the next interaction; no need to refresh.

Verify it's up

# 8765 unless you set serve.port — `darkmux doctor`'s `serve address` row prints the resolved one.
curl -s http://127.0.0.1:8765/health
# {"darkmux_version":"5.0.0","flow_schema_version":"2.1.0", ...}

Or run darkmux doctor: the daemon: reachable check reports Pass, and shows the viewer URL. It resolves the address itself, which makes it the reliable answer when you are not certain what this daemon is listening on.

What the doctor's daemon check does

darkmux doctor includes a daemon: reachable check that probes this machine's resolved daemon address at /health with a short timeout — serve.bind and serve.port if you set them, otherwise 127.0.0.1:8765. It does not assume the built-in port. Three outcomes:

• Pass: daemon is up and returning 200 OK. Live viewing works, and the line shows the viewer URL.
• Warn (connection refused): the daemon isn't running. On Homebrew, start it with brew services start darkmux; from source, run darkmux serve. Non-fatal: you can still do work, you just won't see it live.
• Warn (port held): something is on that port but isn't responding HTTP 200. Check whether another process is using it.

The check is Warn, not Fail: the daemon being off doesn't break anything end-to-end. It only disables live viewing.

The pre-dispatch nudge

When you run darkmux dispatch or darkmux mission launch review while the daemon is off, darkmux prints a one-line stderr nudge: the daemon isn't reachable, your flow records still write to disk, and you won't see them live until you start it (brew services start darkmux on Homebrew, or darkmux serve from source).

The dispatch proceeds either way: this is situational awareness, not consent-gating. You won't commit to a multi-minute dispatch only to realize the viewer would have been empty.

If you started the daemon mid-dispatch, the records that landed before you started it are still visible. The viewer fetches the whole day's file first, then opens an SSE stream for new records. No data lost.

Daemon routes

Routes and response shapes are semver contracts. Each JSON response is a Rust type in crates/darkmux-serve/src/wire.rs, its TypeScript twin is generated into ui/src/types/generated/, and the route table is pinned by crates/darkmux-serve/route-table.golden. There are no aliases: a retired route answers 404.

Endpoint	What it returns
GET /	The viewer. GET /play/<YYYY-MM-DD> serves it in playback mode for one day.
GET /health	darkmux version, build, flow schema version, the lifecycle policy, and the listener and live-channel state. Used by doctor. Response type HealthResponse.
GET /flow/<YYYY-MM-DD>	That day's flow records in a FlowRecordsResponse envelope (the same one /flow-mission/:id returns); ?since=<iso-ts> narrows it.
GET /flow/<YYYY-MM-DD>/stream	Server-Sent Events stream: new records appended to that day's file, in real time.
GET /flow-days	One row per day file on disk (FlowDaysResponse).
GET /flow-missions	A cross-day rollup per mission id (FlowMissionsResponse).
GET /flow-mission/<id>	Every flow record of one mission, across days and machines (FlowRecordsResponse).
GET /flow-dispatch/<id>	Every flow record of one dispatch, across days and machines (FlowRecordsResponse). Renamed from /flow-session/<id>.
GET /runs	The flat, kind-tagged run list the viewer's runs lens reads (RunsResponse).
GET /missions, GET /phases	Missions and phases from ~/.darkmux/missions/ (MissionsResponse, PhasesResponse).
GET /mission/<id>/graph.json	The node and edge snapshot of one mission's task graph (MissionGraph).
GET /machine/status	Snapshot of lms ps --json. Backs the darkmux machine status <id> peer read (MachineStatusResponse).
GET /machine/specs	Version, machine id, RAM, CPU brand, OS and loaded models (MachineSpecsResponse). The machine card, which travels on the fleet listener, embeds it.
GET /machine/resources	Live machine resources: per-model RAM commitment vs observed footprint, pressure rows and the host sampler's reading. Read-only, zero model dispatches (MachineResourcesResponse).
GET /fleet/machines/live, GET /fleet/dispatches/live, GET /fleet/roster	The machines beating right now, the dispatches with a live heartbeat (renamed from /fleet/sessions/live), and the declared roster.
GET /fleet/view	One row per roster machine, and always a row for this machine: presence as a display column, the machine's own card (this machine’s built locally, a peer’s fetched from its fleet listener over the verified peer path, in parallel and cached for 5 seconds), its resolved hardware uid, and what the machine accepts from this one (FleetView). accepts is shown only to this machine, and seats only to this machine or a token holder. Backs darkmux machine list. A machine’s card is served to its peers by its fleet listener at GET /fleet/card, not by the daemon.
GET /lab/runs, GET /lab/run/detail, GET /lab/run/events	The lab observer: run directories, one run's reviews and scores, and a poll-based tail of its events.
GET /panel/<id>	An allowlisted CLI command's own rendered output (PanelResponse).