Observability: daemon + viewer

The big picture

Three pieces work together:

• Flow records on disk. Every dispatch, every phase transition, every QA verdict appends a JSON line to ~/.darkmux/flows/<date>.jsonl via the LocalFileSink. Each record carries provenance fields (machine_id, orchestrator) auto-populated at write time. The CLI writes these; no daemon required. Two opt-in sinks compose alongside via TeeSink: AuditFileSink (a BLAKE3 hash-chained log; darkmux flow integrity-check walks the chain and surfaces edits. The chain is un-anchored: it detects edits, absent a full re-chain, rather than giving an OS append-only guarantee. Enable via DARKMUX_AUDIT_DIR) and RedisSink (the cross-machine coordination stream; enable via DARKMUX_REDIS_URL or config.redis).
• The darkmux serve daemon. A small HTTP server (default 127.0.0.1:8765) that reads those JSONL files and exposes them via REST endpoints plus an SSE tail, plus a /flow-status endpoint that introspects the sink composition, Redis health, and schema drift.
• The web viewer. The daemon serves a single drillable viewer at its own root, http://127.0.0.1:8765/: live timeline, filters, run drill-down. It is a static page whose JavaScript fetches from your local daemon. Nothing is uploaded. (darkmux.com/demo is the same viewer in playback mode over a recorded session, which is why it works with no daemon.)

The daemon and viewer are decoupled from the records: records keep landing on disk whether or not the daemon is running. The daemon being up is only required for live viewing.

Open the viewer

Point any browser at the daemon's own address:

http://127.0.0.1:8765/

Because the page is served by your local daemon, it is same-origin with the data it reads: it loads your records directly, with no "connect" step, no account, and nothing uploaded. (The darkmux.com/flow URL from older docs now redirects to the playback demo; for your live data, use the daemon's own address above. Loading a remote page like darkmux.com against your local daemon would be blocked by the daemon's default CORS policy anyway, which is the safe default.)

This guide intentionally ships without static screenshots: the viewer is iterating fast in this phase. Open it next to a real dispatch to see the live shape. Each card shows:

• Tier stripe: the colored bar on the left. Cyan = operator (manual mission/phase actions), violet = frontier (orchestrator narrative), green = local (model work).
• Action + handle: what happened ("dispatch start", "dispatch complete", "verdict: clean") and which entity it acted on (the role id, phase id, or mission id).
• Meta pills: category, tier, stage, source, model (if known), phase, session id. The model pill on dispatch cards shows which LMStudio model handled the work.
• Wall-clock graphic: for dispatch records, an SVG ring shows the duration, sweeping while the dispatch is mid-flight and frozen at the final duration once complete.

A source pill in the toolbar tells you the truth about the data path: live (reading your daemon, Redis-aggregated if configured), no daemon (can't reach the daemon), or replay (a static fixture, as on the demo).

Reading a live dispatch

Run something that emits records and watch them land:

darkmux mission launch <your-config>         # cyan: mission launch record (mints + starts the mission)
darkmux dispatch coder "..."                # green: dispatch start + (later) dispatch complete
darkmux mission finalize <your-mission-id>   # cyan: mission finalize + duration arc

With the viewer open as you run those commands, the records appear in order: a cyan operator-tier launch record, then a green local-tier dispatch pair (with the model pill visible on each green card), followed by the cyan mission-finalize record with its duration arc.

The records pair via session_id: dispatch start and dispatch complete share the same session id, which is how the wall-clock graphic computes duration. The viewer joins them automatically.

Managing the daemon

You started the daemon in getting started. How you operate it depends on how you installed darkmux.

Homebrew install (managed service)

The daemon runs under brew services: it starts at login and respawns on crash. You rarely touch it.

brew services start darkmux       # start now + at every login
brew services restart darkmux     # after `brew upgrade darkmux`
brew services stop darkmux        # stop + unregister from login
brew services info darkmux        # is it running? where are the logs?

The brew service binds 127.0.0.1:8765 and resolves its Redis connection (if any) from config.redis plus the Keychain. For the production-grade always-on setup (Redis hardening, audit substrate, log rotation, daily integrity checks), see the secure hub guide.

From source (manual)

A cargo install build has no brew service, so you run the daemon yourself. It is a foreground process that blocks until you Ctrl-C, so give it its own terminal tab:

darkmux serve                     # foreground; Ctrl-C to stop
darkmux serve --port 9000 --bind 127.0.0.1   # override the defaults

To restart after upgrading: Ctrl-C, cargo install --path . --force, then re-run darkmux serve. The viewer auto-reconnects on the next interaction; no need to refresh.

Verify it's up

curl -s http://127.0.0.1:8765/health
# {"darkmux_version":"1.17.0","flow_schema_version":"1.17.0"}

Or run darkmux doctor: the daemon: reachable check reports Pass, and shows the viewer URL.

What the doctor's daemon check does

darkmux doctor includes a daemon: reachable check that probes 127.0.0.1:8765/health with a short timeout. Three outcomes:

• Pass: daemon is up and returning 200 OK. Live viewing works, and the line shows the viewer URL.
• Warn (connection refused): the daemon isn't running. On Homebrew, start it with brew services start darkmux; from source, run darkmux serve. Non-fatal: you can still do work, you just won't see it live.
• Warn (port held): something is on 8765 but isn't responding HTTP 200. Check whether another process is using the port.

The check is Warn, not Fail: the daemon being off doesn't break anything end-to-end. It only disables live viewing.

The pre-dispatch nudge

When you run darkmux dispatch or darkmux mission launch review while the daemon is off, darkmux prints a one-line stderr nudge: the daemon isn't reachable, your flow records still write to disk, and you won't see them live until you start it (brew services start darkmux on Homebrew, or darkmux serve from source).

The dispatch proceeds either way: this is situational awareness, not consent-gating. You won't commit to a multi-minute dispatch only to realize the viewer would have been empty.

If you started the daemon mid-dispatch, the records that landed before you started it are still visible. The viewer fetches the whole day's file first, then opens an SSE stream for new records. No data lost.

Other daemon endpoints

Endpoint	What it returns
GET /health	darkmux version + flow schema version. Used by doctor.
GET /flow/<YYYY-MM-DD>.jsonl	That day's full flow file as JSONL.
GET /flow/<YYYY-MM-DD>/stream	Server-Sent Events stream: new records appended to that day's file, in real time.
GET /machine/status	Snapshot of lms ps --json, loaded models grouped by ownership. Backs the darkmux machine status <id> peer read. (#1426: renamed from the retired /model/status alongside the machine CLI family.)
GET /machine/resources	Live machine resources: per-model RAM commitment vs observed footprint plus pressure rows. Backs darkmux machine resources and the viewer's machine lens. Read-only, zero model dispatches.
GET /missions	All missions from ~/.darkmux/missions/ (status, phase_ids, timestamps).
GET /phases	All phases from ~/.darkmux/phases/ (status, mission_id, depends_on, timestamps).
GET /flow-status	Diagnostic snapshot of the flow substrate (active sinks, Redis health, disk health, schema drift). Same JSON shape as darkmux flow status --json; consumed by the store-status pill in the shared shell.