06 · machine setup

Join a worker peer

Connect another machine on your tailnet to an existing darkmux hub: many machines, one development environment. This is a clean-machine, brew-only walkthrough: every command after brew install should work as written. Start small (an observability peer you dispatch to directly), and graduate to a full fleet worker when the hardware's ready.

Don't have a hub yet?

A peer connects to a hub, the always-on machine running Redis. If you haven't set one up, do that first: set up an always-on hub. It's a short, secure, copy/paste recipe (password-less Redis bound to your tailnet, never the public internet). Come back here once darkmux doctor is green on the hub. You'll need two values from it: the hub's tailnet address (tailscale ip -4 on the hub, for the raw Redis check below) and its tailnet DNS name (tailscale status, or the Tailscale admin console — for the roster entry you register with machine add).

Two postures: observability peer, or fleet worker

A peer can play two roles, and they're a progression. Start with the first; move to the second when you want it.

How fleet work travels (4.0). A dispatch sent with --machine studio goes straight to the studio's fleet listener, carrying the fleet token. The studio runs it only if the network itself says the request came from a machine on its allow-list, and only on a profile that machine is allowed to use. Anything else is refused at once, with the reason. Nothing is taken off a shared queue: the Redis work queue of earlier releases is retired, and Redis is only the shared record stream now.

Part 1: Join as an observability peer

On the new machine: install Homebrew and join your Tailscale tailnet (tailscale up). If you'll run dispatches on the peer, also do the base getting-started setup (LMStudio + Docker). Then, confirming it can reach the hub:

# 0. Sanity: can this machine reach the hub over the tailnet?
#    HUB_IP is the hub's Tailscale address. Run `tailscale ip -4` ON THE HUB.
HUB_IP=<hub-tailnet-addr>
nc -zvw1 "$HUB_IP" 6379                            # expect "succeeded"

# 1. Install darkmux.
brew tap kstrat2001/tap
brew install darkmux

# 2. Declare this machine a peer and point it at the hub's Redis (password-less).
darkmux config set fleet.mode peer
darkmux config set redis.enabled true
darkmux config set redis.host "$HUB_IP"
darkmux config set machine_id mini-1              # operator-named; your call

# 3. Verify it reaches the hub.
darkmux doctor

darkmux doctor should read ✓ on flow sink health (a Redis-backed sink reaching the hub) and machine_id, and the verdict banner should land on ● ok (a ⚠ on out-of-scope checks like "no profile loaded yet" is fine). Then prove the end-to-end write:

darkmux flow note --text "hello from mini-1"

Open the hub's live viewer (the hub itself, or https://<hub>.<tailnet>.ts.net/ if you set up Tailscale Serve). The note appears, tagged with this machine's id. That's the peer participating: its records flow to the hub, you see the whole fleet in one place.

No daemon required for this. Writing flow records to the hub happens in-process on every darkmux command; the peer does not need to run darkmux serve for its activity to show up in the hub's viewer. (The trade-off: without a local daemon, the peer won't appear in darkmux machine list --deep's live specs probe, which fans out to each machine's daemon. Its flow records are still in the hub viewer.)

Can I dispatch to this peer from another machine?

Short answer for Part 1: no. A darkmux dispatch <role> --machine mini-1 … from another machine is sent to mini-1's fleet listener, which only exists inside a running darkmux serve with fleet.listener.enabled on. A Part 1 peer runs neither, so the dispatch fails at once with "no answer from mini-1". It never runs somewhere else instead: --machine names the one machine that runs the job.

Run work on the peer (locally)

On the peer, dispatch with no --machine. It runs in the peer's own container, sized by you to fit the hardware:

# On the peer:
darkmux dispatch coder "implement the X feature"

This is the reliable way to use a small box: drive it directly (in person, or over SSH / its own Claude Code session), pick model profiles that fit its RAM, and let its activity stream to the hub for the fleet-wide view. The dispatch path is identical to the hub's (same container, same runtime loop, same flow records), just initiated on the machine that runs it.

Part 2: Run it as a fleet worker

A fleet worker takes work from other machines over its fleet listener: a second port the daemon opens on the machine's own tailnet address (never 0.0.0.0, never loopback, and not behind Tailscale Serve, which would make every caller look like the machine itself). A request is run only when both checks pass:

Each allow-list entry also has a scope: the profiles (on the worker) that machine may run. A profile outside it is refused, and so is a profile that runs on the worker's utility model: utility work (compaction, radio routing) always stays on the machine it serves.

# On the worker (say the studio). The fleet token first, if it has none:
security add-generic-password -U -a "$USER" -s darkmux-serve-token -w   # same value on every machine
darkmux config set runtime.daemon_auth_enabled true

# Trust the laptop for two profiles and two roles. The node is looked up on the tailnet by name
# (the laptop's roster address, or --node <its tailnet name>); you never type its id.
darkmux machine trust laptop --profiles host,coder-studio --roles radio-host,coder

# Open the listener (port 8766 by default; use the same port on every machine).
darkmux config set fleet.listener.enabled true
brew services restart darkmux

darkmux doctor 2>&1 | grep -i fleet   # fleet token / identity / listener / trust rows

Then, from the laptop (the fleet token in its Keychain too, and the studio in its roster, below):

darkmux dispatch radio-host "summarize today's runs" --machine studio --profile host

With --wait (the default) the command returns when the studio's dispatch finishes, with its exit code and output. --no-wait returns as soon as the studio accepts the job. A refusal comes back immediately, naming the reason: studio does not accept work from macbook-pro, not in the allow-list scope: profile coder-big, or studio is busy running <session> (a worker runs one submitted job at a time). darkmux machine untrust laptop takes the laptop back off the list; both verbs take effect on the next request, with no restart. Removing a machine from the tailnet revokes it everywhere, and rotating the fleet token is one secret to redistribute.

What an entry grants, exactly. Roles: only the roles listed (a role is a tool palette; there is no "any role", and utility roles are refused). Images: the job runs on darkmux's own runtime image unless it names one the entry lists with --images. Workspace: --workspace true lets the peer mount any directory under this machine's darkmux worktrees base read-write as the job's workspace, including live worktrees your own git and test commands later run in. In effect that lets the peer run code on this machine, so grant it only to a machine you would give a shell; the default is none. (darkmux does not yet exclude worktrees that local missions are using.) A submitted job never mounts this machine's shared toolchain cache, so nothing it writes is run by your own later dispatches. One job at a time: a trusted peer that keeps resubmitting can keep the slot busy; that is accepted in a model where every trusted peer is yours.

Upgrade every machine. A 3.x daemon still claims work off the old Redis queue (darkmux:work) and re-creates it when it starts, so the unauthenticated queue stays open until no machine runs 3.x. darkmux doctor names any daemon still consuming it. Mixed versions also do not talk: a 4.0 machine sending to a 3.x one gets "no answer" (3.x has no fleet listener), and a 3.x machine's --machine dispatch waits on a queue no 4.0 machine reads, until its timeout.

The sender checks the receiver too. Every request that carries the fleet token (a --machine dispatch, machine status/resources <id>, machine list --deep, and the daemon's peer mission-graph proxy) first resolves the roster address, asks Tailscale which node is there, and refuses unless it is a tailnet node; the first contact pins that node in the roster (as machine add does), every later request checks it is the same node, and the connection goes to that verified address. Only this machine's own daemon and loopback (same-host) entries skip the check, and those never receive the token: whatever listens on a loopback port gets nothing (your own daemon does not ask for it from this machine), so reaching a peer through an ssh tunnel with the token is not supported. Anything a peer sends back has its control characters, bidirectional overrides and zero-width characters removed before it is printed; a field shown in a table or on one line also loses its newlines and tabs and is cut to its column, so a peer cannot print a line of its own.

The listener protects itself too: at most 32 connections at once and 3 per peer address, 3 seconds to send the request headers, at most 4 requests in flight per trusted machine once it is identified, and refusals logged at most 5 per peer per minute (the rest are counted and reported each minute). A node that routes a subnet can spread connections over many addresses, so the per-address cap bounds a single address, not a single node; the per-machine cap applies only after the token and identity checks. A job from another machine is never attributed to one of this machine's missions (its phase id is dropped).

Fail closed. If Tailscale cannot say who is calling (it is stopped, logged out, or the connection came over the office LAN rather than the tailnet), the request is refused. A hardware id is never used to decide: every flow record carries one, so any machine could repeat it.

Register each machine in the others' rosters: a sender finds a worker's address there (the host of the roster address, on fleet.listener.port), and machine list shows it. Use each machine's tailnet DNS name, not its IP — a peer that later sits behind Tailscale Serve routes by Host header, so a bare IP gets Tailscale's own 404 even though the daemon is healthy:

# On the hub:
darkmux machine add mini-1 --address <peer-tailnet-dns-name>:8765
# On the peer:
darkmux machine add <hub-id> --address <hub-tailnet-dns-name>:8765
darkmux machine list
The DNS name above resolves — but only reaches the daemon if something is actually listening on that machine's tailnet interface at :8765. The quick-path daemon binds 127.0.0.1:8765 only; the Tailscale Serve recipe proxies a browser to it on :443, a different port, and doesn't expose :8765 on the tailnet. For machine list --deep / machine status to reach a peer at its default port, that peer needs a genuinely non-loopback daemon: darkmux serve --bind <its-tailnet-ip> plus DARKMUX_SERVE_TOKEN set to the fleet's shared token (a non-loopback bind refuses to start without one, and machine list --deep already sends it). Keep the roster entry pointed at the DNS name regardless — it's still what survives Tailscale Serve ever landing in front of that daemon later.

The big picture: how the fleet coordinates

Three layers, so the commands above make sense:

The darkmux console showing flow status: the sink composition, active kinds, and per-day record counts.
darkmux flow status in the console. The sink composition is what makes a fleet a fleet — every machine writing into the same stream.
darkmux machine list --deep
MACHINE        ADDRESS                PROBE      AI-HEADROOM OS              VERSION  MODELS
laptop         laptop:8765            ✓ 23ms     78 GB       macos aarch64   1.17.0   darkmux:qwen3.6-35b-a3b-mlx
studio         studio:8765            ✓ 45ms     12 GB       macos aarch64   1.17.0   darkmux:qwen3-4b-instruct

AI-HEADROOM is the RAM the peer reports as available for model residency (ram_free_for_ai_bytes) — not the machine’s total free RAM, which is larger. Bounded at ~1s per peer on --deep; degraded peers render with specs? rather than failing the whole command. (Only machines running a daemon, meaning Part 2 workers or the hub, appear here; Part 1 observability peers show their records in the viewer but not in this live probe.)

Scope: single operator, multiple machines.

Work submission checks two things itself: the fleet token, and the sending node as Tailscale reports it against the worker's allow-list. Everything else leans on your tailnet as the boundary: the fleet's Redis record stream carries no auth beyond what your tailnet and the Redis bind provide, and provenance fields in records are operator-asserted. This works because everyone on the substrate is you. Multi-tenant deployment is explicitly out of scope; see DESIGN.md.

When things go wrong

The substrate degrades gracefully. Knowing the modes saves you confused debugging.

Hub drops off the network

The most common failure. Peers' flow writes fall back to local-file-only (records still land on disk per-machine; the Redis sink errors are logged + skipped). Work submission does not go through the hub, so a dispatch between two other machines keeps working. darkmux doctor reports the Redis sink unreachable. An open viewer tab's live tail retries for a bounded budget (~5s), emits a synthetic stream.error, and closes cleanly rather than spinning forever. Recovery: bring the hub back; the substrate self-heals on the next operation, no manual intervention.

"My dispatch to another machine was refused"

The refusal names the reason, and each has one fix. The fleet token is missing or does not match: both machines need the same serve token. does not accept work from <node>: on the worker, darkmux machine trust <name> --profiles …. not in the allow-list scope: add the profile to that entry, or send --profile with one in scope. cannot tell which machine sent this request: Tailscale is not answering on the worker. did not come from a node on the tailscale network: the sender reached the worker some other way (the LAN); use the tailnet DNS name in the roster. this is <x>, not <y>: the sender's roster points at the wrong host. A routed dispatch also emits a dispatch route flow record on the sender (target_machine, decision).

No answer at all

"no answer from http://…:8766" means nothing is listening there: the worker's listener is off, its daemon is not running, or the two machines use different fleet.listener.port values. darkmux doctor on the worker has a fleet listener row.

Two operators on the same machine

Out of scope. machine_id is per-machine, not per-user. If you need per-user provenance on a shared Mac, darkmux has outgrown its target: fork it.