A measurement harness, not a config tool dressed up as one. Every darkmux lab run writes a manifest, a trajectory, and per-turn timing under .darkmux/runs/<id>/, so wall-clock variance, compaction events, and per-profile behavior are inspectable after the fact, not in your head.
Local AI claims are usually unfalsifiable. "This profile is faster". Measured how? On which workload? With which compaction settings? With which loaded state at start?
The lab harness exists to make those questions answerable on your own hardware:
templates/builtin/workloads/ (and your own under ~/.darkmux/workloads/)..darkmux/runs/<id>/. No "I think I saw 30 seconds" hand-waving. The numbers are on disk, in JSON, comparable.lab run compare A B diffs two runs by wall-clock, compaction events, and per-turn behavior.WorkloadProvider trait in src/workloads/types.rs.darkmux lab run uses the same internal Docker-bounded runtime as darkmux dispatch. Workloads dispatch through a per-invocation darkmux-runtime container. darkmux pulls the image built for your version on first use; to build one yourself instead, run docker build --build-arg DARKMUX_VERSION=<version> -t darkmux-runtime:latest runtime/ from the darkmux repo root at that version. An image built for another version, or without the build-arg, is not used.
If you don't have Docker, the profile / machine / doctor verbs still work. You just can't run lab dispatches.
quick-q is a single-turn smoke prompt against the active profile. It's the "is this thing wired up at all" test: should complete in ~6–10 seconds if a model is loaded.
darkmux lab run quick-q
# …
# Captured run quick-q-deep-1778302418-1 in .darkmux/runs/quick-q-deep-1778302418-1/Each run prints its id; that's your handle for inspect/compare later.
ls .darkmux/runs/quick-q-deep-1778302418-1/
# manifest.json trajectory.jsonl verify.jsonmanifest.json: workload id, profile, loaded model state, hardware fingerprint, start/end time.trajectory.jsonl: per-turn detail, including prompt, response, tool calls, timing, and compaction events.verify.json: verify-command outcome. Only coding-task workloads define a verify command; a prompt workload like quick-q has none, so its verify is just "a response came back" and reports pass without running any check. Don't read a prompt workload's "pass" as a test result: there's nothing to fail. The real pass/fail signal lives in coding-task verify commands.Cross-layer telemetry is captured automatically (#557): no flag, no sidecar file. The internal runtime and dispatch emit it as category=telemetry flow records on the flow stream (sources: lms, process, detector, runtime, context, compaction), so you can see what LMStudio had loaded, where the runtime process sat across the run, detector signals, and compaction events. That's useful for diagnosing where wall-clock latency is concentrated.
View it in the daemon's observability viewer: run darkmux serve and open http://localhost:8765/. The viewer reads live flow records straight from the daemon. A demo instance lives at darkmux.com/demo.
pepper-grinder is the workload darkmux's own development runs on: a real multi-turn coding task. The model reads a refresh-token rotation module and its tests, decides what isn't covered, writes those tests, runs the suite, reads the failures and fixes them until it goes green. It exercises tool calls across several files, iteration against a real signal, and a test suite as the ground truth.
The fixture is open source (MIT) and has no dependencies, so the suite runs in well under a second. Clone it once and register it:
git clone https://github.com/kstrat2001/pepper-grinder
darkmux lab fixture register ./pepper-grinder
darkmux lab run pepper-grinderThe untouched suite already passes (14 tests), so a green result on its own is a weak bar. The useful signal is how many tests the run added, which lab run stats reports.
darkmux lab run inspect quick-q-deep-1778302418-1Shows: total wall-clock, turn count, compaction events, mode (fast vs slow), notes from the trajectory.
The mode classification reflects the bimodal wall-clock distribution observed in empirical testing: wall-clock per turn splits into two clusters based on whether the prompt happened to hit a state that triggers heavy compaction or not. Inspect tells you which cluster a given run landed in. The output is a small report, best seen by running it on your own machine after a quick-q or pepper-grinder run. The methodology behind this is documented in Part 2 of the lab series if you want the full empirical grounding.
darkmux lab characterize is the one-command "QA my Mac": it dispatches a representative smoke workload, captures the run, and emits a verdict.
darkmux lab characterizeOutput is a structured JSON verdict plus a human-readable summary. Used to:
Single runs lie about local-AI behavior. Wall-clock varies enough that one run isn't a reliable signal. lab tune dispatches N runs against the same workload and clusters the results, surfacing the bimodal "fast" vs "slow" mode shape if it's there.
darkmux lab tune pepper-grinder --runs 6Output: a per-run table + a cluster verdict ("looks bimodal at fast=<mean>s, slow=<mean>s" or "single-mode").
Use this after changing a profile setting (context length, compaction mode, compactor model) to see whether the change actually shifted the distribution rather than just luck on a single run.
darkmux lab run compare A B diffs two runs, typically a "before tuning" baseline vs "after" verification.
darkmux lab run compare quick-q-deep-1778302418-1 quick-q-deep-1778466601-1Reports: wall-clock delta, compaction-event delta, mode classification per side, any structural differences in the trajectory.
The discipline behind it: baseline → single-variable change → re-measure → compare → record in notebook. Each step has a darkmux primitive. Don't skip the baseline. Don't change two variables at once. Without this discipline, the comparison is uninterpretable.
lab run statsdarkmux lab run stats <run> derives the numbers a run's raw artifacts don't hand you directly — tok/s, active time (wall minus rest), energy per token, which gate cut which turn — and runs a set of reconciliation checks before printing any of it. A figure that fails a check still prints (dropping it would be choosing the answer), but the run's checks are listed underneath, unconditionally, so you never copy a number without the reason it may not be quotable.
darkmux lab run stats pepper-grinder-deep-1790239929-1One run prints wall/rest/active time, turns, throughput, both detection gates (the streaming gate and the per-call checkpoint gate — they judge different things and are never collapsed into one line), and host power/energy when telemetry covers the run. Pass --json for the full RunStats record instead of the text.
Name more than one run, or add --baseline, to get the set view: a table with one row per run, ranges (median with min–max, never a bare mean) across the set, and cost per successful run — every run's cost divided by only the runs that passed verify, because a failed run still consumed the GPU.
darkmux lab run stats candidate-1 candidate-2 candidate-3 --baseline base-1 base-2 base-3A comparison prints both arms' figures side by side with the ratio that moved between them, plus a "read before quoting" block for anything the set can't vouch for: a mixed set of models, a run whose pass came from a runtime error, a run listed in both arms (its ratio is 1.00x by construction, not because nothing changed), or a withheld cost figure with the reason.
Each run's table row ends with flags — short codes for a failed check or a notable condition, defined in stats_set::flags(). A single-run view (darkmux lab run stats <run>, no --json) spells the same conditions out as full sentences under "not reconciled, so do not quote these figures without saying so".
STALE-METRICS: this run's metrics.json doesn't belong to it — its own clock lands more than a few seconds before the run's own identity, or more than ten minutes after it, or it claims less wall time than the trajectory spent generating (past a small jitter tolerance either way — two different clocks, so a real run can be a few ms over). On its OWN row, wall/rest/active are still printed (dropping a row is choosing the answer), just flagged and caveated. In a SET, this run is excluded outright from the wall/rest/active/energy ranges and from cost per success, rather than folded into the median — a wrong number corrupts a comparison the same way a missing one does.OVERLAP (set view only): this run's window overlaps another (non-stale) run's in the same set. Host telemetry is matched by time window, so two overlapping runs would each claim the whole host's power; GPU-busy time and package energy are excluded from the set's ranges and from cost per success for both, rather than apportioned. A STALE-METRICS run's window is never trusted enough to accuse another run of overlapping it, and is itself excluded from overlap detection.RUNTIME-ERROR / ESCALATED: the runtime's own terminal result. A run can pass verify without doing the task (an untouched-tree fixture is green from second zero), so this is flagged, never reclassified.TOKENS / COUNTS: per-turn tokens don't sum to the recorded total, or metrics.json's turns/rest disagree with the trajectory's own count.PARSE / VERDICT: the trajectory is missing an event this reading keys on, or the runtime's recorded degeneracy verdicts don't match what the tail-ratio threshold would derive — a sign the constant here no longer matches the build that produced the run.UNBILLED / STREAMS: a gate cut a stream before its usage frame arrived (tok/s then covers only the billed subset), or a stream started but never ended.REST: rest exceeds wall — two counters measuring different windows.NO-TELEM / TELEM-GAP / NO-FLOW: no host telemetry landed in the run's window, telemetry doesn't cover the whole run (busy time and energy are withheld rather than extrapolated from a gap), or no flow records at all — the run's dispatch bounds are unknown.POLICY: the detection policy the checkpoint records say ran differs from the one the host resolved.CHARS: a turn's reasoning-chars-per-token ratio is outside the plausible band — one of the two counters may not measure what its name says.RATCHET / THROTTLE: the thermal ratchet doubled the rest delay mid-run, or a busy sample recorded a CPU speed limit below 100%.UNGATED: this run's verify: pass predates the write-the-tests work gate for its fixture — the old "verify command exited 0" signal, which a dispatch that made zero edits also satisfies. Never reinterpreted after the fact; flagged so a mixed series isn't silently compared across two different definitions of success.darkmux lab run list --limit 10 # most recent 10
darkmux lab run list --limit 50Each row shows: run id, workload, profile, wall-clock, mode, verify outcome. Quick way to spot which run id to inspect or compare.
A single number isn't a measurement; a number you can reproduce is. The fixture system makes every coding-task run start from a known state and end with a verifiable one, so "this profile is faster" becomes a claim you can re-run on demand.
Every darkmux lab run works in its own sandbox, a copy-on-write clone of the source, not the source itself. The source fixture directory is never touched; cross-run contamination is eliminated by construction. The clone is near-instant where the filesystem supports it (clonefile on APFS, --reflink on btrfs/xfs/zfs) and falls back to a deep copy everywhere else.
~/.darkmux/runs/<run-id>/
sandbox/ # this run's isolated COW clone, where the model works
manifest.json # now carries baseline_hash + final_hash (schema_version 6)
trajectory.jsonl # per-turn detailThe manifest gained two content hashes (BLAKE3, formatted blake3:<hex>):
fixture.baseline_hash: the source state before the run, captured at clone time. Proves two runs started from the same place.final_hash: the sandbox state after dispatch, excluding derived dirs (.git, node_modules, target, __pycache__, .coverage, .darkmux-runtime). Two runs with the same final_hash left bitwise-identical output, the strongest reproducibility signal the lab emits.When a fixture declares baseline.test_count (below), manifest.verify carries a work_gate evidence block on top of the plain {passed, details} shape: tests_total, tests_passed, tests_added, tests_failed, tests_skipped, tests_todo, sandbox_changed, coverage_pct / coverage_min_pct, and command_tampered. For a write-the-tests workload, "the suite still passes" is close to meaningless on its own — a dispatch that made zero edits passes it too. The gate (schema_version 6) instead requires the sandbox to have actually changed, the suite to now have MORE PASSING tests than the fixture's declared baseline (not merely more tests — a model that neuters its own failing test with .skip/.todo doesn't count), and the verify command itself to be unaltered from the fixture's — before it will report passed: true.
A fixture is a self-contained directory with a .fixture.json manifest at its root. Only name is required:
{
"name": "demo-tiny-py", // registry key, no path separators
"version": "1.0",
"satisfies": "tiny-python-suite@1.0", // what abstract requirement this fills
"language": "python",
"verify_command": "python3 -m unittest discover -s tests",
"baseline": { "test_count": 5 }, // declares the write-the-tests work gate active; must be a plain integer or doctor + the gate fail loudly
"required_files": ["src/parser.py", "tests/test_parser.py"],
"hash_exclude": ["__pycache__", ".pytest_cache"]
}hash_include / hash_exclude layer on top of the defaults if you need to pull in out-of-tree files or ignore derived ones.
The fixture directory stays wherever it lives; the registry (~/.darkmux/lab-registry.json, or .darkmux/lab-registry.json project-scoped) is just a name → path lookup with integrity metadata. Registering computes and records the content hash so later drift is detectable.
darkmux lab fixture register ./my-fixture # add by path (reads .fixture.json)
darkmux lab fixture register ./my-fixture --force # replace an existing entry
darkmux lab fixture list # list registered fixtures
darkmux lab fixture unregister demo-tiny-py # drop the entry (never deletes the dir)
darkmux lab doctor # offline integrity check (see below)A workload opts into a fixture by declaring requires_fixture in its manifest, a <name>@<version> string. At run time the resolver finds a registered fixture whose satisfies matches and uses it as the source; with no requires_fixture set, the run falls back to {sandboxes}/<workload-id>/ as before. (This replaces the old DARKMUX_SANDBOX_<ID> env-var binding; the registry is now the only persistent fixture binding.)
darkmux lab doctorA cheap, offline check that catches a broken fixture before you waste a dispatch on it. For every registered fixture it verifies the path still exists, the manifest still loads, a declared baseline.test_count is a plain non-negative integer (a mistyped one fails the write-the-tests work gate closed on every run until it's fixed), the required_files are present, and the content hash + manifest version haven't drifted since registration. If there's no registry at all, it points you at scripts/lab-init.sh.
A built-in demo-tiny-py fixture (a trivial Python module + a 5-test suite) ships in the repo under templates/builtin/lab-fixtures/. Populate your registry with it (and any future built-ins) via the standalone init script (idempotent; safe to re-run after a git pull):
scripts/lab-init.sh # register all built-ins
scripts/lab-init.sh --dry # print what would be registered; no writes
scripts/lab-init.sh --force # re-register, accepting upstream driftIt's a plain script, not a CLI verb: run it once, fork it, or skip it entirely. The discoverability path lives in darkmux lab doctor's "no registry" hint.
Built-in workloads live in templates/builtin/workloads/*.json and are embedded into the binary at compile time via include_str!. They work from any directory without the source tree.
To add your own:
~/.darkmux/workloads/<id>.json (user-local), picked up automatically on next lab run.WorkloadProvider trait in src/workloads/types.rs and register it in src/workloads/registry.rs::register_builtins(). Requires a code change + reinstall.Two provider kinds ship out of the box: prompt (single prompt → response) and coding-task (sandbox + verify-command pattern).
darkmux does not write the prose. The bundled darkmux-lab-notebook skill (installed by darkmux init) tells your orchestrator how to draft a lab-style entry from a run's derived numbers, observation first, with the verify outcome stated exactly as recorded:
darkmux lab run stats quick-q-deep-1778302418-1 --jsonThe entry is written wherever your own instructions say your notebook lives; the skill asks when they say nothing. Set a distinct DARKMUX_MACHINE_ID per machine if you collate entries across several, and the header records which machine each came from; darkmux doctor --verbose shows the resolved id. lab run stats takes a run id (resolved under ~/.darkmux/runs/, or $DARKMUX_LAB_DIR / dirs.lab when set) or the run directory's path.