03 · measure

The lab harness

A measurement harness, not a config tool dressed up as one. Every darkmux lab run writes a manifest, a trajectory, and per-turn timing under .darkmux/lab/<id>/, so wall-clock variance, compaction events, and per-profile behavior are inspectable after the fact, not in your head.

The point of the lab

Local AI claims are usually unfalsifiable. "This profile is faster". Measured how? On which workload? With which compaction settings? With which loaded state at start?

The lab harness exists to make those questions answerable on your own hardware:

Prerequisites for lab

darkmux lab run uses the same internal runtime as darkmux dispatch, in the same sandbox (runtime.sandbox): macOS’s own Seatbelt by default on a Mac, Docker on Linux or when set to docker. Under Docker, workloads dispatch through a per-invocation darkmux-runtime container, and darkmux pulls the image built for your version on first use; to build one yourself instead, run docker build --build-arg DARKMUX_VERSION=<version> -f runtime/Dockerfile -t darkmux-runtime:latest . from the darkmux repo root at that version. An image built for another version, or without the build-arg, is not used.

Under runtime.sandbox docker without Docker, the profile / machine / doctor verbs still work; you just can't run lab dispatches until Docker runs, or until you choose a sandbox this machine can run (darkmux doctor’s sandbox row says which).

The smoke workload

quick-q is a single-turn smoke prompt against the active profile. It's the "is this thing wired up at all" test: should complete in ~6–10 seconds if a model is loaded.

darkmux lab run quick-q
# …
# Captured run quick-q-deep-1778302418-1 in .darkmux/lab/quick-q-deep-1778302418-1/

Each run prints its id; that's your handle for inspect/compare later.

What the run wrote to disk

ls .darkmux/lab/quick-q-deep-1778302418-1/
# manifest.json     trajectory.jsonl   verify.json

Cross-layer telemetry (always-on)

Cross-layer telemetry is captured automatically (#557): no flag, no sidecar file. The internal runtime and dispatch emit it as category=telemetry flow records on the flow stream (sources: lms, process, detector, runtime, context, compaction), so you can see what LMStudio had loaded, where the runtime process sat across the run, detector signals, and compaction events. That's useful for diagnosing where wall-clock latency is concentrated.

View it in the daemon's observability viewer: run darkmux serve and open http://localhost:8765/. The viewer reads live flow records straight from the daemon. A demo instance lives at darkmux.com/demo.

The long-agentic workload: pepper-grinder

pepper-grinder is the workload darkmux's own development runs on: a real multi-turn coding task. The model reads a refresh-token rotation module and its tests, decides what isn't covered, writes those tests, runs the suite, reads the failures and fixes them until it goes green. It exercises tool calls across several files, iteration against a real signal, and a test suite as the ground truth.

The fixture is open source (MIT) and has no dependencies, so the suite runs in well under a second. Clone it once and register it:

git clone https://github.com/kstrat2001/pepper-grinder
darkmux lab fixture register ./pepper-grinder
darkmux lab run pepper-grinder

The workload and its grader are not part of darkmux: they ship in the fixture's own eval/ folder, and darkmux loads them from the registered fixture. darkmux processes an eval; the eval's repo defines it. The model's sandbox never holds that folder.

The untouched suite already passes (14 tests), so a green suite on its own is a weak bar. The workload declares a grader (verify.grade), and a run passes only when every check does. Each check quotes the prompt sentence it grades, and the grader refuses to run one the prompt does not say word for word:

The grader's suite runs execute where the verify command does: in a container with only a copy of the run's sandbox mounted, no network and no capabilities, unless runtime.tests_on_host puts them on this machine as you (see "Where the verify command runs" below).

The checks and their evidence are recorded under verify.grade in the run's manifest. darkmux lab grade <run>... grades kept runs again without re-running them, against the fixture content each run started from; --record writes the result into the run's manifest.

A workload may also name what it grades (verify.grade.criteria, #3386): each criterion has an id (a slug), a name, a one-line description and a source, a file in the eval's repo and an anchor, text that occurs there exactly once. A criterion is graded by the planted bugs that name it ("criterion": "<id>" on a mutant: met when every one turns the suite red) and by the tests it lists in tests (met when every one passes in the grader's run of the suite). The grader records each one met or missed under verify.grade.criteria, lab submit shares them (bundle format 6), and the evals site shows which a run missed, each linked to its anchor at the eval's commit. A registry the grader cannot grade (an undeclared or ungraded criterion, an id used twice, an anchor not in its file exactly once) is refused before any model loads, naming every problem, and again when a run is graded. To add criteria to runs shared before the eval declared any, grade each kept run again with darkmux lab grade <run> --record and share it again.

Inspect a run

darkmux run inspect quick-q-deep-1778302418-1

Shows: total wall-clock, turn count, compaction events, mode (fast vs slow), notes from the trajectory.

The mode classification reflects the bimodal wall-clock distribution observed in empirical testing: wall-clock per turn splits into two clusters based on whether the prompt happened to hit a state that triggers heavy compaction or not. Inspect tells you which cluster a given run landed in. The output is a small report, best seen by running it on your own machine after a quick-q or pepper-grinder run. The methodology behind this is documented in Part 2 of the lab series if you want the full empirical grounding.

Characterize your hardware

darkmux lab characterize is the one-command "QA my Mac": it dispatches a representative smoke workload, captures the run, and emits a verdict.

darkmux lab characterize

Output is a structured JSON verdict plus a human-readable summary. Used to:

Tune: detect bimodal variance

Single runs lie about local-AI behavior. Wall-clock varies enough that one run isn't a reliable signal. lab tune dispatches N runs against the same workload and clusters the results, surfacing the bimodal "fast" vs "slow" mode shape if it's there.

darkmux lab tune pepper-grinder --repeat 6

Output: a per-run table + a cluster verdict ("looks bimodal at fast=<mean>s, slow=<mean>s" or "single-mode").

Use this after changing a profile setting (context length, compaction mode, compactor model) to see whether the change actually shifted the distribution rather than just luck on a single run.

Compare runs

darkmux run compare A B diffs two runs, typically a "before tuning" baseline vs "after" verification.

darkmux run compare quick-q-deep-1778302418-1 quick-q-deep-1778466601-1

Reports: wall-clock delta, compaction-event delta, mode classification per side, any structural differences in the trajectory.

The discipline behind it: baseline → single-variable change → re-measure → compare → record in notebook. Each step has a darkmux primitive. Don't skip the baseline. Don't change two variables at once. Without this discipline, the comparison is uninterpretable.

Derived metrics: run stats

darkmux run stats <run> derives the numbers a run's raw artifacts don't hand you directly — tok/s, active time (wall minus rest), energy per token, which gate cut which turn — and runs a set of reconciliation checks before printing any of it. A figure that fails a check still prints (dropping it would be choosing the answer), but the run's checks are listed underneath, unconditionally, so you never copy a number without the reason it may not be quotable.

darkmux run stats pepper-grinder-deep-1790239929-1

One run prints wall/rest/active time, turns, throughput, both detection gates (the streaming gate and the per-call checkpoint gate — they judge different things and are never collapsed into one line), and host power/energy when telemetry covers the run. Pass --json for the full RunStats record instead of the text.

Comparing a set of runs

Name more than one run, or add --baseline, to get the set view: a table with one row per run, ranges (median with min–max, never a bare mean) across the set, and cost per successful run — every run's cost divided by only the runs that passed verify, because a failed run still consumed the GPU.

darkmux run stats candidate-1 candidate-2 candidate-3 --baseline base-1 base-2 base-3

A comparison prints both arms' figures side by side with the ratio that moved between them, plus a "read before quoting" block for anything the set can't vouch for: a mixed set of models, a run whose pass came from a runtime error, a run listed in both arms (its ratio is 1.00x by construction, not because nothing changed), or a withheld cost figure with the reason.

Run flags

Each run's table row ends with flags — short codes for a failed check or a notable condition, defined in stats_set::flags(). A single-run view (darkmux run stats <run>, no --json) spells the same conditions out as full sentences under "not reconciled, so do not quote these figures without saying so".

List your runs

darkmux run list --kind lab --limit 10        # most recent 10
darkmux run list --kind lab --limit 50

Each row shows the run's kind, status, start time, duration, token count, and id. Quick way to spot which run id to inspect or compare. Drop --kind lab to see mission and dispatch runs in the same list; darkmux run inspect, stats and compare read lab runs only.

Share runs on darkmux evals

darkmux evals collects lab results for public fixtures on Apple Silicon. Sign in there, make an upload token on your account page, and store it once:

darkmux lab submit login                      # paste the token; it is checked with the site
darkmux lab submit <run> [<run>...]           # runs of one fixture checkout
darkmux lab submit <run> --dry-run            # write the bundle, send nothing

The fixture must be an eval the site has approved (propose one there). submit sends only what it can vouch for as of the run: your chip, RAM and OS, the engine, runtime and quantization, a hash of the model's weights, which darkmux build ran it, the verdict and the measurements. A fact that may have changed since the run is sent as "not reported", and the command says why. It never sends a path, a machine name or your prompts. The token lives in the login keychain on macOS and in a file only you can read on Omarchy; darkmux lab submit logout forgets it. A token is kept for one site and sent only to it: login --site, logout --site and submit --site each name the site (evals.darkmux.com by default), and a site with no token of its own is refused, never sent another's.

A run on an endpoint says which machine served it, so its numbers are read against the right hardware. A model LM Studio loaded here is this machine's. A host on darkmux's built-in provider registry (Anthropic, OpenAI, Azure OpenAI and Azure AI Foundry, the Gemini API, Google Vertex AI, Amazon Bedrock, OpenRouter, xAI) is the provider's: the run's hardware is unknown and its engine is the provider's name. Any other url endpoint declares it in profiles.json, and darkmux doctor warns about one that does not:

"endpoints": { "flint": { "url": "http://studio.local:8080/v1", "hardware": "unknown" } }

"submitting-machine" means this machine serves it; "unknown" means another machine does. A run records the value when it starts, so editing the endpoint later does not change it. For a run recorded before darkmux kept it, or one that recorded the wrong value, pass --hardware submitting-machine or --hardware unknown. It never overrides an LM Studio model or a provider's host, it never turns a run recorded as unknown into the submitting machine's, and it applies to every run of the submit, so submit runs that need it apart from those that don't. submit refuses an endpoint run with nothing to go on, and every dispatch, mission launch and lab run refuses an endpoint whose host darkmux cannot read unambiguously (a percent-encoded host, a backslash, whitespace, or a non-ASCII character other than IDNA's full stops), or one that sets hardware where darkmux already knows it. On macOS, a claim the run's own GPU reading disagrees with gets a note for you to check; the reading is the whole machine's, so a hosted run that overlapped a local one can trip it. Your machine's chip, RAM and OS are still sent, and the site decides what to show. The bundle never carries the endpoint's URL or host. A bundle with a run whose hardware is unknown is format 7, which the site reads once it supports it; until then it refuses the upload. Each run of it also names the model the endpoint said answered (so an alias such as a model family name resolves to the exact version) and the reasoning_effort the dispatch actually sent (only a call to a role with no tools sends one; a container's runtime never does), or null for either when there is none to name.

The whole path, from a fresh Mac to a shared run, with a prompt that hands it to an AI agent: Run an eval and share it.

Reproducible sandboxes: fixtures

A single number isn't a measurement; a number you can reproduce is. The fixture system makes every coding-task run start from a known state and end with a verifiable one, so "this profile is faster" becomes a claim you can re-run on demand.

Per-run copy-on-write isolation

Every darkmux lab run works in its own sandbox, a copy-on-write clone of the source, not the source itself. The source fixture directory is never touched; cross-run contamination is eliminated by construction. The clone is near-instant where the filesystem supports it (clonefile on APFS, --reflink on btrfs/xfs/zfs) and falls back to a deep copy everywhere else.

~/.darkmux/lab/<run-id>/
  sandbox/          # this run's isolated COW clone, where the model works
  manifest.json     # now carries baseline_hash + final_hash (schema_version 6 and later)
  trajectory.jsonl  # per-turn detail

The manifest gained two content hashes (BLAKE3, formatted blake3:<hex>):

When a fixture declares baseline.test_count (below), manifest.verify carries a work_gate evidence block on top of the plain {passed, details} shape: tests_total, tests_passed, tests_added, tests_failed, tests_skipped, tests_todo, sandbox_changed, coverage_pct / coverage_min_pct, and command_tampered. For a write-the-tests workload, "the suite still passes" is close to meaningless on its own — a dispatch that made zero edits passes it too. The gate (schema_version 6) instead requires the sandbox to have actually changed, the suite to now have MORE PASSING tests than the fixture's declared baseline (not merely more tests — a model that neuters its own failing test with .skip/.todo doesn't count), and the verify command itself to be unaltered from the fixture's — before it will report passed: true.

Where the verify command runs. The verify command runs the tests the model wrote, so it runs in a container: the workload's image, else the darkmux runtime image, with only the run's sandbox mounted at /workspace, no network and no capabilities. ${SANDBOX_DIR} in the command names /workspace there. A fixture whose tests need a tool the runtime image lacks names its own image. To run it on this machine as you instead, for fixtures and models you trust, darkmux config set runtime.tests_on_host true; darkmux doctor warns while that is on.

The fixture manifest

A fixture is a self-contained directory with a .fixture.json manifest at its root. Only name is required:

{
  "schema_version": "1.2",          // 1.2: the eval owns its roles, under eval_dir (below)
  "name": "demo-tiny-py",          // registry key, no path separators
  "version": "1.0",
  "satisfies": "tiny-python-suite@1.0", // what abstract requirement this fills
  "language": "python",
  "verify_command": "python3 -m unittest discover -s tests",
  "baseline": { "test_count": 5 },   // declares the write-the-tests work gate active; must be a plain integer or doctor + the gate fail loudly
  "required_files": ["src/parser.py", "tests/test_parser.py", "eval/roles/coder.json", "eval/roles/coder.md"],
  "eval_dir": "eval",               // the eval's own folder; never copied into the model's sandbox
  "hash_exclude": ["__pycache__", ".pytest_cache"]
}

hash_include / hash_exclude layer on top of the defaults if you need to pull in out-of-tree files or ignore derived ones.

Register once, resolve by requirement

The fixture directory stays wherever it lives; the registry (~/.darkmux/lab-registry.json, or .darkmux/lab-registry.json project-scoped) is just a name → path lookup with integrity metadata. Registering computes and records the content hash so later drift is detectable.

darkmux lab fixture register ./my-fixture            # add by path (reads .fixture.json)
darkmux lab fixture register ./my-fixture --force    # replace an existing entry
darkmux lab fixture list                         # list registered fixtures
darkmux lab fixture unregister demo-tiny-py          # drop the entry (never deletes the dir)
darkmux lab doctor                           # offline integrity check (see below)

A workload opts into a fixture by declaring requires_fixture in its manifest, a <name>@<version> string. At run time the resolver finds a registered fixture whose satisfies matches and uses it as the source; with no requires_fixture set, the run falls back to {sandboxes}/<workload-id>/ as before. (This replaces the old DARKMUX_SANDBOX_<ID> env-var binding; the registry is now the only persistent fixture binding.)

An eval owns its roles

A lab run is a measurement, so its system prompt is built only from pinned inputs: the eval's own role files, pinned by the eval's commit, or darkmux's compiled-in role, pinned by the darkmux release. It never takes anything from ~/.darkmux/roles/, the preamble override, identity.md or the workspace notes, and it always uses darkmux's compiled-in preamble.

Where a workload bound to a fixture finds its role depends on the schema the fixture declares in .fixture.json, not on your darkmux version:

Names are matched exactly, on every OS: ROLES/ or Coder.md is not roles/coder.md, even on macOS, whose filesystem would open it. A schema_version darkmux cannot read (" 1.2", "v1.2", "1.2-rc1") is never read as the legacy rule: lab run and lab fixture register refuse it and lab doctor and darkmux doctor report it, each naming the file and the value. When a registered fixture's manifest does not load, for this or any reason, lab run of a workload its eval defines names that fixture and the problem instead of "not found". An eval_dir the checkout holds only under another case (Eval/ for eval) is refused. lab run reads the fixture's manifest once, before the first run, and every run's sandbox leaves out the eval_dir read then.

A workload bound to no fixture runs on darkmux's compiled-in role. lab run reads the role once, before the run starts, and every dispatch of that run uses those bytes, so nothing the run writes into its sandbox can change the role it sends.

What a run records. Each dispatch's dispatch.start record carries a role block: the role's id, its source (eval for a schema-1.2 eval's <eval_dir>/roles/, legacy_root for an older eval's root roles/, builtin for darkmux's compiled-in role), and the SHA-256 of the role's .json, its .md and the preamble (as the release ships it), next to the SHA-256 of the assembled prompt. darkmux 5.0 through 5.8 refuse a schema-1.2 fixture at lab run's preflight ("written by a newer darkmux ... Upgrade darkmux"), though lab fixture register accepts it, so a 1.2 eval never runs there on the home folder's roles. Model selection is not pinned yet: it still reads the skill manifests a role names from ~/.darkmux/skills/.

darkmux lab doctor

A cheap, offline check that catches a broken fixture before you waste a dispatch on it. For every registered fixture it verifies the path still exists, the manifest still loads, a declared baseline.test_count is a plain non-negative integer (a mistyped one fails the write-the-tests work gate closed on every run until it's fixed), the required_files are present, and the content hash + manifest version haven't drifted since registration. If there's no registry at all, it points you at scripts/lab-init.sh.

Bootstrap the built-ins

A built-in demo-tiny-py fixture (a trivial Python module + a 5-test suite) ships in the repo under templates/builtin/lab-fixtures/. Populate your registry with it (and any future built-ins) via the standalone init script (idempotent; safe to re-run after a git pull):

scripts/lab-init.sh          # register all built-ins
scripts/lab-init.sh --dry    # print what would be registered; no writes
scripts/lab-init.sh --force  # re-register, accepting upstream drift

It's a plain script, not a CLI verb: run it once, fork it, or skip it entirely. The discoverability path lives in darkmux lab doctor's "no registry" hint.

Adding a workload

Built-in workloads live in templates/builtin/workloads/*.json and are embedded into the binary at compile time via include_str!. They work from any directory without the source tree.

To add your own:

  1. Drop a JSON manifest at ~/.darkmux/workloads/<id>.json (user-local), picked up automatically on next lab run.
  2. For deeper extension (new workload kinds, not just new prompts), implement the WorkloadProvider trait in src/workloads/types.rs and register it in src/workloads/registry.rs::register_builtins(). Requires a code change + reinstall.

Two provider kinds ship out of the box: prompt (single prompt → response) and coding-task (sandbox + verify-command pattern).

Notebook entries from runs

darkmux does not write the prose. The bundled darkmux-lab-notebook skill (installed by darkmux init) tells your orchestrator how to draft a lab-style entry from a run's derived numbers, observation first, with the verify outcome stated exactly as recorded:

darkmux run stats quick-q-deep-1778302418-1 --json

The entry is written wherever your own instructions say your notebook lives; the skill asks when they say nothing. Set a distinct DARKMUX_MACHINE_ID per machine if you collate entries across several, and the header records which machine each came from; darkmux doctor --verbose shows the resolved id. run stats takes a run id (resolved under ~/.darkmux/lab/, or $DARKMUX_LAB_DIR / dirs.lab when set) or the run directory's path.