darkmux evals collects lab runs of public evals on Apple Silicon, so you can see how a model did on a Mac like yours before you download it. This page takes you from a fresh machine to a shared run: set up, run one approved eval, and send the result. Or hand the whole job to an AI agent with the prompt at the end.
A failed run is a result too. Share it as it is; don't re-run to get a pass.
You need LM Studio with the model you want to measure downloaded. Each lab run works in macOS’s own sandbox (Seatbelt), so there is no Docker to install; with runtime.sandbox docker it works in a small container instead, and Docker must be running. Then:
brew install kstrat2001/tap/darkmux
darkmux --version # a release, not a development build
darkmux init # writes your config and profiles
darkmux doctor
Getting started covers each of these in detail.
On Omarchy, darkmux uses neither Homebrew nor LM Studio, so the setup is longer. The Omarchy page covers three things:
Its endpoint config includes "hardware": "submitting-machine", which says this Mac serves the model. Without it, darkmux lab submit refuses the run and asks for --hardware.
Pick a model that fits your RAM with room to spare. darkmux doctor warns on its macOS-only checks there; that is expected.
Browse the approved evals on evals.darkmux.com (or read evals.darkmux.com/api/v1/evals, the same list as JSON). Each one names its repository and its approved commit. Clone it, check out that exact commit, and register the checkout:
git clone https://github.com/<owner>/<repo>
cd <repo>
git checkout <approved commit>
darkmux lab fixture register .
Leave the checkout unchanged. The eval's workloads are the files in its workloads/ folder, inside the eval folder its .fixture.json names (eval/ in the evals listed so far); the workload name is the file name without .json.
Pick the profile that names your model (darkmux profile list; edit ~/.darkmux/profiles.json to add one). On macOS, unload every model in LM Studio first, so the run has the memory and GPU to itself (darkmux loads its own copy under its own name). Then start it and leave the Mac to it:
darkmux lab run <workload> -p <profile>
darkmux run inspect <run id> # the verdict, time and turns
A coding eval can take a long time on a small model. The run prints its id when it finishes. The lab page says what the run records and how it is graded.
darkmux lab submit login # paste the token; darkmux checks it with the site
The token lives in the login keychain on macOS and in a file only you can read on Omarchy. Treat it like a password. darkmux lab submit logout forgets it.
A token is kept for the site that made it, and sent only there. To try a preview of the site, log in to it by name: darkmux lab submit login --site http://evals.localhost:3419 keeps that token beside your evals.darkmux.com one, and darkmux lab submit <run id> --site http://evals.localhost:3419 sends the preview's. A --site you have not logged in to is refused before anything is sent, naming the login to run; logout --site forgets that site's token only.
darkmux lab submit <run id> --dry-run # write the bundle, send nothing
darkmux lab submit <run id>
Look at the dry run first: it writes darkmux-evals-<run id>.json in the current folder, exactly what would be sent. If submit refuses a run, it says why; don't work around it.
Your chip, RAM and OS, the engine, runtime and quantization, a hash of the model's weights, the darkmux build that ran it, the eval and its commit, the verdict, and the measurements. A fact darkmux can't vouch for as of the run is sent as "not reported", and the command says why: re-download the model between the run and the upload, for example, and its quantization and weights hash go up as not reported. The bundle carries no file path, no machine or user name, and none of the conversation: submit refuses to build one that would. The site knows which account uploaded a run, because your token signs it, but it never shows who shared one.
A run waits for the maintainer's review before it is shown. Your runs are listed on Contribute › Runs, where you can see their status and archive one.
Each prompt below walks an AI agent through the whole job. It tells an agent that can run commands on your computer to run them one at a time and to ask before it installs, downloads, changes your darkmux config or uploads anything. An agent that can only chat is told to give you each command and wait for the output. Either way you choose the model, and you store the upload token yourself, so the agent never has to see it. If you don't have upload access yet, the prompt has you ask for it first, so it can come through while the run goes. The same prompts are on your Contribute page once you have upload access.
You are helping me run a darkmux eval on this computer and share the result on
darkmux evals (https://evals.darkmux.com). darkmux is a command-line tool for
running AI models locally. Its guide for this is
https://darkmux.com/guide/share-runs.html
How to work:
- If you can run commands on this computer, run them one step at a time and show
me what each one printed. If you can't, give me each command to run and wait
for me to paste back what it printed.
- Ask me before you install anything, download a model, change anything in
~/.darkmux, or upload anything.
- If darkmux refuses something, tell me what it said. Don't work around it.
Rules for a run that counts:
- This must be an Apple Silicon Mac on macOS. If it isn't, stop and tell me.
- Use the Homebrew release of darkmux (brew install kstrat2001/tap/darkmux),
never a build from source. darkmux --version must not say it is a
development build.
- I choose the model. If the one I pick looks too big for this Mac's memory,
say so, but don't choose a different one for me.
- One model at a time: before the run, unload every model in LM Studio, so the
run has the memory and GPU to itself (darkmux loads its own copy), and start
no other model work on this computer until the run finishes.
- Don't set the context length or the reply length lower than the eval asks for.
- Never edit the eval's files or the run's sandbox, and don't run it again to
get a pass. A failed run is a result, and it can be shared.
Steps:
1. Check darkmux --version, that LM Studio is installed, and that doctor's
sandbox row passes (Docker matters only if runtime.sandbox is docker). Run darkmux init if darkmux has no config yet, then darkmux doctor.
2. If I don't have upload access on darkmux evals yet, ask me to request it now
on the Tokens tab of https://evals.darkmux.com/account. The maintainer grants
it, and I get an email either way. The run can go ahead while I wait.
3. Show me the evals on https://evals.darkmux.com/evals (as data:
https://evals.darkmux.com/api/v1/evals). Ask me which eval to run and with
which model. Each eval names its GitHub repository and its approved commit.
4. Clone that repository, check out that exact commit, and leave the checkout
unchanged. Register it: darkmux lab fixture register <checkout folder>
5. Find or make a darkmux profile for the model I chose (darkmux profile list).
The workload names are the files in the workloads folder inside the eval's
folder (its .fixture.json names that folder, usually eval). Run:
darkmux lab run <workload> -p <profile>
It can take a long time. Wait for it to finish.
6. Show me the result: darkmux run inspect <run id>
7. Show me what would be sent: darkmux lab submit <run id> --dry-run
8. Ask me to make an upload token on https://evals.darkmux.com/account (Tokens
tab) and to run darkmux lab submit login myself, so you never need to
see the token.
9. When I say so, send it: darkmux lab submit <run id>
10. Tell me the run waits for the maintainer's review before it is shown, and
that I can see it at https://evals.darkmux.com/account?tab=runs
You are helping me run a darkmux eval on this computer and share the result on
darkmux evals (https://evals.darkmux.com). darkmux is a command-line tool for
running AI models locally. This computer runs Omarchy Linux on an Apple Silicon
Mac. Set darkmux up by following https://darkmux.com/guide/omarchy.html and
share the run by following https://darkmux.com/guide/share-runs.html
How to work:
- If you can run commands on this computer, run them one step at a time and show
me what each one printed. If you can't, give me each command to run and wait
for me to paste back what it printed.
- Ask me before you install anything, download a model, change anything in
~/.darkmux, or upload anything.
- Never run sudo yourself. When a step needs it, give me the command to run.
- If darkmux refuses something, tell me what it said. Don't work around it.
Rules for a run that counts:
- This must be Omarchy on an Apple Silicon Mac (Asahi Linux). If it isn't, stop
and tell me.
- There is no Homebrew here. Build darkmux from the latest release tag exactly
as the Omarchy guide's "Build and install" section says, so it builds as a
release. darkmux --version must name the release and must not say it is a
development build.
- The model runs on the local model server from the Omarchy guide, not on
LM Studio. I choose the model. If the one I pick looks too big for this Mac's
memory, say so, but don't choose a different one for me.
- One model at a time: nothing else may use the GPU while the run is going.
- Don't set the context length or the reply length lower than the eval asks for.
- Never edit the eval's files or the run's sandbox, and don't run it again to
get a pass. A failed run is a result, and it can be shared.
Steps:
1. Check darkmux --version, that Docker is running and I am in the docker
group, and that the model server answers. Fix anything missing by following
the Omarchy guide: the server must listen on the Docker bridge and the
profile must point at it, so the run's container can reach it. Then run
darkmux doctor. Warnings on checks that only apply to macOS are expected.
2. If I don't have upload access on darkmux evals yet, ask me to request it now
on the Tokens tab of https://evals.darkmux.com/account. The maintainer grants
it, and I get an email either way. The run can go ahead while I wait.
3. Show me the evals on https://evals.darkmux.com/evals (as data:
https://evals.darkmux.com/api/v1/evals). Ask me which eval to run and with
which model. Each eval names its GitHub repository and its approved commit.
4. Clone that repository, check out that exact commit, and leave the checkout
unchanged. Register it: darkmux lab fixture register <checkout folder>
5. Make sure a darkmux profile serves the model I chose (darkmux profile list).
The workload names are the files in the workloads folder inside the eval's
folder (its .fixture.json names that folder, usually eval). Run:
darkmux lab run <workload> -p <profile>
It can take a long time. Wait for it to finish.
6. Show me the result: darkmux run inspect <run id>
7. Show me what would be sent: darkmux lab submit <run id> --dry-run
8. Ask me to make an upload token on https://evals.darkmux.com/account (Tokens
tab) and to run darkmux lab submit login myself, so you never need to
see the token.
9. When I say so, send it: darkmux lab submit <run id>
10. Tell me the run waits for the maintainer's review before it is shown, and
that I can see it at https://evals.darkmux.com/account?tab=runs