12 · contribute

Run an eval and share it

darkmux evals collects lab runs of public evals on Apple Silicon, so you can see how a model did on a Mac like yours before you download it. This page takes you from a fresh machine to a shared run: set up, run one approved eval, and send the result. Or hand the whole job to an AI agent with the prompt at the end.

What counts

A failed run is a result too. Share it as it is; don't re-run to get a pass.

Set up on macOS

You need LM Studio with the model you want to measure downloaded. Each lab run works in macOS’s own sandbox (Seatbelt), so there is no Docker to install; with runtime.sandbox docker it works in a small container instead, and Docker must be running. Then:

brew install kstrat2001/tap/darkmux
darkmux --version     # a release, not a development build
darkmux init          # writes your config and profiles
darkmux doctor

Getting started covers each of these in detail.

Set up on Omarchy

On Omarchy, darkmux uses neither Homebrew nor LM Studio, so the setup is longer. The Omarchy page covers three things:

Its endpoint config includes "hardware": "submitting-machine", which says this Mac serves the model. Without it, darkmux lab submit refuses the run and asks for --hardware.

Pick a model that fits your RAM with room to spare. darkmux doctor warns on its macOS-only checks there; that is expected.

Pick an eval and check it out

Browse the approved evals on evals.darkmux.com (or read evals.darkmux.com/api/v1/evals, the same list as JSON). Each one names its repository and its approved commit. Clone it, check out that exact commit, and register the checkout:

git clone https://github.com/<owner>/<repo>
cd <repo>
git checkout <approved commit>
darkmux lab fixture register .

Leave the checkout unchanged. The eval's workloads are the files in its workloads/ folder, inside the eval folder its .fixture.json names (eval/ in the evals listed so far); the workload name is the file name without .json.

Run it

Pick the profile that names your model (darkmux profile list; edit ~/.darkmux/profiles.json to add one). On macOS, unload every model in LM Studio first, so the run has the memory and GPU to itself (darkmux loads its own copy under its own name). Then start it and leave the Mac to it:

darkmux lab run <workload> -p <profile>
darkmux run inspect <run id>     # the verdict, time and turns

A coding eval can take a long time on a small model. The run prints its id when it finishes. The lab page says what the run records and how it is graded.

Get upload access

  1. Sign in on evals.darkmux.com with a free darklyenergized.com account, and accept the current Terms and Privacy Policy.
  2. On the Tokens tab of Contribute, ask for upload access. The maintainer grants it, and you get an email either way.
  3. Once you have it, make an upload token on that tab and store it in darkmux:
darkmux lab submit login     # paste the token; darkmux checks it with the site

The token lives in the login keychain on macOS and in a file only you can read on Omarchy. Treat it like a password. darkmux lab submit logout forgets it.

A token is kept for the site that made it, and sent only there. To try a preview of the site, log in to it by name: darkmux lab submit login --site http://evals.localhost:3419 keeps that token beside your evals.darkmux.com one, and darkmux lab submit <run id> --site http://evals.localhost:3419 sends the preview's. A --site you have not logged in to is refused before anything is sent, naming the login to run; logout --site forgets that site's token only.

Share it

darkmux lab submit <run id> --dry-run    # write the bundle, send nothing
darkmux lab submit <run id>

Look at the dry run first: it writes darkmux-evals-<run id>.json in the current folder, exactly what would be sent. If submit refuses a run, it says why; don't work around it.

What is sent, and what never is

Your chip, RAM and OS, the engine, runtime and quantization, a hash of the model's weights, the darkmux build that ran it, the eval and its commit, the verdict, and the measurements. A fact darkmux can't vouch for as of the run is sent as "not reported", and the command says why: re-download the model between the run and the upload, for example, and its quantization and weights hash go up as not reported. The bundle carries no file path, no machine or user name, and none of the conversation: submit refuses to build one that would. The site knows which account uploaded a run, because your token signs it, but it never shows who shared one.

After you share

A run waits for the maintainer's review before it is shown. Your runs are listed on Contribute › Runs, where you can see their status and archive one.

Hand it to an AI agent

Each prompt below walks an AI agent through the whole job. It tells an agent that can run commands on your computer to run them one at a time and to ask before it installs, downloads, changes your darkmux config or uploads anything. An agent that can only chat is told to give you each command and wait for the output. Either way you choose the model, and you store the upload token yourself, so the agent never has to see it. If you don't have upload access yet, the prompt has you ask for it first, so it can come through while the run goes. The same prompts are on your Contribute page once you have upload access.

macOS

You are helping me run a darkmux eval on this computer and share the result on
darkmux evals (https://evals.darkmux.com). darkmux is a command-line tool for
running AI models locally. Its guide for this is
https://darkmux.com/guide/share-runs.html

How to work:
- If you can run commands on this computer, run them one step at a time and show
  me what each one printed. If you can't, give me each command to run and wait
  for me to paste back what it printed.
- Ask me before you install anything, download a model, change anything in
  ~/.darkmux, or upload anything.
- If darkmux refuses something, tell me what it said. Don't work around it.

Rules for a run that counts:
- This must be an Apple Silicon Mac on macOS. If it isn't, stop and tell me.
- Use the Homebrew release of darkmux (brew install kstrat2001/tap/darkmux),
  never a build from source. darkmux --version must not say it is a
  development build.
- I choose the model. If the one I pick looks too big for this Mac's memory,
  say so, but don't choose a different one for me.
- One model at a time: before the run, unload every model in LM Studio, so the
  run has the memory and GPU to itself (darkmux loads its own copy), and start
  no other model work on this computer until the run finishes.
- Don't set the context length or the reply length lower than the eval asks for.
- Never edit the eval's files or the run's sandbox, and don't run it again to
  get a pass. A failed run is a result, and it can be shared.

Steps:
1. Check darkmux --version, that LM Studio is installed, and that doctor's
   sandbox row passes (Docker matters only if runtime.sandbox is docker). Run darkmux init if darkmux has no config yet, then darkmux doctor.
2. If I don't have upload access on darkmux evals yet, ask me to request it now
   on the Tokens tab of https://evals.darkmux.com/account. The maintainer grants
   it, and I get an email either way. The run can go ahead while I wait.
3. Show me the evals on https://evals.darkmux.com/evals (as data:
   https://evals.darkmux.com/api/v1/evals). Ask me which eval to run and with
   which model. Each eval names its GitHub repository and its approved commit.
4. Clone that repository, check out that exact commit, and leave the checkout
   unchanged. Register it: darkmux lab fixture register <checkout folder>
5. Find or make a darkmux profile for the model I chose (darkmux profile list).
   The workload names are the files in the workloads folder inside the eval's
   folder (its .fixture.json names that folder, usually eval). Run:
   darkmux lab run <workload> -p <profile>
   It can take a long time. Wait for it to finish.
6. Show me the result: darkmux run inspect <run id>
7. Show me what would be sent: darkmux lab submit <run id> --dry-run
8. Ask me to make an upload token on https://evals.darkmux.com/account (Tokens
   tab) and to run darkmux lab submit login myself, so you never need to
   see the token.
9. When I say so, send it: darkmux lab submit <run id>
10. Tell me the run waits for the maintainer's review before it is shown, and
    that I can see it at https://evals.darkmux.com/account?tab=runs

Omarchy

You are helping me run a darkmux eval on this computer and share the result on
darkmux evals (https://evals.darkmux.com). darkmux is a command-line tool for
running AI models locally. This computer runs Omarchy Linux on an Apple Silicon
Mac. Set darkmux up by following https://darkmux.com/guide/omarchy.html and
share the run by following https://darkmux.com/guide/share-runs.html

How to work:
- If you can run commands on this computer, run them one step at a time and show
  me what each one printed. If you can't, give me each command to run and wait
  for me to paste back what it printed.
- Ask me before you install anything, download a model, change anything in
  ~/.darkmux, or upload anything.
- Never run sudo yourself. When a step needs it, give me the command to run.
- If darkmux refuses something, tell me what it said. Don't work around it.

Rules for a run that counts:
- This must be Omarchy on an Apple Silicon Mac (Asahi Linux). If it isn't, stop
  and tell me.
- There is no Homebrew here. Build darkmux from the latest release tag exactly
  as the Omarchy guide's "Build and install" section says, so it builds as a
  release. darkmux --version must name the release and must not say it is a
  development build.
- The model runs on the local model server from the Omarchy guide, not on
  LM Studio. I choose the model. If the one I pick looks too big for this Mac's
  memory, say so, but don't choose a different one for me.
- One model at a time: nothing else may use the GPU while the run is going.
- Don't set the context length or the reply length lower than the eval asks for.
- Never edit the eval's files or the run's sandbox, and don't run it again to
  get a pass. A failed run is a result, and it can be shared.

Steps:
1. Check darkmux --version, that Docker is running and I am in the docker
   group, and that the model server answers. Fix anything missing by following
   the Omarchy guide: the server must listen on the Docker bridge and the
   profile must point at it, so the run's container can reach it. Then run
   darkmux doctor. Warnings on checks that only apply to macOS are expected.
2. If I don't have upload access on darkmux evals yet, ask me to request it now
   on the Tokens tab of https://evals.darkmux.com/account. The maintainer grants
   it, and I get an email either way. The run can go ahead while I wait.
3. Show me the evals on https://evals.darkmux.com/evals (as data:
   https://evals.darkmux.com/api/v1/evals). Ask me which eval to run and with
   which model. Each eval names its GitHub repository and its approved commit.
4. Clone that repository, check out that exact commit, and leave the checkout
   unchanged. Register it: darkmux lab fixture register <checkout folder>
5. Make sure a darkmux profile serves the model I chose (darkmux profile list).
   The workload names are the files in the workloads folder inside the eval's
   folder (its .fixture.json names that folder, usually eval). Run:
   darkmux lab run <workload> -p <profile>
   It can take a long time. Wait for it to finish.
6. Show me the result: darkmux run inspect <run id>
7. Show me what would be sent: darkmux lab submit <run id> --dry-run
8. Ask me to make an upload token on https://evals.darkmux.com/account (Tokens
   tab) and to run darkmux lab submit login myself, so you never need to
   see the token.
9. When I say so, send it: darkmux lab submit <run id>
10. Tell me the run waits for the maintainer's review before it is shown, and
    that I can see it at https://evals.darkmux.com/account?tab=runs