Skip to content

Queue · runs on your machines

Run the queue. Two scripts, one shared folder.

The coordinator and the worker are plain Node scripts (Node 20 or newer, zstd needs 22.15 or newer) with no dependencies, in examples/lan-queue of the Noeten source. They were tested on one machine with two workers, a lying worker, an abandoned job and a hostile shared folder. This website cannot see your network and does not try.

Prototype

Step 0

Get the code.

The public repository is not open yet; its link appears here when it is. The scripts are in examples/lan-queue of the project source. Put the folder on every machine that will take part. Nothing to install beyond Node.

Step 1

Start the coordinator.

On the machine that holds the logs. The token is a shared secret of at least 16 characters; the shared folder is any folder every machine mounts.

node coordinator.mjs \
  --inbox ./testnet-logs \
  --share /mnt/team-share/noeten \
  --token "$NOETEN_TOKEN" \
  --host 192.168.1.20 \
  --chain-id 46630 --from-block 154000000 --to-block 154012000
# listening on http://192.168.1.20:7420  codecs zstd, brotli, gzip, deflate
# queued 42 file(s) from ./testnet-logs
  • By default it listens on 127.0.0.1 only. Pass your LAN address to let other machines in.
  • Callers outside loopback and private ranges (10/8, 172.16/12, 192.168/16, link-local, fc00::/7) are refused unless you pass --allow-any-ip.
  • --min-workers N waits until N machines have checked in, so one fast machine does not take every file.

Step 2

Start a worker on each idle machine.

node worker.mjs \
  --coordinator http://192.168.1.20:7420 \
  --token "$NOETEN_TOKEN" \
  --share /mnt/team-share/noeten \
  --name studio-mac --max-load 0.6
# [studio-mac] run-07.jsonl: 48211904 -> 3120775 bytes with zstd in 1840 ms, checked
  • It takes a job only while the 1-minute load average per core is under --max-load. On Windows the load cannot be read, so it always counts as idle.
  • It hashes the log as it streams in, compresses, opens the archive again and compares sha256 and size before it reports anything.
  • Temporary files are created new with a random name and no-follow, and the folder must still resolve inside the share: a planted symlink cannot redirect a write.

Step 3

When the queue is empty.

The coordinator re-checks every archive it accepted (size cap, decompression capped at the original size, time limit), then writes:

share/runs/2026-11-18T09-12-44-120Z/manifest.json   # every file, both hashes, both sizes, codec, worker
share/runs/2026-11-18T09-12-44-120Z/anchor.json     # the fields the ledger takes

The run is complete at this point. Anchoring it on chain is optional and is the only step that leaves your network (hashes and sizes only).

Step 4

Check a manifest here.

The root is recomputed from the file list in this tab. Drop archives too and each is matched by sha256.

Step 5

The coordinator's API.

POST /claim

A worker asks for a job, naming the codecs it has. One job per worker at a time; 204 when nothing is free, 410 when the run is done.

GET /jobs/:id/input

Streams the original log to the worker that holds the lease, over your LAN.

POST /jobs/:id/heartbeat

Keeps the lease while a long file compresses. A lease that lapses puts the job back in the queue.

POST /jobs/:id/done

The worker reports the archive's sha256. The coordinator reads the archive from the shared folder and checks it itself.

POST /jobs/:id/fail

The worker gives the job back with a reason. After three failed attempts the file is reported as failed.

GET /status

Counts per state and the workers seen, for a quick look while a run is going.

Every call carries the shared token (Authorization: Bearer) and the worker's name (x-noeten-worker). Bodies are capped at 16 KB.

Anchor a runHow archives are checked