Docs · Everyday use

Batch inference

A finite set of requests, one model, and a results file when it is done. Your records never leave the machine.

The whole flow#

nodeau batch submit requests.jsonl --model qwen3.5-4b-q4km --name nightly
nodeau batch wait nightly
nodeau batch results nightly -o answers.jsonl

Or, composed:

nodeau batch wait nightly && nodeau batch results nightly

wait exits 0 when the job finished and 1 when it failed, was cancelled or timed out, so a shell can branch on it.

The input file#

One JSON object per line. Not a JSON array — Nodeau reads JSONL.

{"custom_id":"req-1","method":"POST","url":"/v1/chat/completions","body":{"messages":[{"role":"user","content":"Summarise this paragraph: ..."}],"max_tokens":2048}}
{"custom_id":"req-2","method":"POST","url":"/v1/chat/completions","body":{"messages":[{"role":"user","content":"And this one: ..."}],"max_tokens":2048}}

Nodeau checks the file before uploading it#

The first twenty records are read locally, so the two commonest mistakes become an answer rather than a per-record error after a large upload:

• A file starting with [ is refused: *"looks like a JSON array. Nodeau reads JSONL."*
• A line that is not valid JSON is refused, by line number.
• If no record in the sample has a custom_id, it is refused.
• An empty file is refused.

It also notes — without refusing — records asking for a small completion budget, because a reasoning model spends its budget thinking before any answer exists. That is advice, not a rule: plenty of records legitimately want a hard cap, and the budget in your record is your decision.

Workers and GPUs#

These answer different questions, and mixing them up costs throughput rather than raising an error.

nodeau batch submit in.jsonl --model M --workers 2 --gpus 1

Two independent workers, each a complete model instance on a card of its own, sharing out the records between them. Faster, because the records do not depend on each other.

nodeau batch submit in.jsonl --model M --workers 1 --gpus 2

One worker whose model spans two cards. For a model too large to fit on either card alone — not a way to go faster.

A worker count is a request. Workers that cannot get a card wait, and the job still completes with the workers it got.

Measured on a 24-record job with two workers: 47 s against 61 s. That is a floor, not a rate — a short job is dominated by loading the model, and both workers pay that cost.

Never divide a model's memory by the worker count. Each worker holds the model's full memory on its own card. See several GPUs in one machine.

Submitting#

nodeau batch submit FILE --model MODEL [flags]

Queuing and capacity#

A batch job takes a whole GPU while it runs, exactly as a service does — Nodeau does not share a card between workloads.

If every GPU is busy, your job waits in a queue and starts by itself when one frees. That is QUEUED, and it is not a failure. Nothing is preempted: Nodeau has no priority queues and does not stop a serving model to make room.

nodeau ps          # batch jobs appear here alongside services
nodeau ps --all    # including finished ones
nodeau batch list

Watching one#

nodeau batch status nightly
nodeau batch status nightly --json
nodeau logs nightly            # a worker's own output

Results#

nodeau batch results nightly                 # writes nightly-results.jsonl
nodeau batch results nightly -o answers.jsonl
nodeau batch results nightly --stdout        # only if you mean it

{"custom_id":"req-1","response":{"status_code":200,"body":{}}}
{"custom_id":"req-2","error":{"code":"InferenceFailed","message":"..."}}

Every record you submitted has exactly one line here, in the order you sent them, each carrying the custom_id it came from — so results are matched by identity rather than by position.

Results are written to a file by default. These are your model's answers, and printing them to a terminal by accident is how they end up in scrollback, a screenshot or a pasted issue.

If a result arrives whose custom_id cannot be resolved, it is reported by line number and says so, rather than being dropped or attributed to the wrong request.

Cancelling#

nodeau batch cancel nightly

Results already produced are kept and remain available with nodeau batch results. Nodeau waits for the model server to actually exit before releasing the GPU, so a cancelled job may take a few seconds to finish stopping.

The local dashboard can also stop a batch job — it is the one action that page has.

Where your records live#

On the control-plane machine's disk, at /var/lib/nodeau/batch.

• They are not sent to Nodeau Cloud.
• They are never written into a Kubernetes object or a log.
• nodeau uninstall keeps them unless you pass --remove-batch-data, because unlike model weights they cannot be downloaded again.

Failure cases#

A worker that cannot start does not hold its GPU indefinitely. Nodeau suspends the job, waits until nothing holds the device, and only then goes terminal — that order is what releases the reservation.

What batch is not#

Nodeau's batch is its own thing. It is not the OpenAI Batch API, there is no /v1/batches endpoint, and it does not accept OpenAI batch files as a drop-in.

There is no preemption, no GPU sharing, no job resume, no batch failover, no scheduled or recurring batch, and no priority queueing. A job runs on the machine it was placed on; it does not span machines.

Documentation for the current published build, the beta channel. Something here wrong or missing? Tell us — a report from a machine we have never seen is the most useful thing we get.