Docs · Everyday use

Several GPUs in one machine

Two completely different things share the phrase "multi-GPU", and confusing them costs throughput rather than raising an error. This page separates them.

Needs FeatureMultiGPU — Home Pro (up to 2 cards in any one machine) or Business (unlimited).

The two modes#

One model across several cards#

nodeau run qwen3.8-27b-q4km --gpus 2

Nodeau names the exact cards, Kubernetes allocates exactly those, and per-process telemetry proves which card carried the work.

It is not a speedup#

Measured on this project's own hardware, with a model that fits on either card:

Generation is bandwidth-bound and a layer split runs the layers sequentially, so throughput lands between the two cards rather than above the faster one.

What it is for#

Models that fit on no single card you own. The strong case is a model whose weights are larger than either card's memory — roughly 19 GiB of weights on a pair of cards that are 10 GiB and 16 GiB. Split, it runs. On either card alone, it does not.

GPU memory is not pooled#

Two 8 GB cards are not a 16 GB card.

Per-device overhead is replicated, not shared, so each card pays the runtime's own buffers again. Nodeau adds a further 128 MiB per device allowance for a multi-device workload. A pair whose total looks ample can have one card that cannot hold its share — and the smaller card runs out first.

Admission evaluates the fit per device, as a conjunction. It never sums.

Choosing how the split is made#

nodeau run <model> --gpus 2 --topology auto    # the default
nodeau run <model> --gpus 2 --topology layer

auto lets Nodeau choose; layer asks for a layer split explicitly. This is advanced and unnecessary for ordinary use.

What is not supported#

• Row split — it does not run on this hardware (`device CUDA0 does not support split buffers`).
• Peer-to-peer between cards — reported unavailable in both directions on the only pair this project owns. No NVLink.
• Across machines — no cross-machine sharding, no distributed tensor or pipeline parallelism.
• Sharing a card — no GPU sharing, time-slicing, MIG, preemption, migration, live repartitioning or automatic tensor-split tuning.
• Batch — --gpus on a batch job sets how many cards one worker uses, and is currently pinned to 1 by its own type.

Heterogeneous cards#

A split across two different models of card is qualified: one pair — an RTX 3080 and an RTX 5060 Ti — running a model whose weights fit on neither alone. Other combinations are untested rather than refused; admission still does the per-device arithmetic and will say what it finds.

If only one of the cards is free#

Atomic set allocation is inherited, not built. A multi-device claim with one member already held is refused outright — "cannot allocate all claims" — and reserves nothing. Nodeau never holds a partial set, because Nodeau never holds devices at all: Kubernetes does.

So a workload asking for two cards when one is busy is refused, holds nothing, and can be run again when the card frees.

--gpus-auto#

nodeau run <model> --gpus-auto --max-gpus 2

Independent workers#

The other mode. Instead of one model on several cards, run several complete model instances, each on a card of its own, each with its own weights and its own KV cache.

For a finite job, that is batch:

nodeau batch submit requests.jsonl --model qwen3.5-4b-q4km --workers 2

Two workers share out the records between them, and Nodeau proves which card ran which worker. Measured on a 24-record job: 47 s against 61 s.

For serving, it is just two workloads:

nodeau run <model> --name a --port 8080
nodeau run <model> --name b --port 8081

workers and gpus are orthogonal#

This is the most dangerous arithmetic available here.

• --workers N is N independent model replicas. Each loads its own weights, holds its own KV cache and reserves its own accelerator set.
• --gpus G is how many accelerators are inside one worker.

Never divide a model's memory by the worker count. Each worker needs the model's full memory on its own card. Doing that arithmetic the other way undercharges every card, in the direction that lets a second workload onto memory already spoken for.

A worker count is a request: workers that cannot get a card wait, and the job still completes.

Seeing what happened#

nodeau ps                          # which workload is on which card
nodeau placement explain <name>    # why those cards, and what the alternatives cost
nodeau service explain <name>      # the per-device VRAM arithmetic

Documentation for the current published build, the beta channel. Something here wrong or missing? Tell us — a report from a machine we have never seen is the most useful thing we get.