Docs · Your machines

Health and diagnostics

Five commands answer five different questions. Using the wrong one is the commonest reason a real problem looks invisible.

Which command answers which question#

CommandQuestionPlatform
nodeau doctorIs anything wrong, and what do I do about it?Both
nodeau healthHow are my machines doing — processor, memory, storage, networkLinux
nodeau statusWhat is Nodeau running right now, and does the endpoint answer?Both
nodeau psWhat is holding a GPU — services and batch jobsBoth
nodeau logsWhat is a workload printing?Linux

doctor is the one to start with. It is read-only, it names problems in typed codes, and every result that is not a pass carries something to do about it.


nodeau doctor#

Terminal
nodeau doctor
nodeau doctor --json          # every result, code, explanation and remedy
nodeau doctor --host-only     # this machine only, without contacting the cluster
nodeau doctor --node nodeau-c # one machine, plus installation-wide checks

Every check is read-only. Doctor never installs, configures, mounts, starts or stops anything; it only looks.

Two halves:

  • The host — is this machine capable of running Nodeau: operating system, processor, memory, whether it is bare metal, storage safety, the state directory, Secure Boot, the GPU on the PCI bus, the driver, VRAM, network.
  • The installation — is Nodeau itself working: its control plane, the agent on each machine, whether every GPU report is fresh enough to schedule against, whether the models on disk still verify, and what your services and batch jobs are doing.

Exit 0 everything passed or only warnings · 1 something failed · 2 called wrongly.

Doctor codes#

Stable, machine-readable, and the same in --json as in the human report.

Reaching the cluster

CodeMeaning
CLUSTER_UNREACHABLENodeau cannot talk to Kubernetes at all
API_FORBIDDENThe cluster answered and refused this identity
NODEAU_NOT_INSTALLEDA Nodeau component is absent. On a machine that has not been set up, this is the correct answer rather than a fault
NODEAU_CRDS_MISSINGThe Nodeau resource types are not registered
CONTROLLER_NOT_RUNNINGThe control plane is not available
AGENT_NOT_RUNNINGNo hardware agent is reporting on some machine
COMPONENT_DEGRADEDA component is running below its desired count

Hardware

CodeMeaning
NO_GPU_NODESNo machine has published a GPU
HARDWARE_REPORT_STALEA machine's GPU report is older than the controller will accept, so admission fails closed for that machine
GPU_UNHEALTHYA reported GPU says it is not healthy
DEVICES_NOT_INDEPENDENTA machine holds more than one GPU and cannot schedule them independently. Not a fault — the state every machine is in until it opts in
DEVICE_DRIVER_MISSINGA machine is switched to per-device scheduling and nothing offers its GPUs
DEVICE_ALLOCATORS_OVERLAPA machine is serving GPU work through both allocators. This one is a fault, and the dangerous kind — two independent allocators over the same cards
HOST_CHECK_FAILEDA host-level precondition failed
STORAGE_LOWThe model cache filesystem is running out of room

Models

CodeMeaning
NO_MODELSNothing has been downloaded yet
MODEL_CACHE_NOT_OBSERVEDA GPU machine has no model-cache observation
MODEL_CACHE_ERRORThe observer ran and could not read the cache
MODEL_CACHE_VERIFYINGA hash is in progress. Clears by itself
MODEL_CORRUPTA cached artifact does not match the catalog digest

Workloads

CodeMeaning
SERVICE_NOT_ADMITTEDA service exists and Nodeau refused it. Not an installation fault — the product working, reported so you can see it without going looking
SERVICE_NOT_READYAdmitted and not yet serving
WORKLOAD_STOPPEDA workload Nodeau has a record of is not running
BATCH_JOB_FAILEDA batch job ended in failure

Local endpoints

CodeMeaning
ENDPOINT_NOT_ANSWERINGA local endpoint is recorded and does not answer
ENDPOINT_UPSTREAM_LOSTIt answers and cannot reach the model behind it
ENDPOINT_ORPHANEDEndpoint units exist for workloads that do not
ENDPOINT_UNRECORDEDAn endpoint is running and Nodeau has no record of it, so nodeau status does not list it
ENDPOINT_BINARY_REPLACEDA local endpoint is running a program that has since been replaced on disk
NETWORK_EXPOSURESomething makes a model reachable from off this machine. Nodeau never creates one

Account and plan

CodeMeaning
ENTITLEMENT_INVALIDAn entitlement exists and was not accepted
PLAN_REFRESH_STALLEDThis installation is linked to an account and has stopped learning what that account entitles it to
FLEET_NOT_REPORTINGThe fleet connector is not reporting. A warning at worst — a cloud outage costs you nothing
FLEET_REPORT_STALEReporting, and the last report is older than the cadence the server asked for
FLEET_PROTOCOL_MISMATCHThis build and Nodeau Cloud cannot exchange fleet state. An update fixes it; a network never will
CONNECTOR_BINARY_REPLACEDThe connector is running a binary that has since been replaced on disk

macOS

CodeMeaning
RUNTIME_MISSINGNo model runtime is installed, so nothing can be executed
RUNTIME_CORRUPTThe installed runtime's bytes no longer match the digests recorded when it was installed
RUNTIME_UNVERIFIEDInstalled, and its integrity was not checked on this run. Deliberately not a pass: "was not checked" and "is intact" are different claims
RUNTIME_UNMANAGEDA runtime Nodeau did not install, so it has no version, no backend and no digests
NOT_ON_PATHThe binary is installed and a shell will not find it

nodeau health#

Terminal
nodeau health
nodeau health --json
nodeau health history --machine nodeforge --since 6h

The current state of every machine — processor, memory, storage and network — alongside the GPU readings already there, together with anything that needs attention.

Every reading was taken by the machine it describes, and stays on your own hardware. Telemetry is not sent to Nodeau Cloud, and there is no field for it in the protocol that talks to Nodeau Cloud at all.

Missing is not zero#

A value shown as "not collected" is missing, which is a different thing from zero. A card reporting 0% utilisation is idle; a card Nodeau could not read is unknown, and the two must never look the same.

health history follows the same rule: an interval with no observations is absent rather than zero, so a gap in the output is a gap in what was observed. Nodeau does not fill one in.

history flags#

FlagDefaultMeaning
--machine <name>the first with any historyWhich machine
--since <d>1hHow far back to look
--resolution <w>—A named window, e.g. 1m or 15m
--metric <name>allRestrict to a metric. Repeatable
--device <uuid>allRestrict to a GPU UUID. Repeatable
--max-points <n>—The most intervals to print per series
--jsonoffMachine-readable

Retention#

24 hours, uniformly. It is not a plan difference, and no plan gets more.

History lives in the Kubernetes datastore — which is the control-plane machine's disk. It is not replicated anywhere. If that disk is lost, the history goes with it.

What health does not do#

  • No disk health and no SMART. Nodeau reads no drive attribute. It reports how full a filesystem is, and says so. It will not tell you a disk is failing.
  • No per-process CPU or memory, no process list, no command lines, no connection table.
  • No telemetry to Nodeau Cloud. The machines appear in the hosted console; these readings do not.
  • No external alert delivery. No email, no webhook, no pager.
  • Not qualified on Apple Silicon.

Alerts#

The controller reconciles a small set of conditions and reports what needs attention:

ConditionAbout
MachineOfflineA machine has stopped reporting
HostTelemetryUnavailableThe machine is there and its readings are not
HostMemoryPressureSystem memory
HostFilesystemLowA filesystem filling up
AcceleratorUnhealthyA card reporting a fault
AcceleratorMemoryExhaustedA card's memory
AcceleratorTemperatureHighTemperature
AcceleratorThermalThrottlingThe card reducing clocks to stay within limits
InsufficientVRAMA workload that cannot be placed
WorkloadNotReadyA workload that is not serving

An alert has three states:

StateMeaning
PendingThe condition is holding and counting toward its debounce window
ActiveIt held for the full window
ResolvedIt was there and stopped

Only Active is counted anywhere — no badge, tile or nodeau health line moves for a Pending alert. That is what makes the third state additive rather than a claim.


nodeau status#

Terminal
nodeau status

Nothing here is read from a cache. The workload state comes from the cluster, the endpoint state from the service manager, and "answering" from an actual connection to the port — so a stale record cannot make this report something that is not true.

It also lists anything that makes a model reachable from your network, if something on the machine does. Nodeau does not create those, and saying so is the point.


nodeau ps#

Terminal
nodeau ps            # everything holding a GPU
nodeau ps --all      # including finished batch jobs
nodeau ps -A         # across all namespaces

Services and batch jobs alike, with where each one is and what it is doing.


nodeau logs#

Terminal
nodeau logs qwen-local -f
nodeau logs my-batch
nodeau logs --system nodeau-controller

Nodeau's own components are under --system; everything else is looked up in your workload namespace.

Prompts, completions and batch records are never written to a log, so they cannot appear here.


The local dashboard#

Terminal
nodeau dashboard

The same information in a browser: hardware, models, running services, alerts and plan. Bound to loopback, protected by a token, compiled into the binary, and it works with no internet connection at all.

It is read-only apart from one action — it can stop a batch job.

Documentation for the current published build, the beta channel. Something here wrong or missing? Tell us — a report from a machine we have never seen is the most useful thing we get.