Sandbox nodes

A hypervisor runs sandboxes through the sandbox engine (hypervisor-microvmd), a service that ships with the node agent and is updated with it. After a node’s agent is updated, restart the engine (systemctl restart hypervisor-microvmd): the agent update does not restart it, and an engine that was running before the update keeps its old behaviour, including how environment variables and the tools gateway reach a sandbox, until it is restarted. This page covers what an operator needs to run it; enabling the feature end to end is on Enable sandboxes.

Before you begin

The node is a KVM hypervisor added to the panel and provisioned by the agent installer (it needs /dev/kvm and cgroup v2 with the cpu, memory and pids controllers).
The node has at least one hypervisor storage attached in Compute > Hypervisors > node > Storage.

Where to find it

Compute > Hypervisors, open the node, click the MicroVM tab, MicroVM Engine card.

What the node card shows

Field	Meaning
Enable Engine	The toggle in the card header. Lets the panel place sandboxes on this node. The engine service must already be running.
Reserved vCPUs / Reserved RAM / Reserved Disk (optional cap)	Ceilings the sandbox placement scheduler never exceeds on this node.
MicroVM Storage	The hypervisor storage that holds sandbox data (kits, running disks, paused-state snapshots). An Auto-selected badge shows when the panel picked it for you; a warning shows if sandboxes would otherwise land on the node’s root filesystem.
State directory	The path under the bound storage where the engine keeps its state, shown once a storage is bound. A Drifted badge means that path no longer matches the bound storage, which makes the node unschedulable.
Public IP (ingress)	The node’s own internet-reachable IPv4, the address the engine’s ingress proxy listens on for ports 80 and 443. Published as the A record for the sandbox wildcard and custom app domains. Must be the node’s default-route source address (or the `MICROVMD_INGRESS_ADDR` override), never a customer or VPC IP. See Enable sandboxes.
Last heartbeat	When the engine last reported to the panel, shown below the Save button once the node has a profile. A stale heartbeat turns the readiness check to warn.
Running VMs / Used vCPU	Live counts the engine last reported, shown next to the heartbeat.

The Firecracker release a node runs is not shown on this card; it is summarized platform-wide by the Readiness page’s Firecracker engine reporting check. Sandbox domain and apps domain are not exposed as fields on this card either; see the note on Enable sandboxes. The domain-related indicator here is an ingress-certificate-coverage badge that names the apps domain when its wildcard certificate does not cover it.

Where sandbox data lives

The engine keeps everything it owns (base kits, per-sandbox disks, paused-state snapshots, build workspaces) under a microvm/ directory inside the storage bound on the node card. The panel schedules against the real free space of that storage. A sandbox paused across a Firecracker upgrade cannot resume from its old snapshot; it is stopped and starts fresh from its kit on the next start.

What a current node adds

Some features need a current node agent, and a node reports each one to the panel on its heartbeat. The panel treats a node that does not report a feature as not supporting it:

Environment variables and secrets in a sandbox. A node without this refuses a launch that needs them, with a clear message, rather than starting a sandbox with missing keys.
The tools gateway. See Tools gateway.
Kit catalog relay. Lets the panel fetch the signed kit catalog through the node when Docker Hub rate limits the panel. See Kits.

Update the node agent, then restart the engine on that node (below). The agent update does not restart hypervisor-microvmd, so until you do, the old engine keeps running without these features.

Restart the engine

systemctl restart hypervisor-microvmd

Safe while sandboxes run: the engine process restarts, the sandboxes keep running and the engine picks them up again when it comes back.

Logs

journalctl -u hypervisor-microvmd -f

Per-sandbox console output is in vms/<id>/console.log under the engine’s data directory, and in the panel on the sandbox’s Logs tab.

Common problems

The engine never starts. Check systemctl status hypervisor-microvmd; the usual causes are a missing /dev/kvm or a bound storage path that is not mounted.
The node card shows no heartbeat. The engine is stopped or cannot reach the panel; start with the service status and the journal.
Sandbox create fails naming a cgroup controller. The node’s cgroup v2 setup lacks that controller; fix the host, then retry.