What everyone gets wrong about building sandboxes
This title could be clearer and more informative.Sign up to try Clickbait Shieldfor free.
Inngest is launching Sandboxes, a built-in primitive for running agent code without a separate vendor or dashboard. The team rejected both Kubernetes with gVisor and plain Firecracker, building instead on cloud-hypervisor for CPU, memory and device hotplug, virtio-fs, and snapshot/restore. Warm pools of snapshotted VMs boot before knowing the workload, then get hotplugged with network, CPU and filesystem mounts on assignment. Networking runs as custom eBPF programs with VXLAN isolation instead of TAP-and-bridge, and storage uses ZFS for fast copy-on-write snapshots, targeting sub-250ms boots and a future of rewindable 'durable sandboxes'.
Table of contents
Building for speedIf you want millisecond speed, get off the critical pathCloud-hypervisor, and the networking problemA high-level overview of our sandbox architectureWhere this goes: durable sandboxesWhat's nextQuestions this post answers
Why isn't Firecracker enough for building fast, full-featured sandboxes?
Firecracker lacks CPU hotplug, virtio-fs for sharing a host filesystem, device passthrough (no VFIO for GPUs or NVMe), and only has network hotplug in developer preview. Firecracker is intentionally minimal, ideal for short-lived Lambda functions, but those missing features block building persistent, resizable, richly networked sandbox VMs. Evaluating hypervisors for agent sandboxes? Track these infra trade-offs on daily.dev.
Why are TAP interfaces a bottleneck when booting many VMs per second?
Creating and configuring each TAP interface takes kernel-wide locks, so they're created one at a time. Bringing up 1,000 sandboxes in a second on one machine causes them to queue behind each other, making warm pool refills the bottleneck. An eBPF-based datapath with VXLAN tagging at line rate avoids this per-VM bridge and interface overhead. Building high-throughput sandbox infrastructure? Follow networking deep dives like this on daily.dev.
What hypervisor does Inngest use for its sandbox VMs and why?
Inngest uses cloud-hypervisor, built from the same rust-vmm components as Firecracker but with the extra features sandboxes need: CPU, memory, and device hotplug, virtio-fs, VFIO passthrough, and snapshot/restore. This lets a generic pre-warmed VM be hotplugged into a specific customer's VPC, CPU shape, and filesystem on assignment, avoiding a separate warm pool per customer configuration. Comparing microVM options for your own sandbox or agent runtime? Keep up with posts like this on daily.dev.
Share this post