Before describing how to run AI agents on an Oxide Cloud Computer, it’s important to understand why AI agents need CPUs in the first place.

AI agents are just systems that perform inference and execute tool calls in a loop. Put another way, AI becomes "agentic" when GPU-based inference needs CPU-based tool execution.

The AI agent harness (e.g., Claude Code, Codex, Amp) is responsible for running this loop. The harness runs as a process on a CPU, communicating with an LLM over the network. When the LLM requests a tool call, the harness executes it on the CPU it’s running on and sends the result back to the LLM.

Without a CPU, you can’t run the harness or execute tool calls.

AI agents need strong isolation for both security and performance. You don’t want an AI agent to delete important data or duel with other AI agents and disrupt their execution.

Virtual machines provide stronger isolation than containers, making them well-suited for AI agents. This isolation comes at a cost, summarized by this table:

Using virtual machines instead of containers for workload orchestration raises the following questions:

These questions are surprisingly difficult to work through, especially when documentation leaves out the details. Look at this quote from the Claude Code self-hosted environments documentation.

Not only does this push readers towards containers, and possibly worse isolation as a result, but it also has a tone of "just do the right thing" without telling readers how to do it. Excuse me, Claude, did you just tell me to go eff myself?

Needless to say, when our customers ask how to run AI agents on Oxide, they also ask how to do it specifically with virtual machines, not containers.

Let’s see how to run AI agents on our Oxide Cloud Computer using virtual machines, specifically using Claude Code self-hosted environments and Amp self-hosted orbs. These were chosen due to their popularity in the industry and to showcase how different AI agents use certain Oxide features.

Our Oxide Cloud Computer has the right primitives for today’s AI agents—​virtual machines with strong isolation, disks with solid performance, control over VPC networking, and an elastic API to dynamically provision everything. Even so, we’re looking ahead to meet the needs of tomorrow’s AI agents.

Conversations with current and prospective customers about AI agents have centered around the following themes:

Security is where most conversations start, especially as the number of reported CVEs increases across the industry. We’ve already discussed the strong isolation provided by virtual machines, but our security story doesn’t stop there. Our Oxide Cloud Computer has silos for tenancy and resource quota enforcement, projects for network isolation, and VPC firewall rules to block undesirable network traffic. Additionally, our removal of the BIOS, along with the virtual machine attestation work discussed in RFD 605, provides a secure foundation for AI agent workloads.

If you run many AI agents, you may be wondering about scale. You can create as many virtual machines as you can fit on your Oxide Cloud Computer, but we’re working on multi-rack capabilities to expand a silo’s capacity beyond a single rack. Once released, this will simplify the management of AI agents across multiple racks.

Our API makes virtual machine provisioning trivial from an operational standpoint, but it can take tens of seconds or even minutes after an API request before a virtual machine is scheduled, booted, and running a workload. This affects users in the form of cold-start latency, which a warm pool can help mitigate, but not entirely eliminate. We accepted the latency in this blog post, but our goal is to reduce it as much as possible.

Cleaning up virtual machines is also trivial using the Oxide API, but, depending on the AI agent, there isn’t always a simple way to know when the workload has finished. In this blog post, we handled this by treating a stopped virtual machine as the signal that the workload has finished. This required us to wrap our AI agent process in a systemd service that runs poweroff after the AI agent process exits, along with a reaper process to poll the Oxide API for virtual machines to clean up. Not the best user experience, especially if the AI agent architecture doesn’t provide a delete operation, but we’re exploring how to make this experience behave like a container that exits when the main process is finished.

I’ve made it this far without talking about microVMs, but the microVM primitive is exactly what some infrastructure providers are using to improve the user experience of running AI agents without sacrificing the strong isolation they’ve come to rely on from virtual machines.

The definition of microVMs can vary across the industry, but discussing them in the context of AI agents has led to internal conversations about what a microVM primitive could look like for Oxide. Our hypervisor team even spiked on work to see how quickly we could launch a virtual machine and have it running its workload. Turns out we’re not far off from popular microVM implementations!

We don’t have anything set in stone, but the conversations about microVMs are proving to be fruitful and are being validated by customer requests. We’re well-positioned to release a better primitive for running AI agent workloads with a focus on security, scalability, cold-start latency, and cleanup, taking inspiration from microVMs.

If you’re interested in running AI agents on Oxide, we’d love to hear from you! Contact us, tell us about your use case, and stay tuned for what’s next.

To see self-hosted AI agents in action, watch the video below, in which I run Claude Code and Amp AI agents on Oxide.

EMERYVILLE, Calif., and NEW YORK — October 19, 2026 — Oxide Computer Company and L Labs set 18 STAC-M3 world records in a single virtual machine on the Oxide Cloud Computer. Audited by STAC, the results open a partnership between the two companies to bring L to trading firms that need the self-service model of the public cloud in their own data centers.

In Shasta, the entire database was pre-loaded into memory. The other two records came in the Antuco suite, where the benchmarks force the database to read from storage. There, L 1.0 achieved “the highest storage efficiency …​ of any published STAC-M3 Antuco result, 364%,” and outperformed all publicly disclosed results on Year High Bid Re-run. It was also “faster in 11 of 17 mean response-time benchmarks than all other publicly disclosed benchmarks using a cloud-based deployment architecture.”

STAC audited L tick database 1.0 on one Oxide virtual machine with 128 vCPUs, 1.25TiB of memory, and five 3.44TB NVMe disks, running Ubuntu 24.04.4 LTS. The STAC-M3 test clients were also hosted on the same virtual machine. L Labs configured and optimized the full stack under test with infrastructure optimization support from the Oxide engineering team. Oxide sponsored the audit and supplied the Oxide Cloud Computer for testing, including servers, storage, networking, and cloud control plane.

As partners, Oxide and L Labs will jointly sell and support a custom tuned version of L on the Oxide Cloud Computer, with availability beginning today. The full STAC reports for Antuco and Shasta are available with free registration.

Jacob Loveless will present the results at the STAC Summit in New York on October 21 to an industry audience of technologists from banks, hedge funds, proprietary trading firms, and exchanges.

Oxide Computer Company is the creator of the Cloud Computer, a rack-scale system with fully unified hardware and open-source software purpose-built to deliver hyperscale cloud computing to on-premises data centers. Oxide gives developers the elasticity, programmability, and unified control of the cloud while enabling enterprises to retain ownership of their infrastructure for greater security, efficiency, sovereignty, and cost predictability. Founded in 2019, Oxide is based in Emeryville, California. To learn more, visit oxide.computer.

L Labs is a public benefit corporation on a mission to make powerful computing more accessible, more efficient and more useful to the people building what comes next. L Labs is the creator of L, a new runtime for k4, q, Parquet, and qSQL that runs existing code unchanged on a modern, compressed SIMD-native execution engine. To learn more, visit l-labs.co.

Media contact (Oxide and L Labs): media@oxide.computer

STAC and all STAC names are trademarks or registered trademarks of Strategic Technology Analysis Center.

Do you remember the game Q*bert? It was one of the first wildly popular isometric games, and looked a little like this:

Why am I showing you a ridiculously small game screenshot? Well, each cube in that image represents a 'voxel,' or a point in 3D space. It’s actually the 3D version of a pixel. Volume Element + Pixel = Voxel. Neat, right?

Anyway, for Oxide, Virtual OXidE Labs (or Voxel) means an incredible amount of fidelity packed into a single piece of hardware. It is the spiritual successor to our prior virtual labs (known as a4x2) which allowed us to hack out decent racks running in VMs.

Pay attention to that word "fidelity" by the way. For it is, as Bryan Cantrill puts it, the Prime Directive: labs should use stock Oxide software without hack or modification.

This is what Voxel gives us. A complete software emulation stack, built on illumos in Rust, where we can build and test a variety of realistic configurations:

And what’s more, it gives us all these features in a fairly small footprint. At home, I run Voxel labs on a Minisforum MS-A2 mini-PC with a 32-thread AMD Ryzen 9 9955HX and 96GB RAM. On that computer, I can spin up 6 sleds with ease, either in the form of one rack or in a multirack configuration (two separate racks, one management plane). All it requires is Oxide Helios and a decently fast NVMe drive. Launch, and in about 10 minutes you have a full Oxide rack with each "sled" in its own Propolis VM — alongside switches (also in Propolis) and other infrastructure.

This is the bit I mentioned a moment ago. A big part of what makes our racks unique is the embedded RoT and SP in every sled. We’ve long had an SpSim construct that served as a stub for the service processor and would return the right info at the right times. But that’s simulation, not emulation. It’s always going to be happy path, and that’s just not enough fidelity for real testing cases.

As such, we built actual emulated Cortex-M7 and Cortex-M33 devices for illumos and Linux (x86 and ARM). The repo for this tool (sp-emu) is open source. Each of the two devices runs in a single thread, and (mostly) faithfully emulates the three main requirements: instructions, time, and interrupts.

From there, you can flash our Hubris firmware directly to these emulated devices, and then communicate by way of an ordinary socket. The Voxel lab can speak with these virtual devices via the Management Gateway Service (MGS) and Inter-Processor Communication Channel (IPCC). You can learn everything you want and more about these concepts in the Management Gateway Service repo and Hubris docs.

There really is no "run real microcontrollers as virtual devices with this one neat trick!" I can think of. But the real challenge of sp-emu’s job is time. Remember all of the code we’re running there is meant for silicon. Hubris schedules off a 1ms SysTick and the emulator doesn’t use wall time; as such, we had to invent a clock. That said, emulated hardware is also slower than the real thing. So there’s…​ well, less time per time. So instead of trying to get all temporal about it, time became instructions. For example, I give you maybe_tick:

In this case, we count instructions against the kernel’s own reload value and fire exceptions anywhere the single-priority design makes it safe to take it — in thread mode, interrupts unmasked. By the same token, peripheral access costs one tick of our clock (an instruction), whereas on silicon it might stall for dozens of cycles on the bus.

You may have noticed self.exception_entry which is defined elsewhere in the code:

fn exception_entry(&mut self, vecnum: u32, bus: &mut Bus)

In short, this function models everything the hardware does on a Cortex-M core taking an exception, all the way between "exception becomes active" to the first handler instruction. vecnum here is an ARMv7-M exception number (also the vector table index for that exception’s handler). bus is the memory system which is used to actually push the frame and read the table.

Hubris has a non-preemptible kernel, though it schedules tasks preemptively. SVCall, PendSV, SysTick and all IRQs run at the same priority level, while faults keep a higher priority. As such, the emulator doesn’t have to nest handlers. It processes pending IRQs based on the PRIMASK, FAULTMASK, and BASEPRI settings. If one of them pended PendSV, that will perform the deferred context switch for the kernel. For more details on how it works internally, you can check this lovely bit of documentation.

There’s a lot more we can talk about here, but to sum it all up you can look at this pseudo-ish loop:

Note: yes I’m aware I said there’s no cycles then have a field named cycles. That’s the "fake it till you make it" part of emulation.

That’s the overall heartbeat of sp-emu, and it runs ad infinitum. Everything else exists to make those calls seem real.

Past the actual working of the emulated sleds and microcontrollers, it’s all good ol' fashion user experience. When you’re running that many components on a single system, it can get really confusing trying to figure out what pieces are where, how to connect to one sled on one rack versus another, etc. So the voxel command that we use to launch and destroy virtual racks also serves as a connectivity tool. For example:

That command logs the user straight into the g0 sled in the global zone. Alternatively:

tp in this case acts like our technician ports, and allows the user to connect straight to a switch zone for rack-wide commands and networking. There are also numerous commands to manage things like images (export/import, delete), config (network topology, sled sizing, multi-rack, etc.), and more features coming as we build.

Voxel is available under the MPL 2.0 license for download. You’ll need to run Helios as your OS to run Voxel. I’d recommend 16+ threads and 64GB+ RAM, plus a good premium NVMe with onboard DRAM.

Oxide Computer Company, the on-premises cloud computing company, and Carahsoft Technology Corp., The Trusted Government IT Solutions Provider®, today announced a partnership. Carahsoft will act as Oxide’s Master Government Aggregator®, to make the Oxide Cloud Computer available to Public Sector customers through Carahsoft’s reseller partners and NASA Solutions for Enterprise-Wide Procurement (SEWP) V and OMNIA Partners contracts.

Oxide is the industry’s first rack-scale system that integrates advanced computing hardware and cloud control plane software to deliver cloud computing within an organization’s own data center or colocation facility. Oxide provides a vertically integrated system, operational within hours of delivery. The Cloud Computer provides elastic compute, storage and networking through first-class APIs, CLI and a web console, enabling developers and cloud infrastructure teams to provision compute in a self-service fashion rather than reacting to IT tickets or leaning on manual configuration scripts.

Oxide’s Cloud Computer provides critical Government capabilities including hardware root of trust, an auditable open source software stack and eliminates dependency on third-party virtualization licensing. The platform supports workloads spanning Agentic AI, High-Performance Computing, classified data processing and general-purpose IT operations.

Oxide’s solutions are available through Carahsoft’s SEWP V contracts NNG15SC03B and NNG15SC27B and OMNIA Partners Contract #R240303. For more information, contact the Carahsoft team at (888) 964-7379 or Oxide@carahsoft.com.

Oxide Computer Company is the creator of the world’s first commercial cloud computer, a rack-scale system with fully unified hardware and software, purpose-built to deliver hyperscale cloud computing to on-premises data centers. With Oxide, enterprises and Government agencies can realize the economic and operational benefits of cloud ownership, accessing the same self-service development experience of public cloud without the public cloud cost. Oxide empowers developers to build, run and operate any application with enhanced security, latency and control, freeing organizations to focus IT operations on strategic initiatives. To learn more, visit oxide.computer.

Contact: media@oxide.computer

Carahsoft Technology Corp. is The Trusted Government IT Solutions Provider, supporting Public Sector organizations across Federal, State and Local Government agencies and Education and Healthcare markets. As the Master Government Aggregator for our vendor partners, we deliver solutions for Hardware, Cybersecurity, MultiCloud, DevSecOps, Artificial Intelligence, Customer Experience and Engagement, Open Source and more. Working with resellers, systems integrators and consultants, our sales and marketing teams provide industry leading IT products, services and training through hundreds of contract vehicles. Visit us at www.carahsoft.com.

Contact: Mary Lange, (703) 230-7434, PR@carahsoft.com

No single provisioning approach fit all customers' workflows, so we ended up publishing three integrations.

Provisioning integrations create and manage Oxide instances, but they do not reconcile those instances with Kubernetes Node objects. Without this reconciliation, a cluster cannot determine the status of an unreachable Kubernetes node.

We needed a component that ran in each cluster and used the Oxide API to reconcile Oxide infrastructure with Kubernetes state. Kubernetes provides the cloud controller manager (CCM) for exactly this purpose. A CCM integrates Kubernetes resources with an infrastructure provider’s API.

We built the Oxide cloud controller manager to integrate Kubernetes with Oxide. Its node controller keeps Kubernetes Node objects synchronized with their Oxide instances, recording details such as instance IDs, network addresses, and instance state. Kubernetes uses this information to initialize nodes and remove them from the cluster when their Oxide instances are deleted.

The CCM does not create instances or provision clusters. That remains the job of provisioning integrations such as the Rancher node driver, the Omni infrastructure provider, and CAPOx. The CCM works with all the provisioning integrations and gives us an integration point to build upon. As Oxide evolves, we can add new controllers to the CCM rather than update every provisioning integration.

With that runtime extension point in place, we could address another layer of the Kubernetes experience: exposing applications. The CCM architecture also defines a service controller for Kubernetes LoadBalancer services, giving us a place to address the next customer problem.

One of the capabilities customers expect from cloud-integrated Kubernetes is support for Service objects of type LoadBalancer. When a user creates one, Kubernetes asks the CCM’s service controller to provision the load balancer and publish its address in the Service status. There was just one problem: Oxide did not yet offer a native load balancer.

Oxide did, however, have floating IPs. Floating IPs are external addresses that can be attached to and detached from instances, exposing those instances outside the VPCs. I decided to use floating IPs for LoadBalancer services to work around Oxide’s lack of a native load balancer. The idea was that a floating IP would deliver traffic to a single Kubernetes node, and the Kubernetes Service would distribute that traffic to the appropriate pods.

Making that work required accounting for how Oxide floating IPs appear to an instance. They are transparent to the guest in two important ways. First, Oxide translates the destination address of inbound traffic to the instance’s internal IP before sending the traffic to the instance. Second, the instance has no network interface configured with the floating IP.

The resulting traffic flow looks like this:

Since traffic arrives at the instance using its internal IP, the service controller publishes two entries in status.loadBalancer.ingress:^[4]

The status entries look like this:

This architecture also makes the kubectl output look a little unusual:

Users see both the floating IP and the node’s internal IP in the EXTERNAL-IP column, even though only the floating IP is externally reachable. This is an imperfect abstraction, but it allows us to support a common Kubernetes workflow while waiting for a native Oxide load balancer.

This implementation currently supports externalTrafficPolicy: Cluster,^[5] which allows the selected node to forward traffic to a Service endpoint anywhere in the cluster. If that node disappears, the CCM moves the floating IP to another eligible node and updates the internal address in the Service status.

When we introduce a native Oxide load balancer, we can update the service controller to use it without changing the Kubernetes interface. Customers will continue creating the same LoadBalancer services, this time powered by the Oxide load balancer.

To install the Oxide CCM on your cluster, see our CCM guide to get started.

With clusters provisioned, reconciled with Oxide, and reachable from outside their VPCs, storage for stateful workloads became the next problem to address. Kubernetes users request persistent storage through PersistentVolumeClaim objects and expect a Container Storage Interface (CSI) plugin to create, attach, and mount the underlying volumes. Oxide had disks, but Kubernetes had no native way to manage their lifecycle.

Without an Oxide CSI plugin, customers could deploy a third-party Kubernetes storage system such as Longhorn. Longhorn provides its own CSI plugin that uses existing disks and handles its own replication. The only disk Oxide supported at the time was a distributed disk which replicates data across three sleds. Using Longhorn with distributed disks created substantial write fan-out that customers wanted to eliminate.

Oxide local disks gave us a way to reduce write amplification when using Longhorn. Local disks have no built-in replication, making them well suited for Longhorn. Our Rancher showcase uses this architecture to reduce write amplification. While this works today, customers still wanted a native Oxide CSI plugin.

My teammate Luiz wrote RFD 595 Oxide CSI Plugin, describing how an Oxide CSI plugin would work. Everything seemed straightforward on paper, a user creates a PersistentVolumeClaim and the CSI controller creates an Oxide distributed disk, attaches it to the node running the pod, and formats it for use. If the pod is rescheduled to another node, the CSI controller detaches the disk and reattaches it to the new node.

Further discussion and prototyping exposed a blocker. Oxide requires an instance to be stopped before attaching or detaching disks. Kubernetes, however, expects a CSI plugin to attach storage to a running node. Stopping the node would disrupt every other workload and could trigger cascading scheduling and attachment operations.

Before we can release our CSI plugin, we need to add support for disk hot-plug throughout the Oxide stack, from the hypervisor all the way up to the API. What began as a Kubernetes integration has turned into a project spanning multiple layers of the Oxide software stack.

Disk hot-plug and the Oxide CSI plugin remain under active development as of this writing. In the meantime, customers can use software such as Longhorn with Oxide local disks for dynamically provisioned persistent storage without stacking two layers of replication. When the native CSI plugin ships, customers will be able to use familiar Kubernetes storage APIs backed directly by Oxide distributed disks with replication and durability built in.

We now have a growing ecosystem of Kubernetes integrations that work well with one another. Rancher, Omni, and Cluster API solve Kubernetes provisioning, the Oxide CCM reconciles Kubernetes nodes with Oxide instances and handles LoadBalancer services. Customers already use some of these integrations in production, and we dogfood several in our own production workloads. Together, they provide a solid foundation to build on.

Our next step is to expand our dogfooding with the newly released Cluster API provider. Using it to provision and operate more of our clusters will test how these integrations work together day to day.

We still have plenty to build and polish. Our near-term work includes completing disk hot-plug and shipping the CSI plugin, adding autoscaling support, and extending the CCM service controller to support external subnets. Longer term, as we ship resource tagging, OIDC support, and native load balancing, we’ll extend our Kubernetes integrations to take advantage of them.

Building these integrations showed how the architectures of Kubernetes and Oxide complement one another. Kubernetes gives infrastructure providers standard extension points, while Oxide exposes infrastructure primitives through APIs. Oxide’s hardware and software co-design lets us address integration blockers at the layer where they belong and carry the necessary changes through the full stack.

This work also lets us exercise our SDKs and APIs from our customers' perspectives and turn customer friction into product improvements. That feedback loop is how we will continue growing this ecosystem. Customer needs shaped each integration in this post, and they will shape the next one, too.

To see the Cluster API and cloud controller manager integrations in action, watch the video below, in which I deploy a Kubernetes cluster on Oxide.

Over our last couple of releases, we’ve focused heavily on the network; namely, the introduction of end-to-end IPv6 plumbing and jumbo frames. Introducing these to a self-contained rack involved a lot of decisions, along with thorough investigation of the expected outcomes. We started by looking at the layers and knobs we could influence. From the wire up:

Each of these knobs helps one or more types of workload. The rest of this post walks them, roughly in the order a packet would encounter them, and notes what each one is good for.

Start with the most visible knob, and the one that took quite a bit of work to turn: the size of a packet.

Size matters because the network does a roughly fixed amount of work per packet, mostly independent of how many bytes the packet carries. Every packet has its headers parsed at several layers. Every packet runs the gauntlet of firewall rules, route lookups, address resolution, and connection-tracking state, much of it in software in the data plane. So if you want to move a megabyte and each packet holds 1500 bytes, you pay that per-packet tax around 700 times. Raise the packet to 9000 bytes and you pay it roughly 120 times. Same data, one-sixth the overhead.

Inside a rack, we have always taken advantage of this. Guests advertise TCP segmentation offload (TSO), so a guest hands us buffers up to 64 KB at a time over virtio. We carry them across a 9000-byte underlay, set the segment size to leave room for encapsulation, and let the sled’s physical NIC slice the big buffer back into wire-sized packets on the way out. This is tunneled TSO, and it is why traffic inside a VPC has effectively been riding jumbo frames for a long time, and why internal throughput has always been high.

The conservative part was at the rack’s edge. Traffic leaving the rack travels onto networks we do not control, and the safe assumption for an arbitrary external path is the Internet-standard 1500-byte MTU. So that is what we advertised to guests for external traffic.

The major public clouds make the same choice, but it is worth being precise about why, because the word “external” means something very different to them than it does to us. For a hyperscaler, the internal network is an entire availability zone: a building, or several, stitched together by a fabric they own. Almost everything a workload talks to lives inside that boundary, and the only thing that is genuinely “external” is the public Internet. Falling back to a 1500-byte MTU at that edge costs them little, because the edge is far away and rarely on the fast path.

For a single Oxide rack, the boundary sits much closer in. “External” is everything past one rack, and a lot of what lives there is not the public Internet at all: it is the rack in the next row, reached over a fabric the same operator owns and provisioned. That operator often does control the upstream and knows exactly what it can carry. Applying the hyperscaler’s conservative default to that link treats a known, fat, operator-owned path as if it were the open Internet, and leaves a great deal of performance on the table.

External jumbo frames let those operators opt out of the conservative default. The end-user-visible part is a single boolean on an instance. The interesting part is everything that boolean has to coordinate underneath it, because a frame size is only real if every layer of the stack agrees on it:

One boolean on the surface; a dozen coordinated turns underneath. It was great being able to bring this capability to our customers. It also meant we found the rough edges ourselves. For example, testing the jumbo path prototype surfaced an interrupt-handling issue in the virtual NIC when oversize frames arrived, and because the driver is ours, we tracked it down and fixed it before the feature shipped, rather than opening a ticket with a vendor and waiting.

If frame size is about how much work there is, this layer is about where that work happens. Two knobs live here, and neither is one a customer turns directly.

The first is offloads, which we touched on above. Offloads decide which layer does the heavy per-packet lifting. Segmentation offload lets a guest hand off a big buffer and let a lower layer chop it into wire-sized packets in bulk, rather than the guest doing it one packet at a time. Getting that right across both IPv4 and IPv6 is part of what it means to bring a protocol to full parity.

The second is CPU placement: which cores actually run the network stack’s work. A guest with many vCPUs is only as fast as the stack’s ability to spread its packet processing across cores. We refined how the virtual NIC’s worker threads are placed relative to a guest’s vCPUs, and the result is that throughput now scales cleanly as guests get larger. On the 64-core, 128-thread parts we test on, a 96-vCPU guest runs wider than the physical cores, onto the second hardware thread of many of them. With the worker threads placed to handle exactly that, a 96-vCPU guest sustains as much (or more) external throughput as a 64-vCPU one.

A single connection has limitations. Many of our tests on the 100G fabric reached that limit at around 60 Gbps, one way, per VM-pair, and it held steady whether we pushed 32, 64, or 128 streams through one connection, or ran several iperf3 processes side by side.

The way past it is not a bigger single flow; it is more flows. Spread the work across multiple connections with distinct 5-tuples, landing on different sleds, and ECMP fans them across the fabric automatically. Aggregate throughput then scales with the number of independent flows rather than bumping into a ceiling. This happens to match how production systems actually behave: many connections, not one giant one.

The practical guidance on this layer:

IPv6 is not a protocol we bolted on for compatibility. The rack’s underlay is IPv6: every sled-to-sled hop, every Geneve-encapsulated guest frame, rides IPv6 by default. It is the rack’s native tongue. So the work across these releases was not teaching the rack to speak IPv6, it was extending that first-class status all the way into the guest, end-to-end, and then tuning until there was no tax for using it.

In our tests on the 40G uplink, at the same MTU, on the same boundary path, IPv4 and IPv6 now deliver the same throughput: about 34.5 Gbps for IPv4 and about 34.9 for IPv6, per pair, at jumbo. There is no inherent IPv6 penalty, on a rack whose fabric was IPv6 from the start there should not be. If you are designing an IPv6-first environment, and many of our customers are, that distinction matters.

One IPv6 caveat: path-MTU discovery is less forgiving for IPv6 than for IPv4, so the jumbo path-audit rule from the frames section matters a little more when you are running IPv6.

Pull back and the through-line is not any single number. It is that a platform built from the ground up has a knob at every layer, and a team that owns every layer can keep turning them. The fabric, the frame size, the offloads, the CPU placement, the flow distribution, the protocol handling: these are not separate products bolted together; they are one system, and tuning one with full knowledge of the others is something only the owner of the whole stack can do.

External jumbo frames are the clearest example. What a user sees is one boolean. What it took was a coordinated change across the virtual NIC, the OPTE data path, the kernel driver, the routing protocol, and the control plane, plus a deliberately chosen frame size with the headroom reasoning written in, plus a decision about where in the system the knob even belongs.

That is also why this is not a one-time announcement. There is no final version of a network stack, or an end to tuning. There is the next layer, and the one under it, all the way to the core. We will keep peeling.

With Rust’s standard library maps, keys are stored separately from values. Let’s say you want to store a map of records keyed by an email address. With std::collections::BTreeMap, you might write something like:

This approach has what I consider to be a pretty major downside: the key (email) is not stored in the same struct as the value (the rest of the record). How would you handle this? One way is to pass around both the email and the user, for example with get_key_value:

As an extension, you could maybe have a struct which combines both at fetch time:

In practice, this gets quite unwieldy when you have lots of different types of records that need this kind of treatment.

Alternatively, you could duplicate the email across the key and the value:

But that has the risk that the emails stored in the key and the value fall out of sync.

iddqd provides a better alternative. With IdOrdMap, you can write something like:

At Oxide, this has proven to be an invaluable pattern: many of the records in our control plane are quite large (think database lookups), and iddqd is a great fit for them. It also comes with several other features that directly address pain points we’ve dealt with at Oxide. A few worth calling out:

Like many of our other crates, iddqd is built for Oxide’s needs but is generally useful to the Rust community. You’re welcome to use it in your own projects as well.

Before I move on, I want to talk about what it means for unsafe to exist in a memory-safe language. The big concern is undefined behavior (UB): a program behaves in an unpredictable way because core assumptions made by the language or compiler have been violated. Rust calls an abstraction sound if no safe code can use it to cause UB, and unsound otherwise.

The vast majority of Rust code is safe, which (assuming that any unsafe used by the safe code is sound) means that no UB can occur. However, due to the fundamental undecidability of static analysis (see Rice’s theorem), it is impossible for the Rust compiler—or any kind of algorithm that terminates—to accept all programs without UB and reject all programs with it. Therefore, when writing such an algorithm, its authors have to make a decision: do they reject some programs without UB, accept some programs with UB, or both? The Rust compiler does the first: within the context of safe Rust, it rejects all programs with UB but also some without UB. (This is the correct choice!)

What if your program is in the no-man’s-land where it gets rejected even though you know it doesn’t have UB? To express those kinds of programs, the Rust compiler provides an escape hatch: the unsafe keyword. By writing it, the author takes responsibility for soundness: they vouch that no safe code can use this to cause UB, and that the compiler should trust them.

What does it mean for some unsafe code to not have UB? Informally, it must follow the rules of Rust—all the rules that the compiler proves for safe Rust. Some examples of these rules are:

The rules of Rust are very hard to reason about! I consider them to be significantly harder than C, for example, particularly due to the rules around mutable aliasing^[2]. So judicious use of unsafe Rust usually involves encapsulating it behind a safe abstraction.

iddqd's architecture is pretty straightforward: it uses a combination of a list of items (internally called an ItemSet) and a table with indexes into the slots. For example, IdHashMap<T> consists of:

For example, an IdHashMap holding three items might look like the diagram below. The top row is the hash table over ItemIndex; the middle row is the Vec<ItemSlot<T>>.

Adding a new item consists of checking free_head to see if any Vacant slots are available. If so, we read the next index out of that Vacant slot, overwrite the slot with Occupied(…​), then set free_head to the next value we read. Otherwise, we push a new Occupied slot to the end. Finally, the hash table is updated, fetching the key and computing the hash, then recording the new index in the hash table.

Starting from the state above: Let’s say we want to insert a new item t₃. Calling insert(t₃) recycles slot 2 from the free chain, advances free_head to 1, and records the new hash entry:

Removing an item given a key involves this process in reverse: first, consult the hash table using the hash computed from the key, and remove the found index from the hash table. With the index in hand, replace the ItemSlot with Vacant, setting the internal next to the current value of free_head. Finally, set free_head to the index that was just replaced with Vacant.

Continuing from the post-insert state above: Calling remove(k₀) drops the hash entry for k₀, marks slot 3 as Vacant with its next pointing at the old free_head (1), and updates free_head to 3:

This pattern generalizes to the other map types:

iddqd has a few different kinds of unsafe Rust within it. Most uses of unsafe follow well-established patterns also used within Rust’s standard library—these are not dependent on user code, and a sufficiently smart borrow checker would make these uses of unsafe unnecessary. But there are a few that stand out.

Perhaps the most challenging one to reason about is mutable iteration over items. For IdOrdMap::iter_mut, iteration is ordered by the key. One might start off by writing something like:

But this would immediately return an error:

The reason this occurs is that Rust’s iterators require that the returned values do not borrow from the iterator itself^[6]. Many uses of iterators do not need this capability—in particular, most for loops only operate on one item at a time:

But Rust’s iterators also let you create values that outlive the iterator, via e.g. the collect combinator:

items_mut cannot outlive items itself, but can outlive the iterator returned by items.iter_mut().

Crucially, this relies on knowing that all of the &mut T returned by the iterator are disjoint—or, in other words, that the iterator never yields the same value twice. (Remember that one of the rules of Rust is that there must never be multiple &mut references to the same memory.) But the Rust borrow checker just sees a succession of accesses into a list, and it has no way of knowing that the indexes are all different. If the human writing the code has knowledge that all of the indexes are different, though, they can use an unsafe pattern called lifetime extension to tell the borrow checker to let this code through:

Because iddqd is such a foundational data structure at Oxide, we go through great lengths to validate its correctness, including the soundness of its abstractions. There is a great deal of analytical reasoning performed by experienced Rust authors and reviewers, as described in the section above. But we also empirically validate iddqd along several different dimensions. None of the layers is sufficient by itself, but by rigorously doing all of them we can be more confident that iddqd is correct.

The layers of validation we apply are:

In this section, I’ll provide a brief overview of each layer. Click through the links to see more details and code samples.

Writing unsafe generic Rust is extremely difficult. Each invariant that the unsafe code relies upon has to be carefully upheld over arbitrary trait implementations, including adversarial ones. This post covers one such example: how an Ord implementation could be carefully written to trick the map into creating mutable aliases to the same memory, and how we fixed it.

No single technique can hope to catch all bugs, which is why iddqd uses several layers of validation. Humans carefully reasoning about every line of unsafe code goes a long way; example-based, pathological, and randomized tests provide empirical evidence; and frontier models can find new and surprising ways to break code that humans might not have thought of. This is a lot of machinery, but at Oxide we believe that for foundational infrastructure, this level of rigor is justified. And if you agree, we’re hiring!

Diagrams courtesy Ben Leonard.