The first gap we tackled was provisioning. Our immediate goal was to unblock the customer who had submitted the Rancher node driver pull request. Working through their use case would also give us firsthand experience creating Kubernetes clusters on Oxide and help us uncover the next problems to solve.

No single provisioning approach fit all customers' workflows, so we ended up publishing three integrations.

Provisioning integrations create and manage Oxide instances, but they do not reconcile those instances with Kubernetes Node objects. Without that reconciliation, a cluster could not reliably determine whether an unreachable Kubernetes node was temporarily unavailable or whether its backing Oxide instance had been deleted.

We needed a component that ran in each cluster, spoke to the Oxide API, and continuously reconciled Oxide infrastructure with Kubernetes state. Kubernetes provides a standard extension point for this purpose: the cloud controller manager (CCM). A CCM lets infrastructure-specific controllers integrate Kubernetes resources with an infrastructure provider’s API without adding provider-specific code to Kubernetes itself.

We built the Oxide cloud controller manager to connect Kubernetes with Oxide. Its node controller keeps Kubernetes Node objects synchronized with their backing Oxide instances, recording details such as instance IDs and network addresses, and reporting whether each instance is running, shut down, or no longer exists. Kubernetes uses this information to initialize nodes and safely remove them when their backing instances are deleted.

The CCM does not create instances or provision clusters. That remains the job of provisioning integrations such as the Rancher node driver, the Omni infrastructure provider, and CAPOx. Instead, it provides a runtime integration shared across those provisioning workflows.

Importantly, building the CCM gave us a durable extension point inside each cluster. As Oxide evolves, we can add new infrastructure-aware controllers to the CCM rather than update every provisioning integration.

With that runtime extension point in place, we could address another layer of the Kubernetes experience: exposing applications. The CCM architecture also defines a service controller for Kubernetes LoadBalancer services, giving us a place to address the next customer problem.

One of the capabilities customers expect from cloud-integrated Kubernetes is support for Service objects of type LoadBalancer. When a user creates one, Kubernetes asks the cloud provider’s service controller to provision the necessary infrastructure and publish its address in the Service status. There was just one problem: Oxide did not yet offer a native load balancer.

Oxide did, however, have floating IPs. Floating IPs are addresses from a rack’s external IP pools that can be attached to and detached from instances, making those instances reachable from outside their VPCs. Using floating IPs offered a way to unblock LoadBalancer services. A floating IP would deliver traffic to a single Kubernetes node, and the Kubernetes Service dataplane could distribute that traffic to the appropriate pods.

Making that work required accounting for how Oxide floating IPs appear to an instance. They are transparent to the guest in two important ways. First, Oxide translates the destination address of inbound traffic to the instance’s internal IP before sending the traffic to the instance. Second, the instance has no network interface configured with the floating IP.

The resulting traffic flow looks like this:

That address translation created a subtle integration problem. The Kubernetes Service dataplane needed to treat the node’s internal IP as a Service frontend because that was the destination address packets actually carried when they reached the guest. The service controller therefore publishes two entries in status.loadBalancer.ingress:^[4]

The status entries look like this:

As a result, the kubectl output looks a little unusual:

Users see both the floating IP and the node’s internal IP in the EXTERNAL-IP column, even though only the floating IP is externally reachable. This is an imperfect abstraction, but it allows us to support a common Kubernetes workflow while waiting for a native Oxide load balancer.

This implementation currently supports externalTrafficPolicy: Cluster,^[5] which allows the selected node to forward traffic to a Service endpoint anywhere in the cluster. If that node disappears, the CCM moves the floating IP to another eligible node and updates the internal address in the Service status.

When Oxide introduces a native load-balancing service, we can update the service controller to use it without changing the Kubernetes interface. Customers will continue creating the same LoadBalancer services and only the infrastructure behind them will change.

To install the Oxide CCM on your cluster, see our CCM guide to get started.

With clusters provisioned, reconciled with Oxide, and reachable from outside their VPCs, storage for stateful workloads became the next layer to address. Kubernetes users request persistent storage through PersistentVolumeClaim objects and expect a Container Storage Interface (CSI) driver to create, attach, and mount the underlying volumes. Oxide had disks, but Kubernetes had no native way to manage their lifecycle.

Without an Oxide CSI driver, customers could deploy a third-party Kubernetes storage system such as Longhorn. Longhorn provides its own CSI driver and replicates data across disks attached to Kubernetes workers. However, using Longhorn meant backing its replicas with Oxide distributed disks, which already store three replicas on distinct sleds.

Layering one replicated storage system on another can create substantial write fan-out. When a three-replica Longhorn volume is backed by three-way-replicated Oxide distributed disks, one application write can fan out to as many as nine disk writes. The exact physical write amplification depends on the workload and configuration, but customers wanted to avoid that duplicated replication.

The introduction of Oxide local disks provided a way to remove the second layer of replication. Local disks have no built-in replication and remain tied to their sled, making them well suited to systems such as Longhorn that replicate data across Kubernetes nodes. Our Rancher showcase uses this approach today. It avoids stacking two replicated storage systems, though Longhorn still manages the storage lifecycle rather than a native Oxide integration.

For a native integration, my teammate Luiz wrote RFD 595 Oxide CSI Plugin. The workflow seemed straightforward on paper. When a user creates a PersistentVolumeClaim, the CSI controller creates an Oxide distributed disk. After Kubernetes schedules the pod, the controller attaches that disk to the selected Oxide instance, and the CSI node plugin formats and mounts it for the pod. If the pod is rescheduled onto another node, the controller detaches the disk and reattaches it to the new node.

Prototyping that workflow immediately exposed a blocker. Oxide requires an instance to be stopped before attaching or detaching a disk. Kubernetes, however, expects a CSI driver to attach storage to a running worker after scheduling a pod. Stopping the worker would disrupt every other workload on the node and could trigger cascading scheduling and attachment operations.

Before we can release our CSI plugin, we need to add support for disk hot-plug throughout the Oxide stack, from the hypervisor all the way up to the API. What began as a Kubernetes integration has turned into a project spanning multiple layers of the Oxide software stack.

Disk hot-plug and the Oxide CSI plugin remain under active development as of this writing. In the meantime, customers can use software such as Longhorn with Oxide local disks for dynamically provisioned persistent storage without stacking two layers of replication. When the native CSI plugin ships, customers will be able to use familiar Kubernetes storage APIs backed directly by Oxide distributed disks with replication and durability built in.

The result is not a single Kubernetes integration but a growing ecosystem. Rancher, Omni, and Cluster API provide different paths for provisioning, while the Oxide CCM provides a shared runtime integration for node reconciliation and LoadBalancer services. Customers already use some of these integrations in production, and we dogfood several in our own production workloads. Together, they provide a solid foundation to build on.

Our next step is to expand our dogfooding with the newly released Cluster API provider. Using it to provision and operate more of our clusters will test how these integrations work together day to day.

We still have plenty to build and polish. Our near-term work includes completing disk hot-plug and shipping the CSI plugin, adding autoscaling support, and extending the CCM service controller to support external subnets. Longer term, as we ship resource tagging, OIDC support, and native load balancing, we’ll extend our Kubernetes integrations to take advantage of them.

Building these integrations showed how the architectures of Kubernetes and Oxide complement one another. Kubernetes gives infrastructure providers standard extension points, while Oxide exposes infrastructure primitives through APIs. Oxide’s hardware and software co-design lets us address integration blockers at the layer where they belong and carry the necessary changes through the full stack.

This work also lets us exercise our SDKs and APIs from our customers' perspectives and turn customer friction into product improvements. That feedback loop is how we will continue growing this ecosystem. Customer needs shaped each integration in this post, and they will shape the next one, too.

To see the Cluster API and cloud controller manager integrations in action, watch the video below, in which I deploy a Kubernetes cluster on Oxide.

Over our last couple of releases, we’ve focused heavily on the network; namely, the introduction of end-to-end IPv6 plumbing and jumbo frames. Introducing these to a self-contained rack involved a lot of decisions, along with thorough investigation of the expected outcomes. We started by looking at the layers and knobs we could influence. From the wire up:

Each of these knobs helps one or more types of workload. The rest of this post walks them, roughly in the order a packet would encounter them, and notes what each one is good for.

Start with the most visible knob, and the one that took quite a bit of work to turn: the size of a packet.

Size matters because the network does a roughly fixed amount of work per packet, mostly independent of how many bytes the packet carries. Every packet has its headers parsed at several layers. Every packet runs the gauntlet of firewall rules, route lookups, address resolution, and connection-tracking state, much of it in software in the data plane. So if you want to move a megabyte and each packet holds 1500 bytes, you pay that per-packet tax around 700 times. Raise the packet to 9000 bytes and you pay it roughly 120 times. Same data, one-sixth the overhead.

Inside a rack, we have always taken advantage of this. Guests advertise TCP segmentation offload (TSO), so a guest hands us buffers up to 64 KB at a time over virtio. We carry them across a 9000-byte underlay, set the segment size to leave room for encapsulation, and let the sled’s physical NIC slice the big buffer back into wire-sized packets on the way out. This is tunneled TSO, and it is why traffic inside a VPC has effectively been riding jumbo frames for a long time, and why internal throughput has always been high.

The conservative part was at the rack’s edge. Traffic leaving the rack travels onto networks we do not control, and the safe assumption for an arbitrary external path is the Internet-standard 1500-byte MTU. So that is what we advertised to guests for external traffic.

The major public clouds make the same choice, but it is worth being precise about why, because the word “external” means something very different to them than it does to us. For a hyperscaler, the internal network is an entire availability zone: a building, or several, stitched together by a fabric they own. Almost everything a workload talks to lives inside that boundary, and the only thing that is genuinely “external” is the public Internet. Falling back to a 1500-byte MTU at that edge costs them little, because the edge is far away and rarely on the fast path.

For a single Oxide rack, the boundary sits much closer in. “External” is everything past one rack, and a lot of what lives there is not the public Internet at all: it is the rack in the next row, reached over a fabric the same operator owns and provisioned. That operator often does control the upstream and knows exactly what it can carry. Applying the hyperscaler’s conservative default to that link treats a known, fat, operator-owned path as if it were the open Internet, and leaves a great deal of performance on the table.

External jumbo frames let those operators opt out of the conservative default. The end-user-visible part is a single boolean on an instance. The interesting part is everything that boolean has to coordinate underneath it, because a frame size is only real if every layer of the stack agrees on it:

One boolean on the surface; a dozen coordinated turns underneath. It was great being able to bring this capability to our customers. It also meant we found the rough edges ourselves. For example, testing the jumbo path prototype surfaced an interrupt-handling issue in the virtual NIC when oversize frames arrived, and because the driver is ours, we tracked it down and fixed it before the feature shipped, rather than opening a ticket with a vendor and waiting.

If frame size is about how much work there is, this layer is about where that work happens. Two knobs live here, and neither is one a customer turns directly.

The first is offloads, which we touched on above. Offloads decide which layer does the heavy per-packet lifting. Segmentation offload lets a guest hand off a big buffer and let a lower layer chop it into wire-sized packets in bulk, rather than the guest doing it one packet at a time. Getting that right across both IPv4 and IPv6 is part of what it means to bring a protocol to full parity.

The second is CPU placement: which cores actually run the network stack’s work. A guest with many vCPUs is only as fast as the stack’s ability to spread its packet processing across cores. We refined how the virtual NIC’s worker threads are placed relative to a guest’s vCPUs, and the result is that throughput now scales cleanly as guests get larger. On the 64-core, 128-thread parts we test on, a 96-vCPU guest runs wider than the physical cores, onto the second hardware thread of many of them. With the worker threads placed to handle exactly that, a 96-vCPU guest sustains as much (or more) external throughput as a 64-vCPU one.

A single connection has limitations. Many of our tests on the 100G fabric reached that limit at around 60 Gbps, one way, per VM-pair, and it held steady whether we pushed 32, 64, or 128 streams through one connection, or ran several iperf3 processes side by side.

The way past it is not a bigger single flow; it is more flows. Spread the work across multiple connections with distinct 5-tuples, landing on different sleds, and ECMP fans them across the fabric automatically. Aggregate throughput then scales with the number of independent flows rather than bumping into a ceiling. This happens to match how production systems actually behave: many connections, not one giant one.

The practical guidance on this layer:

IPv6 is not a protocol we bolted on for compatibility. The rack’s underlay is IPv6: every sled-to-sled hop, every Geneve-encapsulated guest frame, rides IPv6 by default. It is the rack’s native tongue. So the work across these releases was not teaching the rack to speak IPv6, it was extending that first-class status all the way into the guest, end-to-end, and then tuning until there was no tax for using it.

In our tests on the 40G uplink, at the same MTU, on the same boundary path, IPv4 and IPv6 now deliver the same throughput: about 34.5 Gbps for IPv4 and about 34.9 for IPv6, per pair, at jumbo. There is no inherent IPv6 penalty, on a rack whose fabric was IPv6 from the start there should not be. If you are designing an IPv6-first environment, and many of our customers are, that distinction matters.

One IPv6 caveat: path-MTU discovery is less forgiving for IPv6 than for IPv4, so the jumbo path-audit rule from the frames section matters a little more when you are running IPv6.

Pull back and the through-line is not any single number. It is that a platform built from the ground up has a knob at every layer, and a team that owns every layer can keep turning them. The fabric, the frame size, the offloads, the CPU placement, the flow distribution, the protocol handling: these are not separate products bolted together; they are one system, and tuning one with full knowledge of the others is something only the owner of the whole stack can do.

External jumbo frames are the clearest example. What a user sees is one boolean. What it took was a coordinated change across the virtual NIC, the OPTE data path, the kernel driver, the routing protocol, and the control plane, plus a deliberately chosen frame size with the headroom reasoning written in, plus a decision about where in the system the knob even belongs.

That is also why this is not a one-time announcement. There is no final version of a network stack, or an end to tuning. There is the next layer, and the one under it, all the way to the core. We will keep peeling.

With Rust’s standard library maps, keys are stored separately from values. Let’s say you want to store a map of records keyed by an email address. With std::collections::BTreeMap, you might write something like:

This approach has what I consider to be a pretty major downside: the key (email) is not stored in the same struct as the value (the rest of the record). How would you handle this? One way is to pass around both the email and the user, for example with get_key_value:

As an extension, you could maybe have a struct which combines both at fetch time:

In practice, this gets quite unwieldy when you have lots of different types of records that need this kind of treatment.

Alternatively, you could duplicate the email across the key and the value:

But that has the risk that the emails stored in the key and the value fall out of sync.

iddqd provides a better alternative. With IdOrdMap, you can write something like:

At Oxide, this has proven to be an invaluable pattern: many of the records in our control plane are quite large (think database lookups), and iddqd is a great fit for them. It also comes with several other features that directly address pain points we’ve dealt with at Oxide. A few worth calling out:

Like many of our other crates, iddqd is built for Oxide’s needs but is generally useful to the Rust community. You’re welcome to use it in your own projects as well.

Before I move on, I want to talk about what it means for unsafe to exist in a memory-safe language. The big concern is undefined behavior (UB): a program behaves in an unpredictable way because core assumptions made by the language or compiler have been violated. Rust calls an abstraction sound if no safe code can use it to cause UB, and unsound otherwise.

The vast majority of Rust code is safe, which (assuming that any unsafe used by the safe code is sound) means that no UB can occur. However, due to the fundamental undecidability of static analysis (see Rice’s theorem), it is impossible for the Rust compiler—or any kind of algorithm that terminates—to accept all programs without UB and reject all programs with it. Therefore, when writing such an algorithm, its authors have to make a decision: do they reject some programs without UB, accept some programs with UB, or both? The Rust compiler does the first: within the context of safe Rust, it rejects all programs with UB but also some without UB. (This is the correct choice!)

What if your program is in the no-man’s-land where it gets rejected even though you know it doesn’t have UB? To express those kinds of programs, the Rust compiler provides an escape hatch: the unsafe keyword. By writing it, the author takes responsibility for soundness: they vouch that no safe code can use this to cause UB, and that the compiler should trust them.

What does it mean for some unsafe code to not have UB? Informally, it must follow the rules of Rust—all the rules that the compiler proves for safe Rust. Some examples of these rules are:

The rules of Rust are very hard to reason about! I consider them to be significantly harder than C, for example, particularly due to the rules around mutable aliasing^[2]. So judicious use of unsafe Rust usually involves encapsulating it behind a safe abstraction.

iddqd's architecture is pretty straightforward: it uses a combination of a list of items (internally called an ItemSet) and a table with indexes into the slots. For example, IdHashMap<T> consists of:

For example, an IdHashMap holding three items might look like the diagram below. The top row is the hash table over ItemIndex; the middle row is the Vec<ItemSlot<T>>.

Adding a new item consists of checking free_head to see if any Vacant slots are available. If so, we read the next index out of that Vacant slot, overwrite the slot with Occupied(…​), then set free_head to the next value we read. Otherwise, we push a new Occupied slot to the end. Finally, the hash table is updated, fetching the key and computing the hash, then recording the new index in the hash table.

Starting from the state above: Let’s say we want to insert a new item t₃. Calling insert(t₃) recycles slot 2 from the free chain, advances free_head to 1, and records the new hash entry:

Removing an item given a key involves this process in reverse: first, consult the hash table using the hash computed from the key, and remove the found index from the hash table. With the index in hand, replace the ItemSlot with Vacant, setting the internal next to the current value of free_head. Finally, set free_head to the index that was just replaced with Vacant.

Continuing from the post-insert state above: Calling remove(k₀) drops the hash entry for k₀, marks slot 3 as Vacant with its next pointing at the old free_head (1), and updates free_head to 3:

This pattern generalizes to the other map types:

iddqd has a few different kinds of unsafe Rust within it. Most uses of unsafe follow well-established patterns also used within Rust’s standard library—these are not dependent on user code, and a sufficiently smart borrow checker would make these uses of unsafe unnecessary. But there are a few that stand out.

Perhaps the most challenging one to reason about is mutable iteration over items. For IdOrdMap::iter_mut, iteration is ordered by the key. One might start off by writing something like:

But this would immediately return an error:

The reason this occurs is that Rust’s iterators require that the returned values do not borrow from the iterator itself^[6]. Many uses of iterators do not need this capability—in particular, most for loops only operate on one item at a time:

But Rust’s iterators also let you create values that outlive the iterator, via e.g. the collect combinator:

items_mut cannot outlive items itself, but can outlive the iterator returned by items.iter_mut().

Crucially, this relies on knowing that all of the &mut T returned by the iterator are disjoint—or, in other words, that the iterator never yields the same value twice. (Remember that one of the rules of Rust is that there must never be multiple &mut references to the same memory.) But the Rust borrow checker just sees a succession of accesses into a list, and it has no way of knowing that the indexes are all different. If the human writing the code has knowledge that all of the indexes are different, though, they can use an unsafe pattern called lifetime extension to tell the borrow checker to let this code through:

Because iddqd is such a foundational data structure at Oxide, we go through great lengths to validate its correctness, including the soundness of its abstractions. There is a great deal of analytical reasoning performed by experienced Rust authors and reviewers, as described in the section above. But we also empirically validate iddqd along several different dimensions. None of the layers is sufficient by itself, but by rigorously doing all of them we can be more confident that iddqd is correct.

The layers of validation we apply are:

In this section, I’ll provide a brief overview of each layer. Click through the links to see more details and code samples.

Writing unsafe generic Rust is extremely difficult. Each invariant that the unsafe code relies upon has to be carefully upheld over arbitrary trait implementations, including adversarial ones. This post covers one such example: how an Ord implementation could be carefully written to trick the map into creating mutable aliases to the same memory, and how we fixed it.

No single technique can hope to catch all bugs, which is why iddqd uses several layers of validation. Humans carefully reasoning about every line of unsafe code goes a long way; example-based, pathological, and randomized tests provide empirical evidence; and frontier models can find new and surprising ways to break code that humans might not have thought of. This is a lot of machinery, but at Oxide we believe that for foundational infrastructure, this level of rigor is justified. And if you agree, we’re hiring!

Diagrams courtesy Ben Leonard.