Migrating K3s from kube-router to Cilium, live

Every app on velixir runs in a Kubernetes namespace with a NetworkPolicy around it. Those policies decide where each app can send traffic. A free app can only talk to the internet on the ports its plan allows, and no app can reach into another customer's services or the cluster's internal network. Until this month those rules were enforced by kube-router, the policy controller that ships inside K3s.

As the cluster grew, we found a gap between a pod starting and its policy existing. On a small cluster it was too short to measure, but it grew with every app we added, until a new pod could run for a few seconds with no policy around it at all.

This post is about replacing the network underneath a live, multi-tenant cluster with Cilium, in place. It covers the rehearsals, a bug in how two programs share 32 bits of packet metadata, and the one problem the rehearsals didn't catch.

Why does kube-router leave new pods open?

kube-router turns NetworkPolicies into iptables rules and ipsets. When a pod starts, the rules that should cover it don't exist yet. They appear on kube-router's next sync, which rebuilds the policy chains for the whole node, and every policy you add makes that sync a little longer. Until it finishes, a new pod gets the default, and for a pod nobody has written rules for, the default is to let traffic through.

Most clusters never notice, because the pods that start in them are trusted. On a platform where anyone can deploy code, every new pod has to be treated as untrusted from its first packet. Our end-to-end tests caught it when a freshly deployed probe app made outbound connections in its first seconds that its egress policy should have refused. Once the policy landed it was enforced as written, so the gap was timing, not a missing rule, but a few seconds of unenforced egress is still not what we'd promised.

Cilium compiles policy into eBPF programs attached to each pod's network interface, and sets them up while it creates the pod's network, keyed on an identity derived from the pod's labels. There's no node-wide sync to wait for. The gate described below checks that a forbidden path is shut from a new pod's very first packet, and by the end of the migration it passed on every node.

The rules we set ourselves

We wanted to do it in place, with no second cluster to move everything across to. Apps, their volumes and their managed databases would stay on the nodes they were already on, and no host would be rebooted. Customers shouldn't see anything beyond what a normal deploy already costs them, so a zero-downtime app stays zero-downtime and an app with a volume gets the few seconds a redeploy takes. Every step had to prove itself before the next one started, and every step needed a way back.

How two CNIs share one cluster

Cilium documents a path for this. You install it alongside the existing CNI with its own pod range (we used 10.44.0.0/16, next to flannel's 10.42.0.0/16) and its own VXLAN port, and tell it not to write a CNI config yet. At that point nothing changes. Then, one node at a time, a CiliumNodeConfig that matches a node label tells that node's agent to write 05-cilium.conflist:

kind: CiliumNodeConfig
spec:
  nodeSelector:
    matchLabels: { io.cilium.migration/cilium-default: "true" }
  defaults:
    write-cni-conf-when-ready: /host/etc/cni/net.d/05-cilium.conflist
    cni-exclusive: "false"

The container runtime uses whichever config sorts first, so from that moment new pods on that node get a Cilium address while existing pods keep their flannel address and carry on. Because cni-exclusive is off, flannel's config stays on disk, and rolling a node back means removing the label. (That detail about sort order comes back later.)

Once every node is switched, you restart the workloads in a sensible order so each comes back on Cilium, then turn flannel and kube-router off and clean up what they leave behind. That's where the documented path ends, and where our own work started.

What the first rehearsal found

I built a copy of the cluster in k3d with the same K3s release, our real tier policies, a workload that behaves like ours, and a curl loop hitting every app through the edge five times a second. I ran the migration against it from start to finish and wrote down everything that broke.

Both networks masquerade at each other. Cilium SNATs a migrated pod's traffic to flannel pods to the node's address, and flannel does the same in the other direction, so each policy engine sees "a node" instead of Traefik or the app. In the rehearsal, the moment Traefik moved to Cilium, every app returned 502. The fix is a small DaemonSet I'll call the shim, which adds a no-SNAT rule between the two pod ranges at the top of nat POSTROUTING.

Then routing went asymmetric. With SNAT gone, a SYN from a flannel pod to a Cilium pod left over flannel's VXLAN, and the SYN-ACK came back through Cilium's tunnel. The host's conntrack saw half a connection, marked the ACK invalid, and kube-proxy dropped it. The shim now routes flannel-to-Cilium traffic over flannel in both directions.

Every rolled app failed its readiness probe. kube-router accepts node-to-pod traffic before Cilium gets a chance to recognise it as coming from the host, so Cilium labelled the kubelet's probe as world, which our ingress policies don't let in. The shim sets Cilium's own host mark on that traffic in mangle OUTPUT, as Cilium would have done itself if it owned the node.

ipBlock doesn't match pods. Some of our tiers reached their managed databases through ipBlock: 10.0.0.0/8. Under kube-router that matches pod addresses; under Cilium, ipBlock is for things outside the cluster and never matches a pod. Every one of those apps would have lost its database the moment the database moved. The tiers now also name pods with namespaceSelector: {}, which means the same thing to both engines, and a test keeps the two policy sets identical.

Turning flannel off moves the CNI directory. With flannel-backend: none, K3s writes a containerd config with no CNI section, so containerd falls back to /etc/cni/net.d and /opt/cni/bin instead of K3s' own paths. A node restarted while Cilium still wrote to the old paths came up NotReady, so the final Cilium config moves to the standard paths before any K3s restart.

By the end there were eleven of these, each either fixed or covered by a gate. The rehearsal then ran clean, with no failed requests through the install, the node switches and the zero-downtime apps' rollout, and 13 seconds for a volume app, the same as one of its deploys.

The gate

The gate that runs after each node switch caught the first real bug, so it's worth describing.

For each egress tier, it starts a fresh pod pinned to the node that just switched. From its first instant, every half second, that pod tries every path that matters. It tests the cluster's internal services, its own team's database, another tenant's app on another node, another tenant's app on the same node that's still on flannel, the internet on 443 and on 25, DNS, and a kubelet readiness probe coming in through the app's ingress policy.

Passing is asymmetric:

• A path that should be blocked must be blocked from the first sample. Open for half a second is a fail.
• A path that should be allowed may open late, within a grace period, because kube-router only admits a new client to a flannel-era pod on its next sync. Once open, it must stay open.

If anything is wrong, the node stays cordoned until it's rolled back.

The bug the rehearsal couldn't see

The first production node switch failed its gate. The gate keeps a tenant of its own on every node, a test app whose ingress policy should refuse everything from a stranger, and some of our probe pods got through to it. Not all of them and not for long, but a tenant being reachable at all is a fail, so the switch was rolled back. The node stayed cordoned throughout, so the only pods that could have used the gap were our own probes, and no customer app was ever on the Cilium side of it.

The rehearsal had never shown this. The obvious difference was scale. The rehearsal cluster had the right shape but nowhere near production's volume of namespaces, policies and running apps. So I wrote a loader that fills a k3d cluster until it looks like production from the network's point of view, and ran the migration again.

The failure came back. I started more probe pods and lined their results up against everything Cilium knew about them, and one column matched perfectly, the pod's security identity. Every pod with an odd identity number got through, and every pod with an even one was blocked.

32 bits, two owners

Linux gives each packet a 32-bit mark, a scratch field that netfilter rules and eBPF programs can read and write as the packet moves through the host. Nothing allocates its bits, so each networking component picks some and hopes nobody else did.

Cilium writes the sending pod's numeric identity into the upper 16 bits, so that later stages on the same host know who sent the packet without looking it up again.

kube-router keeps its policy verdict in the mark too:

Its final rule rejects only packets without 0x10000. That's bit 16, the lowest bit of the upper half, which makes it the lowest bit of Cilium's identity.

So a packet from a Cilium pod with an odd identity arrived at kube-router's filter already carrying "a policy allowed this", and kube-router never evaluated the tenant's ingress policy. It found its own verdict on the packet, written there by a different program, and trusted it.

The small rehearsal missed it because its handful of identities happened to be even, whereas with enough identities roughly half are odd. Other nodes were never affected, because the mark stays with the packet inside one host and doesn't cross the tunnel.

The fix is one rule

The two programs only meet on a node during the migration, and only for traffic from a Cilium pod to a flannel pod. So the shim clears those two bits for exactly that traffic, in mangle FORWARD, which runs before kube-router's filter rules:

iptables -t mangle -I FORWARD \
  -s 10.44.0.0/16 -d 10.42.0.0/16 \
  -m comment --comment velixir-cilium-migration-clear-identity \
  -j MARK --and-mark 0xfffcffff

kube-router now starts from a clean mark and makes its own decision. Cilium loses nothing, because by the time a packet reaches the host's FORWARD chain on its way to a flannel pod, Cilium has finished with it. With the rule in place, eight fresh probe pods, four of them with odd identities, were all blocked.

A rule that went missing every two seconds

With the mark fixed, a long run at scale still failed once, on a single probe sample.

The shim runs in a loop. Every two seconds it checks its rules are present and in the right position, because kube-router and Cilium both rewrite these chains and will happily push our rules down or out. The check compared the rule text the shim wanted with the text iptables listed.

iptables doesn't store --and-mark as an and-mark. It stores the equivalent --set-xmark with a value and mask, and lists it that way:

-j MARK --set-xmark 0x0/0x30000

The text never matched, so every two seconds the shim decided its rule was missing, deleted "the old one" and inserted a fresh copy, leaving a moment in between with no rule at all. Nearly every packet missed that moment; the failing sample was one that didn't.

The shim now identifies its rules by their comment, never by their text, and when it has to move one it inserts the new copy before deleting the old, so the rule is never absent. It also leaves its rules alone when its own DaemonSet is being updated rather than removed. The first version cleaned up on shutdown, which opened a gap during every rollout.

Making sure the gate tests the right thing

One more failure turned out to be the gate's own fault. It created its same-node tenant just before switching a node, then probed it, but a brand-new flannel pod is exactly the pod kube-router hasn't protected yet. The gate was partly measuring the bug we were migrating away from. The tenants now exist well before any switch, and the gate only probes one that has been running for a while.

The same principle caught a third-party dependency. The gate's internet check pointed at a public port-testing service, which began refusing some of our connections once we probed it hard enough. A gate that fails because someone else is rate-limiting you soon gets ignored, so the common case now goes to an endpoint built to take that load.

With that, the whole migration passed at production scale, end to end, including a final gate with no grace period at all.

The one that reached production

One problem did get past the rehearsals. On 7 November, partway through moving workloads on the real cluster, requests to web apps in the EU Central region started failing at our edge with a 502, and it took about 45 minutes to restore traffic. The apps themselves kept running throughout and no data was affected. The incident is on our status page.

What made it slow to pin down was that every check we ran looked healthy. Traefik answered, the apps answered, and each node's public address returned 200. Our edge doesn't use those addresses, though. It reaches the cluster over a private tunnel, at addresses that aren't the Service's load balancer IPs. kube-proxy handles the public addresses itself and sends that traffic straight to Traefik, whereas traffic to the private addresses goes through K3s' ServiceLB, which publishes ports 80 and 443 on every node using a pod with a hostPort. The CNI portmap plugin wires a hostPort up as a DNAT rule. The rehearsal's edge came in through the public path, so this one had never been exercised.

The first thing we found was that our Cilium config didn't chain portmap, so ServiceLB pods that moved to Cilium got no port mapping. Fixing that didn't bring traffic back, and neither did moving ServiceLB back to flannel, which pointed at something stateful on the hosts.

It was in the hostPort chain on every node:

-A CNI-HOSTPORT-DNAT ... "dnat name: \"cbr0\""    --dports 80,443 -j CNI-DN-3d1d...
-A CNI-HOSTPORT-DNAT ... "dnat name: \"portmap\"" --dports 80,443 -j CNI-DN-fb11...
CNI-DN-3d1d...: -j DNAT --to-destination 10.42.5.6:80
CNI-DN-fb11...: -j DNAT --to-destination 10.44.8.41:80

The first entry belonged to a flannel pod that no longer existed. When the container runtime tears a pod down, it hands the job to whichever CNI config sorts first at that moment, not the one that set the pod up. On a switched node that's Cilium's. So when the old ServiceLB pods were deleted, Cilium's plugin handled their teardown, and it knew nothing about flannel's DNAT rules. Those rules stayed at the head of the chain, so connections to port 80 or 443 on the private address were sent to an address with nothing behind it. Moving ServiceLB back to flannel did the same thing in the other direction, leaving dead Cilium entries behind, which is why it didn't help.

We restored traffic by adding the private addresses to the Services as externalIPs, which makes kube-proxy take that traffic and skip ServiceLB. With apps serving again, we added a private network and a stand-in edge to the k3d rehearsal, and the failure reproduced straight away in both directions.

The fix lives in the shim, which already runs on every node with the host's iptables. On each loop it now removes any hostPort chain whose targets are pod addresses with no live pod on that node. It acts straight away, or after ten seconds if a pod on the node is still waiting for its IP. In the rehearsal, rolling ServiceLB now costs one failed probe per node instead of breaking the path. On the real cluster it cleared every dead chain in its first pass, we removed the workaround, and the roll finished on the normal path.

The same review turned up a second instance of the trap. Our final cleanup script treated every CNI-HOSTPORT rule as a flannel leftover, but with Cilium chaining portmap, those rules are ServiceLB's live ports. It now removes only rules that still point into flannel's range. The portmap binary also needed a permanent home in /opt/cni/bin, which is where containerd looks once flannel is gone and where Cilium doesn't install one. A small DaemonSet now keeps it there on every node, including nodes that join later.

Finishing on the real cluster

With the shim fixed, the rest went to plan. Every node switch passed its gate, zero-downtime apps surged onto Cilium before their old pods stopped, and each managed cache and Postgres instance was back on Cilium in under twenty seconds.

The one step k3d couldn't rehearse was restarting K3s itself with flannel and kube-router switched off. K3s runs with KillMode=process, so a restart leaves the containers alone, and we compared every healthy pod on each node before and after. None of them restarted. A cleanup then removed kube-router's leftover rules and flannel's interfaces from each host, keeping ServiceLB's live ports.

The migration ends with a stricter version of the gate on every worker, with no grace period, so allowed paths have to be open from the first sample too.

final gate: strict probes (no grace for late opens) on every worker
  ok   vx-canary-unclaimed: api ok, db ok, w443 ok, w25 ok, tenant ok, tenant2 ok, dns ok, kubelet ok
  ok   vx-canary-sealed:    api ok, db ok, w443 ok, w25 ok, tenant ok, tenant2 ok, dns ok, kubelet ok
  ok   vx-canary-web:       api ok, db ok, w443 ok, w25 ok, tenant ok, tenant2 ok, dns ok, kubelet ok
  ok   vx-canary-smtp:      api ok, db ok, w443 ok, w25 ok, tenant ok, tenant2 ok, dns ok, kubelet ok
Cilium migration complete.

The same first-packet check now runs in our production end-to-end suite. A freshly deployed, unclaimed app tries to reach the cluster's internal network every half second for its first thirty seconds, and its egress policy refuses all sixty attempts.

If you're doing this yourself

Rehearse at your real size and through your real traffic path. Our first rehearsal had the right topology and still missed the packet-mark bug, which needed enough identities for some to be odd, and the issue that reached production, which only showed on the path our edge actually uses.

Probe from a pod's first packet. A check that waits for the pod to be Ready can't see a policy that lands a few seconds late.

While two CNIs share a node, treat its netfilter state as shared too. Most of our problems lived on the same-node path, in rules one plugin wrote and the other removed, rewrote or misread.

Every new pod on velixir now has its policy before it sends its first packet, which is the promise we started with. The rehearsals caught most of what could go wrong, and the one thing they missed is why we now test changes through the same path our customers' traffic takes.