Part 3 of the MCP series.
- add → tolerate → mint → enforce
- cross-tenant negative tests
- multiple replicas
- drained deploys
The problem
Part 2 describes where the edge ended up: identity from claims, a trust tier per token, permissions that fail closed. Getting there meant changing the rules on a system people were using all day, from Claude, Cursor and the product's own AI.
Two things made that hard. A stricter rule, switched on in the wrong order, denies everyone at once. And the edge itself was fragile: it ran as a single replica, so every deploy was a short outage. You can't ship security changes often and safely if every ship drops connections.
Add, tolerate, mint, enforce
Every rule that depends on a new claim went out in four steps, each a separate deploy:
1. add the code understands the new claim, nothing requires it
2. tolerate every hop passes it through and accepts tokens without it
3. mint the issuer starts putting it in new tokens
4. enforce missing or wrong values are deniedReverse any two steps and something breaks. Enforce before mint and every request is denied. Mint before tolerate and an older hop rejects a header it has never seen.
I used two flags for enforcement, not one, because "wrong value" and "missing value" behave differently. A wrong trust tier is never legitimate, so it could be enforced right away. A missing one is normal until every token issued before the change has expired. That flag waited for old tokens to age out.
One more check before calling a step done: is the flag actually wired into the deploy? A flag that exists in code but in no chart or pipeline cannot be turned on. I now grep the delivery config for every new flag as part of the review.
Testing the case that matters
Most auth tests check bad input: a malformed token, the wrong audience, a missing claim. Those are worth having, but they miss the dangerous case. That case is a perfectly valid token for one store, pointed at another.
So I wrote a negative matrix. Each row is a well-formed credential used where it should not work:
- a valid token for store A, asking for store B
- a valid token on one trust tier, claiming another
- a client sending its own copy of an identity header
- a model writing another store's name into a tool argument
- a token with permissions, calling a server those permissions don't admit
Every row must be denied, and every denial must show up in the audit log. A test that passes because the response was "some error" is not enough. It has to be the right refusal from the right hop.
I also merged all the in-flight changes into one branch and deployed it to one sandbox before production. That found two problems no single change showed: tests pinning config that another change removed, and a deploy setting that only went wrong when two changes were combined.
Making deploys boring
The edge started as one replica with no disruption budget and no drain. Each deploy killed the only pod while it still had requests in flight.
The fixes were standard, and each one mattered:
- Multiple replicas, spread across nodes, so a deploy or a node drain never takes all of them.
- A pod disruption budget, so Kubernetes won't voluntarily evict below a safe count.
- A preStop pause and a grace period, so a pod stops receiving new traffic before it stops serving. I measured how long the load balancer kept sending to a terminating pod and set the pause above that, instead of guessing.
- Atomic deploys, so a failed rollout rolls itself back.
Then two quieter problems. The load balancer's default timeout was shorter than a long tool call, so it cut calls while the server kept working and the client retried. A longer backend timeout fixed it. And the edge and the gateway used the same keep-alive idle limit, so the edge sometimes reused a connection just as the other side closed it. I made the client side expire first, and allowed exactly one retry, only for a fast connection failure before any response. A slow failure is never retried, because the tool call may already have run.
What I'd do differently
- Fix replicas first. I'd make the edge and the gateway multi-replica before any security work. Every later rollout would have been cheaper.
- Write the negative matrix before the features. I'd write those rows first and let them drive the design, instead of adding them alongside it.
Takeaway
Security on a live system is mostly about order. Add, tolerate, mint, then enforce. Test with valid credentials in the wrong place, not just broken ones. And make deploys safe first, because you'll be shipping a lot of them.