A full platform environment in Terraform

• GKE + managed SQL
• Gateway API ingress
• no static keys
• one environment per engineer

The problem

The platform was a managed hosting platform for online stores, running in two regions. Every store was a Kubernetes tenant with its own web stack, cache and database, created and driven by a provisioner service and a dashboard. That shape worked in production. What it lacked was a safe place to change the platform itself.

Chart changes could only be checked against live stores. The store lifecycle (create, reset, delete) touched real databases, so nobody wanted to run it. And engineers who shared one test cluster blocked each other every time someone broke a deploy.

So I built a complete new environment from scratch in Terraform, in its own GCP project, without touching a line of the production configuration.

Same primitives as production

The rule for every choice was: if it behaves differently from production, the bugs it finds are not real bugs.

• GKE for the cluster, zonal and small, because the environment was meant to be disposable.
• Managed SQL on the same engine and version as production, so migrations behaved the same way.
• Gateway API with managed certificates for ingress, the same routing primitives production used, so routing bugs reproduced.
• Workload Identity everywhere, no static keys. Pods got cloud permissions through their service accounts, and CI signed in as a short-lived OIDC identity restricted to specific branches, not with a downloaded key file.

One module per engineer

The decision that mattered most was not a shared sandbox but a per-user namespace module. The Terraform takes a list of users and loops over it. For each one it creates a namespace, that user's databases and database user, service accounts with their Workload Identity bindings, secrets, a resource quota, a network policy and the role bindings the provisioner needs.

users = [ana, ben, demo, qa]
          │
          └─ for_each ─▶ namespace + databases + service accounts
                         + quota + network policy + bindings

Adding an engineer became a one-line diff. The same module later produced the standing demo and QA environments the rest of the company used. I also wrote down what each one was for: personal ones were free to break, the demo one was for release candidates only.

SSH and SFTP through SSHPiper

Stores offered SSH and SFTP through SSHPiper, an SSH reverse proxy. Gateway API only carries HTTP, so it could not front SSH. Instead of paying for a TCP load balancer on a disposable environment, I ran SSHPiper on the host network of one node, on a high port so it did not fight the node's own SSH daemon or need extra privileges.

Keeping it alive

The environment broke in the same ways production could. Each break taught one rule.

Node pools get recreated. The SSH gateway was bound to a node's external IP, and DNS pointed at that IP. When the node pool was recreated, the node vanished, the pod could not schedule, and DNS pointed at a dead address. The fix was a reserved static IP attached to the node. Lesson: never point DNS at anything ephemeral.

Signing keys expire. One day every terraform init in CI failed with an expired key error. Nothing in our code had changed. The signing key that older Terraform versions used to verify providers had expired upstream, and upgrading the pinned version picked up the renewed key. Lesson: pinned tool versions are a dependency too, and they age.

Missing PriorityClasses hang deploys. Every Helm upgrade sat in a pending state until it timed out. Production defined a PriorityClass that the cache pods asked for; the new environment did not. The pods could not schedule, so the upgrade never finished. I applied it by hand to unblock the team the same day, then put it in Terraform. Lesson: anything production has that the chart assumes must be in code, not in someone's memory.

Default-deny drops health checks. Routes started returning "no healthy upstream". The default-deny network policy also blocked the load balancer's health checks, so every backend looked dead. Allowing the health check ranges fixed it. Lesson: a deny-all policy has to list the platform's own probes.

Smaller fixes followed the same pattern. Deploys to one namespace were serialised so two branches could not interleave a Helm upgrade. A CI step that picked "the first pod" was changed to pick a running one, because a dead pod made a successful deploy report failure. And the environment stopped copying production's high-availability shape for every store, which freed most of its capacity.

What I'd do differently

• Separate database users per environment from day one. One database user with full privileges per tenant was simple, but scoped users per purpose are the safer default. Adding them later meant touching every tenant.
• Make chart defaults nil-safe from the start. Upgrades that reuse stored values skip new chart defaults, so a template that checks a new key panics on older releases. I fixed it each time a new subsystem shipped. A lint rule would have stopped the whole class.

Takeaway

A second environment is only useful if it uses the same primitives as production, and only cheap if adding a person is one line. After that, the work is keeping it alive, and every outage there is a rehearsal for one you no longer have in production.