EKS Cluster Setup for Enterprise Application

The client was running a containerised product on hand-built EC2 hosts and had reached the point where every release needed a person watching it. They wanted EKS, in three availability zones, with the deployment path good enough that nobody had to babysit a Friday afternoon push.

What we walked into

Two environments that had drifted apart, an ingress setup nobody could fully explain, and secrets sitting in environment variables baked into images. None of that is unusual. It just meant the first job was not building the cluster, it was writing down how the current thing actually worked.

How the cluster is put together

A single EKS cluster per environment, spread across three AZs in a purpose-built VPC with private subnets for nodes and public subnets only for load balancers.

• Managed node groups, split into an on-demand group for anything stateful and a Spot group for stateless workloads
• Cluster Autoscaler driven by real pod pressure, with Pod Disruption Budgets so scale-down cannot take a service below quorum
• Separate node group for system components, so a noisy application pod cannot starve the ingress controller

The Spot split is where most of the savings came from. It only works because the workloads that run there were made genuinely stateless first, which was a week of application changes, not a cluster setting.

Security

• IRSA, so each workload gets its own IAM role instead of sharing the node role
• Pod Security Standards enforced at the namespace level, restricted by default
• Network policies denying cross-namespace traffic unless it is declared
• Secrets moved out of images into AWS Secrets Manager, mounted at runtime
• ECR image scanning wired into the pipeline, with a blocking threshold on new criticals

The IRSA migration was the fiddly part. Every workload had been inheriting node permissions, so splitting them apart meant finding out what each one had quietly been allowed to do.

Monitoring

Prometheus and Grafana for metrics, Fluent Bit shipping logs to CloudWatch, and Container Insights for the cluster-level view. Alerts were deliberately kept short: if it pages, a human has to act. Everything else lives on a dashboard.

Everything in Terraform

The VPC, cluster, node groups, IAM, and add-ons are all Terraform, with per-environment state and a plan that runs on every pull request. Applications are deployed separately through Argo CD from a Git repo, so cluster changes and application changes never contend for the same lock.

Where it landed

• Deploys went from a scheduled event to something that happens several times a day without an announcement
• Compute spend dropped substantially once the Spot node group took the stateless workloads
• Rebuilding an environment from scratch is a Terraform apply, and we have done it to prove it
• Node patching happens on a rolling schedule instead of during incidents

What was hard

Connectivity back to on-premises systems set the networking design more than anything on the AWS side. One legacy service assumed a fixed local disk and had to keep a StatefulSet with EBS rather than be forced into a stateless shape. And the compliance requirement on log retention meant a second logging destination with a longer lifecycle, which is cheap to build and easy to forget.

What we still do

We run the cluster: monitoring and on-call, Kubernetes and add-on version upgrades on a planned cadence, cost review each month, and a capacity conversation before anything the business knows is coming.