Kubernetes in Production: What We Learned

From Tutorial to Production

Just over two years ago, we deployed our first workloads to Kubernetes. Minikube was running, the first YAML files were written, kubectl apply worked. It felt like we understood Kubernetes. Reality quickly taught us otherwise.

The difference between a functioning cluster and a production-ready cluster is enormous. We learned this difference over the past two years, step by step -- and sometimes painfully. What follows are the lessons that shaped us the most.

Resource Requests and Limits: The Underestimated Foundation

Our first lesson came in the form of an OOMKill. A Spring Boot service without configured resource limits consumed all the memory on a node and dragged three other pods down with it. The node became unresponsive, Kubernetes marked it as NotReady -- and suddenly all pods had to be accommodated on the remaining nodes. Cascade effect.

The solution sounds simple: define resource requests and limits for every pod. In practice, however, finding the right values is anything but trivial.

Requests determine how many resources Kubernetes reserves during the scheduling decision. Limits define the hard upper bound. If you set requests too high, you waste cluster capacity. If you set them too low, too many pods land on one node and compete for resources.

For Java applications, we've found the following rule of thumb works well: the memory request should match the actual average consumption under load. We set the memory limit about 20 to 30 percent above that to absorb short-term spikes while preventing uncontrolled memory consumption. Crucially: the JVM heap setting (-Xmx) must be below the container limit, with sufficient headroom for Metaspace, thread stacks, and native memory.

resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
  limits:
    memory: "768Mi"
    cpu: "1000m"
env:
  - name: JAVA_OPTS
    value: "-Xms256m -Xmx512m -XX:+UseG1GC"

An important note on CPU limits: we learned that overly tight CPU limits can be counterproductive for Java applications. The JVM needs significantly more CPU briefly during startup and garbage collection. A limit set too low leads to CPU throttling, which dramatically slows down startup and causes probes to fail.

Liveness and Readiness Probes: The Art of Proper Configuration

Misconfiguring probes was our second most common problem. The typical trap: liveness probe set too aggressively, combined with a Java application that responds more slowly under load. Kubernetes interprets the timeout exceedance as "container is dead" and restarts it. The restart takes 30 to 60 seconds with Spring Boot, during which even more load falls on the remaining pods. Result: a cascade of restarts.

The solution: liveness and readiness probes serve different purposes and need different configurations.

The liveness probe answers the question "Is the process still fundamentally alive?" -- it should only fail for genuine deadlocks or irrecoverable states. The readiness probe answers "Can the pod currently handle traffic?" -- it may fail under high load or during a database connection interruption.

For Spring Boot applications with Actuator, the following configuration has proven effective for us:

livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
  initialDelaySeconds: 60
  periodSeconds: 15
  timeoutSeconds: 5
  failureThreshold: 5
readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3

The generous values for the liveness probe are intentional. A container that doesn't respond for 75 seconds has a serious problem. But a container that once takes 5 seconds because the garbage collector kicked in should not be immediately restarted. The failureThreshold of 5 gives the pod room to breathe.

Monitoring: No Visibility, No Operations

In the first months, we diagnosed problems using kubectl get pods and kubectl logs. That doesn't scale. Once you have ten or more services in the cluster, you need proper monitoring.

Prometheus and Grafana have established themselves as the de facto standard in the Kubernetes ecosystem -- rightly so. The Prometheus Operator makes installation surprisingly easy, and the community provides ready-made dashboards for almost everything.

Which metrics to monitor naturally depends on context. But some have proven indispensable for us:

• Container restarts: A gradually rising restart counter is an early warning signal. Often there's a memory leak behind it leading to OOMKill.
• CPU throttling: Indicates that CPU limits are set too tightly. Especially relevant for Java applications.
• Request latency (p95/p99): Averages lie. Percentiles show how the slowest requests are doing.
• Pod scheduling wait time: If pods are stuck in Pending status for long, cluster capacity is lacking.
• Persistent volume utilization: Full volumes are a surprisingly common cause of outages.

We additionally export JVM metrics via Micrometer directly from our Spring Boot applications. Heap utilization, GC pauses, and thread counts are invaluable when diagnosing memory problems.

Namespace Strategy and RBAC

In the beginning, we deployed everything to the default namespace. That works with three services. Not with twenty.

We settled on a combination of environment-based and team-based namespaces: staging, production, plus team-specific namespaces for development and testing. Resource quotas per namespace prevent a team from accidentally consuming the entire cluster.

RBAC (Role-Based Access Control) belongs in place from day one. The question "Who is allowed to do what in the cluster?" should not be answered only after someone accidentally deletes pods in production. In our setup, developers get read access to the production namespace and full access to their development namespaces. Deployments to production run exclusively through the CI/CD pipeline.

Rolling Updates and Graceful Shutdown

Kubernetes supports rolling updates by default. But a rolling update without graceful shutdown means dropped requests. For Java applications, this is particularly relevant because the JVM does not cleanly finish in-flight requests without explicit configuration.

Two things solved this problem for us. First: a preStop hook that gives Kubernetes a few seconds to remove the pod from the service endpoint before the SIGTERM signal arrives. Second: Spring Boot's server.shutdown=graceful combined with a timeout that gives in-flight requests time to complete.

Pod Disruption Budgets (PDBs) are the complementary counterpart. They guarantee that during planned maintenance -- such as a node upgrade -- a minimum number of pods from a deployment always remains available. Without a PDB, kubectl drain can in the worst case terminate all pods of a service simultaneously.

Conclusion

Running Kubernetes in production is a continuous learning process. The platform offers powerful tools, but it does not forgive carelessness in configuration. Resource limits, correct probes, monitoring, and well-thought-out namespace strategies are not optional extras -- they are the foundation for stable operations.

The most important insight after two years: Kubernetes makes many things easier, but it shifts complexity rather than eliminating it. If you didn't know how much memory your application needs before, you'll have to learn it with Kubernetes -- or face the consequences in the form of OOMKills.

Our advice: take the time to get the operational fundamentals right before introducing the next layer of abstraction (service meshes, operators, multi-cluster). A well-configured, well-monitored cluster with clean probes and resource limits is worth more than an overloaded setup with the latest features.