Kubernetes is famously easy to start with and famously hard to operate well. The gap shows up around day 90 and is usually painful by day 300. Here are the decisions you can revisit cheaply early on, and the ones that will follow you for the life of the cluster.
Decisions that are cheap to revisit
The shape of your Deployment and Service manifests, the ingress controller, the logging agent, even your observability stack — all of these are replaceable with focused project work. Don't agonise over them on day 1. Pick reasonable defaults and move on.
Decisions that are expensive to revisit
Cluster topology. Single multi-tenant cluster or a cluster per environment? This decision quietly shapes your IAM, your network policies, your CI pipelines, and your incident blast radius. Changing it later is a quarter-long project.
Namespace and RBAC strategy. A flat namespace structure looks fine with five services and is a security audit finding with fifty. Plan a hierarchy that maps to teams and environments before you have to.
Secret management. Plain Kubernetes Secrets are a stepping stone, not a destination. Pick a real secret store — AWS Secrets Manager, GCP Secret Manager, Vault — on day 1 and integrate it via External Secrets or a sidecar. Refactoring later means rotating every credential in the cluster.
Cost model. Without a cost-allocation discipline (labels, namespaces aligned to teams, a tool like OpenCost) you will lose visibility into spend within a quarter and never recover it without a project.
The shape of a healthy day-300 cluster
The clusters that age well share a handful of properties. Workloads are pinned to specific node pools. Pod Disruption Budgets are set on everything stateful. Horizontal Pod Autoscalers are driven by real signals, not just CPU. The platform team owns a small, opinionated set of base manifests, and product teams compose on top of them.
The single biggest predictor of a healthy cluster at day 300 is whether someone owns it. "Kubernetes" should appear on someone's roadmap, not just on their on-call rotation.