Goutham K

TIL: the ingress-nginx 404 page is just another Service

Mon, 24 Aug 2026 00:00:00 +0000

I’d always assumed a 404 from ingress-nginx was nginx doing something built-in and untouchable. It isn’t — it’s routing you to an actual Kubernetes Service, and you can point it at your own.

Every Ingress rule you write matches on a host and a set of paths. If a request comes in that doesn’t match any rule on any Ingress in the cluster, ingress-nginx doesn’t invent a response — it forwards the request to whatever Service is configured as its default backend, and that Service’s response is what the client sees. Out of the box, that’s the small defaultbackend-amd64 pod the chart installs for you, which does nothing but serve a plain 404.

TIL: matchPolicy: Equivalent means your webhook still runs on versions you never listed

Wed, 19 Aug 2026 00:00:00 +0000

I assumed a webhook registered against one API version simply didn’t see requests on any other version. Turns out that’s only true if you set matchPolicy: Exact — and that’s not the default.

A MutatingWebhookConfiguration declares rules for the group/version/ resource it cares about:

rules:
 - apiGroups: ["example.io"]
 apiVersions: ["v1beta1"]
 resources: ["widgets"]

Fine if the resource only has one version. But plenty of resources — CRDs especially — have several, wired together by a conversion webhook. What happens on a version your rules didn’t list is exactly what matchPolicy decides.

TIL: why containers need a real init process

Tue, 18 Aug 2026 00:00:00 +0000

I’d always waved away the “use tini” advice in Dockerfiles as boilerplate cargo-culting. Turns out PID 1 is genuinely special to the kernel, and skipping it is exactly why containers hang on shutdown and quietly accumulate zombie processes.

Two things are true about PID 1 that aren’t true for any other process:

It’s immune to the default action of unhandled signals. For every other process, an unhandled SIGTERM just kills it. PID 1 ignores it unless the program explicitly installs a handler.
It’s responsible for reaping orphaned children via wait() — if it doesn’t, they pile up as zombies.

Most application binaries do neither, because they were never written to be an init system. So when your image runs the app directly as PID 1:

TIL: Kubernetes doesn't restart pods when a Secret changes — sometimes it doesn't even need to

Fri, 14 Aug 2026 00:00:00 +0000

I assumed updating a Secret would eventually reach every pod using it, one way or another. It doesn’t — and the two ways pods consume Secrets behave completely differently once the Secret changes.

If you’re pulling a Secret in as an environment variable, it’s resolved once at pod creation and frozen. Update the Secret, and the running container’s environment never changes. Nothing short of a new pod picks up the new value.

TIL: Argo CD's dry-run can fail before your manifests even get a chance

Tue, 11 Aug 2026 00:00:00 +0000

I’ve hit this exact failure on so many Argo apps that I probably should have figured out what was actually going on years ago: a CRD and a custom resource that depends on it, installed in the same sync, blowing up before either one gets a real chance.

Before Argo CD actually applies anything, it runs each resource through kubectl apply --dry-run=server as a sanity check. That’s normally a good thing — it catches broken manifests before they touch the cluster. But the dry-run for the CR can hit the API server before the CRD has registered. The API server has no idea what kind you’re talking about, so it rejects the dry-run with something like no matches for kind "Foo" in version "v1". The error reads like a Kubernetes problem — because it is one — but it’s easy to go looking for it in the chart templating first.

TIL: a Service doesn't know what a Deployment is

Mon, 10 Aug 2026 00:00:00 +0000

I’d always assumed a Service was wired to a specific Deployment. It isn’t — a Service only matches on pod labels. It has no concept of Deployment or ReplicaSet at all, which means nothing stops two different Deployments from feeding the same Service.

That’s enough to fake a canary rollout without any extra tooling. Give your canary Deployment a single replica, your stable Deployment nine, and make sure both pod templates carry the label your Service selects on:

TIL: making CRD lists behave like maps under server-side apply

Thu, 06 Aug 2026 00:00:00 +0000

I’d been quietly assuming Kubernetes lists were just lists. Turns out some of them are undercover maps, and not knowing which is which explains a whole class of server-side apply conflicts I used to debug by vibes.

Here’s the actual mechanism: when a CRD field is a list of objects, SSA treats the whole list as one atomic value by default. It doesn’t know entry 3 in my update and entry 3 in yours are supposed to be different things — as far as it’s concerned we’re both just yelling into slot 3. Two controllers each adding or updating their own entry look like they’re fighting over the same field, so you get spurious conflicts, or one write silently clobbering the other.

some notes on kubernetes ingress

Mon, 14 Jul 2025 00:00:00 +0000

I keep re-learning how ingress actually routes traffic, so here are the notes I wish I’d written the first time.

the mental model

An Ingress object is just configuration. On its own it does nothing — you need an ingress controller running in the cluster to read those objects and actually route traffic.

a minimal example

Here’s the smallest thing that does something useful:

how does systemd actually start things?

Sun, 22 Jun 2025 00:00:00 +0000

I’ve typed systemctl start thousands of times without really knowing what happens after I hit enter. Here’s what I pieced together.

units all the way down

Everything systemd manages is a unit. A service is a unit, a mount is a unit, a timer is a unit. They’re defined in plain text files, which is nice — you can read them.

systemctl cat sshd.service

dependencies are the interesting part

The thing I didn’t appreciate: systemd builds a dependency graph and starts things in parallel wherever it can. Wants= and Requires= describe ordering intent, and After= / Before= describe actual ordering.

about

Mon, 01 Jan 0001 00:00:00 +0000

I’m Goutham. At Akamai I build an opinionated Kubernetes platform as a product — upstream Kubernetes and Cluster API underneath, a set of deliberate choices on top.

The opinions are the product. Kubernetes will let you do very nearly anything, which is not the same thing as helping you. A platform has to decide on its users’ behalf: what to expose, what to hide, which defaults to enforce, and which escape hatches to leave unlocked. Hold too few opinions and you’ve shipped raw Kubernetes with a support contract. Hold too many and you’ve built a cage with good docs. Finding the balance between those two is the job.

community

Mon, 01 Jan 0001 00:00:00 +0000

Talks, CNCF organizing, and conference appearances. Newest first — see work for the paid stuff.

Cloud Native Computing Foundation (CNCF)

Event Coordinator — Jun 2026 to present — Austin, TX (Remote)

Volunteer organizer for the CNCF local chapter in Austin.

work

Mon, 01 Jan 0001 00:00:00 +0000

The résumé version, for people who want dates and bullets instead of my opinions about control planes. Newest first — see about for the opinions, or community for talks and CNCF stuff.

Akamai Technologies

Software Engineer II — Sep 2024 to present — Cambridge, MA (Remote)

Member of the Kubernetes Product Platform (KPP) team. We build a curated, opinionated Kubernetes platform — the platform I ramble about on the about page is this one, so at least I have to live with my own opinions. Built-in infrastructure provisioning, metrics, logging, monitoring, and alerting. My job is to keep it highly available, easily scalable, and boring in the specific ways PCI and CIS auditors like.