Tag: Kubernetes

Last week, OpenAI has suffered a several hours long outage and published a detailed postmortem about it. Highly recommend reading it. These technical reports are usually a gold mine for all large-scale Kubernetes users, as we all go through similar set of reliability issues running Kubernetes in production. Read More →

This is the analysis of a low severity incident that took place in the Kubernetes clusters at the company I work at that taught me a lot about how to think about the off-the-shelf components we bring from the ecosystem into the critical path and operate at a scale much larger than these components are intended. Read More →

A quick code search query reveals at least 7,000 Kubernetes Custom Resource Definitions in the open source corpus,¹ most of which are likely generated with controller-gen —a tool that turns Go structs with comments-based markers into Kubernetes CRD manifests, which end up being custom APIs served by the Kubernetes API server.

At LinkedIn, we develop our fair share of custom Kubernetes APIs and controllers to run workloads or manage infrastructure. In doing so, we rely on the custom resource machinery and controller-gen heavily to generate our CRDs. Read More →

I’ve been leading the Krew “kubectl plugin manager” (a Kubernetes sub-project). Today, Krew is used to distribute over 70 kubectl plugins. This week, I finally took some time to write my first proper plugin. Read More →

Knative project is usually explained as building blocks for “serverless on Kubernetes”. As a result of this implication, most Kubernetes users are not aware of what Knative can do for their non-serverless workloads: Better autoscaling and networking for stateless microservices on Kubernetes. Read More →

If you need to inspect kubectl network traffic, you can add verbose logging options (-v=8 or higher) to any kubectl command and you can see the URLs, request and response body/headers (except authorization). These headers usually are not complete, because more headers are added after the request is logged. To have a complete view, you need to intercept traffic using a local proxy like mitmproxy. Read More →

If you are developing containers you must have heard the “single process per container” mantra. Inherently, there’s nothing wrong¹ with running multiple processes in a container, as long as your ENTRYPOINT is a proper init process. Some use cases are having processes are aiding each other (such as a sidecar proxy process) or porting legacy applications.

Recently, I had to spawn a sidecar process inside a container. Docker’s own tutorial for running multiple processes in a container is a good place to start, but not production-ready. So I outsourced my quest on Twitter to find an init replacement that can:

1. run multiple child processes, but do not restart them
2. exit as soon as a child process terminates (no point of restarting child processes, let the container crash to be restarted by docker or Kubernetes)
3. fulfill PID 1 (init process) responsibilities like zombie child reaping and signal forwarding.

In this article I explored pros and cons of some of the options like supervisord, runit, monit, tini/dumb-init, s6 (audience favorite), and tini+bash4.x combo (personal favorite). Read More →

If you’re using Google Kubernetes Engine and deploying to it from headless environments like CI/CD, you’re probably installing the gcloud command-line tool (perhaps every time) you run a build. There’s a way to authenticate to GKE clusters without gcloud CLI! Read More →