Did you know you can create and distribute your own kubectl commands? As of Kubernetes 1.12, kubectl now allows adding external executables as subcommands.

fun fun fun

In this blog post, I’ll explain how kubectl plugin mechanism works, why plugins are useful, how you can write your own plugins, and current challenges in the plugin ecosystem.

30-second intro to kubectl plugins

kubectl has adopted the git approach to allow extensions via subcommands: If you have an executable file named kubectl-foo somewhere in your $PATH, you can invoke it as kubectl foo [...] as of kubectl 1.12.

This is pretty practical and doesn’t require any extra configuration to register plugins to kubectl as subcommands.

Why are plugins useful

Technically, instead of making a tool/script a kubectl subcommand like kubectl foo, you can distribute it as foo and you wouldn’t be losing anything.

However, the plugin mechanism makes these commands look like they’re part of kubectl, hence making it a cohesive experience for users of kubectl. I’ll give two use cases like this:

  1. Extend kubectl with new functionality: Plugins can help you add commands that you think are missing from kubectl. For example: kubectl view-cluster-utilization, kubectl rbac-test…

  2. Encapsulate repetitive workflows: If you have a lot of in-house scripts around kubectl, you can make them your kubectl commands. For example, your ./prune-orphaned-pods.sh can be kubectl prune orphaned-pods. This is especially useful if you can install these plugins to all developer machines at your company.

How to write your own plugins

As I described above, any executable named like kubectl-foo[.exe] you drop in $PATH will show up as a plugin in kubectl plugins list command, with the exception that you cannot override existing kubectl commands (like “get”, “apply”, “delete”).

You don’t need any boilerplate or libraries to write plugins. If you want to give a try to writing a plugin in Go, fork the sample-cli-plugin –or simply write a bash script like:

#!/usr/bin/env bash
echo "Hello, I'm a kubectl plugin with arguments: $@"

So which languages are more suitable for kubectl plugin development?

  • Go: Since it’s the choice of cloud-native/kubernetes ecosystem, and it can distribute without runtime/library dependencies as a single binary. Also, client-go is the primary client library for Kubernetes.
  • Bash scripting: It works on macOS/Linux out-of-the-box. You can rely on the fact that user has a working kubectl, so you can call it in your script. No need to parse the kubeconfig file, or make REST API calls like you would in Go.

How are plugins invoked

When your plugin is invoked through kubectl as a subcommand, your executable will replace the kubectl process as it will be started by calling execve system call (as opposed to running as a child process of kubectl forked from it).

Any arguments your plugin is invoked with will be passed to your process as arguments after the plugin name is stripped off. So if users call kubectl foo a1 a2 ..., your executable will be called with argv[1]=a1, argv[2]=a2.

If you want to have a subcommand in your plugin, such as kubectl foo bar [...], you can either distribute an executable named kubectl-foo-bar. Otherwise, your kubectl-foo executable will be invoked with argv=[bar, ...].

So an executable distributed as kubectl-a-b can actually be called as both kubectl a-b or kubectl a b. To disambiguate this and force kubectl a-b invocation, you can name your executable with an underscore like kubectl-a_b.

You can read more details about the plugins mechanism at this enhancement proposal.

Where are we at

Most people don’t know we have the kubectl plugins feature, and it’s okay! It’s still very early and it will take time for the community to understand the value of plugins.

There are about 20 open source plugins today on GitHub. This may seem like a very small number, but given this is a fairly new concept that became stable recently, it is promising.

Since Kubernetes project’s position is shifting towards encouraging development of “new features as extensions” (through CRDs, plugins), this concept will gain more popularity.

There are many great kubectl-based projects on GitHub and over time I think we will see these projects developed and distributed as plugins, so they feel more natural to kubectl users.

Current challenges with plugins

I’ve been looking at the kubectl plugin ecosystem for over 8 months now and here’s a summary of what I am seeing.

Users of kubectl:

  • are not aware of the plugin system (hence this blog!)
  • don’t know where to find plugins
  • don’t know how to install plugins
  • how to keep installed plugins up-to-date

Developers of kubectl plugins:

  • cannot easily make their plugins discoverable
  • don’t know best practices around developer plugins
  • don’t know how to look like kubectl: cli-runtime project provides common logic for printing, kubeconfig parsing and API rest client helpers that you should use
  • don’t know how to package plugins: brew, apt, dnf, or just have people extract things from a tarball/zip to somewhere on $PATH.
  • can’t easily package/distribute: just to support major Linux distros, you’d need to support at least 3 packaging formats for your plugin, but a lot of kubectl users are on macOS and Windows, too.

At Google, we have been brewing a solution to plugin management challenges.

You can take a peek here, or stay tuned for my next blog post where I will explain how we’re approaching these problems in the kubectl plugin ecosystem.