Inspecting kubectl traffic with mitmproxy

If you need to inspect kubectl network traffic, you can add verbose logging options (-v=8 or higher) to any kubectl command and you can see the URLs, request and response body/headers (except authorization). These headers usually are not complete, because more headers are added after the request is logged. To have a complete view, you need to intercept traffic using a local proxy like mitmproxy.

mitmproxy is pretty much the Swiss-army knife of “man in the middle”debugging. It works with protocols and encodings like https, http/2, websockets, protobuf, and gives you even a web interface.

After you install mitmproxy, you actually don’t need to trust its root certificate, as we can ignore server verification in kubectl. Then start mitmproxy as:

mitmproxy -p 5000 --ssl-insecure

-p denotes port number the proxy will be listening on, --ssl-insecure/-k allows us to skip verifying the TLS certificate of the Kubernetes apiserver. Since most clusters self-sign their TLS certificates (you can tell by presence of certificate-authority-data in kubeconfig file), you will need this.

Once mitmproxy starts running, you need to run kubectl with HTTPS_PROXY environment variable to make the traffic flow through mitmproxy^{1 2}. (This variable is widely recognized by most http clients, including the one used by Kubernetes client-go):

HTTPS_PROXY=:5000 kubectl get namespaces --insecure-skip-tls-verify

You will need the --insecure-skip-tls-verify, unless you trust mitmproxy’s root CA certificate on your machine.

Then, you should be able to view and navigate the requests captured by mitmproxy:

Similarly, instead of mitmproxy, you can run mitmweb command and use its web interface, which I find to be much more intuitive than learning mitmproxy’s TUI shortcut keys.

Behind the scenes, mitmproxy presents a self-signed certificate which we do not verify (--insecure-skip-tls-verify) and it therefore decrypts the request made by kubectl. Then it repeats this request to the Kubernetes apiserver, and proxies the response back.

kubectl                 mitmproxy                      apiserver
   |----https handshake---->|
   |<---present fake cert---|
   |------->request-------->|
                            |---------https handshake------>|
                            |<--------present cert----------|
                            |----->decrypt request, https-->|
                            |<-----------response-----------|
   |<-------response--------|

This way, you can fully inspect what’s going on the wire from kubectl.