If you need to inspect kubectl
network traffic, you can add verbose logging
options (-v=8
or higher) to any kubectl
command and you can see the URLs,
request and response body/headers (except authorization). These headers usually
are not complete, because more headers are added after the request is logged. To
have a complete view, you need to intercept traffic using a local proxy like
mitmproxy.
mitmproxy is pretty much the Swiss-army knife of “man in the middle”debugging. It works with protocols and encodings like https, http/2, websockets, protobuf, and gives you even a web interface.
After you install mitmproxy, you actually don’t need to trust its root certificate, as we can ignore server verification in kubectl. Then start mitmproxy as:
mitmproxy -p 5000 --ssl-insecure
-p
denotes port number the proxy will be listening on, --ssl-insecure
/-k
allows us to skip verifying the TLS certificate of the Kubernetes apiserver.
Since most clusters self-sign their TLS certificates (you can tell by presence
of certificate-authority-data
in kubeconfig file), you will need this.
Once mitmproxy starts running, you need to run kubectl with HTTPS_PROXY
environment variable to make the traffic flow through mitmproxy1 2. (This
variable is widely recognized by most http clients, including the one used by
Kubernetes client-go):
HTTPS_PROXY=:5000 kubectl get namespaces --insecure-skip-tls-verify
You will need the --insecure-skip-tls-verify
, unless you trust mitmproxy’s
root CA certificate on your machine.
Then, you should be able to view and navigate the requests captured by mitmproxy:
Similarly, instead of mitmproxy
, you can run mitmweb
command and use its
web interface, which I find to be
much more intuitive than learning mitmproxy’s TUI shortcut keys.
Behind the scenes, mitmproxy presents a self-signed certificate which we
do not verify (--insecure-skip-tls-verify
) and it therefore decrypts the
request made by kubectl
. Then it repeats this request to the Kubernetes
apiserver, and proxies the response back.
kubectl mitmproxy apiserver
|----https handshake---->|
|<---present fake cert---|
|------->request-------->|
|---------https handshake------>|
|<--------present cert----------|
|----->decrypt request, https-->|
|<-----------response-----------|
|<-------response--------|
This way, you can fully inspect what’s going on the wire from kubectl.