If you need to inspect
kubectl network traffic, you can add verbose logging
-v=8 or higher) to any
kubectl command and you can see the URLs,
request and response body/headers (except authorization). These headers usually
are not complete, because more headers are added after the request is logged. To
have a complete view, you need to intercept traffic using a local proxy like
mitmproxy is pretty much the Swiss-army knife of “man in the middle”debugging. It works with protocols and encodings like https, http/2, websockets, protobuf, and gives you even a web interface.
After you install mitmproxy, you actually don’t need to trust its root certificate, as we can ignore server verification in kubectl. Then start mitmproxy as:
mitmproxy -p 5000 --ssl-insecure
-p denotes port number the proxy will be listening on,
allows us to skip verifying the TLS certificate of the Kubernetes apiserver.
Since most clusters self-sign their TLS certificates (you can tell by presence
certificate-authority-data in kubeconfig file), you will need this.
Once mitmproxy starts running, you need to run kubectl with
environment variable to make the traffic flow through mitmproxy1 2. (This
variable is widely recognized by most http clients, including the one used by
HTTPS_PROXY=:5000 kubectl get namespaces --insecure-skip-tls-verify
You will need the
--insecure-skip-tls-verify, unless you trust mitmproxy’s
root CA certificate on your machine.
Then, you should be able to view and navigate the requests captured by mitmproxy:
Similarly, instead of
mitmproxy, you can run
mitmweb command and use its
web interface, which I find to be
much more intuitive than learning mitmproxy’s TUI shortcut keys.
Behind the scenes, mitmproxy presents a self-signed certificate which we
do not verify (
--insecure-skip-tls-verify) and it therefore decrypts the
request made by
kubectl. Then it repeats this request to the Kubernetes
apiserver, and proxies the response back.
kubectl mitmproxy apiserver |----https handshake---->| |<---present fake cert---| |------->request-------->| |---------https handshake------>| |<--------present cert----------| |----->decrypt request, https-->| |<-----------response-----------| |<-------response--------|
This way, you can fully inspect what’s going on the wire from kubectl.