How would you debug a Docker container that has no shell, no tools, but a single
statically-compiled executable? Creating docker images
FROM scratch is a trend
on the rise, but is it really taking away our debugging capabilities?
will tell you to use
FROM scratch (empty base image) to package your
statically compiled app. Others will say you should use
alpine, so at least you get a shell,
lsof and other tools that can help you debug. But
images aren’t entirely non-debuggable.
To debug a running container with empty filesystem, you need to get into the Linux namespace of the container. Some tools let you do this, such as nsenter(1) (and a containerized version of it).
For Kubernetes, I discovered an easier solution that works smoothly.
scratch-debugger script, developed by Google engineer @tallclair.
Specially hacked for Kubernetes, it gets you a shell in
containers, without needing to SSH to the node and installing new stuff.
It works this by executing the following steps:
- locates the Docker container from the Pod name
- deploys a busybox container on the same Kubernetes node as the pod.
- uses the
docker cpcommand (mounted from the host) to copy the
busyboxbinary into the target pod
kubectl execto run
busybox --install -s /tmp/debug-toolswhich installs symlinks for 390+ tools brought by busybox.
- runs a
kubectl exec [...] -- /tmp/debug-tools/shusing to get a shell within the container.
To run it, simply download the script from the link and run:
(I lately patched some issues with it. Give it a try and open an issue if you run into any problems with it!)
Requires privileged access: If your cluster restricts you from doing
hostPathmounts, you can’t get docker.sock into your pod. PodSecurityPolicies allow you to specify such rules.
Modifies the running pod: You should probably delete the pod and recreate it to discard the changes you made to the pod by adding the busybox binary.
Does not work on read-only containers: If container has
securityContext.readOnlyRootFilesystem: trueset on the Pod spec (which is a good practice), adding busybox binary with
docker cpwill fail.
Works only for
dockercontainer runtime: It would require some changes to make it work with rkt, cri-o and other runtimes.
Future: Debug containers
Debug containers is a
feature request for Kubernetes, gaining some steam. Through a
kubectl debug command it’ll give you a way to
get a container launched in a Pod’s namespace.
(If you’re interested in this feature, read the design proposal and participate in community discussions.)
This way, you’ll be able to bring in a container image with your favorite
debugging tools to troubleshoot a problem and don’t need hacks like the
Looks like we are incrementally making scratch containers more convenient!
Thanks Robert Kubis for proofreading this article.