How would you debug a Docker container that has no shell, no tools, but a single statically-compiled executable? Creating docker images FROM scratch is a trend on the rise, but is it really taking away our debugging capabilities?

Some will tell you to use FROM scratch (empty base image) to package your statically compiled app. Others will say you should use alpine, so at least you get a shell, ps, top, lsof and other tools that can help you debug. But scratch-based images aren’t entirely non-debuggable.

To debug a running container with empty filesystem, you need to get into the Linux namespace of the container. Some tools let you do this, such as nsenter(1) (and a containerized version of it).

For Kubernetes, I discovered an easier solution that works smoothly.

Kubernetes scratch-debugger

Meet the scratch-debugger script, developed by Google engineer @tallclair.

Specially hacked for Kubernetes, it gets you a shell in FROM scratch containers, without needing to SSH to the node and installing new stuff.

It works this by executing the following steps:

  1. locates the Docker container from the Pod name
  2. deploys a busybox container on the same Kubernetes node as the pod.
  3. uses the docker cp command (mounted from the host) to copy the busybox binary into the target pod
  4. uses kubectl exec to run busybox --install -s /tmp/debug-tools which installs symlinks for 390+ tools brought by busybox.
  5. runs a kubectl exec [...] -- /tmp/debug-tools/sh using to get a shell within the container.

To run it, simply download the script from the link and run:

./debug.sh [POD_NAME]

(I lately patched some issues with it. Give it a try and open an issue if you run into any problems with it!)

Disadvantages of scratch-debugger

  • Requires privileged access: If your cluster restricts you from doing hostPath mounts, you can’t get docker.sock into your pod. PodSecurityPolicies allow you to specify such rules.

  • Modifies the running pod: You should probably delete the pod and recreate it to discard the changes you made to the pod by adding the busybox binary.

  • Does not work on read-only containers: If container has securityContext.readOnlyRootFilesystem: true set on the Pod spec (which is a good practice), adding busybox binary with docker cp will fail.

  • Works only for docker container runtime: It would require some changes to make it work with rkt, cri-o and other runtimes.

Future: Debug containers

Debug containers is a feature request for Kubernetes, gaining some steam. Through a kubectl debug command it’ll give you a way to get a container launched in a Pod’s namespace.

(If you’re interested in this feature, read the design proposal and participate in community discussions.)

This way, you’ll be able to bring in a container image with your favorite debugging tools to troubleshoot a problem and don’t need hacks like the scratch-debugger.

Looks like we are incrementally making scratch containers more convenient!

Thanks Robert Kubis for proofreading this article.