dexec is a small Go library allowing you to run commands inside Docker containers as if you are running them locally using the os/exec package.

With dexec, you can containerize your external program invocations locally or remotely, using the Docker Engine. Your application will benefit from isolation, security, resource limiting and other advantages of using Docker.

dexec effectively is an abstraction layer that hides the fact that commands are executed within a container, as it gives you the same process semantics such as I/O streams and exit codes.

Check out dexec on GitHub.

Some benefits of using dexec:

  • commands are executed securely and isolated
  • invoke commands remotely as if they are running locally
  • parallelize workloads to a fleet of Docker Engines
  • very easy migration from os/exec package

Switching to dexec

If you have Go program that calls /bin/echo to print Hello, world! and collect its output, it probably looks like this:

func main(){
	cmd := exec.Command("/bin/echo", "Hello, world!")
	b, err := cmd.Output()
	if err != nil {
		panic(err)
	}
	fmt.Printf("Output: %s", b)
}

Now let’s run the same command in a busybox container using dexec by adding a few lines:

client, _ := docker.NewClientFromEnv()
d := dexec.Docker{client}

m, _ := dexec.ByCreatingContainer(docker.CreateContainerOptions{
	Config: &docker.Config{Image: "busybox"},
})

cmd := d.Command(m, "/bin/echo", "Hello, world")
// the rest is the same

and with that, your command will run inside a container and your program will capture the standard output stream of it.

Similar to os/exec

Go standard library offers the marvelous os/exec package to run external commands. This gives you control over standard input/output streams of the commands you invoke.

If you noticed in the example above, dexec API is strikingly similar to the os/exec package —and this is intentional.

Fortunately enough, Docker Remote API offers the same semantics even though the container might be running on a remote machine.

That is why, migrating to dexec is easy. Even though your commands are running inside a container which you specify, you get to use the same methods in os/exec with nearly the same semantics.

Security and Isolation

Wouldn’t it be cool to run a command in an isolated sandbox and limit its resource (memory, CPU, network etc.) use? Well, Docker is exactly for that.

Here is the same Hello World example, running in a container that is memory-limited and has no network connectivity using dexec:

dexec.ByCreatingContainer(docker.CreateContainerOptions{
	Config: &docker.Config{
		Image: "busybox",
		Memory: 200,
		MemoryReservation: 100,
		NetworkDisabled: true,
	}
})

Boom! There it is.

Stream Processing

dexec allows you to read from and write to STDIN/STDOUT/STDERR of the container as if it is running natively on your program’s host operating system. This allows passing data to and reading output from the containerized command.

Yet, the interface remains the same as os/exec, you do not need to learn anything new.

See the stream processing example on GitHub for more.

For something just a little bit crazier, see the video processing example on GitHub where we download a video and extract its audio (using youtube-dl and ffmpeg), then collect the binary output from STDOUT.

Parallel Remote Computation

Sometimes you have this one command that is computationally-expensive and you just want to parallelize it. Well, if you install Docker Engine on your machines, you can write programs in Go using dexec and parallelize your computation.

dexec can do that for you. As long as you point it to a set of Docker Engines or a Docker Swarm cluster, the short-lived containers hosting your commands will be created remotely. This way you can offload or parallelize the computation to remote machines easily.

Check out the example on GitHub where we download bunch of large blobs over the network and compute their checksum. The actual downloading and computation occurs on the remote fleet of machines, not within your program which just coordinates the overall computation.

Learn more

dexec is open source: You can check out its source code and find more examples on GitHub.

I would love to hear your use cases, contributions and what your imagination tells you about what you could do with dexec!

I am hoping to see some contributions to refine the API and caller-experience as I think there is room for improvement that would make the caller code a lot more readable and precise.

I presented dexec at Docker Seattle Meetup on April 27, 2016. You can find the brief slide deck here:

Side note: Turns out I was not the only one who thought about this idea. Heck I was not even the only one who came up with the same name. 😬 There is another dexec which kind of does a similar job, but instead of invoking commands like my project , it builds and runs your source code on a Docker Engine, which might be suitable for some other use cases.