Special case of building multi-arch container images: Distroless, Go and Podman

Photo by Kristin Hillery on Unsplash

This post addresses a special case of arriving at a multi-architecture container manifest, i.e., the container image name and tag that work the same across multiple OS architectures such as x_86 and arm.

Imagine you are pulling ubuntu:21.04 container image on two separate OS architectures, say linux/amd64 and linux/arm64. As you would expect, the contents of container image on these two architectures would differ despite the fact that both were pulled using the same container image name and tag. Let’s see this in action:

On linux/amd64 machine we can display the arch on the command line and then pull ubuntu:21.04 image. You will also notice that I am using podman tool to pull the image, which is part of container tools from RedHat. I won’t go into details of podman but do check this link for more details:

Inspecting this image we find the signatures for the root file-system layers as follows:

Now let’s do the same on linux/arm64 machine:

Inspecting image gives us different signatures for rootfs layers compared to x_86 , which implies that what we pulled on arm64 is a completely different image tarball despite pulling using the same name and tag.

There is obviously some kind of indirection which fetches the right container image based on the target architecture. This is defined in a manifest, which we can inspect directly:

As you can see this manifest defines specific digests for different supported architectures. This brings us to the main point of discussion on how we can go about putting together container images for our applications that support more than one target architectures.

Special v/s General case

A general case of this problem consists of building any application, that can be built, for a target platform and then adding it to the base image and packing it into a container. For instance, if we have a C app, then we would need to first build it and produce different binaries for each of the architectures and then put together containers for each. While this sounds simple and repetitive, it can get tedious to manage building container images for different architectures because it requires configuring development environment for each of these target machines, moving code between them and pushing to registries from those machines. Not to mention the work scales with increasing number of platforms to support.

A special case of this problem, however, is a simpler approach. It involves building pure-Go based application that can run on distroless base image. In such cases we can short circuit and execute all required steps on one development machine and use cro-compiling feature of Go to put together different container images.

Cross Compiling Go-code

Go is easy to cross-compile. For instance, we could add following RUN commands in Dockerfile to build separate binaries for each of the target architctures in need:

The workflow has two distinct steps:

  • Build a container image with multiple binaries in it, one for each supported target architecture
  • Use the container image built in previous step as the base image from where binaries are copied and repeat this step for each of the target architecture using arch flag of podman build command.

We are essentially avoiding building our application separately on different target platforms, since build steps are the most resource consuming steps also requiring copying code into the container each time.

Using arch flag

podman build command has --arch flag which allows us to operate as if we are working on that particular arch. Now we can repeat the command below for different architecture packing separate container images and then pushing them to the registry.

As you can see, we build separate tags for separate architecture and then push them to the registry. The only thing left to do now is to pack these container images in a manifest. A manifest is essentially a list of pointers to these images.

Building manifest

Building manifest involves creating a new manifest and then adding various target specific images to it. Note that all these steps are happening on a single development machine so far (which in my case is a Linux running Fedora with podmaninstalled)

Finally we can push the manifest to the registry! At this point pulling this container image with the name and tag ${IMG}:${VERSION} would work across different OS architectures.

Caveats

It is important to note that we never needed to execute commands in the Dockerfile that were architecture specific. For instance, the cross-compiling step ran go build command but did so only on the development machine. If your application requires third party libraries that need to be pulled using apt-get or yum, then this workflow will not work for you. The trouble is that apt-get or yum are command executions and are OS/architecture specific. Since the special case I highlighted here only requires copying the previously cross-compiled binary, we are able to leverage the arch flag of podman build command to put together binaries and container images for various target architectures.

Summary

I have been spending a lot of time lately working on Kubernetes cluster on a Raspberry Pi, which is arm64based. At the same time I am working with x_86 or amd64 based Kubernetes cluster on the cloud and needed an easy way to put together container images that would work seamlessly on both architectures.

I must mention that upstream recommendation from docker is probably a formal way to achieve multi-arch container images, but I have been happy with my special case scenario using podman, hence this post documenting that workflow.

Software engineer and entrepreneur currently building Kubernetes infrastructure and cloud native stack for edge/IoT and ML workflows.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store