Building Cross Platform Images of Operator

[tunahanertekin]

Hey there!

I want to build cross platform images of operator. When I execute make build and make images, it only creates images for amd64. How could I build images supporting amd64 and arm64? Any kind of help is appreciated. Thanks!

[skitt]

To build multiarch images, run

make multiarch-images

The default platforms for multiarch images are linux/amd64 and linux/arm64.

See https://submariner.io/development/shipyard/images/#multi-arch-images for details. You’ll need to ensure that builders are available for the platforms you’re interested in; the images will be built as tarballs in the package directory. See https://github.com/submariner-io/submariner-operator/blob/devel/.github/workflows/multiarch.yml for inspiration.

[tunahanertekin]

Thank you very much for your instant reply! Have a great day :)

[tunahanertekin]

Hi again!

TL;DR: I built images and imported tarballs successfully with the command above however there has been some troubles when I tried to use those images. Container runtime cannot start entrypoint since it cannot find submariner-operator binary in right path.

Normally, if I guarantee that there is a submariner-operator binary under bin/ directory, make images works properly. I assume make build puts this binary, or it can be put with go build.

When I try to run multiarch image in my local machine, I got the following error meaning no binary found named submariner-operator:

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "submariner-operator": executable file not found in $PATH: unknown.
ERRO[0000] error waiting for container: context canceled

Here is the Dockerfile that builds both single arch and multiarch images:

ARG BASE_BRANCH
ARG SOURCE=/go/src/github.com/submariner-io/submariner-operator

FROM --platform=${BUILDPLATFORM} quay.io/submariner/shipyard-dapper-base:${BASE_BRANCH} AS builder
ARG SOURCE
ARG TARGETPLATFORM

COPY . ${SOURCE}

RUN make -C ${SOURCE} LOCAL_BUILD=1 bin/${TARGETPLATFORM}/submariner-operator

FROM --platform=${TARGETPLATFORM} scratch
ARG SOURCE
ARG TARGETPLATFORM

ENV USER_UID=1001 PATH=/

# install operator binary
COPY --from=builder ${SOURCE}/bin/${TARGETPLATFORM}/submariner-operator /submariner-operator

ENTRYPOINT ["/submariner-operator"]

USER ${USER_UID}

So I figured it should be okay if I can put binaries under bin/linux/amd64/ and/or bin/linux/arm64/ directories since the Dockerfile expects the argument TARGETPLATFORM. (again the binaries are obtained with make build or go build with GOARCH and GOHOSTARCH)

The error didn't change after I updated the paths of binaries. When I digged the logs of make multarch-images I found these lines that builds images:

[09:47:51.638] [dir=submariner-operator]$ output_flag=--load
[09:47:51.640] [dir=submariner-operator]$ output_flag=--output=type=oci,dest=package/submariner-operator.tar
[09:47:51.754] [dir=submariner-operator]$ default_platform=linux/amd64
[09:47:51.756] [dir=submariner-operator]$ go env GOOS
[09:47:51.815] [dir=submariner-operator]$ go env GOARCH
[09:47:51.873] [dir=submariner-operator]$ 'buildargs_flags=(--build-arg BUILDKIT_INLINE_CACHE=1 --build-arg "BASE_BRANCH=${BASE_BRANCH}")'
[09:47:52.309] [dir=submariner-operator]$ docker buildx use buildx_builder
[09:47:52.772] [dir=submariner-operator]$ docker buildx build --output=type=oci,dest=package/submariner-operator.tar -t quay.io/submariner/submariner-operator:dev --cache-from quay.io/submariner/submariner-operator:release-0.14 --cache-from quay.io/submariner/shipyard-dapper-base:release-0.14 -f package/Dockerfile.submariner-operator --iidfile package/.image.submariner-operator --platform linux/amd64,linux/arm64 --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg BASE_BRANCH=release-0.14 .

I think the problem is that docker buildx build command doesn't set the TARGETPLATFORM argument. So the binary paths not right inside multiarch images.

Also I'm facing with issues when I try to generate binaries by make build too as you can see the logs below:

docker network create -d bridge kind
54cc5fdf1ca3c63c32b825ba78e28cbdae3a65c1500acabd79ec8787a421264b
./.dapper   -- make  build
[+] Building 0.0s (8/8) FINISHED                                                                            
 => [internal] load build definition from Dockerfile.dapper                                            0.0s
 => => transferring dockerfile: 45B                                                                    0.0s
 => [internal] load .dockerignore                                                                      0.0s
 => => transferring context: 2B                                                                        0.0s
 => [internal] load metadata for quay.io/submariner/shipyard-dapper-base:release-0.14                  0.0s
 => [1/4] FROM quay.io/submariner/shipyard-dapper-base:release-0.14                                    0.0s
 => CACHED [2/4] WORKDIR /go/src/github.com/submariner-io/submariner-operator                          0.0s
 => CACHED [3/4] RUN git config --global --add safe.directory /go/src/github.com/submariner-io/submar  0.0s
 => CACHED [4/4] RUN git config --global --add safe.directory /go/src/github.com/submariner-io/submar  0.0s
 => exporting to image                                                                                 0.0s
 => => exporting layers                                                                                0.0s
 => => writing image sha256:6732910f97a7d4d97dd1afb47fd14e2bb3fa5eecaef652404a2ce7017870d509           0.0s
 => => naming to docker.io/library/submariner-operator:robolaunch-devel                                0.0s
Checking if Makefile.dapper is up to date
Checking if Dockerfile.dapper is up to date
Makefile:104: warning: overriding recipe for target 'e2e'
/opt/shipyard/Makefile.inc:192: warning: ignoring old recipe for target 'e2e'
make: Nothing to be done for 'build'.

I wonder if I'm skipping some of the necessary steps I should do before I build multiarch images. Kinda stuck here.

[tunahanertekin]

Testing New Image in Kubernetes

Here's my steps after running make multiarch-images:

docker import package/submariner-operator.tar --platform linux/amd64
docker tag <HASH> robolaunchio/submariner-operator:dev-v27-amd64
docker push robolaunchio/submariner-operator:dev-v27-amd64

Then I update Helm chart values as below:

--set operator.image.repository="docker.io/robolaunchio/submariner-operator" \
--set operator.image.tag="dev-v27-amd64" \

Here's the events when describing pod created by submariner-operator deployment:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  14s   default-scheduler  Successfully assigned submariner-operator/submariner-operator-6b676bf8bd-tc2c4 to ip-172-31-180-137.eu-central-1.compute.internal
  Normal   Pulled     14s   kubelet            Container image "docker.io/robolaunchio/submariner-operator:dev-v27-amd64" already present on machine
  Normal   Created    14s   kubelet            Created container submariner-operator
  Warning  Failed     14s   kubelet            Error: failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "submariner-operator": executable file not found in $PATH: unknown
  Warning  BackOff    12s   kubelet            Back-off restarting failed container

Testing New Image Locally (just to understand if entrypoint fails or not)

To make this test quicker, I run the image in my computer:

docker run -it --name test robolaunchio/submariner-operator:dev-v27-amd64 submariner-operator

And here's the logs:

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "submariner-operator": executable file not found in $PATH: unknown.
ERRO[0000] error waiting for container: context canceled 

[tunahanertekin]

Isn't TARGETPLATFORM argument supposed to be set in this line, in shipyard?

[skitt]

TARGETPLATFORM is set by buildx, see https://docs.docker.com/engine/reference/commandline/buildx_build/. When you try to run the operator manually, note that it’s installed in /, so if you’re specifying the command to run, you need to use /submariner-operator; you can also rely on the default entry-point.

[tunahanertekin]

TARGETPLATFORM is set by buildx, see https://docs.docker.com/engine/reference/commandline/buildx_build/.

Yeah sorry, my bad.

I still couldn't manage to run images anywhere though. Maybe I made a mistake when importing tarball to Docker memory.

It seems that everything is ok until I run or push the multiarch images. How can I run or push package/submariner-operator.tar to any image registry after it's generated?

[tunahanertekin]

Finally I managed to extract images and run in their platforms. I will post here my steps as soon as possible but it's still a mystery for me how one could push/run multiarch OCI tarball images without using --load or --push arguments of docker buildx build. (This was the case with shipyard's build method, no --load or --push arguments were used) I also asked this here for the reference.

Thank you very much for your support @skitt!