I am encountering an intermittent error when using GitLab runners to build Docker images, specifically during multistage builds. This issue occurs sporadically; however, when it manifests on a particular machine, subsequent builds on that same machine consistently fail, indicating a persistent issue with that specific machine environment. Interestingly, the same build job may succeed on a different machine.

To address this, we have found that completely wiping the Docker directory serves as a temporary workaround.

We have also attempted to implement this workaround (adding RUN true between COPY commands as discussed in this GitHub issue), but it did not resolve the problem.

Build ends with this error:

#25 [base 4/7] RUN rm -rf /flyway/jre
#25 CACHED
#26 [base 5/7] RUN ln -s /flyway/flyway /usr/local/bin
#26 CACHED
#27 [base 6/7] COPY ./docker/testcontainer-postgres-xxx/onInit/flyway-migrate.sh /usr/local/bin
#27 CACHED
#28 [final 3/3] RUN echo ${PIPELINE_ID} > /tmp/pipeline_id
#28 CACHED
#29 exporting to image
#29 exporting layers done
#29 preparing layers for inline cache done
#29 writing image sha256:b9c9b7b5436bfab4b7f94d73f9501c2ac2cf26d68ab63feedcbe4c5b86061cfb done
#29 ERROR: failed to get layer sha256:4572d629188142b6cf001bec82a4f44f795854b40d7432942a77a5308d58254e: layer does not exist

We have found this error in runners syslog:

Mar 21 06:15:44 gitlab-runner-3 dockerd[1650152]: time="2024-03-21T06:15:44.732808293Z" level=info msg="Layer sha256:a9fb5fd507b30c7a01deba587053acc920cc115435ce5fae405438c88b48a60e cleaned up"

Execution command:

docker build
      --build-arg BUILDKIT_INLINE_CACHE=1
      --cache-from $TEST_CONTAINER_POSTGRES_ADATA_TAG_BASE:develop
      --cache-from $TEST_CONTAINER_POSTGRES_ADATA_COMMIT_REF_SLUG
      --build-arg POSTGRES_IMAGE_NAME=$POSTGRES_IMAGE_NAME
      --build-arg FLYWAY_BASE_IMAGE=$DOCKER_IMAGE_TAG
      --build-arg type=adata
      -t $TEST_CONTAINER_POSTGRES_ADATA_COMMIT_REF_SLUG
      -f ./docker/testcontainer-postgres-xxx/Dockerfile .

Dockerfile:

ARG POSTGRES_IMAGE_NAME=registry.xxx.com/xxx/postgres/postgres-xxx:16-2-xxx-1-2-0
ARG FLYWAY_BASE_IMAGE=develop
ARG type=app-schema

# hax so we can define tag via variable
FROM registry.xxx.com/xxx/core/flyway-base:$FLYWAY_BASE_IMAGE as flyway-base

FROM $POSTGRES_IMAGE_NAME AS base

ENV POSTGRES_USER postgres
ENV POSTGRES_PASSWORD passwd

ENV FLYWAY_OUT_OF_ORDER=true
ENV FLYWAY_BASELINE_ON_MIGRATE=true
ENV FLYWAY_PLACEHOLDER_REPLACEMENT=false
ENV FLYWAY_CONNECT_RETRIES=120
# Number of flyways that are checked for repeatability in each schema, starting with the newest one
ENV REPEATABILITY_CHECK_SIZE=6
# Default password for every type
ENV FLYWAY_PASSWORD passwd

# default flyway connection config
ENV DB_NAME xxx
ENV DB_USER xxx



## Get flyway script
# java version must match the one used by flyway cmd
RUN apt-get update && apt-get install -y --no-install-recommends openjdk-17-jre
COPY --from=flyway-base /flyway /flyway
# do not include jre provided in flyway. Installed JRE will be used to simplify java setup
RUN rm -rf /flyway/jre
RUN ln -s /flyway/flyway /usr/local/bin
########################

COPY ./docker/testcontainer-postgres-xxx/onInit/flyway-migrate.sh /usr/local/bin
RUN ["chmod", "+x", "/usr/local/bin/flyway-migrate.sh"]

######### Stages based on defined type #########
FROM base AS app-schema
ARG DB_SCHEMAS="app admin quartz translations"
# Regular Flyways
COPY ./app/app-db/src/main/resources/db/app/ ./docker-entrypoint-initdb.d/app/
COPY ./app/app-db/src/main/resources/db/translations/ ./docker-entrypoint-initdb.d/translations/
COPY ./app/app-db/src/main/resources/db/admin/ ./docker-entrypoint-initdb.d/admin/
COPY ./app/app-db/src/main/resources/db/qrtz/ ./docker-entrypoint-initdb.d/quartz/

# Extra flyways for IT
COPY ./app/app-db/src/main/resources/db/it-app/ ./docker-entrypoint-initdb.d/app/

# used for verifying flyway scripts in gitlab pipeline test stage and as source of configured DB for MR namespaces
FROM base AS app-data
ARG DB_SCHEMAS="app admin quartz translations app_data admin_data app_seed"
# Regular Flyways
COPY ./app/app-db/src/main/resources/db/app/ ./docker-entrypoint-initdb.d/app/
COPY ./app/app-db/src/main/resources/db/translations/ ./docker-entrypoint-initdb.d/translations/
COPY ./app/app-db/src/main/resources/db/admin/ ./docker-entrypoint-initdb.d/admin/
COPY ./app/app-db/src/main/resources/db/qrtz/ ./docker-entrypoint-initdb.d/quartz/
COPY ./app/app-db/src/test/resources/db/app_data/ ./docker-entrypoint-initdb.d/app_data/
COPY ./docker/flyway-seed-xxx-app/sql/ ./docker-entrypoint-initdb.d/app_seed/
COPY ./app/app-db/src/test/resources/db/admin_data/ ./docker-entrypoint-initdb.d/admin_data/
...