I have a Gitlab CI pipeline that runs 3 jobs in parallel and caches files created in those jobs to use further in the pipeline, but only files created in one of the 3 parallel jobs are actually being cached & available in subsequent jobs.
Here’s basically what my pipline code looks like:
default:
cache:
key: $CI_COMMIT_SHA
paths:
- release/**/*
stages:
- stage1
- stage2
- stage3
job1:
stage: stage1
before_script:
- mkdir release
script:
- echo 'job1' > release/job1
job2:
stage: stage2
parallel:
matrix:
- ITEM:
- item1
- item2
- item3
script:
- echo "job2-$ITEM" > release/job2-$ITEM
job3:
stage: stage3
script:
- echo 'job3' > release/job3
post:
stage: .post
script:
- ls release/
In the post
job, the only file from stage2
is job2-item1
. It should have also created job2-item2
and job2-item3
.
In the item2
and item3
parallel jobs in stage2
, it’s creating those files and caching them but for some reason job3
isn’t picking them up. also, the previously-cached items from job1
are being restored in job2 [item1]
but not item2
or item3
job2 [item1]
:
Successfully extracted cache
...
Creating cache 4e6ab4bcc8d3d0b12c60ee71276fd205ae555268-non_protected...
release/**/*: found 2 matching artifact files and directories
Note that it says “found 2 matching artifact files”–the job1
file and the newly-created job2 [item1]
file.
job2 [item2]
:
WARNING: Cache file does not exist
Failed to extract cache
...
Creating cache 4e6ab4bcc8d3d0b12c60ee71276fd205ae555268-non_protected...
release/**/*: found 1 matching artifact files and directories
Note that it only found 1 file–the newly-created job2 [item2]
file, but it’s not receiving the cached job1
file.
job2 [item3]
:
WARNING: Cache file does not exist
Failed to extract cache
...
Creating cache 4e6ab4bcc8d3d0b12c60ee71276fd205ae555268-non_protected...
release/**/*: found 1 matching artifact files and directories
What’s the right way to accomplish this?