I have a repository that builds and supplies artifacts to other projects. In order to get the latest artifacts the [GitLab Job Artifacts API][1] offers two options based on the source of the request – inside a CI job or elsewhere. The URL I am using is
https://gitlab.example.com/api/v4/projects/${proj_id}/jobs/artifacts/${branch_tag}/download?job=${job_name}
I have a separate download.cmake
files inside a cmake
directory of my project (all of which is configured using CMake), where FetchContent_Declare()
and FetchContent_MakeAvailable()
are being executed to ensure that the project is setup for both development (locally) and deployment (on-premise premium GitLab instance).
Among other function inside download.cmake
the following is what I use for downloading any artifact from my GitLab:
function(download_file_gitlab_latest name token proj_id branch_tag job_name)
set(_URL "https://gitlab.example.com/api/v4/projects/${proj_id}/jobs/artifacts/${branch_tag}/download?job=${job_name}")
set(_HEADER "PRIVATE-TOKEN: ${token}")
message("GET Request: ${_URL}")
message("Destination prefix: ${name}")
FetchContent_Declare(
${name}
#QUIET
URL "${_URL}"
HTTP_HEADER ${_HEADER}
DOWNLOAD_NO_EXTRACT 0
DOWNLOAD_EXTRACT_TIMESTAMP 1
DOWNLOAD_NO_PROGRESS 0
)
if(NOT ${name}_POPULATED)
FetchContent_MakeAvailable(${name})
endif()
endfunction(download_file_gitlab_latest)
The header of the request contains an single parameter PRIVATE-TOKEN
that contains a group/project/personal access token required for authenticating any request against the GitLab API (more on that parameter later on).
The token is supplied by a token.txt
file, which has a single line with the respective token and is loaded in the CMakeLists.txt
that calls the download function from above líke this:
file(READ "${CMAKE_SOURCE_DIR}/token.txt" TOKEN) #<---- the token is loaded from the token.txt and stored inside the TOKEN variable
...
download_file_gitlab_latest(
"${DOWNLOAD_DIR_PREFIX}"
${TOKEN} #<---------------------------------------- supplied by token.txt
68522
"${config}"
"build-all-${config}"
)
where config
‘s value is a single value from CMAKE_CONFIGURATION_TYPES
(debug
, release
etc.).
Both locally and in my CI job I configure the project (which also triggers the fetching of the artifacts) using
cmake -Bbuild -G "Visual Studio 16 2019" -S.
The only difference is the passing of CMAKE_CONFIGURATION_TYPES
in the CI job to ensure that only the artifacts for the respective build type are downloaded.
Configuring the project locally shows no signs of a problem and the artifacts are downloaded, extracted, copied to a specific directory, included in the project and the build step is executed, producing a working EXE. However, remotely I am getting the following error:
CMake Error at D:/Software/CMake/share/cmake-3.30/Modules/FetchContent.cmake:1416:EVAL:1:
Parse error. Function missing ending ")". Instead found unterminated
bracket with text "JOB-TOKEN: ��g".
Call Stack (most recent call first):
D:/Software/CMake/share/cmake-3.30/Modules/FetchContent.cmake:1416 (cmake_language)
cmake/download.cmake:32 (FetchContent_Declare)
CMakeLists.txt:41 (download_file_gitlab_latest)
CMake Error at D:/Software/CMake/share/cmake-3.30/Modules/FetchContent.cmake:1416 (cmake_language):
cmake_language unknown error.
Call Stack (most recent call first):
cmake/download.cmake:32 (FetchContent_Declare)
CMakeLists.txt:41 (download_file_gitlab_latest)
-- Configuring incomplete, errors occurred!
Respectively the build steps never takes place.
There are two crucial steps that I have in my CI job that ensure the proper setup for the download and can be seen in the CI job below:
build-imgui-dx11-demo:
stage: build
rules:
- if: $CI_COMMIT_REF_NAME == "main"
allow_failure: true
artifacts:
untracked: true
paths:
- build
- cmake # Allows me to verify that PRIVATE-TOKEN has been replaced with JOB-TOKEN
- token.txt # Allows me to verify that the CI_JOB_TOKEN is stored in the token.txt and it's not just an empty file
when: always # Allows me to see the artifacts even if the CI job fails
before_script:
- echo $CI_JOB_TOKEN > token.txt
- |
(Get-Content ./cmake/download.cmake).Replace('PRIVATE-TOKEN', 'JOB-TOKEN') | Set-Content ./cmake/download.cmake
script:
- cmake -Bbuild -G "Visual Studio 16 2019" -S. -DCMAKE_CONFIGURATION_TYPES="release"
- cmake --build build --target ALL_BUILD --config release
For local download I use PRIVATE-TOKEN
as also described in the download.cmake
function. Since it’s really bad practice to upload passwords and personal tokens to a repo, the token.txt
is listed in .gitignore
and therefore needs to be generated in the CI job
echo $CI_JOB_TOKEN > token.txt
Further, since the project is configured in a CI job, I also replace PRIVATE-TOKEN
with JOB-TOKEN
in the header for FetchContent_Declare()
, which I have verified by checking the changed download.cmake
that indeed has
set(_HEADER "JOB-TOKEN: ${token}")
Since I have a premium subscription I do know that I can use needs: project
to interconnect multiple repositories and their artifacts. However, I would like to do it this way since it does not depend on premium features and (with minimal changes) mirrors the local setup.
[1]: https://docs.gitlab.com/ee/api/job_artifacts.html#get-job-artifacts