I am very new to Powershell especially in the context of GitLab CI configuration.
My repo has the following structure:
*
|
+ - .gitlab-ci.yml
|
+ - CMakeLists.txt
|
+ - .gitmodules
|
+ - .configs
| |
| + - .basic_config.yml
| |
| + - .dep_1_config.yml
| |
| ...
| |
| + - .dep_N_config.yml
|
+ - deps
|
+ - submodule_1
|
...
|
+ - submodule_N
In my CMakeLists.txt
I add (based on a -D
flag for the CMake command call) each submodule 1..N with add_subdirectory("${CMAKE_SOURCE_DIR}/deps/submodule_X")
.
The .configs
directory contains a basic CI configuration used by all others and a single configuration file per dependency (represented by a corresponding git submodule, e.g. .dep_3_config.yml
takes care of the CI configuration for submodule_3
). All dependency configs are then called inside the main gitlab-ci.yml
.
.basic_config.yml
# Currently used only for experimentation
.build-setup:
before_script:
- git config --system core.longpaths true
# The main base configuration for any building CI job
.build:
variables:
DepName: ""
# DepCmakeArg will be overwritten by inheriting CI config of a dependency and set to the CMake cmd arguments required for its configuration
DepCmakeArg: -DBuildDEFAULT=ON
rules:
- if: $CI_COMMIT_REF_NAME == "debug" | $CI_COMMIT_REF_NAME == "main"
script:
- |
New-Item -Path "." -Name "build" -ItemType "directory"
# main == release but we have to rename it to keep it consistent
# This check should be dropped in the future (perhaps) for the tag
# check and proper release (special type of CI job)
if($env:CI_COMMIT_REF_NAME -eq "main"){
$env:CI_COMMIT_REF_NAME = "release"
}
# Configuration step with CMake
Start-Process cmake -ArgumentList "-Bbuild -G ""Visual Studio 16 2019"" -S$PWD -DCMAKE_CONFIGURATION_TYPES=""$env:CI_COMMIT_REF_NAME"" $env:DepCmakeArg" -NoNewWindow
# Building step with CMake
Start-Process cmake -ArgumentList "--build build --target ALL_BUILD --config $env:CI_COMMIT_REF_NAME" -NoNewWindow
# Sets up the artifacts directory to ensure common structure for all artifacts so that they can be easily integrated as dependencies in other projects
.artifact-setup:
variables:
DepName: "DEFAULT" # Will be overriden by the local CI job variable with the same name
ArtifactRoot: "artifacts"
script:
- |
if($CI_COMMIT_REF_NAME -eq "main"){
$ArtifactRoot = "${ArtifactRoot}/release"
}else{
$ArtifactRoot = "${ArtifactRoot}/${CI_COMMIT_REF_NAME}"
}
mkdir -p ${ArtifactRoot}lib
mkdir -p ${ArtifactRoot}bin
mkdir -p ${ArtifactRoot}include${DepName}
# Copy the artifacts from ArtifactRoot to a specific location set by ArtifactRootCopyDestination
# Can be used e.g. to provide artifacts locally to CI jobs instead of forcing those to trigger download
.artifact-copy:
script:
- |
Copy-Item -Path "$env:ArtifactRoot" -Destination "$env:ArtifactRootCopyDestination" -Recurse -Container:$true -force
Get-Date | Out-File "$env:ArtifactRootCopyDestination/$env:CI_COMMIT_REF_NAME/artifacts.txt" -NoNewLine -Force
An example for a dependency is OpenCV, which I build (using some CMake variables inside my CMakeLists.txt
:
.opencv_config.yml
# Add base configuration
include:
- local: .configs/.base_config.yml
.build-opencv:
stage: build
variables:
DepName: opencv
# Specifying CMAKE_INSTALL_PREFIX makes sure that the generated artifacts are referencing relatively the installation dir
# In case of OpenCV if not used and even if the files are properly copied, the ".cmake" config files inside the installed
# lib dir will reference <opencv-install-dir>/<arch>/<compiler>
DepCmakeArg: -DBuildOpenCV=ON -DCMAKE_INSTALL_PREFIX="./install/"
ArtifactRoot: !reference [.artifact-setup, variables, ArtifactRoot]
GIT_SUBMODULE_STRATEGY: recursive
GIT_SUBMODULE_PATHS: deps/opencv
ArtifactRootCopyDestination: "${COMMON_DEPS}/opencv/4.10.0/cpu/"
artifacts:
# Upload everything until issue fixed
untracked: true
#paths:
# - artifacts
# - "./deps/build/*"
when: always
script:
- ls
- !reference [.build, script]
The call for that dependency inside my CI configuration:
.gitlab-ci.yml
stages:
- build
include:
- local: .configs/.opencv_config.yml
build-opencv-release:
stage: build
rules:
- if: $CI_COMMIT_REF_NAME == "main"
variables: !reference [.build-opencv, variables]
artifacts: !reference [.build-opencv, artifacts]
script:
- !reference [.build-opencv, script]
I had it all working until I started adding the command line arguments for CMake with the option to adjust those using CI variables. Then I was forced to switch from just calling CMake, e.g.
script:
- cmake -Bbuild -G "Visual Studio 16 2019" -S. ..........
to using a multiline script as described in my .basic_config.yml
.
When I run the CI job I get
$ # Main == release but we have to rename it to keep it consistent # collapsed multi-line command
-- Selecting Windows SDK version 10.0.17763.0 to target Windows 10.0.14393.
Error: could not load cache
...
without the actual building step producing any binaries (that was not the case before I switched to Start-Process
!).
I am not sure what’s going on.
PS: The ArtifactRootCopyDestination
is for creating a local copy of the artifact. Some of the dependencies produce artifacts that are too big to upload so project that are using those have the option to use a local version (everything is stored and executed on an executor I have set up myself so the CI has access to these resources).