GitLab

This story presents the deployment of the new CI. The description may evolve in the future. Also, a more practical guide will be added soon.

• Test data

  • Move test data from otb-data into main repository using Git-LFS (OTB/Data)

• Dashboard

  • Deploy CDash 2.6 to new URL

• Gitlab Runners

  • Use Docker Machine to run jobs on linuxes
  • Deploy gitlab-runner on a macOS server
  • Deploy gitlab-runner on Windows VMs

• CI steps:

  • Structure: 4 pipeline profiles have been defined:
    • wip: default pipeline for a push on any branch
    • mr: merge request pipeline, when the branch is the source of a merge request
    • develop: pipeline for pushes on develop branch
    • release: pipeline for pushes on a release-X.Y branch
  • These profiles are used to enable/disable some steps in the pipeline stages. The stages are:
    • precheck: all profiles
    • prepare: enabled for [mr, develop, release]
    • build: enabled for [mr, develop, release]
    • deploy: enabled for [develop, release]
    • external: all profiles, corresponds to the CDash submissions made during previous steps
  • precheck
    • fast compilation on a Ubuntu image with native dependencies
  • prepare
    • Build OTB XDK with SuperBuild on Ubuntu
    • Build OTB XDK with SuperBuild on Centos 6
    • Build OTB XDK with SuperBuild on macOS
    • Build OTB XDK with SuperBuild on Windows 8.1
    • Build OTB XDK with SuperBuild on Windows 10
  • build
    • Build OTB on Debian with native dependencies
    • Build OTB on Ubuntu with XDK
    • Build OTB on Centos 6 with XDKstatus
    • Build OTB on macOS with XDK
    • Enable LargeInput on macOS
    • Build OTB on Windows 8.1 with XDK
    • Build OTB on Windows 10 with XDK
    • Build Documentation on Ubuntu
    • Build standalone package on Ubuntu
    • Build standalone package on Centos 6
    • Build standalone package on macOS
    • Build standalone package on Windows 8.1
    • Build standalone package on Windows 10
    • Build Docker image with OTB
      • Current status: the develop branch is compiled on a Ubuntu native image to get a warm ccache
    • Report link to CDash submission ⚠ (see Caveats)
  • quality
    • Launch quality tests
    • Check contributions
    • Check licences
    • Check Copyright
  • deploy
    • Deploy documentation
    • Deploy packages from develop branch
    • Deploy packages from release branches
    • Deploy a release to production

• Remote modules

  • Setup for Travis integration
  • Setup for AppVeyor integration

Caveats:

• Twin pipelines: Gitlab-CI spawns 2 pipelines when a push is made on a branch with a merge request (one for normal branches, one for merge requests). Since we don't enable the same steps for MR and non-MR push, we had to setup a custom script (CI/check_twin_pipelines.py) that will cancel the that we want to skip. This issue should be fixed in Gitlab CE in a future release.
• Builds from forked repositories:
  • At the moment they are not supported, due to the need of custom secret variables that are defined only in orfeotoolbox/otb. They will be reduced to a single variable for an API token. See next point.
  • Builds from forks are supported, provided a CI admin has given you access to the project runners. Ask for it in your first MR.
• API tokens: some parts of the CI scripts need a token to access Gitlab API. For commit in orfeotoolbox group, these tokens are already supplied as secret variables, but not on user's forks. If you want CI to run from your fork, you need to create a personal access token (choose the scope api), and supply it as a secret variable: on the project page of your fork, go to Settings -> CI/CD -> Variables, add the variable K8S_SECRET_API_TOKEN and put the token in this variable. After !536 (merged), this token is actually optional, but 2 features are disabled if no token is given:
  • the publication of external jobs with links to CDash, these links are output in the logs anyway.
  • the handling of twin pipelines.
• CDash links: the links to CDash submissions is posted in the external step. However, due to issues with the twin pipelines, they may not be posted in the right pipeline. (Should be fixed)
• CDash status: the passed/failed status of CDash jobs doesn't always match. The submissions to CDash are asynchronous, and we rely on CDash API to get the number of errors/warnings/tests failed. So it can happen that CDash has not finished processing a submission when the request is made. The status can show passed whereas the jobs actually failed. This behaviour will be changed in order to avoid using these calls to CDash API, we will only request the build ID. (Fixed by !526 (merged) )