Skip to content

Gitlab CI structure

This documentation details how the default pipelines have been organized.

They are all about testing, building, and deploying the Kamea project.

Files structure example

.gitlab-ci.yaml # Main Gitlab file
    ├──   core/apps/azure-functions/.gitlab-ci.yaml        # specific project CI
    ├──   infrastructure/.gitlab-ci.yaml                   # specific project CI
    ├──   website/.gitlab-ci.yaml                          # specific project CI

Main gitlab file

In this file we defined all static configurations like pipeline variables, stages... And it also includes all specific configuration for subprojects.

It defines when the pipeline should run, and in which environment it runs.

Specific project CI

In these Gitlab files we can find templates and specific jobs to build and test the applications.

Pipelines

Pipeline example

Stages

Pipeline stages are defined in the main CI file. Each job can be linked to a stage using the stage: keyword (stages can be test, build, deploy, ...)

DAG pipelines

Usually, GitLab pipelines move on to the next stage once the current stage in completed (these are the "basic pipelines").

However in GitLab DAG (directed acyclic graph) pipelines, it is not needed to wait for the completion of each stage to move on. Jobs can run whatever the stage status. Instead, jobs depend on other jobs thanks to using the keyword needs:. Kamea uses DAG pipelines.

GitLab's general pipeline documentation

GitLab's DAG pipeline documentation

Variables and secrets

Secrets and other pipeline variables are stored in the GitLab CI/CD Variables (available in the project's CI/CD settings). They can be shared across all environments, or separated for one environment.

Predefined GitLab CI values are listed in these docs.

Runners

In order to allow more parallel jobs than our previous setup with one dedicated server to run our jobs, and to allow better cost management and runner sizing compared to Gitlab Shared Runners, we deployed Gitlab runner autoscaling through Docker Machine with VMs deployed in our Azure Dev Resource Group.

Runner Manager

The deployed VM project_name-gitlab-runner-vm was deployed to have the role of Runner Manager, it will trigger VMs deployment and decommissioning, manage the runner fleet and jobs assignment. This VM is a persistent VM, but only requires a small sizing (B1s as for now, it could get reduced to B1ls if its memory usage is confirmed to stay low).

A Network Security Group only allows management traffic to it. It was deployed in another vNet than the other services deployed in this RG.

The only service deployed in it is gitlab-runner, which relies on docker-machine to manage its runners (as documented in this page).

Docker Machine

Docker machine is a feature initially offered by Docker to dynamically provision Docker nodes to supported Cloud providers, but which is no more supported.

Gitlab still supports it because its runner autoscaling relies on it until they deploy their Autoscaling plugin for Azure.

To be able to provision VMs, the gitlab-runner config needs (in its config.toml) credentials to authenticate to Azure (we provided it with a Service Principal, with Contributor role in this RG), a VM configuration template (with its sizing, base image, network, ...), and a scaling strategy.

Scaling strategy

The scaling strategy of gitlab-runner is highly tunable but we tried to keep it simple.

Each time the runner receives a new job, it will either check if it has an idle worker available or deploy a new VM (which takes a few minutes). Once a job ends, the runner will check if it has a pending job to give to this worker or it will wait for an idle timeout before deprovisioning the VM.

We allowed it to deploy a maximum of 10 VMs, with a maximum of 3 at a time (to avoid deploying a lot of VMs in a short period of time for only short jobs).

We also set up a specific rule for the periods 10h-12h and 14h-17h during weekdays (our main work hours) where it will keep an idle VM for faster spin up of new pipelines.

  • Standard rule is: max 10 jobs/VMs at a time, VMs are kept idle for 5 min.
  • Specific rule is: max 10 jobs/VMs at a time, VMs are kept idle for 10 min, one VM is kept idle with no timeout for faster pipeline spin up.

Deployment

Deployment jobs

Environments

Environments help to keep track of which code is deployed, and where.

This project has three environments: dev, qa, prod.

  • dev environment is used for development purposes.

    It should always contain the latest changes that have been merged to main.

  • qa environment is used for quality assurance. It can be used to check if everything works fine before going to production.

    It corresponds to the latest tag starting with qa-.

  • prod is the production environment.

    It corresponds to the latest tag starting with prod-.

Each environment is made of sub-environments. For example infrastructure, management-front-end, management-api are services that can exist in each of the environments. Therefore, the dev environment exists as dev/infrastructure, dev/management-front-end and so on.

Each of the three environments can deploy to its own Azure resource group as needed. You can set the project's CI/CD variables by scoping variables to an environment:

  • AZURE_RESOURCE_GROUP determines which resource group the deployment should go to
  • PROJECT_NAME is used as prefix in the deployed Azure resources, and as name of the corresponding Terraform state stored in GitLab.

You can see how PROJECT_NAME is scoped to each environment in the Variables settings:

Scoped variables in the Variables settings

GitLab's environments documentation

Deployment

Deploying a development version

The latest changes on main are automatically deployed to the dev environment.

Other branches do not deploy on dev. However the init, test and build stages still have access to the dev environment. This is useful for terraform:plan for example.

Deploying a production or QA version

Production or QA releases look like a tag starting with prod- or qa-. Creating a production or QA release goes exactly the same for both.

Procedure:

  1. Make sure the version you want to release completes successfully on a lower level deployment. This minimizes risks of failing release deployments.
  2. On the GitLab repository, go to Repository > Tags.
  3. Click New Tag and fil in the name, starting with prod- or qa-.
  4. Choose the version you want to deploy, optionally fill in some information about your release.
  5. You can watch the pipeline progress on the CI/CI > Pipelines overview.

Deployments on environments

You can see the overview of deployments on your environments in Deployments > Environments.

All CI environments

You can find the exact commits that correspond to your deployment in the environment "Show all" details:

Example production environment

Terraform

All terraform states are stored directly on Gitlab. They are available in the project's Infrastructure > Terraform settings.

A new state is created simply by referencing one with a new name. In Kamea, the Terraform state has the project name, so a new project name will create a new Terraform state.