Gitlab CI-CD

Pipelines are the top-level component of continuous integration, delivery, and deployment.

A pipeline is a set of instructions (ie. Jobs) that are executed in an order that we define

A merge request can have many pipelines, and each pipeline must belong to one merge request.

Normally a pipeline is created when you push to a branch on Gitlab

Pipelines comprise:

• Jobs, which define what to do. For example, jobs that compile or test code.
• Stages, which define when to run the jobs. For example, stages that run tests after stages that compile the code.

If all jobs in a stage succeed, the pipeline moves on to the next stage.

If any job in a stage fails, the next stage is not executed and the pipeline ends early.

• we can set a job to allow_failure: true if we want to override this behaviour.

A typical pipeline might consist of four stages, executed in the following order:

• A test stage, with two jobs called unittest and lint.
• A build stage, with a job called compile.
• A staging stage, with a job called deploy-to-stage.
• A production stage, with a job called deploy-to-prod.

Types of Pipeline

Pipelines come in various configurations

Basic pipelines (ie. Branch Pipelines)

run everything in each stage concurrently, followed by the next stage. Run the pipeline each time changes are pushed to a branch.

Basic pipelines have some characteristics:

• they run when you push a new commit to a branch.
• they have access to some predefined variables.
• they have access to protected variables and protected runners.

Merge Request Pipelines

Run only when commits associated with an MR are pushed (rather than for every commit).

• Merge Request jobs run in a separate pipeline from Commit jobs.

Pipelines for merge requests run when you:

• Create a new merge request.
• Commit changes to the source branch for the merge request.
• Select the Run pipeline button from the Pipelines tab in the merge request.

This means that if you use a squash-based workflow, jobs in the MR pipelines (ie. those associated with each commit to that MR) will have run on a different commit than the commit that actually gets merged into the main branch.

• remember that in a squash-based workflow, all the commits that went into that MR will not make its way into the repo's history. Consider that if we have a job that depends on a git commit SHA (e.g. a contract-test job that records the publication of the contract by the commit SHA), we cannot perform this job in the MR pipeline alone, because the contract testing framework (here, Pact) will only be aware of the commit SHA-related contracts that are associated with those commits that will never make its way into the main commit history line.

Merge Request Pipelines don't run by default; they must be configured with rules or only/except (old method)

Jobs of a merge request pipeline only run on the contents of the source branch, ignoring the target branch.

Merge Request Pipelines also have access to additional pre-defined variables

Merge Request Pipelines display the merge request tag:

Git Tag Pipelines

• note: "git tag pipeline" is not an official type of pipeline

A pipeline that runs once a git tag has been created

.git_tag_trigger:
  only:
    variables:
      - $CI_COMMIT_TAG

Merged Requests Pipeline

• note the merged requests

merge request pipelines that act as though the changes from the source branch have already been merged into the target branch.

Merge Trains

Merge trains use pipelines for merged results to queue merges one after the other.

Directed Acyclic Graph Pipeline (DAG)

pipelines are based on relationships between jobs and can run more quickly than basic pipelines.

• we achieve this with the needs keyword, which allows us to be explicit about the job's dependencies
  • ex. a post-install stage needs an install stage to have succeeded first

Multi-project pipelines

combine pipelines for different projects together.

Parent-Child pipelines

break down complex pipelines into one parent pipeline that can trigger multiple child sub-pipelines, which all run in the same project and with the same SHA. This pipeline architecture is commonly used for mono-repos.

The order that we list our stages defines the execution order for jobs:

• Jobs in the same stage run in parallel.
• Jobs in the next stage run after the jobs from the previous stage complete successfully.

If stages is not defined in the .gitlab-ci.yml file, the default pipeline stages are:

• .pre
• build
• test
• deploy
• .post

If a job does not specify a stage, the job is assigned the test stage.

To make a job start earlier and ignore the stage order, use the needs keyword.

What is a Job?

A job is an atomic unit that is picked up by the runner.

• If we were acting as the runner, we would simply run a script, like npm run test or npm run lint. These are 2 different jobs, and they are picked up by the runner.
• jobs are executed in the environment of the runner.

Multiple jobs in the same stage are executed in parallel

Jobs are defined at the top-level of the gitlab-ci.yml file.

• you might have jobs called Lint, Test, Deploy Preview, etc

What is important is that each job is run independently from each other.

with rules, we can modify the order that jobs within a stage will be run.

Gitlab loads everything from top to bottom, allowing us to override any job, just by changing the sequential order.

The rules or only/except keywords are what determine whether or not a job is added to a pipeline.

• if a job is not added to the pipeline, the issue is probably related to this.
• only/except is the older way to do it

Non-zero exit codes

When script commands return an exit code other than zero, the job fails and further commands do not execute.

Store the exit code in a variable to avoid this behavior:

job:
  script:
    - false || exit_code=$?
    - if [ $exit_code -ne 0 ]; then echo "Previous command failed"; fi;

Job Templates

Jobs prepended with a . will be hidden, and will not be processed by Gitlab CI/CD.

• You can use hidden jobs as templates for reusable configuration with extends keyword or YAML anchors.

Running jobs conditionally (rules)

default-job:
  script:
    - yarn test
  rules:
    - if: $CI_COMMIT_BRANCH

Groups of jobs

build ruby 1/3:
  stage: build
  script:
    - echo "ruby1"

build ruby 2/3:
  stage: build
  script:
    - echo "ruby2"

build ruby 3/3:
  stage: build
  script:
    - echo "ruby3"

Each job can run in a separate isolated Docker container

• docs

Resource Group

Allows us to make sure only one job per resource group is working at a time

• ex. consider 2 different pipelines of the same repo that both deploy to the same place. Of course, deployments should happen sequentially, so we put both of those jobs in the same resource_group, guaranteeing that they will run one after the other.

A job can be specified as part of a resource_group which ensures a job is mutually exclusive across different pipelines for the same project.

• if multiple jobs that belong to the same resource group are queued simultaneously, only one of the jobs starts. The other jobs wait until the resource_group is free.
• ex. resource_group=prod puts a limitation that only one job in the prod resource group may run at a time.
• Resource groups behave similar to semaphores in other programming languages.

Service

When we use services, we specify 2 keywords:

• services
• images

A job can define a Docker image

---

You can’t use these keywords as job names:

• image
• services
• stages
• types
• before_script
• after_script
• variables
• cache
• include