Introduce GitlabCI

Abstract

This template which store short yaml to show how can work with GitlabCI and note some specify for use case

More information about Gitlab CI/CD and Variables. Go check

General

Gitlab Documentation - General Documentation about Gitlab
GitLab Runner - Information about Gitlab Runner and configuration
GitLab CI/CD examples - GitlabCI/CD Collections
Gitlab Permissions and roles - About Role and Permissions of account in GitLab organization

Syntax

Gitlab CI/CD Syntax - Introduce about Gitlab Syntax and usage
Variables & Predefine variables - GitlabCI Variables and Predefine variables
Scripts and job logs - Use for format script and log inside job

Features

GitlabCI Services - Use service keyword GitlabCI
Run your CI/CD jobs in Docker containers - Setup GitlabCI in docker+machine
Use Docker to build Docker images - Run Dind to build image inside GitlabCI
DRY development: A cheatsheet on reusability throughout GitLab - DRY Mindset
Optimize GitLab CI/CD configuration files - Optimize and DRY Mindset
CI/CD components - GitlabCI Components
Downstream pipelines - Use Downstream pipelines for trigger
Gitlab CI/CD Pass artifacts/variables between pipelines - Methodology for resue artifacts
GitLab CI/CD artifacts reports types - Report type of Gitlab Artifact
GitLab CI/CD Environments and deployments - Environments describe where code is deployed.
GitLab container registry - You can use the integrated container registry to store container images for each GitLab project.
How to modify a variable, used as name for docker image - Dynamic image to using for gitlabci

Use cases and scenarios

Case 1: Completely pipeline for Container Services

Info

React build app, test and build Image for container services

Info

This pipeline definition will cover for me and you with mostly cases when you work with GitlabCI

Run Script

Perform test

Build code and upload artifact

Build docker image and push to currently registry

.gitlab-ci.yml

# Define stage for running job inside
stages:
  - 'synchronize-code'
  - 'test'
  - 'build-code'
  - 'build-image'
 
# Choose the default image if not set, it will use it
default:
  image:
    name: node:18-bullseye
 
# Default, run script before you run the script on job
before_script:
  - npm i -g yarn --force
  - yarn install
 
# Job is define with rule, another before_script and script
synchronize-code-job:
  stage: synchronize-code
  retry: 1
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      changes:
        - src/config/i18n/data/*
  image: debian:11.7
  before_script:
    - apt update && apt install -y curl jq
  script:
    - bash -c "./utils/synchronize_gitlab.sh $(echo $PRIVATE_GLAB_TOKEN) $(echo $GLAB_USERNAME)"
 
test-code-job:
  stage: test
  retry: 1
  script:
    - yarn lint:fix
  allow_failure: false
 
build-code-job:
  stage: build-code
  retry: 1
  script:
    - yarn build
    - cp -a dist/. public/
# Upload artifact to gitlab, on path and keep on time
  artifacts:
    untracked: false
    when: on_success
    expire_in: "30 days"
    paths:
      - "./public"
  allow_failure: false
 
# For some reason, Docker in Docker was tough suggestion with need to manipulated agent run. So alternative for that is using kaniko
# Read on: https://docs.gitlab.com/ee/ci/docker/using_kaniko.html
build-image-job:
  stage: build-image
  retry: 1
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: ['']
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  variables:
    IMAGE_NAME: '$CI_REGISTRY_IMAGE:$CI_COMMIT_BRANCH'
  before_script: # Login into registry to push image
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}"
  after_script:
    - rm -rf /kaniko/.docker
  allow_failure: false

Case 2: Build & push docker image to private registry

Info

Use private ECR of AWS with dind (Local Include)

Before run and execute the script, you need to expose some variables into CI/CD variables because of AWS authentication, and one more to ensure anything secrets will not expose

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_ACCOUNT

Remember when you want to your variables have effected with your .gitlab-ci.yml pipelines, you need to concern about use variables in mode protected or not because if your branch is not protected, you variables will not have export anything, that sick 😄

center

And then you can execute pipelines like usual, but for become special, I will use add on include keyword to import somewhere template gitlab, and try to use this one via !reference

You need to create template in your repo, for example /templates/ci/ecr-auth.gitlab-ci.yml

ecr-auth.gitlab-ci.yml

.ecr_auth:
  before_script:
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - export AWS_ACCOUNT=$AWS_ACCOUNT
    - apk add aws-cli
    - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com

Info

With template will help us to authentication AWS Account, we will use that via !refernce tags inside your .gitlab-ci.yml and attach with template define with name and which one to use inside template. For example, I have .ecr_auth template with before_script , base on those one you can use with !reference [.ecr_auth, before_script]

And to use !reference in yaml file of editor, you need to configure inside settings of them for bypass the error

.vscode/settings.json

{
    "yaml.customTags": [
        "!reference sequence"
    ]
}

Info

YAML has a feature called ‘anchors’ that you can use to duplicate content across your document. For example, read here: YAML anchors for scripts

Warning

But You can’t use YAML anchors across multiple files when using the include keyword. Anchors are only valid in the file they were defined in. To reuse configuration from different YAML files, use !reference tags or the extends keyword.

.gitlab-ci.yml

stages:
  - "build-docker-image"
 
include:
  - local: "/templates/ci/ecr-auth.gitlab-ci.yml"
 
image: docker:27
# Remember if you use docker:dind from another registry, set alias for service to prevent the error
# Documentation: https://docs.gitlab.com/ee/ci/services/#available-settings-for-services
services:
  - docker:dind
 
before_script:
  - !reference [.ecr_auth, before_script]
 
build-python-image:
  stage: build-docker-image
  when: manual
  script: |
    docker build -t $AWS_ACCOUNT.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/base-image:python-generic -f ./docker/python/Dockerfile .
    docker push $AWS_ACCOUNT.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/base-image:python-generic
  allow_failure: false
 
build-node-image:
  stage: build-docker-image
  when: manual
  image:
    name: docker:dind
  script: |
    docker build -t $AWS_ACCOUNT.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/base-image:node-generic -f ./docker/node/Dockerfile .
    docker push $AWS_ACCOUNT.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/base-image:node-generic
  allow_failure: false

Case 3: Authentication Job with private ECR from remote repository

Info

Purpose: Create a general template, that help job can authenticate to private ECR from remote repo, following that job will pull and use private docker image for jobs instead of use the public version

Following this idea, Gitlab offers for us one methodology to help us authentication private CR via CI/CD variables by DOCKER_AUTH_CONFIG. Read more at documentation

Note

Remember, DOCKER_AUTH_CONFIG will only help you authenticate your service for pull image from docker registry, If you want to make interaction with private registry, please use docker login or crane auth login

Create a CI/CD variable DOCKER_AUTH_CONFIG with the content of the Docker https://docs.gitlab.com/ee/ci/yaml/script.html#multiline-commands-not-preserved-by-folded-yaml-multiline-block-scalarconfiguration file as the value
- For protected, general repository will use and wrap authentication, such as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, we only authenticate private ECR via only general repository.
- Therefore, we will use downstream techniques of Gitlab, with one help us trigger general pipeline from remote repository, and in the last that will submit authentication successfully as ~/.docker/config.json to remote repository
You can now use any private image from private ECR defined in image or services in your .gitlab-ci.yml file

Create Upstream Pipeline

Upstream pipeline where we configure to generate token of ECR or private dockerhub, you can do same with each others

.gitlab-ci.yml

stages:
  - "build-docker-image"
 
include:
  - local: "/templates/ci/ecr-auth.gitlab-ci.yml"
 
image: docker:27
services:
  - docker:dind
 
before_script:
  - !reference [.ecr_auth, before_script]
 
after_script: docker logout $AWS_ACCOUNT.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
 
# Export ECR_AUTH_CONFIG for remote repository
ecr_auth:
  stage: .pre
  image: docker:27
  services:
    - docker:dind
  when: always
  script:
    - apk add jq
    - echo "AUTH_CONFIG=$(cat /root/.docker/config.json | jq -r ".auths.\"private.container.registry.com\".auth")" >> auth.env
 
  rules:
    - when: on_success
 
  artifacts:
    reports:
      dotenv: auth.env

Follow the technique in the second part, you can reuse template for login ECR, and try to import that with include and !reference keyword
After run, before script, It mean your docker+machine runner work and authenticate to private ECR, now we add jq for help us manipulate and analysis json file, which one format of /root/.docker/config.json and help you retrieve the token when you use docker login command (NOTE: Base64 is actually work and you not need to do anything).
After analysis and retrieve the token, Gilab offer us the methodology to export that result into .env because file is tough for process,therefore with simple string we have easily ways to apply that in other repository. Read more at GitLab CI/CD artifacts reports types

Warning

Remember expose rules for any situation because need or trigger will base on that for condition which execute command in remote repository

Lastly, we expose that via artifacts, and you will create downstream pipeline which one retrieve that value

But wait, you need do one more thing. Because of expiration of AWS ECR, you need have strategy to renew that. You don’t have much time to prepare what time we need to trigger pipeline again. That why we need schedule to handle, and Gitlab provide it for us inside crontab and very easily. Read more at Scheduled pipelines

For example, I create pipeline schedule with strategy

Rerun after every 55 minutes, practice with cron guru
Select time zone, for example Singapore but it not important in situation
Select branch, I choose main branch
You can add more custom variable when run pipeline
Click create and enjoy

Create Downstream pipeline

With downstream pipeline you have two option

Use trigger to call remote repository, but that tough and to control
Use need to receive artifact from remote repository

Follow we will use need to define downstream pipeline, for easily expose your token. You will need to practice with curl command with GitlabAPI Project-level CI/CD variables API

.gitlab-ci.yml

# Downstream use need to receive artiface type
stages:
  - auth
  - ci-stuff
 
authentication:
  stage: auth
  image: alpine:3 # recommend for cut of time
  needs:
    - project: "gitlab/upstream/project"
      job: private_auth # same name with job expose token
      ref: "main" # branch you want to receive the artifact
      artifacts: true
  script:
    - apk add curl jq
    - DOCKER_AUTH_CONFIG=$(echo "{\"auths\":{\"private.container.registry.com\":{\"auth\":\"$(printf "%s" "$AUTH_CONFIG")\"}}}")
    - 'UPDATE_CONDITION=$(curl -XGET --header "PRIVATE-TOKEN: $GITLAB_AUTH_TOKEN" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/variables/DOCKER_AUTH_CONFIG")'
    - >
      if [ "$UPDATE_CONDITION" == "{\"message\":\"404 Variable Not Found\"}" ]; then
        echo "Add new DOCKER_AUTH_CONFIG variable"
        curl -XPOST --header "PRIVATE-TOKEN: $GITLAB_AUTH_TOKEN" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/variables" --form "key=DOCKER_AUTH_CONFIG" --form "value=$DOCKER_AUTH_CONFIG" --form "raw=true"  > /dev/null 2>&1
      else
        echo "Update DOCKER_AUTH_CONFIG variable"
        curl -XPUT --header "PRIVATE-TOKEN: $GITLAB_AUTH_TOKEN" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/variables/DOCKER_AUTH_CONFIG" --form "value=$DOCKER_AUTH_CONFIG" --form "raw=true"  > /dev/null 2>&1
      fi
 
# Job for check successfull authentication ECR
recheck:
  stage: ci-stuff
  image: private.container.registry.com/image:latest
  script:
    - echo "I am in private container !!! bruh"

With this CI, we will do twice stuff, authentication and check pull private image successfully or not
Use need to retrieve the artifact in dotenv in Gitlab, because gitlab provide us many type of artifact and dotenv is one of them. Read more at GitLab CI/CD artifacts reports types
Use script to install some we need curl because alpine:3 is lighweigh image and do not include anything inside. After that, we trigger curl command to execute API of Gitlab to set string DOCKER_AUTH_CONFIG inside machine with created via artifact to CI/CD variables
Now your recheck job will pass with in-use private image, how cool is this 🙌🙌🙌

For optimize pipeline, and easily reuse by other repository

With idea cut off the effort for user who want to implement pipeline, you just change job from execute to hidden with . form with become template for reuse by remote repositories, like these

ecr_auth.gitlab-ci.yml

.ecr_auth:
  before_script: &ecr_auth
    - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
    - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
    - export AWS_ACCOUNT=$AWS_ACCOUNT
    - apk add aws-cli
    - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
 
.ecr_remote_auth:
  stage: test
  image: alpine:3
  needs:
    - project: "gitlab/upstream/project"
      job: private_auth # same name with job expose token
      ref: "main" # branch you want to receive the artifact
      artifacts: true
  script:
    - apk add curl jq
    - DOCKER_AUTH_CONFIG=$(echo "{\"auths\":{\"private.container.registry.com\":{\"auth\":\"$(printf "%s" "$AUTH_CONFIG")\"}}}")
    - 'UPDATE_CONDITION=$(curl -XGET --header "PRIVATE-TOKEN: $GITLAB_AUTH_TOKEN" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/variables/DOCKER_AUTH_CONFIG")'
    - >
      if [ "$UPDATE_CONDITION" == "{\"message\":\"404 Variable Not Found\"}" ]; then
        echo "Add new DOCKER_AUTH_CONFIG variable"
        curl -XPOST --header "PRIVATE-TOKEN: $GITLAB_AUTH_TOKEN" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/variables" --form "key=DOCKER_AUTH_CONFIG" --form "value=$DOCKER_AUTH_CONFIG" --form "raw=true"  > /dev/null 2>&1
      else
        echo "Update DOCKER_AUTH_CONFIG variable"
        curl -XPUT --header "PRIVATE-TOKEN: $GITLAB_AUTH_TOKEN" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/variables/DOCKER_AUTH_CONFIG" --form "value=$DOCKER_AUTH_CONFIG" --form "raw=true"  > /dev/null 2>&1
      fi

And from remote, you can use both include and extends keywords for retrieve this template, super easy 😄

.gitlab-ci.yml

# Downstream use need to receive artiface type
stages:
  - auth
  - ci-stuff
 
include:
  - project: "gitlab/upstream/project" # Not include gitlab.com
    file: "/templates/ci/ecr-auth.gitlab-ci.yml"
    ref: "main"
 
authentication:
  # Easily reuse with no one understand syntax inside user gitlab
  extends: .ecr_remote_auth
 
# Job for check successfull authentication ECR
recheck:
  stage: ci-stuff
  image: private.container.registry.com/image:latest
  script:
    - echo "I am in private container !!! bruh"

xeusnguyen.xyz

Explorer

Recent Notes

First EKS Cluster with Milvus DB

Integration unit test with GitLab CI

GitlabCI

Introduce GitlabCI

General

Syntax

Features

Use cases and scenarios

Case 1: Completely pipeline for Container Services

Case 2: Build & push docker image to private registry

Case 3: Authentication Job with private ECR from remote repository

Create Upstream Pipeline

Create Downstream pipeline

For optimize pipeline, and easily reuse by other repository

Graph View

Table of Contents

Backlinks