DevOps Blog - Nicolas Paris

Gitlab CI/CD example with Laravel, Kubernetes and Helm

DevOpsGitlab

Here is a full configuration file for Continuous Integration / Continuous Delivery (CI/CD) pipeline with Gitlab.

This is a three part article, meaning if you want explanations, just go down.

Here are some technical consideration used in the post

Entire Configuration file

# .gitlab-ci.yaml
variables:
IMAGE: eu.gcr.io/xxx/xxx
TAG: "${CI_COMMIT_SHORT_SHA}"
CONNECT_K8S_PREPROD_CMD: gcloud container clusters get-credentials xxx --zone europe-west1-b --project xxx
CONNECT_K8S_PROD_CMD: gcloud container clusters get-credentials xxx --region europe-west1 --project xxx
GIT_DEPTH: 1

stages:
- build
- test
- maintenance_down
- migration
- delivery

build:
stage: build
tags:
- gcp-shell
except:
- tags
script:
- docker build -t ${IMAGE}:${TAG} .
- gcloud auth activate-service-account gitlab-push-container@xxx.iam.gserviceaccount.com --key-file=/gitlab.json
- gcloud auth configure-docker
- docker push ${IMAGE}:${TAG}

test:
stage: test
image: ${IMAGE}:${TAG}
services:
- mysql:latest
variables:
MYSQL_DATABASE: nci_authentication
MYSQL_ROOT_PASSWORD: root
tags:
- gcp-docker
except:
- tags
before_script:
- curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
- cd /var/www/html
- composer install
- apk add gettext # envsubst
- envsubst < /var/www/html/devops/.env.test > /var/www/html/.env
- php artisan key:generate
- php artisan migrate
script:
- ./vendor/phpunit/phpunit/phpunit

maintenance_down:
stage: maintenance_down
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
rules:
- if: $CI_COMMIT_BRANCH == "master"
when: on_success
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PROD_CMD}
- if: $CI_COMMIT_BRANCH == "develop"
when: on_success
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
- when: never
tags:
- gcp-shell
script:
- ${CONNECT_CLUSTER}
- kubectl get po -l app.kubernetes.io/name=xnet -o name | xargs -I{} kubectl exec {} -c xnet-backend -- php artisan down

migration:
stage: migration
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
ENV_FILE: /var/www/html/devops/.env.stage
rules:
- if: $CI_COMMIT_BRANCH == "master"
when: on_success
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PROD_CMD}
ENV_FILE: /var/www/html/devops/.env.prod
- if: $CI_COMMIT_BRANCH == "develop"
when: on_success
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
ENV_FILE: /var/www/html/devops/.env.stage
- when: never
image: ${IMAGE}:${TAG}
tags:
- gcp-docker
before_script:
- apk add gettext # envsubst
- envsubst < ${ENV_FILE} > /var/www/html/.env
- cd /var/www/html
- php artisan key:generate
script:
- cd /var/www/html
- php artisan migrate

delivery:
stage: delivery
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
VALUES_FILE: devops/helm/stage.yaml
rules:
- if: $CI_COMMIT_BRANCH == "master"
when: on_success
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PROD_CMD}
VALUES_FILE: devops/helm/prod.yaml
- if: $CI_COMMIT_BRANCH == "develop"
when: on_success
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
VALUES_FILE: devops/helm/stage.yaml
- when: never
tags:
- gcp-shell
script:
- ${CONNECT_CLUSTER}
- envsubst < ${VALUES_FILE} > devops/helm/dist.yaml
- helm upgrade -f devops/helm/dist.yaml xnet devops/helm/

About the runners and installation

Before even talk about the CI/CD himself, let's see how runners was setup. I'm sure I could come up with several ways to handle this.

The installation of the gitlab-runner is done with docker.
Two runners was registred, one with the docker mode, the other with shell mode.

On a fresh compute instance, docker was installed

curl https://baltocdn.com/helm/signing.asc | apt-key add -
apt-get install apt-transport-https --yes
echo "deb https://baltocdn.com/helm/stable/debian/ all main" | tee /etc/apt/sources.list.d/helm-stable-debian.list

Few more binary was install, it will be available on the shell runner.

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
apt-get update
apt-get install -y apt-transport-https ca-certificates curl
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubectl

We need gcloud as the container registry is on Google Cloud Platform.

apt-get install apt-transport-https ca-certificates gnupg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
apt-get update && apt-get install google-cloud-sdk
cat <<EOT >> /gitlab.json
{
"type": "service_account",
"project_id": "xxx",
"private_key_id": "2xxxd",
"private_key": "-----BEGIN PRIVATE KEY-----\xxx\n-----END PRIVATE KEY-----\n",
"client_email": "xxx@xxx.iam.gserviceaccount.com",
"client_id": "1xxx2",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/xxxiam.gserviceaccount.com"
}
EOT

gcloud auth activate-service-account xxx@xxx.iam.gserviceaccount.com --key-file=/gitlab.json
gcloud auth configure-docker

This will usefull to manipulate some files like envsubst < ${VALUES_FILE} > devops/helm/dist.yaml.

apt-get install gettext-base

Interact with the runner

You'll might need interact with the gitlab-runner container. You can go inside the container and executes commandes like this.

docker exec -it gitlab-runner /bin/bash
# Now you are inside the container
cat /etc/gitlab-runner/config.toml
gitlab-runner register
gitlab-runner reload
gitlab-runner restart

Or send commande from outside the container like this, it will output the result.

docker exec gitlab-runner gitlab-runner verify

Configuration file explanation

variables

Build

build:
stage: build
tags:
- gcp-shell
except:
- tags
script:
- docker build -t ${IMAGE}:${TAG} .
- gcloud auth activate-service-account gitlab-push-container@xxx.iam.gserviceaccount.com --key-file=/gitlab.json
- gcloud auth configure-docker
- docker push ${IMAGE}:${TAG}

test

test:
stage: test
image: ${IMAGE}:${TAG}
services:
- mysql:latest
variables:
MYSQL_DATABASE: xnet
MYSQL_ROOT_PASSWORD: root
tags:
- gcp-docker
except:
- tags
before_script:
- curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
- cd /var/www/html
- composer install # on a besoin des dépendences de dev (contrairement à la prod)
- mv /var/www/html/devops/.env.test /var/www/html/.env
- php artisan key:generate
- php artisan migrate
script:
- ./vendor/phpunit/phpunit/phpunit
# .env.test
APP_NAME=Laravel
APP_ENV=local
APP_KEY=

DB_CONNECTION=mysql
DB_HOST=mysql
DB_PORT=3306
DB_DATABASE=xnet
DB_USERNAME=root
DB_PASSWORD=root

This .env.test will contains what is needed for running tests.

Maintenance down

maintenance_down:
stage: maintenance_down
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
rules:
- if: $CI_COMMIT_BRANCH == "master"
when: on_success
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PROD_CMD}
- if: $CI_COMMIT_BRANCH == "develop"
when: on_success
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
- when: never
tags:
- gcp-shell
script:
- ${CONNECT_CLUSTER}
- kubectl get po -l app.kubernetes.io/name=xnet -o name | xargs -I{} kubectl exec {} -c xnet-backend -- php artisan down

Migration

migration:
stage: migration
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
ENV_FILE: /var/www/html/devops/.env.stage
rules:
- if: $CI_COMMIT_BRANCH == "master"
when: on_success
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PROD_CMD}
ENV_FILE: /var/www/html/devops/.env.prod
- if: $CI_COMMIT_BRANCH == "develop"
when: on_success
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
ENV_FILE: /var/www/html/devops/.env.stage
- when: never
image: ${IMAGE}:${TAG}
tags:
- gcp-docker
before_script:
- apk add gettext # envsubst
- envsubst < ${ENV_FILE} > /var/www/html/.env
- cd /var/www/html
- php artisan key:generate
script:
- cd /var/www/html
- php artisan migrate
APP_NAME=Laravel
APP_ENV=local
APP_KEY=

DB_CONNECTION=mysql
DB_HOST=10.10.10.10
DB_PORT=3306
DB_DATABASE=xnet
DB_USERNAME=foo
DB_PASSWORD=$DB_PASSWORD_PREPROD

Delivery

delivery:
stage: delivery
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
VALUES_FILE: devops/helm/stage.yaml
rules:
- if: $CI_COMMIT_BRANCH == "master"
when: on_success
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PROD_CMD}
VALUES_FILE: devops/helm/prod.yaml
- if: $CI_COMMIT_BRANCH == "develop"
when: on_success
variables:
CONNECT_CLUSTER: ${CONNECT_K8S_PREPROD_CMD}
VALUES_FILE: devops/helm/stage.yaml
- when: never
tags:
- gcp-shell
script:
- ${CONNECT_CLUSTER}
- envsubst < ${VALUES_FILE} > devops/helm/dist.yaml
- helm upgrade -f devops/helm/dist.yaml xnet devops/helm/
env:
LOG_CHANNEL: stderr
DB_PASSWORD: $DB_PASSWORD_PREPROD

Summary

This configuration allow a full build and deployment from a Laravel build to a production Kubernetes cluster.
There is other way to approch the problem, here is the one we will have in production.