DevOps Blog - Nicolas Paris

Kubernetes, Helm, Istio, Laravel, PHP-FPM, Nginx, GitLab the DevOps Way

KubernetesDevOpsLaravelIstio

I'll try to explain a full setup from the DevOps side of a full stack web application based on Laravel. Here is technologies that we will discuss here.

The application will be a single application with front and backend on the same git repository.

I made some updates on this post. This post is the big picture, and I go in some more details in the following related post.

Nginx, PHP-FPM and a Laravel application

The approch here is each pod of a ReplicaSet will contains the following.

This mean it will scale as much nginx/php-fpm that we have pods.
The Nginx container and the PHP-FPM will need either JS/PNG/CSS or PHP.
The PHP-FPM will be your container that contains everythings, the one you build on day to day basis with unit tests and everything. The Nginx container will receive the needed files on Init, will make a simple copy at start of the pod. This mean very low maintenance on the nginx container.

We need two parts, two Dockerfile for two containers.

Laravel / php-fpm Dockerfile

FROM node:lts-alpine as node_build
WORKDIR /app
COPY package.json ./
RUN npm install
COPY webpack.mix.js ./
COPY resources/ ./resources/
COPY public/ ./public/
# fail du purge sinon
RUN mkdir -p /public/css
RUN touch /public/css/app.css
RUN mkdir -p /public/js
RUN touch /public/js/app.js
RUN npm run prod

FROM composer:2.1.9 as composer_build
# voir pour le .lock
COPY ./composer.json /app/
RUN composer install --no-dev --no-autoloader --no-scripts
COPY . /app
RUN composer install --no-dev --optimize-autoloader

FROM php:8.0-fpm-alpine
RUN mv "$PHP_INI_DIR/php.ini-production" "$PHP_INI_DIR/php.ini"
RUN docker-php-ext-install pdo pdo_mysql
COPY devops/docker/php/*.conf /usr/local/etc/php-fpm.d/
COPY --chown=www-data --from=composer_build /app/ /var/www/html/
COPY --from=node_build /app/public/ /var/www/html/public/
RUN php artisan view:cache
# && php artisan route:cache \ fonctionne pas pour le moment

This is a multi-stage build of Docker

I add the following into the PHP configuration, to make php-fpm work.

A more in depth explaination is done here Docker Best Practice, Multi-Stage Build

security.limit_extensions = php

Nginx Dockerfile

I'll make some basic configuation inside a build.

FROM nginx:1.21.3-alpine

COPY vhost.conf /etc/nginx/conf.d/default.conf

With a vhost that contains this.

server {
listen 80;
listen [::]:80;
server_name localhost;

autoindex on;
index index.php;

# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}


location / {
try_files $uri $uri/ /index.php?$args;
}

location ~ \.php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /var/www/html/public/$fastcgi_script_name;
include fastcgi_params;
}

location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
expires max;
add_header Cache-Control public;
access_log off;
}
}

I does not have lots of experience with this setup, but this is what I came up with, and it works.

With this done, I have a Laravel up and running with php-fpm and nginx. You build both independently.

Kubernetes, first approch

Let's build something that work on minikube, and lets make it simple, a single replica of 2 containers inside a pod. Let's call my-nginx, the customized nginx that make php calls to the other container build with Laravel, they both contains code, but it will be mounted on runtime.

apiVersion: apps/v1
kind: Deployment
metadata:
name: app
labels:
app: web
spec:
replicas: 1
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: nginx
image: "my-nginx:1"
ports:
- containerPort: 80
volumeMounts:
- mountPath: /etc/nginx/html/
name: assets
- name: php
image: "my-app"
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9000
initContainers:
- name: init-assets
image: "my-app"
imagePullPolicy: IfNotPresent
command: ['sh', '-c', "cp -r /var/www/html/public/* /assets/"]
volumeMounts:
- mountPath: /assets/
name: assets
volumes:
- name: assets
emptyDir: {}

---

apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
type: NodePort
selector:
app: web
ports:
# By default and for convenience, the `targetPort` is set to the same value as the `port` field.
- port: 80
targetPort: 80

It's just a service on Port 80 (we'll see later the TLS support), it point on the nginx container that will send php request to my-app.

Helm, Istio, Let's encrypt

This is a real life approch, let' have some tools working for us. It will be quite long as it contains lots of yaml. Let's do it.
First, we setup the certificate issuers with let's encrypt if is not aleady done. This will make it work with istio.
I made an update on ths part (Istio and Let's Encrypt) with a more detail approch.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: kube-system
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: xxx@gmail.xxx
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: istio

Now, the istio gateway.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: cluster-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- my-website.com
- port:
number: 443
name: my-website
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: "my-website-certs" # This should match the Certificate secretName
hosts:
- "my-website.com" # This should match a DNS name in the Certificate

Be sure to apply your gateway change on kubernetes to retreive the certificate right.
You can check with this command.

kubectl get certificates -n istio-system

If something goes wrong, with describe can show with object to check for (order, challanges if I remember well).

Helm

I am not showing every single file, juste the one I added for certificates or istio.
Let's describe the values.yaml.

# values.yaml
replicaCount: 1
namespace: default

image:
repository: xxxxx
pullPolicy: Always
# Overrides the image tag whose default is the chart appVersion.
tag: $TAG

imagePullSecrets:
- name: gcr-io

nameOverride: ""
fullnameOverride: ""

serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""

podAnnotations:
"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
"sidecar.istio.io/proxyCPU": "20m"

virtualService:
url: "xxx.xxx.fr"

podSecurityContext: {}
# fsGroup: 2000

securityContext: {}
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000

service:
type: ClusterIP
port: 80

resources:
limits:
cpu: 300m
memory: 512Mi
requests:
cpu: 20m
memory: 128Mi

env:
LOG_CHANNEL: stderr
APP_ENV: local # FIXME
APP_NAME: xxx
APP_KEY: base64:xxx=
APP_DEBUG: true # FIXME
APP_URL: https://xxx.fr
DB_CONNECTION: mysql
DB_HOST: "xxx"
DB_DATABASE: xxx
DB_USERNAME: xxx
DB_PASSWORD: $DB_PASSWORD_PREPROD


autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
# targetMemoryUtilizationPercentage: 80

nodeSelector: {}

tolerations: []

affinity: {}

If you wender where the ImagePullSecret gcr-io come from, I made a post where more details on this. Kubernetes and Google Container Registry

Now, for the template part of Helm

First, a certificates that works with let's encript and istio.

# certificates.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: {{ include "xxx.fullname" . }}-certs
namespace: istio-system
spec:
secretName: {{ include "xxx.fullname" . }}-certs
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
commonName: {{ .Values.virtualService.url }}
dnsNames:
- {{ .Values.virtualService.url }}

Now, a simple loop for the configMap.yaml, with a simple key / value iteration.

apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "nci-authentication.fullname" . }}-env
namespace: {{ .Values.namespace }}
data:
{{- range $key, $val := .Values.env }}
{{ $key }}: {{ $val | quote }}
{{- end }}

I guess I have to copy/paste my deployment.yaml as it contains the initialization part, and the configMap. I know it will be verbose, and I apologies for this.

apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "xxx.fullname" . }}
namespace: {{ .Values.namespace }}
labels:
{{- include "xxx.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "xxx.selectorLabels" . | nindent 6 }}
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
rollme: {{ randAlphaNum 5 | quote }} # force le roll des pods, surprennant, mais dans la doc officielle
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "xxx.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "xxx.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:

- name: {{ .Chart.Name }}
image: "my-nginx:1"
imagePullPolicy: Always
ports:
- name: http
containerPort: 80
protocol: TCP
volumeMounts:
- mountPath: /etc/nginx/html/
name: assets

- name: {{ .Chart.Name }}-backend
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: fpm
containerPort: 9000
protocol: TCP
# livenessProbe:
# httpGet:
# path: /
# port: http
# readinessProbe:
# httpGet:
# path: /
# port: http
resources:
{{- toYaml .Values.resources | nindent 12 }}
env:
{{- $dot := . }}
{{- range $key, $val := .Values.env }}
- name: {{ $key }}
valueFrom:
configMapKeyRef:
name: {{ include "xxx.fullname" $dot }}-env
key: {{ $key }}
{{- end }}

initContainers:
- name: init-assets
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: IfNotPresent
command: ['sh', '-c', "cp -r /var/www/html/public/* /assets/"]
volumeMounts:
- mountPath: /assets/
name: assets

volumes:
- name: assets
emptyDir: {}

{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}

This is the same approch as earlier, with a copy of code in the InitContainers. I'm not sure it is the best approch, but this is a low maintenance approch with absolutly 100% certenty to run the right code on each node.

I'm not showing the hpa.yaml, and the service.yaml is not very interreseting.

apiVersion: v1
kind: Service
metadata:
name: {{ include "xxx.fullname" . }}
namespace: {{ .Values.namespace }}
labels:
{{- include "xxx.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "xxx.selectorLabels" . | nindent 4 }}

Last, the virtualservice.yaml is specific to the istio part.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: {{ include "xxx.fullname" . }}
spec:
hosts:
- {{ .Values.virtualService.url }}
gateways:
- cluster-gateway
http:
- match:
- uri:
prefix: "/"
route:
- destination:
host: {{ include "xxx.fullname" . }}.{{ .Values.namespace }}.svc.cluster.local
port:
number: 80

I wont explain helm commands to install or upgrade releases, this is not a beginner tutorial on how to deploy with helm.

Now, this is running on a Kubernetes cluster, with Istio.

Gitlab

Let's just add a runner to this, it's not yet complete but I'll add an update soon.

Update: a more in depht post as been added : Gitlab Example with Laravel, Kebernetes and Helm

.gitlab-ci.yaml

 variables:
IMAGE: eu.gcr.io/xxxx/xxxx
TAG: "${CI_COMMIT_SHORT_SHA}"
CONNECT_K8S_PREPROD_CMD: gcloud container clusters get-credentials kube-xxxx --zone europe-west1-b --project xxxx

stages:
- build
- test
- maintenance_down
- migration
- maintenance_up
- delivery_staging

build:
stage: build
tags:
- gcp-shell
script:
- docker build -t ${IMAGE}:${TAG} .
- gcloud auth activate-service-account gitlab-push-container@xxxx.gserviceaccount.com --key-file=/gitlab.json
- gcloud auth configure-docker
- docker push ${IMAGE}:${TAG}

test:
stage: test
image: ${IMAGE}:${TAG}
tags:
- gcp-docker
before_script:
- curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
- cd /var/www/html
- composer install # on a besoin des dépendences de dev (contrairement à la prod)
- apk add gettext # envsubst
- envsubst < /var/www/html/devops/.env.test > /var/www/html/.env
- php artisan key:generate
script:
- ./vendor/phpunit/phpunit/phpunit

maintenance_down:
stage: maintenance_down
tags:
- gcp-shell
script:
- echo "MAINTENANCE=DOWN" >> build.env
- kubectl get po -l app.kubernetes.io/name=my-app-backend -o name | xargs -I{} kubectl exec {} -c my-app-backend -- php artisan down
artifacts:
reports:
dotenv: build.env

migration:
stage: migration
image: ${IMAGE}:${TAG}
tags:
- gcp-docker
before_script:
- apk add gettext # envsubst
- envsubst < /var/www/html/devops/.env.test > /var/www/html/.env
- cd /var/www/html
- php artisan key:generate
script:
- cd /var/www/html
- php artisan migrate

maintenance_up:
stage: maintenance_up
dependencies:
- maintenance_down
tags:
- gcp-shell
when: always
script:
- echo $MAINTENANCE
- ${CONNECT_K8S_PREPROD_CMD}
- (if [ "$MAINTENANCE" == "DOWN" ]; then kubectl get po -l app.kubernetes.io/name=my-app -o name | xargs -I{} kubectl exec {} -c my-app-backend -- php artisan up; fi);


delivery_staging:
stage: delivery_staging
tags:
- gcp-shell
script:
- VALUES_FILE=devops/helm/stage.yaml
- ${CONNECT_K8S_PREPROD_CMD}
- envsubst < ${VALUES_FILE} > devops/helm/dist-stage.yaml
- helm upgrade -f devops/helm/dist-stage.yaml xxxxx devops/helm/

There is a lot going on in this file, i'll explain in more details soon in a post. I'll give some update here soon.
Update: a more in depht post as been added : Gitlab Example with Laravel, Kebernetes and Helm

Conclusion

This is a full setup of an application with Kubernetes, Istio (i'm not naming every technologies again...). I'm not going to deep on explainations, but I'll give some update soon.

I hope it could help someone, as it took me some time to summarize all this.