Skip to content

CI/CD Pipeline

The GitLab pipeline is split into per-area build jobs gated on rules:changes:. Pushes that only touch one area (admin / backend, web frontend, or POS frontend) skip the build jobs and PM2 reloads for the other areas. Pushes that touch nothing build-relevant skip the deploy entirely.

Pipeline structure

Three branches drive deploys:

Branch Environment Runner tag
develop dev develop
uat UAT uat
main live (production) main

This document covers uat and main (live). develop is currently the unchanged legacy pipeline.

Each pushed commit on uat or main evaluates these jobs:

Stage Job Fires when
build build_backend_uat / build_backend_live backend/, scripts/, start_live.sh, ecosystem.config.js, or .gitlab-ci.yml changed
build build_web_frontend_uat / build_web_frontend_live frontend/web/, frontend/common/, or .gitlab-ci.yml changed
build build_pos_frontend_uat / build_pos_frontend_live frontend/pos/, frontend/common/, or .gitlab-ci.yml changed
deploy deploy_uat / deploy_live any of the above paths changed

.gitlab-ci.yml is included in every job's filter as a safety net so a change to the pipeline itself always runs the full pipeline.

Selective deploy mechanism

Each build job writes a one-line dotenv artifact when it succeeds:

Build job Writes file Sets variable
build_backend_* build_backend_scope.env BUILT_BACKEND=1
build_web_frontend_* build_web_scope.env BUILT_WEB=1
build_pos_frontend_* build_pos_scope.env BUILT_POS=1

The filenames are different on purpose. GitLab keeps each job's dotenv artifact and merges them into the deploy job's environment via the artifacts:reports:dotenv mechanism. If two jobs used the same filename, only one would survive — that bug bit us once.

The deploy script (start_live.sh for live, scripts/deploy_uat.sh for UAT) reads the flags and runs only the corresponding sections:

if [ "${BUILT_BACKEND:-0}" = "1" ]; then
    composer install
    composer dump-autoload
    php artisan vendor:publish ...
    php artisan route:clear
    php artisan config:clear
fi
php artisan storage:link            # always (idempotent, cheap)
pm2 start ecosystem.config.js       # always (idempotent, cold-start safe)

APPS=()
[ "${BUILT_BACKEND:-0}" = "1" ] && APPS+=(voucher order webPrinter laravelEcho laravelQueues)
[ "${BUILT_WEB:-0}" = "1" ]     && APPS+=(web)
[ "${BUILT_POS:-0}" = "1" ]     && APPS+=(pos)

# pm2 reload only accepts one process per call in the installed version,
# so we loop instead of passing all names at once.
for app in "${APPS[@]}"; do
    pm2 reload "$app"
done

The pm2 start ecosystem.config.js line is a no-op for already-running processes, but it bootstraps a cold-start server (or after pm2 delete all) so the per-app reload that follows has something to reload.

Build scripts

Live (referenced from .gitlab-ci.yml):

  • scripts/build_backend_live.sh — installs frontend/common/utils first, then builds the Laravel microservices (with webpack) and the Nova components in parallel via nohup ... & plus a wait at the end so the script doesn't exit while builds are still running.
  • scripts/build_web_frontend_live.sh — installs common/utils, then runs npm install, npm run build, npm run generate in frontend/web/.
  • scripts/build_pos_frontend_live.sh — same as above for frontend/pos/.
  • scripts/build_live.sh — DEPRECATED but kept as a manual fallback; no longer wired into CI.

UAT uses the existing build_backend.sh, build_web_frontend.sh, build_pos_frontend.sh from scripts/. They differ from the live scripts mainly in which config dir they rsync (~/config/ vs ~/config_live/).

scripts/deploy_live.sh rsyncs the runner workspace to ~/tops/ and then calls start_live.sh from there. scripts/deploy_uat.sh does the rsync and selective reload inline (no separate start_*.sh).

How frontend/common/utils fits in

The Laravel microservices import from frontend/common/utils/* (e.g. dealSchedule.js, orderProcessor.js), which themselves require moment, slug, moment-timezone. Those dependencies must be installed before the microservice's webpack build runs, otherwise webpack fails with "Module not found: moment".

GitLab runs git clean -ffdx between pipelines (default GIT_STRATEGY: fetch), so node_modules is wiped and a fresh npm install is required every time. Each build script that touches the microservice runs cd frontend/common/utils && npm install up front.

Servers and runners

Environment Runner workspace Deploy target
UAT ~/builds/t3_dxX7hs/0/standbyteam/topspizza.co.uk/ (user uattopspizzaco on svr01) /home/uattopspizzaco/tops/
Live ~/builds/WLymjMOvx/0/standbyteam/topspizza.co.uk/ ~/tops/ (whatever user runs gitlab-runner for tags: main)

Composer is required on the deploy server for the artisan steps in start_live.sh. The runner does not run composer install for live (microservice + nova-components only). UAT's build_backend.sh does run composer on the runner, so composer must be in PATH for the runner user too.

Adding a new PM2 process

If you add a process to ecosystem.config.js, also update the hardcoded app lists in:

  • start_live.sh — the APPS+=(...) lines under the correct BUILT_* flag
  • scripts/deploy_uat.sh — same

If you forget, deploys will still succeed (the pm2 start ecosystem.config.js call bootstraps the new app on first run), but subsequent selective deploys won't reload it.

Adding a new build path

If a new top-level directory needs to participate (e.g. you add frontend/driver-app/ to the live deploy), update:

  1. The corresponding build_*_live.sh script to build it.
  2. The rules:changes: block on the relevant build job in .gitlab-ci.yml to include the new path.
  3. The rules:changes: block on deploy_live / deploy_uat (which lists the union of all build-tracked paths).

First deploy after a CI change

Any push that touches .gitlab-ci.yml, scripts/, or start_live.sh will fire ALL three build jobs because those paths appear in every filter. This is intentional — it forces a full rebuild whenever the pipeline itself changes, so artifacts on the runner stay in sync with the new CI logic. Subsequent pushes will go back to selective behavior.

Recovery procedures

Production is in a partial state (some apps reloaded, some not)

ssh <user>@<host>
pm2 reload all
pm2 list                # verify uptime reset across all 7 apps

This brings every process to the current code on disk. It's a no-downtime cluster reload, safe to run.

Force a full rebuild

Either touch .gitlab-ci.yml (whitespace change), or trigger a manual pipeline run from the GitLab UI. Both fire every job because of the safety-net path filter.

Runner workspace got wiped between pipelines

Same fix — force a full rebuild. After one full rebuild, the workspace has fresh artifacts in every section and selective deploys work again.

Pipeline says SUCCESS but server isn't updated

Usually means one of:

  • The deploy job ran but the relevant build job was skipped, so its BUILT_* flag was unset and that section's reload was skipped. Check pm2 list uptimes per app.
  • The tmux new-session -d ... deploy_*.sh from the CI job exited before the deploy actually finished. The CI marks the job passed based on the tmux launch, not the deploy. Look at the deploy log on the server: tmux capture-pane -p -t tops_deploy_live or check ~/tops/'s file mtimes.
  • The BUILT_* env vars didn't make it through tmux. When a tmux server is already running on the host, tmux new-session inherits env from that server, not from the calling CI job — so dotenv artifacts effectively get stripped. The deploy job inlines the values on the tmux command line specifically to avoid this:

yaml tmux new-session -d -s tops_deploy_live \ "BUILT_BACKEND=${BUILT_BACKEND:-0} BUILT_WEB=${BUILT_WEB:-0} BUILT_POS=${BUILT_POS:-0} ./deploy_live.sh"

If you change this line, preserve the inline assignment. Bash expands the variables in the CI shell before tmux runs, so the values are baked into the command string.

Known limitations

  • Single runner per tag: the live and UAT runners each have one shell executor, so build jobs in a single pipeline run sequentially, not in parallel. Acceptable trade-off given the runner is also the deploy server.
  • Detached deploy: the deploy script runs in tmux new-session -d ..., so GitLab marks the deploy job as passed as soon as tmux is launched. The actual deploy may still be running when the pipeline UI shows green. SSH into the server and check pm2 list or tmux ls to confirm the deploy completed.
  • pm2 reload single-arg constraint: the installed PM2 version doesn't accept multiple process names in one pm2 reload call. The deploy scripts loop instead. If you upgrade PM2 and the loop becomes unnecessary, the syntax pm2 reload "${APPS[@]}" would also work.
  • Composer lock drift: if composer.json is edited without refreshing composer.lock locally, composer install on the deploy server fails with "lock file is not up to date". Always run composer update <package> locally before pushing.