In the ever-evolving landscape of DevOps and continuous integration/continuous deployment (CI/CD), finding the right tools and strategies to streamline your development pipeline is crucial. GitLab has emerged as a powerful platform for managing source code, collaboration, and automation. One of the key features of GitLab CI/CD is its flexibility when it comes to choosing executors for your pipelines.

An executor, in the context of GitLab CI/CD, is essentially the engine that executes the jobs defined in your pipeline. It’s the workhorse responsible for running the tasks specified in your CI/CD configuration. GitLab offers a variety of default executors that cater to different use cases, making it adaptable to a wide range of projects and workflows.

The supported executors by default in GitLab include (this list isn’t exhaustive):

Shell Executor: This is the simplest executor, using the shell to execute jobs on the GitLab Runner server.
Docker Executor: Docker is a popular choice for containerization, and this executor allows you to run your jobs within Docker containers, isolating dependencies and ensuring consistency.
Kubernetes Executor: If you’re using Kubernetes for your infrastructure, this executor enables you to run jobs within Kubernetes pods, taking advantage of the scalability and flexibility of Kubernetes clusters.
Parallels Executor: This executor is designed for macOS-specific CI/CD jobs, providing support for macOS environments.

While these default executors cover many scenarios, GitLab also offers the flexibility to create custom executors tailored to your specific needs. Custom executors can be configured to use different virtualization technologies, cloud providers, or even unique hardware setups. They give you the freedom to adapt GitLab CI/CD to your infrastructure and requirements, ensuring that your pipelines are optimized for your particular use case.

In this two-part blog series, we’ll create one such custom executor that will allow us to run jobs within Multipass virtual machines: the Multipass Executor. That executor will launch a new multipass VM for every job it executes, after which the disk and VM will be deleted. We’ll explore how to harness the power of Multipass as an executor in your GitLab CI/CD workflows, offering a clean build environment for every build, close to your production environment and preventing jobs to access the file system hosting the runner. Whether you’re a seasoned GitLab user or just getting started, this guide will help you leverage Multipass to take your CI/CD processes to the next level. Let’s embark on this journey of optimizing your development pipeline with GitLab CI/CD and Multipass!

Before you begin

Install Gitlab runner

GitLab Runner must be installed on your local computer.

Please note that the GitLab Runner project may have evolved and introduced new installation methods since the day of this write-up. I recommend checking the official GitLab documentation for the most up-to-date information on how to install Gitlab Runner

So far, there are the commands I use on my Ubuntu computer:

sudo curl -L --output /usr/local/bin/gitlab-runner "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64"
sudo chmod +x /usr/local/bin/gitlab-runner
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start

Install Multipass

I think you already guess it, we need Multipass to be installed too for sure:

sudo snap install multipass

After you meet the requirements above, let’s get into the business.

Multipass Executor

According to GitLab documentation, the Multipass executor needs to follow a specific set of stages during the job execution process. The stages run in the following sequence:

Prepare: This initial stage involves setting up the runtime environment, which includes any necessary dependencies, tools, and configurations required for the job’s execution.
Run: This is the stage where the actual job script is executed. The runner runs the commands specified in the job’s script section. The commands perform the tasks defined in the CI/CD pipeline, such as building, testing, or deploying code.
Cleanup: In the final stage, the runner performs any necessary cleanup tasks. This may include removing temporary files, cleaning up resources, or executing post-job actions to ensure that the environment is left in a clean and consistent state.

For each stage, there’s a corresponding script that will be executed by GitLab runner. We will create a folder to store all the scripts needed:

mkdir -p /opt/multipass-driver/

I will detail below the content of each one.

Prepare

The Prepare stage is for setting up the environment. This script will do the following:

Destroy any existing VM with the same name
Start a VM and wait for it to start.
Install prerequisite dependencies.

We need to create the /opt/multipass-driver/prepare.sh file and paste in the content below:

#!/usr/bin/env bash

# /opt/multipass-driver/prepare.sh

currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base.

set -eo pipefail

# trap any error, and mark it as a system failure.
trap "exit $SYSTEM_FAILURE_EXIT_CODE" ERR

start_VM () {
    if multipass info "$VM_ID" >/dev/null 2>/dev/null ; then
        echo 'Found old VM, deleting'
        multipass delete --purge "$VM_ID"
    fi

    # The VM image is hardcoded, but you can use
    # the `CI_JOB_IMAGE` predefined variable
    # https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
    # which is available under `CUSTOM_ENV_CI_JOB_IMAGE` to allow the
    # user to specify the image. The rest of the script assumes that
    # you are running on an ubuntu image so modifications might be
    # required.
    multipass launch --name "$VM_ID" "$VM_IMAGE"

    # Wait for VM to start, we are using multipass list to check this,
    # for the sake of brevity.
    for i in $(seq 1 30); do
        if test "$(multipass list | grep $VM_ID | awk '{print $2}')" = "Running"; then
            break
        fi

        if [ "$i" == "30" ]; then
            echo 'Waited for 30 seconds to start VM, exiting..'
            # Inform GitLab Runner that this is a system failure, so it
            # should be retried.
            exit "$SYSTEM_FAILURE_EXIT_CODE"
        fi

        sleep 1s
    done
}

install_dependencies () {
    # Install Git LFS, git comes pre installed with ubuntu image.
    multipass exec "$VM_ID" -- sh -c 'curl -s "https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh" | sudo bash'
    multipass exec "$VM_ID" -- sh -c "sudo apt-get install -y git-lfs"

    # Install gitlab-runner binary since we need for cache/artifacts.
    multipass exec "$VM_ID" -- sh -c 'sudo curl -L --output /usr/local/bin/gitlab-runner "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64"'
    multipass exec "$VM_ID" -- sh -c "sudo chmod +x /usr/local/bin/gitlab-runner"
}

echo "Running in $VM_ID"

start_VM

install_dependencies

Run

This will run the script generated by GitLab Runner by sending the content of the script to the VM via STDIN. We need to create the /opt/multipass-driver/run.sh file and paste in the content below:

#!/usr/bin/env bash

# /opt/multipass-driver/run.sh

currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base.

multipass exec "$VM_ID" /bin/bash < "${1}"
if [ $? -ne 0 ]; then
    # Exit using the variable, to make the build as failure in GitLab
    # CI.
    exit $BUILD_FAILURE_EXIT_CODE
fi

Cleanup

This will destroy the VM and purge any associated data since the build has finished. We need to create the /opt/multipass-driver/cleanup.sh file and paste in the following content:

#!/usr/bin/env bash

# /opt/multipass-driver/cleanup.sh

currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base.

echo "Deleting VM $VM_ID"

multipass delete --purge "$VM_ID"

Base

You may have noticed that on the top of all the previous scripts, there’s a common piece of code:

currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base.

That code snippet is used in each stage (prepare, run, and cleanup) to get variables from the base script base.sh below. It’s important that this script is located in the same directory as the other scripts, in this case /opt/multipass-driver/.

#!/usr/bin/env bash

# /opt/multipass-driver/base.sh

VM_ID="runner-$CUSTOM_ENV_CI_RUNNER_ID-project-$CUSTOM_ENV_CI_PROJECT_ID-concurrent-$CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID-$CUSTOM_ENV_CI_JOB_ID"
VM_IMAGE="22.04"

Make the scripts executable

sudo chmod -R 775 /opt/multipass-driver/

Now that we got the Multipass executor ready, we need to test it.

Create a project

We will create a small project on GitLab (or any self-hosted instance) and name it for example, test-multipass-executor. In the screenshot below, you can see that the repo just contains two files: README.md and test.txt. The content of these files don’t matter.

Create a pipeline for the project

Next, create a .gitlab-ci.yml file at the root of the project repository and paste in the content below. This is a YAML file where you specify instructions for GitLab CI/CD.

stages:
  - build
  - test

job_build:
  stage: build
  script:
    - echo "Building the project"
    - tar -cf code.tar ./
  artifacts:
    paths:
      - code.tar
    expire_in: 1 days
  tags:
    - multipass

job_test:
  stage: test
  script:
    - echo "Running tests"
    - tar --list -f code.tar
  tags:
    - multipass

In this configuration, there are two jobs that the runner will run: a build job and a test job. The build job will just create an archive of the repo and store it as an artifact, then the test job will pick up the artifact and list its content. (Yeah it’s super simple for the sake of the demo)

Please note that every job of this pipeline use the tag “multipass”

Create and register a project runner

Next, we will create a project runner and register it. You must register the runner to link it to GitLab so that it can pick up jobs from the project pipeline.

To create a project runner:

On the left sidebar of your project, Select Settings > CI/CD.
Expand the Runners section.
Select New project runner.
Select your operating system: Linux.
In the Tags section, type multipass. Tags specify which jobs the runner can run and are optional.
Select Create runner.

The runner authentication token will show up on the screen, something like glrt-8y8EWk342r5DZW5_dsnv. Copy that value, we will use it in the next section.

Now on your local computer create a template-config.toml file which will be used as template to register the runner.

If you use a self-hosted Gitlab instance, replace the URL in the file below by your instance. For example, if your project is hosted on gitlab.example.com/yourname/yourproject, then your GitLab instance URL is https://gitlab.example.com. If your project is hosted on GitLab.com, the URL is https://gitlab.com.

[[runners]]
  url = "https://gitlab.com"
  executor = "custom"
  # Absolute path to a directory where builds are stored in the context of the selected executor. For example, locally, Docker, or SSH.
  builds_dir = "/home/ubuntu/builds"
  # Absolute path to a directory where build caches are stored in context of selected executor. For example, locally, Docker, or SSH. If the docker executor is used, this directory needs to be included in its volumes parameter.
  cache_dir = "/home/ubuntu/cache"
  [runners.custom]
    prepare_exec = "/opt/multipass-driver/prepare.sh"
    run_exec = "/opt/multipass-driver/run.sh"
    cleanup_exec = "/opt/multipass-driver/cleanup.sh"

Execute the following command on your terminal to register the runner:

gitlab-runner register --non-interactive --template-config /opt/multipass-driver/template-config.toml --executor custom --name multipass-runner --token glrt-8y8EWk342r5DZW5_dsnv

Please note that at the end of that command, you should replace the token value by the one you got previously.

If things went smoothly, you will get an output as below:

Runtime platform                                    arch=amd64 os=linux pid=8994 revision=853330f9 version=16.5.0
Running in system-mode.                            
                                                   
Merging configuration from template file "/opt/multipass-driver/template-config.toml" 
Verifying runner... is valid                        runner=8y8EWk342
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
 
Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml"

Trigger a pipeline to run the runner

Next, we will trigger a pipeline in the project, so we can see the multipass executor in action.

On the left sidebar of your project, Select Build > Pipelines.
Select Run pipeline.

While the pipeline is running, you can check on your local machine, you will see that a Multipass VM is created to run the job inside:

$ multipass list
Name                                                      State             IPv4             Image
runner-29251061-project-51866534-concurrent-0-5458984676  Running           10.28.91.183     Ubuntu 22.04 LTS

Select a job to view the job log. The output should look similar to this example, which shows your runner successfully executing the job:

Job_build log

Running with gitlab-runner 16.5.0 (853330f9)
  on test-VM 8y8EWk342, system ID: s_c4fdd2319548
Preparing the "custom" executor 02:11
Using Custom executor...
Running in runner-29251061-project-51866534-concurrent-0-5458984676
Retrieving image: 100%Verifying image:   /-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\| Preparing image for runner-29251061-project-51866534-concurrent-0-5458984676  /-\|/-\|/-\|/-\|/ Configuring runner-29251061-project-51866534-concurrent-0-5458984676  /-\| Starting runner-29251061-project-51866534-concurrent-0-5458984676  /-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\ Waiting for initialization to complete  /-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\ Launched: runner-29251061-project-51866534-concurrent-0-5458984676
Detected operating system as Ubuntu/jammy.
Checking for curl...
Detected curl...
Checking for gpg...
Detected gpg...
Detected apt version as 2.4.10
Running apt-get update... done.
Installing apt-transport-https... done.
Installing /etc/apt/sources.list.d/github_git-lfs.list...done.
Importing packagecloud gpg key... Packagecloud gpg key imported to /etc/apt/keyrings/github_git-lfs-archive-keyring.gpg
done.
Running apt-get update... done.
The repository is setup! You can now install packages.
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  git-lfs
0 upgraded, 1 newly installed, 0 to remove and 8 not upgraded.
Need to get 7244 kB of archives.
After this operation, 16.2 MB of additional disk space will be used.
Get:1 https://packagecloud.io/github/git-lfs/ubuntu jammy/main amd64 git-lfs amd64 3.4.0 [7244 kB]
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin: 
Fetched 7244 kB in 0s (19.2 MB/s)
Selecting previously unselected package git-lfs.
(Reading database ... 64306 files and directories currently installed.)
Preparing to unpack .../git-lfs_3.4.0_amd64.deb ...
Unpacking git-lfs (3.4.0) ...
Setting up git-lfs (3.4.0) ...
Git LFS initialized.
Processing triggers for man-db (2.10.2-1) ...
Running kernel seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 60.7M  100 60.7M    0     0  22.5M      0  0:00:02  0:00:02 --:--:-- 22.5M
Preparing environment 00:00
Running on runner-29251061-project-51866534-concurrent-0-5458984676...
Getting source from Git repository 00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /home/ubuntu/builds/theko2fi/test-multipass-executor/.git/
Created fresh repository.
Checking out 1edb276c as detached HEAD (ref is main)...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:01
WARNING: Starting with version 17.0 the 'build_script' stage will be replaced with 'step_script': https://gitlab.com/groups/gitlab-org/-/epics/6112
$ echo "Building the project"
Building the project
$ tar -cf code.tar ./
tar: ./code.tar: file is the archive; not dumped
Uploading artifacts for successful job 00:01
Uploading artifacts...
Runtime platform                                    arch=amd64 os=linux pid=2960 revision=853330f9 version=16.5.0
code.tar: found 1 matching artifact files and directories 
Uploading artifacts as "archive" to coordinator... 201 Created  id=5458984676 responseStatus=201 Created token=64_pLJY_
Cleaning up project directory and file based variables 00:00
Job succeeded

Job_test log

Running with gitlab-runner 16.5.0 (853330f9)
  on test-VM 8y8EWk342, system ID: s_c4fdd2319548
Preparing the "custom" executor 01:23
Using Custom executor...
Running in runner-29251061-project-51866534-concurrent-0-5458942516
Creating runner-29251061-project-51866534-concurrent-0-5458942516  /-\|/-\|/ Configuring runner-29251061-project-51866534-concurrent-0-5458942516  /-\|/ Starting runner-29251061-project-51866534-concurrent-0-5458942516  /-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\ Waiting for initialization to complete  /-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/- Launched: runner-29251061-project-51866534-concurrent-0-5458942516
Detected operating system as Ubuntu/jammy.
Checking for curl...
Detected curl...
Checking for gpg...
Detected gpg...
Detected apt version as 2.4.10
Running apt-get update... done.
Installing apt-transport-https... done.
Installing /etc/apt/sources.list.d/github_git-lfs.list...done.
Importing packagecloud gpg key... Packagecloud gpg key imported to /etc/apt/keyrings/github_git-lfs-archive-keyring.gpg
done.
Running apt-get update... done.
The repository is setup! You can now install packages.
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  git-lfs
0 upgraded, 1 newly installed, 0 to remove and 8 not upgraded.
Need to get 7244 kB of archives.
After this operation, 16.2 MB of additional disk space will be used.
Get:1 https://packagecloud.io/github/git-lfs/ubuntu jammy/main amd64 git-lfs amd64 3.4.0 [7244 kB]
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin: 
Fetched 7244 kB in 0s (27.9 MB/s)
Selecting previously unselected package git-lfs.
(Reading database ... 64306 files and directories currently installed.)
Preparing to unpack .../git-lfs_3.4.0_amd64.deb ...
Unpacking git-lfs (3.4.0) ...
Setting up git-lfs (3.4.0) ...
Git LFS initialized.
Processing triggers for man-db (2.10.2-1) ...
Running kernel seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 60.7M  100 60.7M    0     0  21.3M      0  0:00:02  0:00:02 --:--:-- 21.3M
Preparing environment 00:00
Running on runner-29251061-project-51866534-concurrent-0-5458942516...
Getting source from Git repository 00:02
Fetching changes with git depth set to 20...
Initialized empty Git repository in /home/ubuntu/builds/theko2fi/test-multipass-executor/.git/
Created fresh repository.
Checking out 1edb276c as detached HEAD (ref is main)...
Skipping Git submodules setup
Downloading artifacts 00:01
Downloading artifacts for job_build (5458984676)...
Runtime platform                                    arch=amd64 os=linux pid=2860 revision=853330f9 version=16.5.0
Downloading artifacts from coordinator... ok        host=cdn.artifacts.gitlab-static.net id=5458984676 responseStatus=200 OK token=64_duyWn
Executing "step_script" stage of the job script 00:00
WARNING: Starting with version 17.0 the 'build_script' stage will be replaced with 'step_script': https://gitlab.com/groups/gitlab-org/-/epics/6112
$ echo "Running tests"
Running tests
$ tar --list -f code.tar
./
./README.md
./test.txt
./.gitlab-ci.yml
./.git/
./.git/lfs/
./.git/lfs/tmp/
./.git/FETCH_HEAD
./.git/HEAD
./.git/objects/
./.git/objects/51/
./.git/objects/51/5b45f9ceeab7a22213f5e4740b2e06babdc6a4
./.git/refs/remotes/
./.git/refs/remotes/origin/
./.git/refs/remotes/origin/main
./.git/logs/
./.git/logs/HEAD
./.git/logs/refs/
./.git/logs/refs/remotes/
./.git/logs/refs/remotes/origin/
./.git/logs/refs/remotes/origin/main
Cleaning up project directory and file based variables 00:00
Job succeeded

As you can see in the logs, all the stages we defined previously are followed. A VM is created, dependencies are installed in it, the repo is cloned inside the VM, the job script is executed and everything is deleted at the end of the execution. That’s it

Next steps

You have now successfully created the Multipass executor on your machine, and you have a runner registered to use it!

However, there are some improvements we can do. The execution time is one of the most valuable aspect of CI/CD pipelines. As you can see below, the whole pipeline took 00:03:50 to run. That’s too long for a simple pipeline which does almost nothing.

We can speed up the process by injecting dependencies into the base VM image used to spin up new instances. So that dependencies are not downloaded every build, reducing the time it takes to provision new virtual machines. Read the next part of this series if you’re interested!

Troubleshoot potential error

If you edited the executor scripts on a Windows machine before uploading them to a Linux machine, you might get the error below:

Running with gitlab-runner 16.5.0 (853330f9)
  on test-VM 8y8EWk342, system ID: s_c4fdd2319548
Preparing the "custom" executor 00:09
Using Custom executor...
/usr/bin/env: ‘bash\r’: No such file or directory
/usr/bin/env: use -[v]S to pass options in shebang lines
WARNING: Cleanup script failed: unknown Custom executor executable exit code 127; executable execution terminated with: exit status 127
ERROR: Preparation failed: unknown Custom executor executable exit code 127; executable execution terminated with: exit status 127
Will be retried in 3s ...

You need to run the following commands to fix it:

apt-get install -y dos2unix
find /opt/multipass-driver -type f -exec dos2unix {} \;

Check here for more details.

Thank you for reading this article all the way to the end! I hope you found the information and insights shared here to be valuable and interesting. Get in touch with me on LinkedIn

I appreciate your support and look forward to sharing more content with you in the future. Until next time!

Kenneth KOFFI

Using Multipass as an Executor in GitLab CI/CD Pipelines - Part 1