In this final part, we will see how to improve the performance of the executor we created in the previous article.

How?

We will use Packer to build a custom Multipass image embedding all the dependencies needed for the executor.

Packer is an open-source tool developed by HashiCorp that is used for creating identical machine images for multiple platforms from a single source configuration. Machine images are templates that contain pre-configured operating system environments and software, and they are used to create virtual machines, containers, or instances in various cloud and virtualization platforms. Multipass can run those images too.

Before you begin

Install Packer

Please note that the Packer project may have evolved and introduced new installation methods since the day of this write-up. I recommend checking the official Packer documentation for the most up-to-date information on how to install Packer for your system.

So far, there are the commands I use on my Ubuntu computer:

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install packer

Install QEMU

QEMU, which stands for Quick Emulator, is an open-source and versatile emulator and virtualization tool that allows you to run a wide range of operating systems and architecture types on a host system. QEMU is often used for hardware virtualization, system emulation, and cross-architecture development and testing.

sudo apt install qemu-system-x86 -y

Setting up the build directory

We will execute the commands below to set up the build architecture:

mkdir -p ${HOME}/packermultipass/cloud-data
touch ${HOME}/packermultipass/cloud-data/meta-data
touch ${HOME}/packermultipass/cloud-data/user-data
touch ${HOME}/packermultipass/ubuntu2204.pkr.hcl

You will get an architecture like this:

${HOME}/packermultipass/
├── cloud-data
│   ├── meta-data
│   └── user-data
└── ubuntu2204.pkr.hcl

1 directory, 3 files

User data

Paste the content below into the ${HOME}/packermultipass/user-data file

#cloud-config
ssh_pwauth: true
users:
  - name: packer
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: users, admin
    passwd: $6$rounds=4096$XxQGokSw4FI8unZF$lAnQ0ZSMuCvSlv.rFjcxOpyAZr/ZDwtaI/X6BSSH0wtKngvprmgr9nvSMV/dBzE.TJ7Tvd8y0.T50dW5Bi1vf/
    lock_passwd: false
apt:
    preserve_sources_list: true
package_update: false

Metadata

We will leave the ${HOME}/packermultipass/meta-data file empty on purpose.

Packer configuration file

Paste the following content into the Packer configuration file (${HOME}/packermultipass/ubuntu2204.pkr.hcl)

packer {
  required_plugins {
    qemu = {
      source  = "github.com/hashicorp/qemu"
      version = "~> 1"
    }
  }
}

source "qemu" "my_qemu_builder" {
  disk_discard        = "unmap"
  disk_image          = true
  disk_interface      = "virtio-scsi"
  disk_size           = "5120M"
  http_directory      = "cloud-data"
  iso_checksum        = "file:http://cloud-images.ubuntu.com/releases/jammy/release/SHA256SUMS"
  iso_url             = "http://cloud-images.ubuntu.com/releases/jammy/release/ubuntu-22.04-server-cloudimg-amd64.img"
  output_directory    = "/root/packerimages"
  qemuargs            = [["-smbios", "type=1,serial=ds=nocloud-net;instance-id=packer;seedfrom=http://:/"]]
  ssh_password        = "packerpassword"
  ssh_username        = "packer"
  use_default_display = true
  vm_name             = "ubuntu2204-gitlab-runner"
}

build {
  sources = ["source.qemu.my_qemu_builder"]

  provisioner "shell" {
    inline = [
      "curl -s 'https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh' | sudo bash",
      "sudo apt-get install -y git-lfs",
      "sudo curl -L --output /usr/local/bin/gitlab-runner 'https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64'",
      "sudo chmod +x /usr/local/bin/gitlab-runner"
    ]
  }

  provisioner "shell" {
    execute_command = "sudo sh -c ' '"
    inline          = [
      "/usr/bin/apt-get clean",
      "rm -r /etc/netplan/50-cloud-init.yaml /etc/ssh/ssh_host* /etc/sudoers.d/90-cloud-init-users",
      "/usr/bin/truncate --size 0 /etc/machine-id",
      "/usr/bin/gawk -i inplace '/PasswordAuthentication/ { gsub(/yes/, \"no\") }; { print }' /etc/ssh/sshd_config",
      "rm -r /root/.ssh",
      "rm /snap/README",
      "find /usr/share/netplan -name __pycache__ -exec rm -r {} +",
      "rm /var/cache/pollinate/seeded /var/cache/motd-news",
      "rm -rfd /var/cache/snapd/*",
      "rm -r /var/lib/cloud /var/lib/dbus/machine-id /var/lib/private /var/lib/systemd/timers /var/lib/systemd/timesync /var/lib/systemd/random-seed",
      "rm /var/lib/ubuntu-release-upgrader/release-upgrade-available",
      "rm /var/lib/update-notifier/fsck-at-reboot",
      "find /var/log -type f -exec rm {} +",
      "rm -r /tmp/* /tmp/.*-unix /var/tmp/*",
      "for i in group gshadow passwd shadow subuid subgid; do mv /etc/$i- /etc/$i; done",
      "rm -r /home/packer",
      "/bin/sync",
      "/sbin/fstrim -v /"
    ]
    remote_folder   = "/tmp"
  }

}

This configuration file is written in HashiCorp Packer’s HCL (HashiCorp Configuration Language) and is used to create a machine image for a virtual machine using QEMU. Here’s a breakdown of what each section of the configuration file does:

Packer Block

This block defines the required plugins for the Packer build. In this case, it specifies that the “qemu” plugin is required from the source on GitHub. It also ensures that the version is approximately 1.

Source Block

This block specifies the source for building the virtual machine image using the “qemu” builder. The various parameters within this block configure the QEMU virtual machine:

http_directory: Specifies the directory containing files to be served over HTTP. In this case, it’s “cloud-data.”
iso_checksum and iso_url: These parameters define the ISO image that will be used to install the operating system. The checksum is checked against the provided SHA256SUMS file. Modify these parameters if you want to specify a different image or image type.
output_directory: Defines where the built image will be stored, in this case, “/root/packerimages”

Build Block

This block specifies how the image is built using the previously defined source:

sources parameter points to the source configuration defined earlier.
provisioners: This section defines one or more provisioners to execute commands within the virtual machine.
- The first shell provisioner installs the dependencies (Git LFS and the GitLab Runner) by running a series of shell commands.
- The second shell provisioner performs various cleanup tasks within the virtual machine. It removes sensitive information and files, configures SSH settings, and cleans up temporary files and directories.

Overall, this Packer configuration file is used to create a QEMU-based virtual machine image with Ubuntu 22.04, installs necessary software and configurations, convert the VM hard drive to image and prepares it for further use, such as GitLab Runner tasks.

Build the image

Now we will put Packer into action

cd ${HOME}/packermultipass
packer init .
packer build ubuntu2204.pkr.hcl

You will get a similar output:

qemu.my_qemu_builder: output will be in this color.

==> qemu.my_qemu_builder: Retrieving ISO
==> qemu.my_qemu_builder: Trying http://cloud-images.ubuntu.com/releases/jammy/release/ubuntu-22.04-server-cloudimg-amd64.img
==> qemu.my_qemu_builder: Trying http://cloud-images.ubuntu.com/releases/jammy/release/ubuntu-22.04-server-cloudimg-amd64.img?checksum=sha256%3A054db2d88c454bb0ad8dfd8883955e3946b57d2b0bf0d023f3ade3c93cdd14e5
    qemu.my_qemu_builder: ubuntu-22.04-server-cloudimg-amd64.img 36.50
    ...
    ...
==> qemu.my_qemu_builder: Provisioning with shell script: /tmp/packer-shell2758249411
    qemu.my_qemu_builder: /: 3.1 GiB (3293908992 bytes) trimmed
==> qemu.my_qemu_builder: Halting the virtual machine...
==> qemu.my_qemu_builder: Converting hard drive...
==> qemu.my_qemu_builder: Error getting file lock for conversion; retrying...
Build 'qemu.my_qemu_builder' finished after 2 minutes 21 seconds.

==> Wait completed after 2 minutes 21 seconds

==> Builds finished. The artifacts of successful builds are:
--> qemu.my_qemu_builder: VM files in directory: /root/packerimages

As you can see in the logs above, Packer retrieved the ISO image we specified, created a QEMU virtual machine, ran the provisioning shell scripts then finally exported the hard drive to the output directory (/root/packerimages).

Let’s test if the built image works well with Multipass:

multipass launch --name test-VM file:///root/packerimages/ubuntu2204-gitlab-runner

Now the above works, we will delete the test instance with:

multipass delete --purge test-VM

Customizing the executor

Now let’s edit the executor we created in the previous article to use that image.

Edit the /opt/multipass-driver/prepare.sh file and remove the install_dependencies function.

#!/usr/bin/env bash

# /opt/multipass-driver/prepare.sh

currentDir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
source ${currentDir}/base.sh # Get variables from base.

set -eo pipefail

# trap any error, and mark it as a system failure.
trap "exit $SYSTEM_FAILURE_EXIT_CODE" ERR

start_VM () {
    if multipass info "$VM_ID" >/dev/null 2>/dev/null ; then
        echo 'Found old VM, deleting'
        multipass delete --purge "$VM_ID"
    fi

    # The VM image is hardcoded, but you can use
    # the `CI_JOB_IMAGE` predefined variable
    # https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
    # which is available under `CUSTOM_ENV_CI_JOB_IMAGE` to allow the
    # user to specify the image. The rest of the script assumes that
    # you are running on an ubuntu image so modifications might be
    # required.
    multipass launch --name "$VM_ID" "$VM_IMAGE"

    # Wait for VM to start, we are using multipass list to check this,
    # for the sake of brevity.
    for i in $(seq 1 30); do
        if test "$(multipass list | grep $VM_ID | awk '{print $2}')" = "Running"; then
            break
        fi

        if [ "$i" == "30" ]; then
            echo 'Waited for 30 seconds to start VM, exiting..'
            # Inform GitLab Runner that this is a system failure, so it
            # should be retried.
            exit "$SYSTEM_FAILURE_EXIT_CODE"
        fi

        sleep 1s
    done
}

echo "Running in $VM_ID"

start_VM

Edit the /opt/multipass-driver/base.sh file and replace the VM_IMAGE value

#!/usr/bin/env bash

# /opt/multipass-driver/base.sh

VM_ID="runner-$CUSTOM_ENV_CI_RUNNER_ID-project-$CUSTOM_ENV_CI_PROJECT_ID-concurrent-$CUSTOM_ENV_CI_CONCURRENT_PROJECT_ID-$CUSTOM_ENV_CI_JOB_ID"
VM_IMAGE="file:///root/packerimages/ubuntu2204-gitlab-runner"

Trigger a pipeline to run the runner

Next, we will trigger again a pipeline in the project, so we can see the executor performance improvement.

As you can see below, the pipeline ran this time in 00:02:48, as opposed to the 00:03:50 observed in part 1 of this series.

That’s it!

Thank you for reading this article all the way to the end! I hope you found the information and insights shared here to be valuable and interesting. Get in touch with me on LinkedIn

I appreciate your support and look forward to sharing more content with you in the future. Until next time!

References

https://multipass.run/docs/building-multipass-images-with-packer

Kenneth KOFFI

Using Multipass as an Executor in GitLab CI/CD Pipelines - Part 2