Jump start CI/CD process using gitlab runner

6 min readNov 22, 2020

What is Continuous Integration?

Consider an application that has its code stored in a Git repository in GitLab. Developers push code changes every day, multiple times a day. For every push to the repository, you can create a set of scripts to build and test your application automatically, decreasing the chance of introducing errors to your app.

This practice is known as Continuous Integration; for every change submitted to an application — even to development branches — it’s built and tested automatically and continuously, ensuring the introduced changes pass all tests, guidelines, and code compliance standards you established for your app.

GitLab itself is an example of using Continuous Integration as a software development method. For every push to the project, there’s a set of scripts the code is checked against.

What is Continuous Delivery?

Continuous Delivery is a step beyond Continuous Integration. Your application is not only built and tested at every code change pushed to the codebase, but, as an additional step, it’s also deployed continuously, though the deployments are triggered manually.

This method ensures the code is checked automatically but requires human intervention to manually and strategically trigger the deployment of the changes.

What is Continuous Deployment?

Continuous Deployment is also a further step beyond Continuous Integration, similar to Continuous Delivery. The difference is that instead of deploying your application manually, you set it to be deployed automatically. It does not require human intervention at all to have your application deployed.

Introduction to GitLab CI/CD

GitLab CI/CD is a powerful tool built into GitLab that allows you to apply all the continuous methods (Continuous Integration, Delivery, and Deployment) to your software with no third-party application or integration needed.

Runners or gitlab Runners are processes running in a remote machine or instance connected to the gitlab instance.

Pipelines are set of predefined tasks triggered by an internal e.g git commit or an external event e.g slack post

Defining a CI/CD use case

The requirement is to validate and deploy the python app into a linux box and s3 buckets.

Validate

Run a few unit test cases on the app
On successful completion continue to next step

Deploy

Copy the application into an linux machine
Upload app related configuration on s3
Trigger an Rest API

Step 1 : Download of prerequisite in a remote/deployment machine

The prerequisite softwares required to be installed on the remote machine/deployment server for the exercise.

git scm

Sync the code from gitlab into the remote machine on predefined event (e.g git commit)

https://git-scm.com/downloads

aws cli

Automate upload/download to aws s3 and can also perform any aws tasks via command line interface

https://awscli.amazonaws.com/AWSCLIV2.msi

pscp.exe

Sync the artifacts from gitlab into multiple linux machines

https://the.earth.li/~sgtatham/putty/latest/w64/pscp.exe

gitlab runner

Run pipelines on git runner process on a remote/deployment machine

https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-windows-amd64.exe

Installation of python , aws cli , pscp.exe(putty) is beyond scope of this web blog.

Step 2 : Installation of gitlab runner in a remote/deployment machine

Download the gitlab runner into a directory (C:\GitLab-Runner)

Rename the binary to gitlab-runner.exe

Register the Runner

Open the command prompt with administrator access

Run the command

gitlab-runner register

Enter the gitlab instance url

e.g. https://gitlab.com

Enter the registration token.

Follow the below steps for gitlab instance url and registration token.

gitlab runner url and registration token

Enter the required tags

The matching tags on pipeline will only be allowed to run on the runner unless specified to ignore.

e.g development , datawarehouse , marketingBU

Enter the executor as “Shell”

The gitlab runner can be configured to run a shell script , docker , docker on kubernetes etc. Detailed documentation can be found on the https://docs.gitlab.com/runner/

Update the config.toml and add the below the “executor” line (If required)

Once the gitlab runner is successfully registered, the config.toml and gitlab webUI should display the registered runner and its relevant details as shown below.

Install and Start the gitlab runner

cd C:\GitLab-Runner

.\gitlab-runner.exe install — user USERNAME — password PASSWORD

.\gitlab-runner.exe start

[DEBUG] : If the gitlab runner service doesn’t start with the network credentials then follow the below steps:

click on start → services.msc → gitlab-runner →LogOn → update the username/password → click on OK

Step 3 : Building the pipeline definition

To define a CI/CD pipeline in a project, it is required to create a .gitlab-ci.yml in the /root directory of the repository. The pipeline contains a bunch of tasks which should be run in a specific order and details. Any command that can be run in “powershell” interface can be configured to run via the pipeline. The git runner felicitates to run those commands on the remote/deployment machine. Below are some of the tasks that can be run in an CI/CD pipeline.

Test a functionality by running a Selenium code
Run a python script using powershell
Build an microsoft application using msbuild
Deploy the build into linux machine/ Hadoop / S3

For this exercise , Below is the .gitlab-ci.yml file.

The below pipeline contains 2 stages (validate and test) and each stage contains one or more task/job to be performed via a powershell command.

Validate

Task/Job 1 → Run the validate.py python script on the remote/deployment machine to perform basic unit testing on the python app.

Deploy

Task/Job1 → Deploy the python code into multiple linux machine
Task/Job2 → Deploy the configuration into s3 using aws cli command
Task/Job3 → Invoke a REST api

For detailed information on each of the keyword refer the page : https://docs.gitlab.com/ee/ci/yaml/

Create variables in gitlab to be consumed across the pipelines;

Shell script;

Step 4 : Pipeline execution

Let’s commit some code and see the output of the pipeline.

The gitlab runner will first sync the code from git into the remote/deployment server under C:\GitLab-Runner\builds\<registration token>\<edit number>\*