Jump start CI/CD process using gitlab runner

What is Continuous Integration?

This practice is known as Continuous Integration; for every change submitted to an application — even to development branches — it’s built and tested automatically and continuously, ensuring the introduced changes pass all tests, guidelines, and code compliance standards you established for your app.

GitLab itself is an example of using Continuous Integration as a software development method. For every push to the project, there’s a set of scripts the code is checked against.

What is Continuous Delivery?

This method ensures the code is checked automatically but requires human intervention to manually and strategically trigger the deployment of the changes.

What is Continuous Deployment?

Introduction to GitLab CI/CD

Runners or gitlab Runners are processes running in a remote machine or instance connected to the gitlab instance.

Pipelines are set of predefined tasks triggered by an internal e.g git commit or an external event e.g slack post

Defining a CI/CD use case

Validate

  • Run a few unit test cases on the app
  • On successful completion continue to next step

Deploy

  • Copy the application into an linux machine
  • Upload app related configuration on s3
  • Trigger an Rest API

Step 1 : Download of prerequisite in a remote/deployment machine

git scm

Sync the code from gitlab into the remote machine on predefined event (e.g git commit)

https://git-scm.com/downloads

aws cli

Automate upload/download to aws s3 and can also perform any aws tasks via command line interface

https://awscli.amazonaws.com/AWSCLIV2.msi

pscp.exe

Sync the artifacts from gitlab into multiple linux machines

https://the.earth.li/~sgtatham/putty/latest/w64/pscp.exe

gitlab runner

Run pipelines on git runner process on a remote/deployment machine

https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-windows-amd64.exe

Installation of python , aws cli , pscp.exe(putty) is beyond scope of this web blog.

Step 2 : Installation of gitlab runner in a remote/deployment machine

Rename the binary to gitlab-runner.exe

Register the Runner

Run the command

gitlab-runner register

Enter the gitlab instance url

e.g. https://gitlab.com

Enter the registration token.

Follow the below steps for gitlab instance url and registration token.

gitlab runner url and registration token

Enter the required tags

The matching tags on pipeline will only be allowed to run on the runner unless specified to ignore.

e.g development , datawarehouse , marketingBU

Enter the executor as “Shell”

The gitlab runner can be configured to run a shell script , docker , docker on kubernetes etc. Detailed documentation can be found on the https://docs.gitlab.com/runner/

Update the config.toml and add the below the “executor” line (If required)

Once the gitlab runner is successfully registered, the config.toml and gitlab webUI should display the registered runner and its relevant details as shown below.

gitlab runner registration

Install and Start the gitlab runner

cd C:\GitLab-Runner

.\gitlab-runner.exe install — user USERNAME — password PASSWORD

.\gitlab-runner.exe start

[DEBUG] : If the gitlab runner service doesn’t start with the network credentials then follow the below steps:

click on start → services.msc → gitlab-runner →LogOn → update the username/password → click on OK

services account update

Step 3 : Building the pipeline definition

  • Test a functionality by running a Selenium code
  • Run a python script using powershell
  • Build an microsoft application using msbuild
  • Deploy the build into linux machine/ Hadoop / S3

For this exercise , Below is the .gitlab-ci.yml file.

The below pipeline contains 2 stages (validate and test) and each stage contains one or more task/job to be performed via a powershell command.

Validate

  • Task/Job 1 → Run the validate.py python script on the remote/deployment machine to perform basic unit testing on the python app.

Deploy

  • Task/Job1 → Deploy the python code into multiple linux machine
  • Task/Job2 → Deploy the configuration into s3 using aws cli command
  • Task/Job3 → Invoke a REST api

For detailed information on each of the keyword refer the page : https://docs.gitlab.com/ee/ci/yaml/

Create variables in gitlab to be consumed across the pipelines;

Shell script;

Step 4 : Pipeline execution

The gitlab runner will first sync the code from git into the remote/deployment server under C:\GitLab-Runner\builds\<registration token>\<edit number>\*

pipeline
pipeline completion
pipeline tasks/jobs completion
pipeline output

[DEBUG] : If the pipeline is in pending state:

click on Settings → CI/CD → edit icon next to the runner

Documentations

Decisive and multi-faceted IT professional with 12 years of experience delivering data management projects with 5+ years experience in cloud and big data tech

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store