DevOps with Docker & Gitlab

Docker
Containerization
Deployment
Gitlab
DevOps
CI/CD
Automation
Github Pages
Author

Bernardo Freire

Published

August 1, 2024

Introduction

My intention with this article is to provide a hands-on demonstration of what I have accomplished so far in DevOps. Rather than just discussing theory, I will show you the following:

  • How to leverage a Dockerfile to create a Docker image.
  • How to push and pull the image to and from a private Docker registry and not from public Docker Hub.
  • How to push the output of a process to GitHub in order use Github Pages to publish a website.

We are going to use GitLab CI/CD to automate all the steps mentioned above.

Please note that I am not an expert in DevOps and am still learning. I am sharing this article to show you what I have done so far and to get feedback from you. :)

Motivation

In my previous jobs, when the code was done, I only needed to push it to a repository while providing a requirements.txt file. The rest was handled by the DevOps team. I never had to worry about how the code was deployed to the server.

But now, I am working on my own projects and have to manage everything myself. I had to set up servers, deploy the code, and ensure everything is working correctly. This is why I started learning DevOps. So far I used Gitlab CI/CD to automate the process of building, pushing, and pulling a Docker image. For those of you who are not familiar with GitLab CI/CD, it is a tool that automates the process of testing and deploying code. It is essentially a pipeline that runs every time you push code to a repository and executes the steps you define in the .gitlab-ci.yml file. You can also leverage Gitlab Runner to run the pipeline on your own server/computer, and not rely on GitLab to run the pipeline for you.

When combined with Docker, it can be used to automate the process of building, pushing, and pulling Docker images and customizing the deployment process. Moreover, when using a private Docker registry, you can ensure that your images are secure and not publicly available.

Enough talking, let’s get started!

Requirements

In order to be able to follow along with this article, you need to have the following installed/registered:

Optional:

These are the only requirements you need to follow along with this article. Next, we will discuss the requirements for the project we are going to work on.

Choose a Project

I have chosen a simple project that I have been working on—this blog 😜

Think about a project that you have been working on and would like to automate the process of building and pushing. It doesn’t really matter what project you choose. The important thing is to understand the process of building, pushing, and pulling a Docker image and how to leverage GitLab CI/CD to automate the process.

Our plan is as follows:

First, we push our code to GitLab and then use GitLab CI/CD to build and push a Docker image to our private Docker registry. We will then pull the image and “do something” with it. In my case, I am going to render this blog using ̀Quarto and push it to GitHub. Finally, we are going to use GitHub Actions to automate the process of pushing the output of the process to GitHub Pages.

Components of Pipeline

In the next sections, we are going to discuss the components of the pipeline we are going to build. Here, a brief overview of the components:

  • Dockerfile: Which ensures that the image is built correctly and reproducibly.
  • GitLab CI/CD: To automate the process of building, pushing, and pulling the Docker image.
  • Docker Registry: To store the Docker image.
  • GitHub Actions: To automate the process of pushing the output of the process to GitHub Pages.

Dockerfile

Again, the Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Using Dockerfile, we can automate the process of building a Docker image.

Here, I demonstrate how I utilized a Dockerfile to build a Docker image for this blog. I used Quarto framework to render HTML pages. I chose to build the image from scratch, starting with the Ubuntu base image and installing all necessary dependencies along with the Quarto application.

# Base image
FROM ubuntu:jammy

# Install the required packages
RUN apt-get update && \
    apt-get install -y \
    bash \
    curl \
    git \
    jq \
    openssh-client \
    ca-certificates \
    dpkg

# Install quarto  
RUN curl -sL -o installer.deb \
    https://github.com/quarto-dev/quarto-cli/releases/download/v1.5.56/quarto-1.5.56-linux-amd64.deb && \
    dpkg -i installer.deb && \
    rm installer.deb

# Create a directory for the app
RUN mkdir -p /app

# Set the working directory
WORKDIR /app

# CHMOD app - give permission to the app directory
RUN chmod -R 777 /app

# Copy the content of the local src directory to the working directory
COPY . .
# Make a directory for the output (empty for now)
RUN mkdir -p ./_output

We start with the base image ubuntu:jammy, which is a Docker image containing the Ubuntu operating system. First, we install the necessary dependencies and download the Quarto installer from GitHub, then proceed with its installation. Next, we create a directory for the app and set the working directory to /app. We copy the contents of the current directory into the /app directory and create an _output directory. This _output directory will store the process output, which will eventually be pushed to GitHub. Currently, _output is empty.

Docker Registry

As we decided to create a private Docker registry to store our Docker images, we use the Docker registry image to create a private Docker registry and also a convenient UI to manage the registry. The easiest way to do this is by using Docker Compose, which is a tool for defining and running multi-container Docker applications.

services:

  registry-server:
    image: registry:2.8
    container_name: registry-server
    ports:
    - "5000:5000"
    restart: always
    volumes:
      - ./registry-data:/var/lib/registry
    
  registry-ui:
    image: joxit/docker-registry-ui:2.5-debian
    container_name: registry-ui
    restart: always
    ports:
      - "9000:80"
    environment:
      SINGLE_REGISTRY: true
      REGISTRY_TITLE: Own Docker Registry UI
      DELETE_IMAGES: true
      SHOW_CONTENT_DIGEST: true
      NGINX_PROXY_PASS_URL: http://registry-server:5000
      SHOW_CATALOG_NB_TAGS: true
      CATALOG_MIN_BRANCHES: 1
      CATALOG_MAX_BRANCHES: 1
      TAGLIST_PAGE_SIZE: 100
      REGISTRY_SECURED: false
      CATALOG_ELEMENTS_LIMIT: 1000
      THEME: dark

First, we define the services that we are going to use. We define two services: registry-server and registry-ui. The registry-server is the private Docker registry that we are going to use to store the Docker image. The registry-ui is a Docker image that provides a UI for the private Docker registry. We can use the UI to see the images that are stored in the private Docker registry.

Gitlab CI/CD

Having a Dockerfile and Registry means we are almost ready to deploy.

The final step in our pipeline is to wrap everything up in a pipeline. This is where Gitlab CI/CD comes into play. The .gitlab-ci.yml file is where we define the pipeline and the steps that are going to be executed when we push code to the repository. Stages, variables and jobs are defined in the .gitlab-ci.yml file.

stages:
  - build
  - render
  - deploy

variables: 
  REGISTRY_HOST: '<host-url>'
  REGISTRY_USER: '<host-user>'
  APP_NAME: '$CI_PROJECT_NAME'
  APP_VERSION: '$CI_COMMIT_BRANCH'
  
## Build the Docker image and push it to the private Docker registry
build:
  stage: build
  image: docker
  services:
    - name: docker:27.1.1-dind
      alias: docker
      command: [ "--tls=false","--insecure-registry=<host-url>"]
  script:
    - docker build -t $REGISTRY_HOST/$REGISTRY_USER/$APP_NAME:$APP_VERSION .
    - docker push $REGISTRY_HOST/$REGISTRY_USER/$APP_NAME:$APP_VERSION
  tags: [your_tag]

## Render the bookdown project using the previous build image
## content is rendered to "_output"
render:
  stage: render
  image: $REGISTRY_HOST/$REGISTRY_USER/$APP_NAME:$APP_VERSION
  script:
    - quarto render
  artifacts:
    paths: ['_output/']
    expire_in: 10 mins
  tags: [your_tag]

# Push the content of "_output" to github (for github pages)
# Fetch output from the previous stage
push_output_to_github:
  stage: deploy
  image: ubuntu:jammy
  before_script:
    - apt-get update && apt-get install -y git
  script:
    - git config --global user.email "<your-email>"
    - git config --global user.name "<your-name>"
    - git clone https://$GITHUB_TOKEN@github.com/<your-name>/<repo-name>.git
    - git checkout main
    - cd blog
    - rm -rf *
    - cp -r ../_output/* .
    - git status
    - git add .
    - git status
    - git commit -m "auto-commit for deployment"
    - git push origin main
  dependencies:
    - render
  only:
    - main
  tags: [your_tag]

Variables like $GITHUB_TOKEN should be defined in the Gitlab settings. You can define them in the CI/CD settings of your project.

This YAML file defines a CI/CD pipeline with three stages: build, render, and deploy and four variables that are used in the pipeline. Let’s review the pipeline:

Variables:

  • REGISTRY_HOST: URL of the Docker registry.
  • REGISTRY_USER: User for the Docker registry.
  • APP_NAME: Name of the application, derived from the CI project name.
  • APP_VERSION: Version of the application, derived from the CI commit branch.

Build Stage:

  • Uses the docker image.
  • Runs a Docker-in-Docker service.
  • Builds a Docker image and pushes it to the private Docker registry.

Render Stage:

  • Uses the Docker image built in the previous stage.
  • Runs quarto render to render the bookdown project.
  • Saves the rendered content to the _output directory as artifacts.

Deploy Stage:

  • Uses the ubuntu:jammy image.
  • Installs Git.
  • Configures Git user details.
  • Clones a GitHub repository using a token.
  • Copies the rendered content from _output to the repository.
  • Commits and pushes the changes to the main branch.
  • The file also includes a tip that variables like $GITHUB_TOKEN should be defined in the GitLab CI/CD settings.

We have seen how to build a Docker image, push it to a private Docker registry, render the content, and push the output to GitHub.

In my case, where I would like to publish a static HTML website I still need to go one step further. I need enable Github Actions to publish a website using Github Pages. Github Pages is a static site hosting service that takes HTML, CSS, and JavaScript files straight from a repository on Github and publishes a website - for more information on how to enable Github Pages for your repository, please refer to the official documentation.

Summary

In this article, we discussed how to automate the process of building, pushing, and pulling a Docker image using GitLab CI/CD. We also discussed how to use a private Docker registry to store the Docker image and how to leverage GitHub Pages to publish the output of the process. By following these steps, you can automate the deployment process for your projects, making it easier to manage and deploy your applications. This not only saves time but also reduces the chances of human error.

In a next article, I will discuss how to use GitLab Runner to run the pipeline on your own server/computer, and not rely on GitLab to run the pipeline for you and as well the specifics of the Docker Registry and the UI of Gitlab.

Remember, the key to successful DevOps is continuous learning and improvement. As you gain more experience, you can refine and optimize your pipeline to better suit your needs.

Thank you for following along, and I hope you found this guide helpful. If you have any feedback or questions, feel free to reach out!