GitLab with Python Workshop

PDF version of this article: pythan-gitlab-ci-cd-development-guide.pdf

Python Backend CI example

We will be creating basic API project using Python language and Flask Framework. Complete code can be found here: https://gitlab.com/devops-training-info/examples/python/flask/simple-gitlab-ci-cd-example and you can just check your code against it if any problem will occur.

Because Gitlab Shared Pipelines were used for the Bitcoin mining to use them you need to add to your account credit cart or register own runner if you plan to use own username space or own Gitlab group for this training

In this case if you can’t or do not want to use own credit cart to activate public shared runners for your project please use this guide: https://gitlab-runner-k3d-guide-1aca27.gitlab.io prior to the next steps.

Core project creation

In this project instead of the standard workflow we will be using pipenv^[1] with the latest at the moment of this guide creation python3^[2]. You need to install them prior to the next steps.

First, you will need to create a git repository in the gitlab.com which will be used for our project, lets name it simple-gitlab-ci-cd-example. You will need to clone this empty repository in the location of your choose.

Requirements

Let’s create our pipenv environment:

pipenv install

this will create our Pipfile and Pipfile.lock files in the main project folder.

Now what we need is to install framework we will be using:

pipenv install flask

and our cors library as an example:

pipenv install flask-cors

Central controller, configuration, and first view

Now we need to prepare out base app core structure. Please create app folder and inside of this folder main sub folder. In the app/main folder we will create a views.py file with this content:

from flask import Blueprint, jsonify

main = Blueprint('main', __name__)


@main.route('/', methods=['GET', 'OPTION'])
def index():
    return jsonify({'message': 'Hello my name is Backend'})

We will treat main folder as a package for the main blueprints, and we also needs a app/main/__init__.py file like this:

from . import views

Now please create app/config.py file like this:

import os
basedir = os.path.abspath(os.path.dirname(__file__))


class Config:
    SECRET_KEY = os.environ.get('SECRET_KEY') or 'hard to guess string'
    SSL_REDIRECT = False

    @staticmethod
    def init_app(app):
        pass


class DevelopmentConfig(Config):
    DEBUG = True


class TestingConfig(Config):
    TESTING = True
    WTF_CSRF_ENABLED = False


class ProductionConfig(Config):
    @classmethod
    def init_app(cls, app):
        Config.init_app(app)


class DockerConfig(ProductionConfig):
    @classmethod
    def init_app(cls, app):
        ProductionConfig.init_app(app)

        # log to stderr
        import logging
        from logging import StreamHandler
        file_handler = StreamHandler()
        file_handler.setLevel(logging.INFO)
        app.logger.addHandler(file_handler)


class UnixConfig(ProductionConfig):
    @classmethod
    def init_app(cls, app):
        ProductionConfig.init_app(app)

        # log to syslog
        import logging
        from logging.handlers import SysLogHandler
        syslog_handler = SysLogHandler()
        syslog_handler.setLevel(logging.INFO)
        app.logger.addHandler(syslog_handler)


config = {
    'development': DevelopmentConfig,
    'testing': TestingConfig,
    'production': ProductionConfig,
    'docker': DockerConfig,
    'unix': UnixConfig,

    'default': DevelopmentConfig
}

Our app folder also will be a package and will have create_app function which will be using our views like created upper and other potential files like models and forms we can add in the future. Will be also our central controller. Please create app/__init__.py file like this in this folder:

import os

from flask import Flask
from flask_cors import CORS

from app.config import config


def create_app(config_name):
    myapp = Flask(__name__)

    myapp.config.from_object(config[config_name])
    config[config_name].init_app(myapp)

    CORS(myapp)

    # Our Ping/Echo - just to have blueprint example done with the Flask standard
    from .main.views import main as main_blueprint
    myapp.register_blueprint(main_blueprint)

    return myapp

app = create_app(os.getenv('FLASK_CONFIG') or 'default')

With this core app files are created, please go to pipenv shell:

pipenv shell

and run flask in a dev mode:

flask run

You should see text like this:

 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit

You can check your app by just going to: http://127.0.0.1:5000/

Before adding anything to the git repository please add also a .gitignore file in the root folder looking like this:

# Byte-compiled / optimized / DLL files
__pycache__/
*.py[codz]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
#   Usually these files are written by a python script from a template
#   before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py.cover
*.lcov
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
#   For a library or package, you might want to ignore these files since the code is
#   intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
#   However, in case of collaboration, if having platform-specific dependencies or dependencies
#   having no cross-platform support, pipenv may install dependencies that don't work, or not
#   install all needed dependencies.
# Pipfile.lock

# UV
#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
#   This is especially recommended for binary packages to ensure reproducibility, and is more
#   commonly ignored for libraries.
# uv.lock

# poetry
#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
#   This is especially recommended for binary packages to ensure reproducibility, and is more
#   commonly ignored for libraries.
#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
# poetry.lock
# poetry.toml

# pdm
#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
# pdm.lock
# pdm.toml
.pdm-python
.pdm-build/

# pixi
#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
# pixi.lock
#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
#   in the .venv directory. It is recommended not to include this directory in version control.
.pixi/*
!.pixi/config.toml

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule*
celerybeat.pid

# Redis
*.rdb
*.aof
*.pid

# RabbitMQ
mnesia/
rabbitmq/
rabbitmq-data/

# ActiveMQ
activemq-data/

# SageMath parsed files
*.sage.py

# Environments
.env
.envrc
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
#   JetBrains specific template is maintained in a separate JetBrains.gitignore that can
#   be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
#   and can be added to the global gitignore or merged into this file.  For a more nuclear
#   option (not recommended) you can uncomment the following to ignore the entire idea folder.
# .idea/

# Abstra
#   Abstra is an AI-powered process automation framework.
#   Ignore directories containing user credentials, local state, and settings.
#   Learn more at https://abstra.io/docs
.abstra/

# Visual Studio Code
#   Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore
#   that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
#   and can be added to the global gitignore or merged into this file. However, if you prefer,
#   you could uncomment the following to ignore the entire vscode folder
# .vscode/
# Temporary file for partial code execution
tempCodeRunnerFile.py

# Ruff stuff:
.ruff_cache/

# PyPI configuration file
.pypirc

# Marimo
marimo/_static/
marimo/_lsp/
__marimo__/

# Streamlit
.streamlit/secrets.toml

Which I found here: https://github.com/github/gitignore/blob/main/Python.gitignore

Please also exit shell and commit code to your repository.

Tests

Normally we should start with them but in this case we just add them now. Please remember that next views and code should be done with the TDD methodology.

Requirements

Let’s start with adding packages we will be using in our development environment.

Pytest and Coverage

pipenv install pytest coverage --dev

Flake8

pipenv install flake8 --dev

isort

pipenv install isort --dev

First test file

We need to prepare tests folder which we will need to treat as the package. For this we need a empty init.py file. After creating this file we just need to add our first test file test_basics.py which will look like this:

import unittest
from flask import current_app
from app import create_app


class BasicsTestCase(unittest.TestCase):
    def setUp(self):
        self.app = create_app('testing')
        self.app_context = self.app.app_context()
        self.app_context.push()

    def tearDown(self):
        self.app_context.pop()

    def test_app_exists(self):
        self.assertFalse(current_app is None)

    def test_app_is_testing(self):
        self.assertTrue(current_app.config['TESTING'])

Again to check all of this we need to activate pipenv shell:

pipenv shell

Now it is possible to run pytest which command

pytest

and coverage with command:

coverage run -m pytest

More examples can be found here:

Flake8

For the next type of the test, lets add configuration for the flake8 which is checking if our code meets PEP standards. We need to create .flake8 file in the root folder looking like this:

[flake8]
# it's not a bug that we aren't using all of hacking, ignore:
# E402 module level import not at top of file
# F401 imported but unused - Flask hacking for the Blueprints
ignore = E402,F401
max-line-length = 120
exclude = app/static

After adding this file we can just check if out code is ok with command:

flake8 app

If result is empty all is OK.

Isort

Last test we planning to use is isort what is even more strict then PEP and checking if our imports are as readable as possible. We can check this by using command:

isort --diff --check-only app

If result is empty all is OK.

Now lets commit our changes and push to the Gitlab.

Docker

We like to have our application in the container what will help us to install it in the modern environment like for example in the Kubernetes cluster. For this we need to have docker^[3] and docker-compose^[4].

Docker files

We will go with a multistage Dockerfile looking like this:

FROM python:3.14-slim as base

# Setup env
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONFAULTHANDLER 1


FROM base AS python-deps

# Install pipenv and compilation dependencies
RUN pip install pipenv
RUN apt-get update && apt-get install -y --no-install-recommends gcc

# Install python dependencies in /.venv
COPY Pipfile .
COPY Pipfile.lock .
RUN PIPENV_VENV_IN_PROJECT=1 pipenv install --deploy


FROM base AS runtime

# Copy virtual env from python-deps stage
COPY --from=python-deps /.venv /.venv
ENV PATH="/.venv/bin:$PATH"

# Create and switch to a new user
RUN useradd --create-home appuser
WORKDIR /home/appuser
USER appuser

# Install application into container
COPY . .

# Run the application
CMD ["gunicorn", "-b", ":5000", "-k", "gevent", "--access-logfile", "-", "--error-logfile", "-", "app:app"]

Now we can build oud Docker image, but let’s do this with Docker Compose.

Docker Compose

Create your compose.yml file which we will use as a starting point looking like this:

services:
#  mongo:
#    image: mongo
#    restart: always
#    environment:
#      MONGO_INITDB_ROOT_USERNAME: root
#      MONGO_INITDB_ROOT_PASSWORD: example
#    ports:
#    - "27017:27017"
#
#  mongo-express:
#    image: mongo-express
#    restart: always
#    ports:
#      - 8081:8081
#    environment:
#      ME_CONFIG_MONGODB_ADMINUSERNAME: root
#      ME_CONFIG_MONGODB_ADMINPASSWORD: example

  api:
    build: .
    command: ["flask", "run", "--host=0.0.0.0", "--port=5000"]
    volumes:
      - ./app:/home/appuser/app
      - ./tests:/home/appuser/tests
      - ./.flake8:/home/api/.flake8
    ports:
      - "5000:5000"

volumes:
  data-volume:

Now we can build our image with the docker-compose build command and start it with the docker-compose up command.

Now lets commit our changes and push to the Gitlab.

CI/CD

Now we are missing CI/CD in our project. In the Gitlab we need to create .gitlab-ci.yml file which will instruct Gitlab how to achieve this. We can even have Auto DevOps with gitlab created but let’s focus on the simpler pipeline. Please create file like this:

# Version: 6.0
stages:
  - configure
  - lint
  - build
  - test
  - release

include:
  - component: gitlab.com/devops-training-info/templates/ci/components/python-component/install@1.0.0
    inputs:
      stage: configure
  - component: gitlab.com/devops-training-info/templates/ci/components/python-component/flake8@1.0.0
    inputs:
      stage: lint
  - component: gitlab.com/devops-training-info/templates/ci/components/python-component/isort@1.0.0
    inputs:
      stage: lint
  - component: gitlab.com/devops-training-info/templates/ci/components/container-component/build@1.0.2
    inputs:
      stage: build
      docker_file: Dockerfile
  - component: gitlab.com/devops-training-info/templates/ci/components/python-component/pytest@1.0.0
    inputs:
      stage: test
  - component: gitlab.com/devops-training-info/templates/ci/components/python-component/coverage@1.0.0
    inputs:
      stage: test
  - component: gitlab.com/devops-training-info/templates/ci/components/security-component/trivy@1.0.0
    inputs:
      stage: test
  - component: gitlab.com/devops-training-info/templates/ci/components/container-component/release@1.0.2
    inputs:
      stage: release
  - component: gitlab.com/devops-training-info/templates/ci/components/container-component/clean@1.0.2

Now commit and push and check pipeline.

This work is licensed under a Creative Commons Attribution 4.0 International License

1. Pipenv: Python Development Workflow for Humans https://pipenv.pypa.io/en/latest/

2. Python download: https://www.python.org/downloads/

3. Docker: https://docs.docker.com/engine/install/

4. Docker Compose: https://docs.docker.com/compose/install/