how to make CI for an OpenSource project with Github Actions / Habr

Short description

Python Project Digest has discussed the revival of its technical project, particularly on the organization of Continuous Integration (CI) for the open source project based on Github Actions. The Github Actions automation tool is essential to bind the execution of any actions to the repository’s interaction events. To automate the routine, they selected CI to deploy the program, run tests, and backup the database. The old system server was replaced with a Docker (Dockerfile) image of the application, while the Python-dotenv library was used to read the variables during the application start using Github Secrets.

how to make CI for an OpenSource project with Github Actions / Habr

Equator talks about the technical revival of Python Project Digest. Earlier I told how to switch from Python 3.4 to Python 3.11 and automatically update all code. In this part, I will talk about the organization of CI for an Open Source project based on Github Actions – how to run tests, check code, dependencies, deploy the program and make backups to external storage.

Contents of the series of articles

The state after formatting the code

  • The program is launched both locally and on the server, the tests pass.
  • The code can be run on Python 3.11 and the dependencies are up to date and added via poetry.
  • There is a server on Ubuntu 14.04, backups, certificates, etc. are updated manually.
  • Changes to the application are rolled out manually via git pull & restart commands.

The task (part 3) is to remove the need to access the server for typical operations

In Python Digest, there are three operations that I would like to automate immediately:

  • Running the tests and checking the code through pre-commit (the second part) — tests.yaml
  • Deployment of the program to the server – ci.yaml
  • Database backup and download to an external drive – backup.yaml

Work plan

  • Select CI to automate the routine
  • Prepare a new server
  • Configure CI for application deployment and backup

How Github chose Actions as CI

5 years ago, Travis CI was the “default CI for Open Source projects”, it was used to run tests, make builds, even work with servers over SSH. It was convenient and free.

Currently, the “default CI for OpenSource on Github” is Github Actions. This is an automation tool that allows you to bind the execution of almost any actions to the events of interaction with the repository. They wrote about him several times on Khabra — once, twice, three times.

Github Actions is cool because the Actions themselves are plugins (often scripts or Docker images) that anyone who is not lazy writes and posts on the Github Marketplace.

It works like this:

  • When we change something in the repository – push, pull request, by button, by schedule, etc. – An event appears;
  • Github Actions catches this event and starts an agent;
  • the agent picks up a container with the specified version of the operating system;
  • further inside the container you can do whatever you want.

And Github Action plugins are ready-made actions that you can use to download the code from the repository, convert it, save it somewhere, and more. There are many plugins.

Which Github Actions did you choose?

Such plugins were enough for my tasks. They cover steps from downloading the repository code to building the image in Docker Hub and connecting to an external SSH server.

An important note. These Github Actions are popular, with thousands of community members using them. However, they are developed by an open community, which does not guarantee the stability, correctness and compatibility of new and old versions. Examine the build, scripts, and Docker images for errors and obvious vulnerabilities before use.

As prepared by the server

The old server was on an outdated Ubuntu 14.04, which failed to find live package repositories and upgrade to above. Also, the system itself was heavily clogged. Therefore, I decided that it is easier to create a new server and configure it.

I didn’t want to end up with a garbage server after another 5 years, so I decided to run the application using docker compose (docker_compose.prod.yml), building a Docker (Dockerfile) image of the application.

But first I set up a server with an up-to-date Ubuntu.
Python Digest has been hosted on FirstVDS for a long time, without any unexpected problems, so it is easier to start a new server there. FirstVDS has detailed documentation for most life cases, even the very first steps with hosting are described.

I bought a server with 2 CPU threads, 4 Gb RAM and 60 Gb SSD. For the eyes and ears, this amount of resources is enough for the project.

After receiving the server, you need:

The server is ready for the next steps.

How to set environment variables

It is good to write programs taking into account 12 factors, we get a more convenient, understandable application in support. One practical aspect of 12 factors is to abandon hardcoded settings in favor of setting through the environment. In the running environment, we register the variables (access to the database, secrets, etc.), and the application reads them at the start using the python-dotenv library.

Github Secrets allow you to store environment variables and then refer to them during CI operations. Moreover, you can make separate sets of variables for each branch. Here I created the production set and specified the variables for SSH access, as well as for the program to work.

Done, now we can start.

How did I get the result?

Made a test commit and processing started:

  • Tested the code using pre-commitpassed the test.
  • The Docker image was assembled and uploaded to Docker Hub.
  • Settings were copied to the server via SCP.
  • Migrations have started, statics have been prepared and the application has launched.
  • nginx accepts the new configuration and the service is ready to receive domain traffic.

Next, in the DNS settings of the domain, an A record was configured with a domain-ip server address pair.

That’s all, now Github Actions will automatically back up the database to Yandex.Disk once a day, tests and code checks will be performed during Pull Request, and the application will be deployed on the server after a commit to the main branch.


Conclusions

  • The combination of Docker compose + CI on a virtual server also works well for projects, after the basic setup of the server there is no need to monitor it.
  • Github Actions has many plugins (some of them very weak) that solve typical tasks of project operation. However, sometimes it is easier to “manually” log in via SSH and do the necessary things instead of learning the innards of actions.
  • Now it’s a pleasure to make changes to Python Digest without fear of breaking dependencies or code.


    A UFO flew in and left a promo code here for our blog readers:
    – 15% on all VDS tariffs (except the Progriv tariff) – HABRFIRSTVDS.

Related posts