About the use of Liquibase in development projects

About the use of Liquibase in development projects

Inspired by a recent discussion of Liquibase with colleagues, and based on an earlier impression of a widespread attitude towards this tool by developers, I decided to write about my vision for the purpose of Liquibase, and tools like it, in enterprise application development projects using databases.

Definition of Liquibase: “Liquibase is an open-source database-independent library for tracking, managing and applying database schema changes”, presentation of the product on the official website: “Accelerate Your Database Deployments”, “CI/CD for the database”, “Releas faster by bringing DevOps to database with Liquibase”.

Definition of Flyway: “Flyway is an open-source database-migration tool”, presentation of the product on the official website: “Increase reliability of deployments by versioning your database”, “Database migrations made easy”, “Made for CI/CD”.

Obviously, the initial perception of the majority is formed as follows: these are tools for DevOps. This perception is complemented by the declared purpose: “Database migrations”, which, in my opinion, is unfortunately propagated further in thematic articles as the main purpose of these tools. That is, the understanding boils down to the fact that when it comes to the application of the mentioned tools, it is assumed that there is already an existing, developed/developed database, changes from which are now perfectly transferred from one stand to another, once or on a regular basis.

I believe that these tools are primarily developer tools, which should be used by developers directly in the development process itself, and from the very beginning; and all other scenarios of their application are secondary and have a private nature.

Let’s try to understand, actually, what is the original problem, why did such tools appear at all, why do you need to understand at all, and why “It is what it is”.

As you know, you cannot simply take and transform a database that is in operation (including previously deployed in a common test environment with many developers working on the same project). In making this statement, for the sake of caution, I will limit myself to relational databases only, and clarify that I am talking about the structure of the database – DDL instructions for tables, in the general case.

In a “normal” application, the structural elements of the project can be added/changed, after which the project is reassembled and deployed on the bench, replacing the previously deployed copy of the program with its new version.

// Версия 0.1

// /incredible-app/src/main/java/net/cuttingedge/SilverBulletSolution.java

package net.cuttingedge;

public class SilverBulletSolution {
    int magicAttribute;

    public static void main(String[] args) {
        System.out.println("This is the initial version!");
    }
}
// Версия 0.2

// /incredible-app/src/main/java/net/cuttingedge/SilverBulletSolution.java

package net.cuttingedge;

public class SilverBulletSolution {
    int magicAttribute;
    String yetAnotherMagicAttribute;
    
    public static void main(String[] args) {
        System.out.println("This is a breakthrough improvement!");
    }
}

In the case of the database, it will not work either: the supplemented/changed definition of the main structural element of the project – the table – is not simply replaced, you have to add separate instructions for the change.

-- Версия 0.1

-- /incredible-app/src/main/database/cuttingedge/silver_bullet_solution.sql

create schema cuttingedge;

create table cuttingedge.silver_bullet_solution(
    magic_attribute integer       
);
-- Версия 0.2

-- /incredible-app/src/main/database/cuttingedge/01-silver_bullet_solution.sql

create schema if not exists cuttingedge;

create table if not exists cuttingedge.silver_bullet_solution(
    magic_attribute integer       
);

-- /incredible-app/src/main/database/cuttingedge/02-silver_bullet_solution-new_attribute.sql

alter table cuttingedge.silver_bullet_solution
    add column yet_another_magic_attribute text
;

It’s good that there is, that is.

After all, the database project can be presented in the form of text files – scripts, arranged in folders and built in the required sequence, which means that it is placed in the repository of the version control system (VCS) and can, as a result, be deployed from the desired branch to the desired stand in the right condition This is the way they go. For example, they build a target database assembly system using Shell scripts and some additional tools. At the same time, developers develop scripts separately, using some convenient database manager (database tool) and using a prepared stand with a deployed DBMS, local and/or distributed, after which they make a Pull/Merge Request and send it to Code Review. The final consolidation of the project, after merging the changes, automated to one degree or another, gives the final result. The scheme of work is clear and does not raise any special questions. Everything is fine if everything is done as intended. At the same time, previously unused tools such as Liquibase and Flyway are often perceived as “yet another tool”, which are probably more convenient and should be switched to, because they acquire the status of a de facto standard for solving similar tasks.

And to my surprise, I’ve repeatedly heard from developers who intend to use Liquibase something like this: “I’ll continue to develop my scripts as before, within the project model I understand. And Liquibase requires that the deployment script be generated somehow in the required format, and further executed by DevOps”. In my understanding, this is equivalent to saying, “I have a software project folder with text files. I need someone (not me, like DevOps) to build the project.” You can probably argue that the developer will always send the correct changes to the deployment. But, in my understanding, the problem of the complexity of real enterprise projects is at such a level that you should not rely on the fact that you remember or do not remember what and where you did or did not do. Confidence in the correctness of the changes being made is based on the success of their application on the current stand? And what is the condition of the stand? And what kind of stand is this? And when and who did the deployment at this or that stand, and which one? It is probably better to rely on an automaton, which provides control of correctness, often not obvious. And the right way is to complete the project of the target database with each change project, which includes deployment on an available bench, i.e. when the deployment stage is inseparable from the development process (deployment is rehearsed) – the design solution developed by the developer includes deployment.

Liquibase can and, in my opinion, should be used as a tool for organizing and compiling a database development project – this is its main purpose. The Liquibase project structure must first be laid out as a tree-like hierarchy of Changelog format files. Generating a Changelog file from an existing database is only used initially for old projects that are converted to Liquibase format, or if some one-time migration is required. The project catalog, as usual, is stored in the version control system (VCS) repository. And under versioning on the part of Liquibase, it is understood that the process of applying those changes as part of the current update, which are necessary for application on a specific stand, is automatically ensured (in the target database, Liquibase registers the applied changes in special service tables (Tracking Tables), which allows it to determine during the next update, which changes have already been applied and which have not yet been applied). At the same time, there is no strict need to use separate branches within the framework of VCS for different stands – the same state of the project can be transferred to different stands, with a different current state, and repeatedly – the issue of dividing project states will be determined only by the need to divide stages for revisions. Liquibase, of course, will not allow you to arrive at the desired model of replacement of the finished table based on its definition localized in one block of code – after a while, the definition of the target table will still be presented in the project as the initial create instruction and a series of subsequent alter instructions (there is a possibility after some time, after updating all the stands involved in the process, “comb” the definition of the table, combining all additions to the table in a single create instruction, but this is not a completely trivial procedure). Nevertheless, such a toolkit allows you to significantly “raise” the project model for the database on the way to high-level development based on Agile principles with CI/CD.

An approach based on the application of metadata allows you to get even closer to the desired high-level design model of the database. It allows you to switch to a declarative description of the database structure with the addition of an object-oriented paradigm to the project (partially). As a result, Liquibase instructions will be reduced to simple instructions for making changes to metadata tables; the project for the development of the target DB, in part of the structure, will turn into a set of declarations (this refers to the code of the target representations, functions and procedures to which the generalized elements can be applied only partially). Of course, all this is provided by a special platform, which must be developed if it does not exist. But the main motivating idea of ​​this approach is to provide an opportunity to concentrate on a logical design solution for the application developer, and to switch from working on an instance of the target system to working on an instance generator for the system developer. And this is the material of a separate article.

Related posts