How we automated and simplified the release management process

How we automated and simplified the release management process

Until the beginning of 2020, our team participated in releases as a group of testers led by a coordinator. The process was completely manual, lengthy and time-consuming. The coordinator was involved in the formation of regression teams, activity tracking, dashboard management and other organizational activities within the testing team. At the same time, the main functionality — formation of the release composition, communication between the testing team and developers, holding meetings and other events related to releases — was under the control of the adjacent management: all this was done by the release managers, and they were the head of the release.

After the transformation, all the functionality described above was transferred to one group – Release Management. Communications, work with the testing team, reporting and meetings – everything was started by our team.

Most of the communication was by mail. Collecting, fixing the composition of the release and reporting – everything had to be done manually. The process of preparing for the release lasted up to five working days. Every day, the release manager saved a pdf file with a list of tasks from confluence and compared the composition of the release for added and excluded tasks. After the task was included in the release, the test managers independently entered it on the “Update Regressions” page on confluence according to the color legend and determined which tasks required changes in the test model and which had not yet been implemented. There were many manual activities of release managers and test managers, and this made us think about partially automating the release management process.

Next, I will tell in detail how everything was “before” and how it became “after”.

QAD — quality assurance dashboard — has already been partially developed in the Department of Load Testing. QAD has a lot of functionality, guides, distribution warehouses, test suites, components, profiles. This tool allows you to use Jira filters to form and track changes in the list of tasks and partially – reporting. It is enough for us to form a task for QAD support with a complete description of the needs and the desired result, and then at intermediate meetings we discuss the nuances, if necessary, make adjustments, agree on the deadlines and wait for implementation.

With the joint efforts of the load and integration testing departments, we are gradually filling QAD with those features that no release can do without.

1. Formation of the composition of the release

We remember that a list of tasks included in the scope of the release is formed at a joint meeting with the teams. The release manager prepares Jira filters for each team, accepts submitted tasks, checks for correct filling of fields, presence of protocols and completion of testing.

Previously, the list was only in confluence. After implementing the revision in QAD, the release manager introduces the filters themselves into it and performs synchronization. Tasks are entered in QAD to the block “On approval”, then after manual checking of statuses and availability of current protocols, the task is accepted. If everything is filled in correctly, then the task is approved and goes to the “Agreed” block. For ease of use and simultaneous viewing of all necessary fields, the release composition page can be configured to display all necessary fields: team, stream, author, dates, commits and versions, etc.

2. Happy path tasks included in the release

After the task is included in the release, the team in which the task was developed must go through the Happy path on the regression environment.

Previously, a page was created on the same confluence, on which the teams posted the corresponding statuses for the tasks: Happy path – passed, not passed, not required. Not always and not everyone did it promptly, and in many projects Jira simply did not have this field to fill in, so someone entered information in comments, and someone put labels. There was no common understanding. Release managers often had to return to this page and point-by-point track each task. We didn’t count how much time it took, we can only guess that it was quite a lot.

What did we do? The first is that all Jira project owners were asked to add the Happy path field. At the meeting to fix the release scope, all teams are told that the Happy path must be completed within the specified time according to the control points of the release schedule, otherwise the task will be excluded from the release. On the composition page of the QAD release, the Happy path field was displayed and the ability to create a table with a list of tasks was added. Next, we are already sending a letter to the authors and performers about the need to complete and complete the Happy path.

3. Information about permission

In addition to a dry list of tasks, we wanted to visualize releases. Tasks are submitted in the release, their number can be different (80, 100, 150…) from different teams. Charts and graphs are generated on the Info page in QAD. For further analysis, we consider how many tasks in the release are submitted by team, as well as how many tasks are included in the release after its fixation.

4. Fixing changes in the composition of distributions

The composition of the release can be changed before the frieze: tasks can be included or excluded. To control the composition, we added the ability to record the reassembly of the distribution, indicating the dates, systems, time and composition of the changes.

5. Updates of regressions

Almost every task involves changes in one or another functionality. It takes communication across our large team to find someone to make changes to the cases. Test managers appoint responsible persons and monitor implementation. It was already mentioned at the beginning of the article that previously test managers maintained a page on confluence separately for each release. We decided to move this activity to QAD. In the current implementation, the page is filled with many filters by projects, statuses, performers, teams, systems. Various options for searching and generating reports have been added for the convenience of test managers and managers.

6. Managing errors from the regression environment and later – shooting errors from the industrial environment

An important activity of the release manager is the control and maintenance of errors during the release and after its release. Using the Jira filter, we add all errors found during release testing to QAD. Here you can leave comments, view statuses and create a report. In the section by releases, all errors and comments are saved, if necessary, you can return to past releases and review the history.

Since release management does not end with its release to the industrial environment, a post-release incident tracking table was added later. It is filled in manually: we take the incident number, description, system and date of detection from the periodic report from monitoring colleagues, then we conduct an analysis and form a conclusion.

7. “Versions of microservices” page

Allows you to compare the versions and commits installed on the test environment with the production environment. Previously, there was a separate tool – rancher-dashboard, which could not display all the information, but showed the composition only within one microservice, and the difference had to be determined by yourself. After the implementation of the next revision in QAD, we can see the full composition and list of microservices, the changes of which were included in the release. If the versions in the task do not match those installed on the environment, this can be seen by the red filling.

8. Limitations during release implementation

There are a number of limitations when installing a release on an industrial environment. Previously, all communication was in the mail and collected from performers manually. After this page was added to QAD, there was no need to collect in the mail: our support in the “Restrictions” block keeps its marks on the status of acceptance of distributions and work time, lists of employees who will go to the installation are formed. All we have to do is click on the “Generate letter” button and send all the collected information to the necessary recipients.

Currently, QAD houses not only Bank-wide Releases, but also Payment System Releases, Unscheduled Patches, and the Online Applications team. Automation of mailings, collection of letters, reporting — this reduced our resources for conducting and supporting the entire release in general. And we don’t stop there!

Related posts