When we switched to another management system with an automated testing team, we had the Russian TMS TestIt as a working tool. We did not deal with deployment or configuration, only active use. Now I will tell you what feelings are like.

Test management systems (also known as TMS) greatly simplify the life of the testing team. There are dozens of them on the market. They are needed for maintaining a test model and describing cases, their versioning, integration with autotests and their launch, collecting data from them, drawing up and passing test plans, generating various reports.

With a large number of cases (and we have several thousand of them), both automated and manual tests in the test plan cannot do without TMS.

It was the optimization of the routine in terms of variations of launches, getting results on the fly, ditching Allure and getting a single report that gave us a time gain that can give the team a few extra days every month to automate something else. For example, so that the percentage of coverage by autotests does not decrease with the constant expansion of functionality and the appearance of new features and services.

Why did we need TMS integration?

A TMS not only helps you assign tasks to specific people, but also allows you to track whether an action has been performed and by whom, in addition to tracking various aspects of testing.

It also provides an opportunity to make the process transparent and understandable and to improve testing plans in the future.

Also for convenience – all test models, test runs, test results and reporting are in a single space and easily accessible.

In addition, any team member can easily run arbitrary sets of autotests simply by selecting them from the available list, without being immersed in the intricacies of implementation and without having direct access to the code.

Why TestIt

TestIt suited our tasks. Developed for testers by the same testers, has a Russian interface and Russian support.

TestIt is delivered either as an Enterprise solution that is deployed locally at the customer’s facilities, or as a cloud-based SaaS solution, while customers do not spend resources on maintaining their own architecture and facilities. In our case, the Enterprise solution was used.

Our colleagues managed to create a flexible tool that allows you to manage data, configure test cases, and set access rights and security policies. What did we like about working with this system?

Both manual and automated tests can be combined in test plans and suites, and plug-in autotests can be built in different frameworks.

TestIt is actively developing and now has integration libraries for various languages ​​and platforms.

At the time of our integration, this was not the case.

You can work with the most popular frameworks: TestNG, Cucumber, JUnit (connected via adapters). At the same time, client libraries are supported for integration using such languages ​​as JavaScript, Python, and C#. There is integration with Jira, Jenkins, GitHub and other tools.

Automated test scenarios can be managed through a single interface. And they can be launched directly from different sections.

Convenient functionality for working with test plans in startups and creating screens with different reporting display formats.

A role-based model of access delimitation has been implemented.

Flexible, convenient, clear.

After overcoming another group of floating errors and stabilizing the results of running the autotests we had at that time, we decided to start integrating our project with TestIt.

Stages of integration, their features

The integration process was divided into three stages.

Stage 1

Learned the TMS API. Implemented binding of autotests to manual tests, launch with CI by tags, unloading of reports.

From that moment on, our autotests were linked to TestIt’s manual tests. The launch ID had to be retrieved manually and sent to the CI request itself. Reports were uploaded after all tests were completed. Unpleasant situations also occurred, for example, the termination of the processing of test run results during plan editing, as a result of which the plan had to be forcibly blocked during the run.

Stage 2

Implemented launches via web hooks. Parameterization.

Immediately after the possibility of using webhooks appeared, we configured the parameters of requests to TeamCity and implemented a script for selected tests with the possibility of running on different circuits.

Next, the possibility of working with parameters was provided, that is, the same cases with different variants of parameter values.

For this, we decided to use the configuration section. For each configuration, a set of parameters (key-value) can be set through the TMS interface, which is used to parameterize the tests.

Stage 3

Various buns were added

into the system (links, screenshots, logs, archives) to make it faster and easier to analyze results and, in particular, crashes.

As a result, we got an additional and most pleasant opportunity to run tests with TestIt.

Advantages and limitations of autotests

One of the benefits of using the TestIt integration is the ability to apply different parameter sets to cases. Integration with TestIt of parameterized tests using configurations

convenient visual representation

results and

flexible startup

test variations.

It is irrational to create a new case for each variant of the parameter values ​​(and to reject parts of them as a result). In TestIt, you can create configurations with an arbitrary number of key-value pairs and use them for the same cases. Information about configurations and their parameters for running parameterized tests can be obtained from the TMS API. Test run results are processed and displayed across different configurations.

Pros and cons of integrating with TestIt

Well, in the end — the distribution of elephants. Here’s a small list of the benefits we’ve seen when integrating with TestIt:

• Ease of selecting tests and running them.
• The ability to see test results not only after a full run is completed or after generating a report in Allure, but also in real time as the tests progress.
• Ability to store all reports in one place, as all analytics can be done in TestIt. Here you can see a detailed history of test runs and an analysis of the reasons for the crash. Summary reports by error category are also available.

As for the shortcomings, we noted the following in our work:

• Problem using error categories and tags. We were unable to use this useful functionality because they are not attached to projects and are visible to everyone inside TestIt. This is associated with the risk of their loss or modification.
• Inability to work with parameterized tests “out of the box”.
• No grouping of test plans when storing and viewing trends. This makes it difficult to compare similar scenarios to be able to analyze biases.

Recommendations for working with TestIt

What recommendations can be made based on our experience with such a system?

You can choose any automated tests, add the necessary configurations and parameters to them and see how it works.

In particular, we developed one small script for the support team to run and monitor functionality between shifts.

When a TMS is needed and when such integration can be dispensed with

No TMS and TestIt, in particular, solve all the problems that testers have. Do not think that it will be possible to completely get rid of manual labor. Which testing tool to choose depends on the size of the team and its needs, the life span of a particular project. If you are stuck on using only manual tests, then such a system will probably be excessive for you.

I think this framework can be recommended for teams that have a lot of test cases and that use both manual and automated tests. The convenience is that this system combines all this in one interface.

All this will definitely simplify your work and make it more efficient. At least it worked for us.