how internal testing helps us refactor Wiki / Habr

how internal testing helps us refactor Wiki / Habr

Hello everybody! My name is Sonia Yevstigneeva, I am a project manager at AGIMA. I will tell an almost detective story about our internal grading system and what metamorphoses are taking place with it. At first, it simply helped us determine the level of knowledge of newcomers, and now we use it to fundamentally update the knowledge base of companies. At the same time, I will show the questions of our grade test. Let’s go!


For convenience, I will divide the article into two parts. The first is about how our project managers work with grades in general. In the second, it is about how this seemingly simple task helps us to optimize the knowledge base.

Part One: How We Define Grade

Grading of managers

When it comes to grading in development, the standard scheme usually comes to mind: junior – middle – senior. Each of them has their own set of skills, experience and salary expectations. And in most cases, we follow this scheme. But later it had to be expanded for project managers.

AGIMA has accumulated a lot of instructions, tips and regulations over 16 years. All this information is stored in an internal knowledge base, aka Wiki. For convenience, all materials there are divided into blocks: for the same juniors, middles and seniors. Therefore, generally accepted grades in our case are attached to very specific (and sometimes specific) knowledge and skills.

Roughly speaking, if you need to get a regular ballpoint pen at company X, the process will be one, but at Y, the process will be different. And thus, each task will have its own subtleties. Therefore, it is important for us to immerse a new employee precisely in our processes and regulations. And the grading system has been refined taking into account our unique experience.

Our Wiki has selections of mandatory materials for specialists of each grade

Globally, our internal system repeats the generally accepted:

  • First, all employees (not just managers) pass zero grade. Passing a zero grade shows that the employee has a basic understanding of the company’s positioning, areas of responsibility and office rules. Depending on the position, we add specific questions: for a developer, there may be a question about the organization of a VPN at the workplace, and for a project manager – about communication with the accounting department.

  • Next, we offer the manager to pass the grade JPM. Completion of this grade is usually included in the list of required achievements before the end of the internship or probationary period. At the time of delivery, the project manager must know what tools we use, distinguish between work formats, adhere to the rules of documentation coordination, work with workshops, time management principles, etc.

  • After passing the starting stages, the handover follows MPM. We expect that the RP of the middle grade knows how to manage a budget, how to correctly decompose systems for evaluation, understands the intricacies of managing the budget, risks and changes. We expect him to confidently navigate the JPM regulations and also be able to name the features and development technologies.

  • The final stage of grading is testing for the level SPM. At this stage the manager already has a good understanding of development standards, features of the stack and infrastructure organization, the specifics of the SLA format, knows the principles of extreme programming, DevOps and CI/CD, information security techniques. He can be entrusted with complex high-load projects with continuous support.

But despite the usual words like “junior” and “senior”, it has many features. All of them are related to the specifics of corporate experience and the tools we use. Our primary task when onboarding a new employee is to painlessly immerse him precisely in our processes. Graded testing helps us in this.

Excerpt from the Wiki article about communication and, in particular, about communication in messengers

Grade testing

Without practice, the theory from our Wiki is learned slowly, there is a lot of information. And in order to allow the intern to the first real tasks, he needs to know the basics. Here the circle closes. Therefore, we assumed that in this situation internal graded, practice-oriented testing would help us.

Many of the tasks in the tests are designed to put an employee in a typical junior situation and see how he would handle it. At the same time, we do not want to “drown” a person or scare him. Our goal is to understand what information he has learned and what he has not. Here are examples of tasks:

Zero grade question

Question from JPM

Testing shows that the employee navigates the knowledge base and understands where to find the answer to any question. A high score in the test means that the manager can independently search for answers to the question. The mentor will, of course, continue to work with him and help him, but now the novice is ready to go on a big swim.

The system with graded testing works not only for trainees. If the manager wants to get a higher grade and more difficult tasks, he also passes the test. But here it is important:

the grade system is not tied to salary

If a person does not pass grade testing, his salary will still increase. The financial motivation of employees generally depends on the grade. But our experience shows that sooner or later any manager starts to want more ambitious tasks. In this case, upgrading is a necessity.

What is testing

In total, our database contains more than 200 questions for testing. But on the test, you need to answer 20. It takes place online in the INDIGO system. Its conduct is monitored by an eichar.

Question from JPM

Questions from SPM

One more question from SPM

The tests have different types of questions: with one answer, with several answers, with a field for free input. Each question has its own weight, they affect the final result in different ways. It is considered that a successfully passed test is 80% of points. Any question can be resolved after testing. And if the employee does not agree with the result, then appeal. It is important!

Tests can be submitted multiple times. Therefore, before the start, the mentor explains to the manager that this is not the EDI and there is no need to worry too much. However, we do expect high scores from interns, at least on the second attempt – the result of the internship depends on this.

This system has obvious advantages:

  • helps adapt trainees;

  • helps managers analyze their own experience;

  • motivates to be interested, read more and immerse yourself in the profession;

  • the system is not tied to the salary, and therefore does not put pressure on the person;

  • allows you to gradually memorize the rules in a playful way.

But, unfortunately, there are also disadvantages:

  • the system does not allow you to reliably determine the employee’s usual grade;

  • this is an additional burden on mentors and eichars;

  • refactoring is required when regulations are updated.

The last point is especially important. As a result, it was he who helped us find a lot of inconsistencies in the regulations and start improving them.

Part two: How we went about refactoring the Wiki

What is the problem

It is difficult to maintain such a system, because it has to be constantly updated. The company is developing, processes are changing. Processes change – regulations change. Regulations change – tasks change. That is why I emphasized that the test result can be appealed.

For example, the regulation of document coordination has changed. Now, instead of Vanya, Petya is responsible for approval, and the approval period has decreased from 5 to 3 working days. We updated the rules in the Wiki, but forgot in the task. The person passed the test and pointed out the discrepancy to us. Or even more: after testing, we realized that both the test and the regulation do not reflect reality. So, it is necessary to change both.

In addition, the Eichars were fully responsible for the tests. And they could not always notice the flaws in the answer. Banal: some norm may no longer apply, but is marked as the correct answer in the test. As a result, we confused both the person taking the test and ourselves. At the same time, checking two hundred questions and all regulations is a huge task.

Refactoring of the grading system

We had to think about refactoring the entire system. We wanted the information in the tests and the Wiki to always match. Otherwise, the grading system itself lost its meaning. In addition, working with the tests revealed the aging of the Wiki itself. So it was an opportunity to update all the data at once.

We made a refactoring scheme:

We outlined this scheme at the first meeting

The main points of refactoring were:

  • formation of a list of responsible persons and an action plan;

  • checking for relevance and updating articles;

  • checking for relevance and updating tests.

Our plan to update materials involved three phases. We have documented them in our Wiki.

The first stage

At this stage, we have completed all the preparatory work:

  • uploaded the questions and answers from INDIGO to the Wiki, compiled the questions after a general check;

  • drew up interaction schemes;

  • checked the articles, compiled a top-level table with articles from grades, indicated the relevance of each.

The first stage took about a month. In addition, we have changed the procedure. Now the eichar only conducted the test, and sent the results to the responsible manager or his assistant, also the project. And they, in turn, updated the materials and took part in denying the results.

When it was necessary to update the rules, the assistant made corrections to the articles himself and indicated it in the Wiki. All responsible people were automatically notified of the change. In the case of more complex adjustments, the manager started a task in the task tracker and appointed a responsible specialist from the workshop.

An example of the interaction scheme during the appeal of the results:

  • a person is undergoing grading, he has questions; he sends comments to the mail, puts responsible persons in the copy;

  • the responsible manager processes these issues: either explains that there are no errors, or takes them to work;

  • when (and if) the responsible manager makes changes to the regulations, he informs this team, and the HR specialist adjusts the test result.

The group head or head of the project office of the person who took the test can help in making a decision.

The second stage

Here we identified the real responsible parties, created a backlog of tasks, monitored their execution and delegation. This was the biggest and key stage. In order to update some of the Wiki articles, it was necessary to bring in specialists from the production shops.

A simple task involving a specialist is to add an example report or replace a broken link in an article.

A difficult task: connect a specialist from the production shop and schedule a task for processing materials. Or, for example, to discover that the real process does not correspond to what is described or is not yet described in the article

We created tasks in the task tracker and made a plan for when they would be completed. At this stage, we have verified that the interaction scheme works and that it is suitable for the first large-scale refactoring.

At the same time, we regulated this process and agreed that we would regularly check the articles for relevance according to our table, so that the grading took place correctly.

The third stage

Here we checked the results of the refactoring of the grading system and summed up the results. Since we didn’t have any hard deadline, the refactoring lasted about 3 months with noticeable pauses before the New Year. Often we prioritized other tasks, but with space and a plan of action in hand, it’s easy not to lose the shift.

Further plans

The development process has not stopped, we are still replenishing the backlog and closing tasks. There are plans not only to regularly refactor grading, but also to expand the system — to make individual grades for group heads and heads of project offices, to fill them with situational tasks from our company’s experience.

We hope that in the future we will be able to issue certificates of compliance with grades from AGIMA to managers. The grade specified in them will not be equal to the generally accepted standard. But for companies that worked with AGIMA, such a certificate will be an objective bonus in the employee’s portfolio.

If you are interested in project management, follow on our Telegram channel.

What else to read

Related posts