Hello, Habre! My name is Kyrylo Pokladov, I am the IT director of corporate, investment and depository business at Rosbank. In this post, I will talk about a tool that helps our bank grow faster and fight the stereotype that all banks are slow and any release there takes a long time. This tool is our engineering culture.

What is engineering culture

Engineering culture is a set of practices, patterns of behavior and decision-making that helps to achieve maximum efficiency of the development process. There are several key goals at the heart of engineering culture.

• Rapid release – the world is changing rapidly, and our products must not lag behind.

• Modern approaches to development – we strive for the level of global technological giants.

• Product Reliability and Security – We implement modern practices such as SRE (Site Reliability Engineering) like Google and Chaos Engineering like Netflix.

• The variety and complexity of tasks is how developers feel more involved in projects.

• Increasing the value of employees as a result of work using engineering culture.

Principles of engineering culture

The goals have been dealt with, now it is the turn of the principles that help to achieve them. The principles are formulated as simply and clearly as possible, but have a deep meaning.

I work in the fields, I set an example

Each of us, from the director to the newcomer in the team, is called not to be afraid to “get our hands dirty” and plunge into the task to the full depth.

I have more than 100 systems in my department, which we constantly refine and support. The volume of changes is huge, decisions about development must be made constantly. I myself supervise and analyze many projects in detail, but of course I give high priority. I carefully approach each of them: I study the architecture, check the composition of the team, draw up the budget together with it and take part in drawing up development plans.

When solving day-to-day problems, I try to be in the context of what is happening: I participate in internal team meetings, conduct calls with the teams and solve the difficulties that the guys have. My example is not the only one for Rosbank, other teams work according to this principle. I give the floor to the “Situation Center” team – the main IT manager Serhiy Tereshchenko and the chief specialist Artem Shnyder:

Our team monitors the bank’s IT systems 24/7, coordinates the resolution and follow-up of incidents, and also ensures smooth operation of telecommunications. Our regular changes use several practices that reflect the principle “I work in the fields, I lead by example”:

• team leaders change almost every month;

• daily “five minutes” are organized for discussion and assistance in solving current issues, proposals, incidents;

• once a week, team leaders go to face-to-face meetings to share experiences and discuss current issues.

I do it for myself

This principle helps us design with quality and with an eye to the future, thinking through every detail. For example, Rosbank’s Service Development and Development Practices Competence Center has developed a technical debt registration system. The head of the center will tell you more about it Zakhar Frolov:

The technical debt is drawn up in the form of tasks marked in different ways: with the help of predefined labels (Labels) or with the help of a separate field Component/s with a predefined name. The technical debt tasks then go through the same evaluation and decomposition cycle as the remaining tasks in the backlog.

Based on our experience, we recommend taking at least 15% of technical debt tasks in each sprint in order to avoid its accumulation, in some cases more, but it is not always possible to agree on this amount with the customer. Teams agree on the scope of technical debt tasks for each sprint, and try to perform them strictly in the previously agreed order from sprint to sprint.

The use of a generally accepted approach to tracking technical debt tasks allows you to determine its scope in terms of the team, the product, directions for making management decisions and planning the work of the team.

By developing the existing one, I offer a new one

We do not consider current solutions as the best by default and always look for the best among the latest approaches and tools. At the same time, we are in no hurry to break the old, which has already proven itself. For example, at Rosbank we are implementing Chaos Engineering – meetings where different IT teams can conduct a controlled experiment as a paper test. Based on the results after the conditional resolution of the incident, the team supplements its monitoring and response scenarios.

Dmytro Stupin, chief specialist of the intrabank systems department, will talk about the experience of Chaos Engineering:

Our bank has a PWC (IFRS9) calculator that calculates the reserve according to the methodologies of international financial reporting standards. The calculator has a user interface for parameterization and control.

Data exchange takes place through ODPP (Online data processing platform) and files downloaded from CDWH. The calculator itself provides the results of its calculations as data in MSSQL database tables. The calculator is developed on the .NET Framework 4.6 platform

In this configuration, for example, I will give a paper test where the situation “Interruption/loss of connection between the database and the application” (inaccessibility of the port) was considered. Hypotheses were put forward:

If the session with the database is interrupted for a short time (loss of connection, unavailability of the port), the program works normally:

1. After restoring the connection with the database, the service also resumes work without problems with the application.

2. After restoring the connection to the database, the service becomes active, but it reconnects to the database, does not process incoming packets, and in the logs we see the absence of a connection.

These hypotheses were tested on the test server of the application. According to the results of the check, it turned out that in the event of a short-term loss of connection between the application and the database, the web application works and is active, while the services are in the Running state. But when the connection with the database is restored, the program does not process requests from the database, although it is in the Running status. In the logs, we can see that the program does not work and does not reconnect after the connection is restored. An error occurs:

WARN NServiceBus.Transports.SQL Server.Sql Server Polling DequeueStrategy Exception occurred when connecting to the configured SQLServer instance System.Data.SqlClient.SqlException (0x80131904)

According to the results of the test, we received confirmation of the second hypothesis. On the basis of this, monitoring was configured in Zabbix, which checks the log for the presence of an error and, when it appears, sends a warning to mail and telegram to restart the service.

I am developing and sharing my experience

By supporting and developing each other, we invest in our future. Knowing what our colleagues are responsible for and what their strengths are, we turn to them for help, creating a single team where everyone is valuable.

Our guilds are based on this principle – professional IT communities where engineers exchange experience, adopt best practices and improve their skills. Some specialists organize and participate in various HR activities, for example, setting goals and evaluating professional development. Others help maintain standards across the technology stack and develop unified profession standards across the bank. Still others help develop the profession through educational meetups and conferences. DevOps, Java, QA, architecture, admins, analytics – these are just some of the areas in which we have guilds.

If you are interested in learning more about guilds, watch the recording of the meeting “Trade unions in corporations: how, why and why?”, which took place in the office of Rosbank in “Moscow City”. In addition to Oleksandr Denisov, director of the IT support department for user services and digital services clients of Rosbank, experts from SBER, Tinkoff and Aston took part in it.

These are far from all the principles of Rosbank’s engineering culture. I will talk about the rest in the next part of the material. In the meantime, I’m happy to discuss with you what internal processes increase development efficiency in your company.