how to change the vendor in the project for 6000 access points in 1.5 months
In the spring of 2022, the usual vendors of network transport infrastructure, with whom everyone is used to working, left the Russian market. We found ourselves in the middle of a large and urgent project, from which the equipment supplier – Huawei – left. I had to choose a new one in a matter of weeks.
Below I will tell you how we urgently looked for a new supplier, why we chose the Maipu brand, which no one in Russia knew, and what came of it.
Contents
Scale of the project
The initial task was to deploy an IT infrastructure that would allow regional and international events to be held at a high level and withstand increased network loads during mass events.
A cluster of dormitories, a large sports facility and a leisure center are located on the territory of the site. In terms of equipment, it is 500 routing switches and 6,000 access points. The initial plan involved Huawei equipment and by the end of winter 2022, we managed to install it at 2 facilities.
At this moment, all the events take place, as a result of which Huawei refuses to work in Russia. We understand that we need to choose another vendor within a few weeks that will be suitable for large-scale implementation, will be convenient, and will work for many years. And, of course, it will be compatible with the infrastructure already deployed on Huawei.
A small timing spoiler for this story:
-
August 2021: approval to start work using Huawei equipment.
-
February 2022 – withdrawal of many foreign vendors from the Russian market and Huawei’s refusal to ship equipment.
-
March-May 2022: search and selection of a vendor, verification of its reliability and scalability, partnership negotiations, remote testing. All this in 1.5 months.
-
By October 2022 — customs clearance, deployment of the test stand on the territory of the customer, final joint decision with him on the implementation of the selected vendor.
-
October 2022-May 2023 – use, commissioning.
So we went.
Vendor selection
When we were left without Huawei, we went to the Internet to search for everything that was available on the world market, since until that moment no one in Russia understood what alternatives there were to the vendors that left, we decided to look for not just a supplier for a separate project, but a long-term one a partner with whom we can continue to work.
We have compiled a shortlist of 20 domestic and foreign vendors. The main criterion for us was a quality product that can compete with Huawei and Cisco. The next step is direct contact with vendors and a conversation about everything that is important from the point of view of both the current urgent project and the long-term partnership.
Based on the results of the selection, we settled on the Maipu vendor. In their home market, they are quite well-known as a manufacturer of network equipment for commercial use, so they had the necessary enterprise-level performance line. Back in 2018, we tested this vendor’s equipment and even tried to implement several projects with it, but unfortunately, the “dark horse” could not defeat Huawei at that time.
As part of the testing, we checked not only Maipu for suitability, but also several alternative solutions. As a result, Maipu showed the most worthy results both with the integration with the installed Huawei and with the reliability indicators of the equipment.
Why did we choose them?
Compliance with technical requirements to the equipment in our project and passing our tests, which I will talk about a little later.
Focus on partnership. We were looking for those who were ready to commit to a long-term partnership. From the partner’s side, we were waiting for priority conditions regarding terms, cost, readiness to add software. We ourselves were ready to promise priority implementation and undertake technical support. Maipu turned out to be very flexible and believed in us (after all, we also had to convince them that we are a reliable partner and worth doing business with).
Independence of the supply and production chain. Maipu has its own production in China and a component supply chain independent of other markets and sanctions. In terms of reducing risks in a long-term partnership, this is an important point.
Delivery speed. Maipu proved to be ready to quickly supply the necessary amount of equipment and, according to our estimates, are ready to increase it as needed. Including the account of warehouse stocks of manufactured equipment.
Simple mastering of the interface and approach to work. No special training is required to work with the equipment. This is important not only for our experts who will deal with the implementation, but also for the customer’s experts who will decide to operate. The Maipu command line uses the syntax adopted from Cisco, and the approach to configuring the equipment is almost Cisco, with a slight touch of Huawei. Anyone who has dealt with Cisco understands easily. Well, the GUI is not Chinese, but English.
By the way, K2Tech engineers became the first to receive Maipu expert certification in Russia.
Testing and supply
When you have enough time, you order equipment and wait for delivery. Then you conduct bench tests, make a pilot and run some combat traffic on it to make sure everything works. And you write separate chapters in the article on Habra about supply and testing.
We were running out of time. Uncertified equipment from China takes 6-12 weeks to clear customs. Therefore, our supply took place in parallel with testing, for this we organized a whole project team. This is what they did:
-
Supply organization. A separate song is chartering an airplane. During the Chinese lockdown in 2022, this is, believe me, a whole month-long story.
-
Notification of the FSB about the maintenance of cryptographic functions in the equipment is a mandatory requirement when importing the equipment to the EAPP. We initiated and helped receive the notification.
-
Customs clearance
Meanwhile, our new Chinese partners have agreed to set up a bench for us to test the equipment. They gave us remote access to review the interface and basic functionality, run synthetic tests. We managed to do all this while the main equipment was on its way to us.
When the equipment arrived, we already knew what to expect. We quickly deployed the stand on the customer’s territory, showed it to him and received good feedback about the quality of the equipment and the ease of working with it. In general, we were given the green light. We implement.
Network architecture
Our network architecture was thought out in advance. We took as a basis the architectural approach used in the construction of L3 distribution campus networks. The key advantage of this solution is resilience to failure on trunk links without using L2 protocols. This automatically limits the impact of broadcast domains on network stability, parasitic traffic is localized by the domain boundary, and the load on trunk channels is balanced due to the ECMP (Equal-Cost Multi-Path) mechanism of dynamic routing protocols.
Since the network was built on the basis of equipment from two vendors, Huawei and Maipu, for transparent integration we chose L2 and L3 level protocols that both vendors support. Many services are deployed on top of LOM, which sometimes need to be isolated from each other. For example, access points in dormitory buildings not only distributed Wi-Fi, but also allowed connecting laptops, landlines, and televisions via Ethernet. We segmented the network using the VRF (Virtual Routing and Forwarding) mechanism in Cisco terminology, or vpn-instance in Huawei terminology, and divided the various services into isolated functional groups that never intersect. If necessary, we organized network interaction of different network segments through NGFW.
Since we had both Huawei and Maipu devices, we implemented a management system that allows us to track all devices online and manage their operation.
Realization
4 months of tireless work at the facility united the team. I want to draw a whole comic or film a TV series about it. But I will limit myself to photos, they fully convey the atmosphere.
To begin with, we entered the facility in parallel with the end of the construction works in order to start the systems as early as possible and hand over the facility on time.
The facility was de-energized, and our engineers walked around it in helmets with flashlights. They carried 70-meter skeins of wire that they pulled from working outlets to poll and adjust equipment.
Separately, I would like to highlight the impressions of our engineers about Maipu devices. Here is the opinion of the chief architect of the project, Volodya Naumov: “They are simpler than the same Huawei, they include minimal and sufficient functionality. There is no need to use any special modes, the interface is intuitive, there are no pitfalls. You take it, set it up, and it works right away: both from setting up the LOM and the wireless network. Huawei is more complex, but because of this, some of the operating modes are mutually exclusive. The simplest example is the support of 1 Gbit/s transceivers on ports 10/25, when for the sake of one connection it is necessary to switch the mode of operation for 4 ports of the switch at once. For simple scenarios like ours, this is overcomplication. Maipu pleasantly surprised us.”
Results of the project
The LOM and wireless network we deployed became the transport for 50-60 adjacent systems, from ACS to complex multimedia solutions. It has several conference rooms for 250 people with mixing consoles and streaming to a 20×30 meter video wall. A sports event with thousands of participants on the operation of the network went off with a bang, and the infrastructure continues to work.
Support
In the main markets of Maipu’s presence, there is a raw material and material base that allows to implement a transparent warranty policy, to ensure SLA. Of course, Russia does not have all this. Any railway from China must be transported. Someone should have organized the infrastructure to support Maipu’s solutions. Guess who got this honor?
We have organized the Maipu support service inside K2Tech. We created a raw material base, a spare parts base, and personal protective equipment. We provide technical support for the Maipu solutions that we implement and guarantee SLA.
At the initial implementation, we did not know the real quality of the equipment, so based on the experience of working with the top global vendors, we set a rather high 5-10% of the marriage. Maipu exceeded our expectations: on the launched project, the marriage turned out to be 0.1% (in our experience, it is even less than Huawei). Moreover, even devices and individual elements failed.
Team
In conclusion, I would like to highlight the contribution of the people who made this project possible. Projects like this become moments of growth and rebirth of teams. You remember it for many years, and those who were close to you become more than colleagues. The short time it took to choose a vendor, how to make decisions, conduct negotiations, create from scratch the internal services of the distributor, warranty service — all this turned out to be possible thanks to them:
-
30 backoffice people who organized vendor selection, negotiations, delivery and everything that needs to be done before implementation.
-
12 debuggers on site at the time of implementation.
-
7 service engineers and support administrators on the site and in the office, who support further operation.
Thank you all for your attention. Write in the comments what else you want to tell about Maipu or in general about the process of attracting new vendors to the domestic market.