how we lost and gave away skins from the game for millions of dollars

how we lost and gave away skins from the game for millions of dollars

Greeting! My name is Mykhailo Malashin, I am the product manager of the CS.MONEY game items marketplace. Today I will tell you how almost 1 billion rubles were stolen from us a year ago – this is a story about the need to change passwords in a timely manner, not to give excessive access to anyone, and to carefully monitor deployed Open Source systems. Let’s go.

I will remember Saturday in August 2022 for a long time. Early in the morning, an alert arrives that our CS.MONEY skin exchange service is being robbed: game items worth almost 1 billion rubles are stolen from the robots. Slack breaks up, new work chats appear one by one in Telegram – me, product and other teams are shocked and upset.

And we are more gathered than ever. A couple of days after the robbery, we will literally, like in the old memes, find the attacker by IP address, find out his mail, real physical address and get access to other information: for example, we will find out that our hero has already been charged with several cases of fraud. Then we will send him directly in the messenger all the screenshots with the composition of the crime, talk to our hearts and find out exactly how he hacked us.

But about everything in order. Let’s rewind and try to understand what happened on the day of one of the biggest hacker attacks in game trading history.

Disclaimer: CS.MONEY is a marketplace for buying, selling and trading in-game items, primarily CS:GO. All CS:GO items are owned by Steam. We earn commissions from direct transactions between users (p2p) or buy items ourselves to later sell them to other players. In the second case, we store all purchased skins in warehouses – our bots, and we provide the logistics of items using Steam accounts.

Here’s how it works: Steam has an API that allows you to exchange skins between Steam accounts. If it is very simplified, then the front of CS.MONEY acts as a shop window, and the backend of the service is a wrapper on top of Steam, which contains the business rules of our service.

Friday, (on the night of) the 13th

On the night of August 13, 2022, unknown persons hacked CS.MONEY. I became aware of the issue on Twitter: users were reporting that our bots were sending them trade offers for Counter-Strike: Global Offensive (CS:GO) skins.

First thought: someone has hacked our API and is exchanging items through it. The team immediately began migrating databases and virtual machines from OVH and self-hosted subsystems to AWS and as a Service. We removed part of the self-hosted database to RDS and closed it from the outside world, limiting access to it for developers (historically, part of our infrastructure was publicly available).

We shut down external access to the main database and dropped all but the most essential accesses, leaving the infrastructure in the inner loop. Using the Hashicorp Vault, the team began to encrypt the secret data that was previously stored in the open, but it did not help.

It soon became clear that the attackers had somehow gained access to the credentials and the second factor, Steam Guard, of the trading bots. The fact is that Steam has a cunning two-factor authentication system: the verification code itself is generated, if necessary, through the mobile application based on one static code. And since the credentials, the code for the second factor, and the jobs themselves were stored in the main base, the conclusion is obvious: it was hacked.

As a result, unknown persons took control of trading bots CS.MONEY, which at that time stored 394 thousand game items. After that, someone made hundreds of transactions, sending the stolen money to their own and other people’s accounts, including the accounts of popular streamers and traders unrelated to the hack. Probably, this is how the attackers tried to divert our attention and involve innocent people in a fraudulent scheme instead of traces.

Skins were sent out until 5 in the morning – in the first wave of the hacker’s attack, the company lost $6,000,000. We spent the whole day trying to figure out what the problem was until we reset the passwords. It seemed to work: skin translations stopped, but only for a while. As it turned out later, the operations stopped not because of our actions, but because the attacker simply got tired and went to sleep.

After waking up, the hacker continued to rob the site: there was a second multi-million wave of theft, during which in-game items were withdrawn to accounts with the ironic name CS.MONEY Recovery, so that these translations looked like a skin protection operation and misled users even more.

Search for the criminal and the bounty on his head

On August 15th, we asked the community for help by announcing a $100,000 reward for information about the evil. In the first few days, dozens of messages arrived on the CS.MONEY social network and in the mail: we were contacted by imposters pretending to be hackers, and those who simply wanted to take the cash, confusing the investigation.

A few days later, our guys calculated the attacker’s IP address and found out where to look for skins stolen from CS.MONEY.

Colleagues checked the IP address of the alleged attacker through our traffic analytics service Amplitude, and found the account and two email addresses of the fraudster in the logs – it turned out that the hacker had visited the CS.MONEY site several months before the hack. Including, with the help of this data, we managed to get all the information that helped to find the criminal.

After finding the attacker’s contacts, it’s time to talk to him personally. A marketing colleague WhatsApped the scammer, sent screenshots proving he was exposed, and convinced the hacker to return the credentials from our bots. The thief returned the controls and explained how he managed to break into us.

How and why the heist of the year became possible?

We do not absolve ourselves of responsibility: we messed up a lot. We have deployed an Open Source system for A/B testing GrowthBook. Once upon a time, we wanted to try it, and simply did not change the settings, leaving them at default.

Later it turned out that there was a bug in GrowthBook: the GrowthBook image uploader allowed you to upload any file to any location instead of an image. In our case, an attacker loaded a Reverse Shell and took control of the machine: the hacker gained access to the environment and found out which database we use. This happened because access to our internal database was added to GrowthBook as a Data Source. Data about Data Source is stored in an encrypted form; unfortunately, in our case, the encryption key remained standard: dev.

But that’s not all. GrowthBook allowed you to create a new user at any time with any password. Usually, when starting the service for the first time, the user gets to the login page. In GrowthBook, the mandatory check for the possibility of creating a new user was only in the frontend. And it was possible to create a new account in the backend at any time – thanks to this bug, a hacker was able to enter our system and download a script instead of a picture, gaining access to the database.

After the incident, our CTO informed the authors of GrowthBook about these vulnerabilities, patch already released

Digital hygiene: what can other companies check now to avoid stepping on the same rake?

The rules of “digital hygiene for the little ones” cost us quite a lot. Here are some important lessons learned from bitter experience.

  1. All that you are show to the outside world (bases, services, infrastructure, etc.), required close with 7 locks.

  2. Necessarily must have two-factor authentication. It may not protect against hacking, but it will make it more difficult for attackers.

  3. The more complex the passwords, the better. No need to remember them: generate random combinations with a length of, for example, 33 characters, and store them in your password manager.

  4. And the iron principle: a unique password for each service.

  5. In modern realities, it makes sense to use the Zero Trust model. Its essence is described in the title: do not trust anyone. Force everyone authorize, authenticate and periodically validate your access.

The more companies that adhere to these principles, the fewer data breaches there will be. And the safer the world we live in will be.

The GrowthBook vulnerability played a key role in the CS.MONEY hack. Key, but not the only one. Here are some other important points to consider:

  1. Check for redundant accesses and create separate accounts for external services. Yes, our company’s analysts had excessive access to product databases: instead of creating a separate profile for a third-party service, we mistakenly gave GrowthBook excessive rights. This happened after CS.MONEY added a new source to enrich the data coming to GrowthBook – our database of confidential information and access to it. All this created confusion when investigating the incident, and it took us a long time to find the account with suspicious behavior. If we had a separate account for external services, it would be much easier to detect a security hole.

  2. Check the Helm abstractions again. If you use Helm Charts, do so very carefully. In our case, Helm abstracted an important key configuration that allowed GrowthBook to run with a configuration that was not acceptable for a production environment.

Let your defenses be unbreakable and let the very smart hackers bypass you. Good luck!

Related posts