Inside the Stack: DDoS Attacks Should Look Like Nothing Ever Happened

Welcome to Inside the Stack, where the people behind Nitrado talk through the real decisions, tradeoffs, and lessons behind running multiplayer games.

This time, we sat down with Ross, Team Lead of SteelShield at Nitrado. From typing game code out of library books in the 90s to managing complex data center networks, he’s now the one tasked with keeping malicious actors away from the server room.

Q: What games are you currently playing?

Dave the Diver on my Steam Deck :)

Q: Is there a game you think is brilliant from a technical perspective?

Having grown up in the 90s, and starting my programming journey then too, I’m still in awe of the Wolfenstein to Doom to Quake evolution.

During that time in South Africa, where we didn’t have access to every global market, the only way I could really get access to games was taking books out of the library that had source code for them. 

But Quake was when gaming started to become this thing where my friends and I would get together regularly and go to tournaments.

Being a kid and seeing games evolve like that was mind blowing.

Q: What is your day-to-day here at Nitrado with SteelShield?

Mostly meetings, planning, and support :) and of course I get a chance to do some actual development!

I spend a fair amount of time speaking with our other teams and our Product Manager (check out Torben's interview!) to figure out the best way we can better serve our customers and products in protecting them from DDoS attacks.

Q: What pulled you into security and DDoS protection?

Luck and admiration for the low level.

I learned to program back when every instruction counted, and after years of writing JavaScript, I wanted to get back to writing code that wasn’t supported by a thousand npm packages and a confusing web of promises (the programming kind, not the kind we tell our managers).

A friend of mine had started working at Nitrado, and a position for a C developer opened up and he suggested I apply. Writing C code for Linux was far more interesting than JavaScript for WebOS and Tizen. 

After a few interviews, they explained how things worked and I thought well that seems like overkill, and then after actually looking at the graphs, you realize well yeah there really is a massive need for DDoS security.

It’s amazing how much I didn’t know and how much I know now.

Q: What does it feel like to have ownership over something as unique as SteelShield?

It’s great. Challenging, but great.

SteelShield is a fantastic innovation that’s often seen as a magical black box. What excites me is having the opportunity to take away the “magical black box” view and try to expose its features and inner workings to more people and customers so they can use it to its full potential.

It’s always interesting to find new and better ways to stop other people’s toxicity from entering, you know, the fun zone.

Q: Why do generic DDoS tools fail on UDP game traffic?

With TCP, you have a handshake process that takes place between the client and server, and you can intercept that to challenge the client to prove that it’s real. 

With UDP, this handshake doesn’t exist, so any UDP packet arriving over the line could be completely legitimate or from a botnet. There really is no way to tell.

The way this is handled by most DDoS protection is they just implement a rate limiter because they can’t validate the connection. This means you’re only protecting the network behind the rate limiter, and not the UDP traffic and the server itself. 

If your rate limiter is dropping half the traffic to keep the bandwidth entering your network below a threshold, you could be dropping just as much good traffic as you are bad traffic.

Q: Why was it important that Nitrado and GameFabric servers offer a solution like SteelShield?

Gamers are passionate people, and sometimes that passion bleeds over into anger and frustration, so they take it out on the game servers. And there’s no shortage of malicious actors online. People have found ways to exploit games by causing DDoS attacks.

These days, you can hire a botnet quite easily to attack a server. If you’re in the right channels, somebody will join and say, “Rent out this botnet for attacks. All you have to do is get your mom’s credit card.”

So the number of attacks, and the size of these attacks, have increased significantly over the years. We can see up to 100+ attacks in a single day, ranging from gigabits to terabits, lasting for hours. 

Without SteelShield, these attacks would have a direct impact on game servers, preventing players from actually being able to play and enjoy their games.

Q: Our 0% false positives claim, how did we achieve that outcome?

This is our guiding light and is always at the forefront of our minds in each line of code we write. We’ve never detected bad traffic getting past SteelShield in a properly configured system.

This is achieved by working with game developers to implement our Proof of Identity system, where the client is validated before attempting to connect to the server. 

SteelShield is then able to use Deep Packet Inspection to ensure that the network traffic it lets through has been validated as a real game client. We do this as close to line rate as we possibly can, ensuring there’s no performance or latency impact on the network.

Originally, what would happen is we would actually take a game and look at how the protocol worked and write mitigation specifically for that game. But because a developer can change how their game is programmed, a simple update could break it. 

That’s why one of the purposes of Proof of Identity is to simplify onboarding. We now just give our users a document and say “implement this specification” or “here’s the Unreal Engine plug-in that we’ve developed for you.” 

You just add it into your game and set up a few parameters and everything will work.

Q: What does a real attack look like from the inside, for us and the customer?

If your server is protected by SteelShield, a real attack looks like nothing ever happened :)

 If it isn’t protected by SteelShield, you’ll see various effects ranging from lag and rubber banding as the server is unable to receive game state updates from your client, to being kicked from the server or being unable to join at all. 

In rare cases, the game server could crash, erasing player progress or corrupting the game state.

When an attack happens against a server, we can see it in our dashboard, like high volumes of traffic hitting the routers and being processed by SteelShield, showing us exactly what traffic is being allowed or dropped, and why.

We can also look at the node itself and ensure that, yes, the traffic levels on the actual game server node hasn’t increased and that accepted traffic is remaining stable.

We always balance our network and our infrastructure to ensure that time isn’t a factor. From the SteelShield perspective, it doesn’t matter how long the attack goes on, we deal with it.

Q: For the studios that are unprotected, what’s the fallout after an attack?

Mostly angry and frustrated players. If one person is able to deny 49 other people from being able to play, you’re going to have 49 people who are considering switching to another game that isn’t always lagging.

Depending on how the game is written, a DDoS might allow a bad actor to take advantage of an exploit or they can get the clients and servers out of sync, giving them an unfair playing field. 

Sometimes you might need to roll back the game state to an earlier version, which can also result in unhappy players who lose their progress. A dedicated person who has set their mind to attacking a server can find ways to screw it up for everyone.

Q: What’s the most underrated game?

Journey on the PlayStation 3. I’m sad it was a PS3 exclusive at the time of launch, because by the time it made it to other platforms, it was already 7 years old and no one seemed as interested.

It was an incredibly immersive and emotional game with random encounters with other players who you could never directly communicate with, but you still relied on them to help you on your own journey. 

It was such a beautiful experience that, as soon as I finished the game, I made my then girlfriend (now wife), who was not a gamer, sit down and play it from beginning to end.

Ross Simpson is Team Lead SteelShield at Nitrado, working on game server security. Connect with him on LinkedIn.