Inside the Stack: ARPGs and Spinning Up the Backend Behind Last Epoch

Welcome to Inside the Stack, where we talk through the real decisions, tradeoffs, and lessons behind running multiplayer games.

Today, we’re excited to be speaking with a long-time GameFabric studio partner, Eleventh Hour Games (EHG). We sat down with their Principal DevOps Engineer, Brandon Nelson, to discuss the engineering mindset behind EHG’s smash-hit ARPG, Last Epoch, how they manage seasonal Cycles, and where GameFabric fits into their daily development workflow.

Q: What’s a game you keep coming back to, even years later?

I have played WoW for over 20 years, and many of my tattoos are themed around WoW.  It will forever hold a special place in my heart. I started playing while I was in the military and it allowed me to connect with people. I’ve also worked on a WoW addon for almost 14 years now, which has been a huge boon for keeping my engineering skills sharp!

(Editor’s note: You can check out the addon, TradeSkillMaster.)

Q: Was there a moment that made you realize you wanted to make games for a living?

It’s always been a dream, but very few game companies use .NET for the backend.  I stumbled quite accidentally into the open role at EHG, left a brand-new job to come here, and the rest is history! Best decision ever.

Q: The hallmark of ARPGs is thrilling itemization and dungeon crawling. What pulled you toward the infrastructure side of making that happen, rather than, say, combat or level design?

Ever since I was a little kid, I have been obsessed with how things work. I would take a screwdriver from my dad’s tool pouch and take screws out of everything I could find before I was even 4 years old. That naturally led to a love of the “nuts and bolts” of things on the backend of software and games!

Q: Is there an online game, not necessarily an ARPG, where the way it handles multiplayer still impresses you?

Any MMO (WoW, FF14, etc.) legitimately blows my mind. I would love to be able to sit down and have dinner with some people on their infra teams.

Q: Last Epoch's crafting system, building the item you want rather than hoping for the right drop, is one of the things that sets it apart in the ARPG space. What's the story behind how that came together?

That was all largely in place before my tenure at EHG! I will agree, though, I love our crafting system. There have been a few changes to it since my time at EHG, such as Corruption or guaranteeing 1 affix when crafting Legendaries. Both have been a lot of fun!

Q: Each new Cycle (season) effectively resets the game world for everyone at once. What's that moment like for your infrastructure, in the lead-up to a season launch?

The flip-over is as simple as a configuration update. We’ve finely honed the process over the years and it's quicker than ever now! There are some things that take longer, like migration of economic data, but that is just a factor of time.

Q: Is there a challenge that's specific to building a session-based ARPG like Last Epoch, something a persistent open-world game wouldn't run into?

Not my particular area of expertise, but I would imagine constantly designing the next X number of seasons, and keeping all of the in-progress pipelines straight, is a huge task vs. a persistent open-world game.

Q: When you're testing new content, classes, items, builds, ahead of a season, what does that process look like, and is there anything about GameFabric's setup that makes iterating easier?

It’s honestly everything you can imagine, from automatic to hands-on testing of various specific chunks of content. GameFabric has been very helpful due to the ease of spinning up new Armadas for new temporary nightly builds of long-lived feature sets. This ensures our Dev and Production teams always have a relatively up-to-date snapshot of work in progress.

Q: From a day-to-day development standpoint, is there something about working with GameFabric, an environment, a workflow, a feature, that's made things easier for your team?

The ever-improving UI/UX of GameFabric has been amazing. The few complaints we’ve had over the years are usually addressed very rapidly. The team is fantastic as well!

Q: When you look at what you and your team have built, the thing players will never see, never know exists, and never thank you for, what are you most proud of?

I’ve made an internal site that shows all kinds of data and analytics about builds, popularity of skills and items, the economy, and so on. It's been a huge help to all manner of teams at EHG. Players will likely never know of its full existence, but I am extremely proud of it!

Brandon Nelson is the Principal DevOps Engineer at Eleventh Hour Games. Connect with him on LinkedIn.

Eleventh Hour Games is the independent studio behind Last Epoch. GameFabric by Nitrado provides the server orchestration, scaling, and infrastructure tools powering their multiplayer environment.