How to avoid network and server infrastructure issues during the development cycle

Mathieu Duperré, CEO, Edgegap

Regardless of whether you’re an indie developer or a AAA publisher, making games is never simple. Taking a game project from the initial concept to launch is a huge undertaking, especially when you begin incorporating multiplayer and social elements. Despite eye-watering budgets and huge teams of developers, countless games fall flat on release.

Thankfully, many of the common obstacles can be easily avoided and disruption to your release timeline kept to a minimum if you’re aware of the potential network and server infrastructure issues at key stages of development: the main development cycle, pre-launch, the launch period itself and once the game has launched.

Problems during the main development cycle

Some of the most common problems you’ll encounter on the tech side of the main development cycle can be the result of:

• Imperfect or hurried CI/CD pipelines. It can take days to push a new release for testing when you should be able to launch a few releases every day.
• Dev teams taking on too much in-house rather than seeking outside assistance (netcode, game engine, backend services).
• Trying to save funds through open-source projects that eventually become outdated, unsupported during your dev cycle or ends up in engineering money pits.
• Developers waiting until post-launch to focus on important elements such as crossplay.
• QA teams lacking constant access to the game build.

Any of these sound familiar? If so, it might be time to rethink your core strategy and whether you’re adhering to the DevOps methodology. Do you have a structured approach or are you lurching leftfield and drifting off course?

While developing your own netcode and game engine in-house can seem appealing, it’s important to consider the potential risks that might emerge further down the road. If problems with your netcode or a proprietary game engine present themselves at a particularly busy stage of development, you’ll wish you’d have outsourced them.

Considering where you can offload certain aspects of development to external partners can help reduce your team’s workload and improve their efficiency so they can focus on more pressing matters.

Mistakes to avoid just before the launch

One of the biggest mistakes you can make just before launch? Underestimating your CCU (concurrent users worldwide) and employing a low percentage buffer of around 10%. Misjudging your CCU is going to put too much strain on your servers and overall backend services and is one of the most common reasons players experience crashes around the launch. Similarly, don’t underestimate the number of regions you’ll need servers in too. A global game launch requires an equal distribution of worldwide servers to avoid latency issues.

When it comes to servers, it’s always best to plan ahead and scale back, rather than the other way around. Prioritise network partners who offer pay-as-you-go options for server access, so if you do need to scale back (or up) you can do so easily. This should always be the preferred option and will prevent you from being charged for servers you’re not using. It’s important to be aware of the risks associated with overestimating, as this can lead to studios signing long-term contracts and ultimately being overresourced.

Finally, one issue that can cause developers and studios a lot of trouble is data protection regulation laws. I’ve heard plenty of horror stories about studios turning a blind eye to privacy laws such as the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA). Make sure you’ve checked the specific rules regarding data collection in your region as early as possible to avoid a frantic rush of paperwork towards the end of development or, worse, incurring costly fines.

Obstacles on launch day

It can be easy to think that you’ve made it through the worst once you reach launch day, but this is usually the stage where major issues can suddenly appear. You may notice a bombardment of negative responses from Twitter, Steam and other platforms addressing critical issues with the game – make sure you address these rather than ignoring them. These responses can be important indicators of how the players are responding to your game and if you’ve factored their feedback in the development phase.

Once the game has launched, you will have the clearest indicator of how many servers you’ll need to support the players. This is where your planning will prove to be correct or wide of the mark if you’d reserved say 1000, but you only need 200.

A 24/7 support plan is an important consideration when the game is live, this will enable rapid fixes to negate as many disruptions as possible. A low-cost DDoS solution is also a crucial facet to bear in mind. A server orchestrator will help developers manage influx of traffic along with capturing crucial data that can be incorporated into the game as part of the observability phase.

Post-launch issues

Now that your game is in the world, there can be a strong temptation to introduce a plethora of new elements to the game to enhance the player experience. While on the face of it these are good ideas, new mods or minor updates can prove problematic and contain major bugs. Responding to these new issues can be distracting and take weeks or even months to solve.

On top of this, if you haven’t properly planned earlier in the process, major updates to your game servers could potentially lead to maintenance issues and significant downtime for your player base. If players have a bad experience on launch, they’ll likely just move on to the next best thing. When EA launched Battlefield 2042, server issues, loading times and game-breaking bugs plagued the game. Within months, player numbers dropped to fewer than 1,000 concurrent players on Steam.

Of course, there are a number of ways to circumvent these issues. It is crucial to choose a backend that can support multi-versioning A/B testing and facilitate rolling updates without outages. You’d be hard-pressed to find a gamer who enjoys an unplayable game. Automation can also help in the production pipeline, this includes deploying quick fix updates and upgrades which will reduce the human error factor.

Another consideration that is often overlooked is outsourcing for network and platform providers to remove a large in-house team dedicated to DevOps or Engineering. This is intensive work that will take up the time of staff, and outsourcing is a viable option and it makes perfect sense for smaller studios, especially to pursue this option purely from a cost-saving perspective.

Making edge count

Edge computing is often overlooked by game developers, but it can alleviate many of the issues described here, particularly around server provisioning and latency issues. Edge technology brings computation and data storage closer to the source of the data, placing it at the edge of the network where performance gain is most optimal. This reduces unnecessary travel and drastically speeds up the process resulting in close to a lag-free experience.

If you’re looking at cross-play for your game, infrastructure requirements become more complex. Edge computing helps to negate some of the most nagging delay issues that come with playing across different platforms. As cross-play becomes more of a consideration for developers, edge computing along with automation can come to be one of the solutions to some of the associated problems. Developers should consider the automation part of the equation when planning their game project. It becomes even more vital when you add more latency-reliant elements to a game.

Fail to plan, plan to fail

There is no one size fits all approach to game development or the challenges associated with it. But taking some of the steps outlined above can help negate potential issues and lessen their impact, preventing total derailment of a game before it even launches.

Small steps like putting your trust in external partners or looking at technologies like Edge computing to squeeze as much performance out of your game as possible might seem small, but their impact on development can’t be underestimated.

Planning for different eventualities at key stages of development will go a long way in ensuring your game is the best it can possibly be at launch. This will give your game the best chance at becoming a lasting hit, not a flash in the pan that’s marred by server issues or bugs.

The logistical side of creating a game might not always be glamorous or exciting, but it’s unavoidable. If your development pipeline is hit by server issues, you’ll be glad you planned ahead.

Related