Since the launch of Diablo II: Resurrected, we have been experiencing multiple server issues, and we wanted to provide some transparency around what is causing these issues and the steps we have taken so far to address them. We also want to give you some insight into how we’re moving forward.
tl;dr: Our server outages have not been caused by a singular issue; we are solving each problem as they arise, with both mitigating solves and longer-term architectural changes. A small number of players have experienced character progression loss–moving forward, any loss due to a server crash should be limited to several minutes. This is not a complete solve to us, and we are continuing to work on this issue. Our team, with the help of others at Blizzard, are working to bring the game experience to a place that feels good for everyone.
We’re going to get a little bit into the weeds here with some engineering specifics, but we hope that overall this helps you understand why these outages have been occurring and what we’ve been doing to address each instance, as well as how we’re investigating the overall root cause. Let’s start at the beginning.
The problem(s) with the servers:
Before we talk about the problems, we’ll briefly give you some context as to how our server databases work. First, there’s our global database, which exists as the single source of truth for all your character information and progress. As you can imagine, that’s a big task for one database, and wouldn’t cope on its own. So to alleviate load and latency on our global database, each region–NA, EU, and Asia–has individual databases that also store your character’s information and progress, and your region’s database will periodically write to the global one. Most of your in-game actions are performed against this regional database because it’s faster, and your character is “locked” there to maintain the individual character record integrity. The global database also has a back-up in case the main fails.
With that in mind, to explain what’s been going on, we’ll be focusing on the downtimes experienced between Saturday October 9 to now.
On Saturday morning Pacific time, we suffered a global outage due to a sudden, significant surge in traffic. This was a new threshold that our servers had not experienced at all, not even at launch. This was exacerbated by an update we had rolled out the previous day intended to enhance performance around game creation–these two factors combined overloaded our global database, causing it to time out. We decided to roll back that Friday update we’d previously deployed, hoping that would ease the load on the servers leading into Sunday while also giving us the space to investigate deeper into the root cause.
On Sunday, though, it became clear what we’d done on Saturday wasn’t enough–we saw an even higher increase in traffic, causing us to hit another outage. Our game servers were observing the disconnect from the database and immediately attempted to reconnect, repeatedly, which meant the database never had time to catch up on the work we had completed because it was too busy handling a continuous stream of connection attempts by game servers. During this time, we also saw we could make configuration improvements to our database event logging, which is necessary to restore a healthy state in case of database failure, so we completed those, and undertook further root cause analysis.
The double-edged sword of Sunday’s outage was that because of what we’d dealt with on Saturday, we had created what was essentially a playbook on how to recover from it quickly. Which was good.
But because we came online again so quickly in a peak window of player activity, with hundreds of thousands of games within tens of minutes, we fell over again. Which was bad.
So we had many fixes to deploy, including configuration and code improvements, which we deployed onto the backup global database. This leads us into Monday, October 11, when we made the switch between the global databases. This led to another outage, when our backup database was erroneously continuing to run its backup process, meaning that it spent most of its time trying to copy from the other database when it should’ve been servicing requests from servers. During this time, we discovered further issues, and we made further improvements–we found a since-deprecated-but-taxing query we could eliminate entirely from the database, we optimized eligibility checks for players when they join a game, further alleviating the load, and we have further performance improvements in testing as we speak. We also believe we fixed the database-reconnect storms we were seeing, because we didn’t see it occur on Tuesday.
Then Tuesday, we hit another concurrent player high, with a few hundreds of thousands of players in one region alone. This made us hit another incident of degraded database performance, the cause of which is currently being worked on by our database engineers. We also reached out to other engineers around Blizzard to work on smaller fixes as our own team focused on core server issues, and we reached out to our third-party partners for assistance as well.
Why this is happening:
In staying true to the original game, we kept a lot of legacy code. However, one legacy service in particular is struggling to keep up with modern player behavior.
This service, with some upgrades from the original, handles critical pieces of game functionality, namely game creation/joining, updating/reading/filtering game lists, verifying game server health, and reading characters from the database to ensure your character can participate in whatever it is you’re filtering for. Importantly, this service is a singleton, which means we can only run one instance of it in order to ensure all players are seeing the most up-to-date and correct game list at all times. We did optimize this service in many ways to conform to more modern technology, but as we previously mentioned, a lot of our issues stem from game creation.
We mention “modern player behavior” because it’s an interesting point to think about. In 2001, there wasn’t nearly as much content on the internet around how to play Diablo II “correctly” (Baal runs for XP, Pindleskin/Ancient Sewers/etc for magic find, etc). Today, however, a new player can look up any number of amazing content creators who can teach them how to play the game in different ways, many of them including lots of database load in the form of creating, loading, and destroying games in quick succession. Though we did foresee this–with players making fresh characters on fresh servers, working hard to get their magic-finding items–we vastly underestimated the scope we derived from beta testing.
Additionally, overall, we were saving too often to the global database: There is no need to do this as often as we were. We should really be saving you to the regional database, and only saving you to the global database when we need to unlock you–this is one of the mitigations we have put in place. Right now we are writing code to change how we do this entirely, so we will almost never be saving to the global database, which will significantly reduce the load on that server, but that is an architecture redesign which will take some time to build, test, then implement.
A note about progress loss:
The progress loss some players have experienced is due to the way we do character locks both in the regional and global databases–we lock your character in the global database when you are assigned to a region (for example, when you play in the US region, your character is locked to the US region, and most actions are resolved in the US region’s database.)
The problem was that during a server outage, when the database was falling over, a number of characters were becoming stuck in the regional database, and we had no way of moving them over to the global database. At that time, we believed we had two options: we either unlock everyone with unsaved changes in the global database, therefore losing some progress due to an overwrite that would occur in the global database, or we bring the game down entirely for an indeterminate amount of time and run a script to write the regional data to the global database.
At the time, we acted on the former: we felt it was more important to keep the game up so people could play, rather than take the game down for a long period of time to restore the data. We are deeply sorry to any players who lost important progress or valuable items. As players ourselves, we know the sting of a rollback, and feel it deeply.
Moving forward, we believe we have a way to restore characters that doesn’t lead to any significant data loss–it should be limited to several minutes of loss, if any, in the event of a server crash.
This is better, but still not good enough in our eyes.
What we are doing about it:
Rate limiting: We are limiting the number of operations to the database around creating and joining games, and we know this is being felt by a lot of you. For example, for those of you doing Pindleskin runs, you’ll be in and out of a game and creating a new one within 20 seconds. In this case, you will be rate limited at a point. When this occurs, the error message will say there is an issue communicating with game servers: this is not an indicator that game servers are down in this particular instance, it just means you have been rate limited to reduce load temporarily on the database, in the interest of keeping the game running. We can assure you this is just mitigation for now–we do not see this as a long-term fix.
Login Queue Creation: This past weekend was a series of problems, not the same problem over and over again. Due to a revitalized playerbase, the addition of multiple platforms, and other problems associated with scaling, we may continue to run into small problems. To diagnose and address them swiftly, we need to make sure the “herding”–large numbers of players logging in simultaneously–stops. To address this, we have people working on a login queue, much like you may have experienced in World of Warcraft. This will keep the population at the safe level we have at the time, so we can monitor where the system is straining and address it before it brings the game down completely. Each time we fix a strain, we’ll be able to increase the population caps. This login queue has already been partially implemented on the backend (right now, it looks like a failed authentication in the client) and should be fully deployed in the coming days on PC, with console to follow after.
Breaking out critical pieces of functionality into smaller services: This work is both partially in progress for things we can tackle in less than a day (some have been completed already this week) and also planned for larger projects, like new microservices (for example, a GameList service that is only responsible for providing the game list to players). Once critical functionality has been broken down, we can look into scaling up our game management services, which will reduce the amount of load.
We have people working incredibly hard to manage incidents in real-time, diagnosing issues, and implementing fixes–not just on the D2R team, but across Blizzard. This game means so much to all of us. A lot of us on the team are lifelong D2 players–we played during its initial launch back in 2001, some are part of the modding community, and so on. We can assure you that we will keep working until the game experience feels good to us not only as developers, but as players and members of the community ourselves.
Please continue to submit your feedback to the Diablo II: Resurrected forum 117, report your bugs to our Bug Report forum 37, and for troubleshooting assistance, visit our Technical Support forum 20. Thank you for your ongoing communication with us across all channels–it’s invaluable to us as we work on these issues.
The Diablo community team will keep you updated on our progress via the forums.
A new patch is now available for Xbox consoles and PlayStation 4 consoles. PC, Nintendo Switch and PlayStation 5 consoles will follow shortly and we will update this thread when they become available on those platforms.
Improvements to character deletion to help users from accidentally deleting their characters.
STABILITY AND PERFORMANCE
Fixed a crash that could occur when using the legacy toggle after extended play sessions
Fixed a crash that could occur when the Korean IGR playtime warning UI would appear (PC Only)
Fixed a crash that could occur after launching the game with no existing settings file (PC Only)
Fixed a crash that could occur when binding passive abilities to the interact button
Stability improvements in the effects system
Stability improvements on game shutdown
Fixes to localizations and subtitles
Other minor crash/stability fixes
Wyatt Cheng Twitter Thread Regarding Diablo III & The Auction House
Wyatt posted an insightful thread on Twitter that debunks the old thought of Diablo III originally being tuned the use of the Auction House.
Gonna set the record straight on a Diablo 3 tidbit. This won't be popular but I feel with D2R on everybody's mind, now is a more relevant time than others to bring this up. With the design lessons I learned at the end /thread.
Hello, and welcome once again to a new Diablo IV Quarterly Update.
I'm Joe Shely from the Diablo IV team. As a design lead who has been working on this dark, shared, open world action role-playing game from the beginning, I’m honored to continue the vision of Diablo IV as its new Game Director, and I’m humbled to represent the team pouring their hearts into this game.
Like many of you, our team has been reflecting upon recent events. A lot has happened since our last blog and the hard work of practicing the values we aspire to must continue. In parallel with that important work, development of Diablo IV continues too.
Over the past few years, we've assembled a strong team with incredible passion for Diablo IV. You, Diablo’s fans, are a critical part of this team. With the help of your valuable feedback, we've steadily refined and deepened the game experience. We have ways to go, and while much has changed, our commitment to the game is unwavering.
Sanctuary should always be items glittering in dark dungeons. Tales of powerful heroes standing against the onslaught of hell. Lands where trials, treasure, and terrible monsters lie around every corner, equal parts familiar and boundless in its possibilities. Doing this world justice is a solemn responsibility. Today, we're taking a deep dive into the sound design of Diablo IV. Sound is a sometimes underappreciated yet integral element of the game's design, serving as a channel for communicating everything from incoming damage, to confirmation that a button press was registered by the game, to the intensity of a combat sequence. Try turning off the sound in a Diablo game sometime; you'll find your eyes have to work a lot harder to follow the action.
Sound also conveys the subtext of the world through which you adventure. It supports the rising and falling action of the campaign and immerses you into the ambient life of a region, transporting you into the game world. While you're reading through the blog, I encourage you to listen to the ambient tracks and pay attention to their effect on your heartrate and emotions. Try closing your eyes while listening to get a better sense of how much is being communicated.
Crafting Diablo's sound requires science, art... and the occasional ball of fire. Sound Supervisor Kris Giampa and his team have fascinating insights to share in this behind-the-scenes look at how it all works.
We hope you enjoy this update and look forward to your thoughts and reactions. As promised, our next blog will cover endgame systems and visual effects. It's been a while since we looked at systems, and there's a lot we're excited to share. Let us know what topics you'd like to hear about in the future too!
We are grateful to have you with us on this journey. Thank you for playing the stuff we make.
-Joe Shely, Game Director, Diablo IV
The Sound Design of Diablo IV
Hello traveler, I’m Kris Giampa, Sound Supervisor for Diablo IV.
The sound team has been steadily cranking away on the soundscape for a while now, and while we aren’t ready to go in depth about the music side of Diablo IV quite yet, we wanted to start to give you some insight into some of the audio processes, content, and motivations behind the sound for the game.
Before we begin, we wanted to give you something to listen to while you are reading this quarterly update blog. Please enjoy the snowy, dark, and stormy ambience of Fractured Peaks below as you start on your journey.
Sound and music in games are the invisible glue that supports the storytelling and ties you to your character and their actions during gameplay. Creating sound for games is an exciting artistic endeavor that you can’t see— only hear. However, you can feel it literally, with the soundwaves against your body depending on what you are listening back on. It’s an amazing medium that can also affect how you feel emotionally while playing a game. A lot of the time it’s subtle, and other times it’s over the top, but always there to support the moment-to-moment gameplay. We hope you enjoy this dive into various aspects of the game’s soundscape, and you’ll have plenty more to look forward to and experience when you finally get to play it!
Naturally, we want anyone and everyone who might be hearing impaired to enjoy the experience of Diablo IV as well. So, we are taking measures to broaden the experience to be inclusive for people with hearing or visual disabilities. There are various accessibility features underway that we hope to talk about more in the future.
The Devil is in the details
For the Diablo IV soundscape we’ve continued the tradition of gratifying combat, expanded upon the ambience to support the epic open world, continued to embrace the darkness in tone and gore while also trying to deliver a cleaner-yet-punchy audio mix that’s adapting to the situations as you play.
One of the biggest goals we try to focus on as sound designers is to make the highest quality sounds to be triggered back in the game real-time and make them seem believable as well as grounded within the game world, tied to what you are experiencing. The randomness of audio playback is of the utmost importance when it comes to gameplay. If you think about real life, nothing is ever heard the exact same way twice due to your listening environment and the positioning of a sound source. Sounds never play back at the same exact sound pressure level along with the reflections within your environment and everything else happening around you at that moment in time. In essence, there are always subtle, real-life reasons as to why nothing ever sounds exactly the same. So, as sound designers for video games, we always strive to introduce subtle, randomized variation to not only the sound design itself, but also for when you hear it in the game. When we are doing our job correctly, it’s something you tend to not notice and supports your immersion into the game world by backing up the incredible visuals, story, and end-to-end experience.
Another huge goal we have when creating audio for Diablo IV is to fill in sounds for just about everything that’s occurring on your screen. Whether it’s the world ambience, monsters idling offscreen, the tiny chunks of wood that are colliding off a wall when you break an object... everything should make a sound. We pour countless hours of effort into covering almost everything you see and don’t see, while also keeping it subtle enough to not be distracting and just feel right. The devil certainly lurks within the details...
However, just because we are filling in as much sound as we can, it doesn’t mean you need to hear it every time. The playback engine within the game will not trigger too many instances of a sound if they are trying to play at the same time based on strict settings we create as basic rules. Because of the isometric camera view and being able to see so much on screen at once, we must limit how many instances of each sound will playback at any given moment. Once dialed in correctly, you tend not to notice that some instances were never triggered, and that helps with the clarity of the audio mix. It’s a fine line we straddle during big moments with a lot going on screen.
How about we get on to more of the creative side of sound design? Naturally, there is no Diablo game without the heroes who do your bidding to vanquish the various evils that lie in your path. Let’s talk about some of the fire-based Sorcerer skills that partly define the class...
Hero Fire Skills
The sound crew luckily gets to record all kinds of neat and unique sounds for the game so that we have plenty of sound source to edit from when it comes time to start sound designing. Sound Design is technically described as taking a recorded audio source, editing it, and processing it to be used in another medium. In the case of Sound Design for games, we’ll record raw audio, re-process them, and edit it for our gameplay needs in various ways to achieve something that sounds clean, usable, and replayable for gameplay. The sound could be exactly what it was originally meant for or end up sounding completely different and used for something else entirely.
Something we always tend to need for a game like Diablo is, of course, fire! When time permits, we plan for some time to record sounds out in the field. For Diablo IV, one of the first big recordings we did as a team was a desert fire session before the lockdown for COVID-19. We traveled far from Blizzard HQ to record various types of fire sounds in the deserts of California, armed with multiple recording rigs and microphones. Thankfully, since it was wintertime, it wasn’t too hot during the day and just a bit cold at night. While our main goal was to capture fire, we ended up capturing all kinds of other sounds we have used during production, like ambience, rocks impacts, foliage movements, wood impacts, door slams, wooden cabin creaks, metal impacts, and scrapes.
Sorcerer Skills
Firebolt and Inferno
Some of the fire recordings were then used specifically for the Sorcerer skills Firebolt and Inferno. For the skill Firebolt, we recorded sets of wispy and smoldering flame whooshes using a fire staff or a dried-out medium-sized log of wood and performed the sound in various ways around sets of microphones. Once we had a nice assortment of different types of fire sounds, we then edited and processed those fire whooshes into game-ready one-shot audio files for the casting and impact sounds, as well as longer loops for the projectile traveling through the air. It all comes together as one cohesive-sounding experience once we get it hooked up in the game to play back as the entire skill sound effect set for Firebolt.
For the Inferno Sorcerer skill, we then used other takes of the fire recordings and processed them to sound more aggressive and powerful for the larger skill. Just like Firebolt, there’s a set of casting one-shot sounds, a set of loops, and another set of one-shots for when the snake form constricts its body. One cool thing about the Inferno skill is that while it’s fire, it’s fire that has taken the form of a snake. Because of this form, we are able to take some liberties on pushing it away from just fire sound design. We added some light snake-rattling SFX paired with a darker-toned ethereal end to the skill sound to make it feel a bit more magical. When all these pieces re-trigger in-game, it will always sound like the same skill, but be slightly different each time—which increases the replayability sound-wise.
Monster Sound Design
Diablo games wouldn't be as fun if you didn't have monsters to slay. One of the most fun things about working on a Diablo game is the vast amount and variance of monsters that exist. This makes monsters ripe for both experimental and more traditional sound design, so let's dig into some monster sound design for foley and voice.
Monster Movements
The combination of expert animation and AI brings life and personality to the creatures as they undertake their nefarious activities. When we start the audio process for a brand-new monster, I always recommend that the sound designers start by adding footsteps and foley (clothing or skin) to their movement animations. The moment the creature has footsteps and foley, the creature's cadence and rhythm of their movement really comes to life. It’s at this point I consider that they are becoming grounded and attached to the world. This also dictates how vocal they might sound based on their patterns of movement.
Monster Voice
The next layer that finishes the birthing of the creature into existence is their voice exertions. These are the grunting or yelling sounds of them exerting at the player as they attack, or the screams of pain as you take them out one-by-one. Each monster family can be quite different from the next, so depending on the type of monster, we might have intense sound design layers of animalistic-type sounds or even everyday objects that we will manipulate to sound like a screech or scream to create a layer within that final voice. Other times it can be simpler, as we’ll hire creature voice actors to help create the core tone of the monster’s voice that we can then build around with other sounds.
In the case of the Wood Wraith, it's almost fully sound designed from wood creaks and strains processed to extreme lengths and choosing the right sounds to convey emotion. The Wood Wraith was a blast to sound design as it’s mostly freaky and creaky wood sounds, with a touch of very low-pitched human tone underneath it all.
Another monster we had the pleasure of working on is the disgustingly awesome Fly Host. This beast walks around birthing flies to attack the player. We ended up using some of our early gore session recordings where we ripped and smacked cabbages and melons, and stirred and squished mayonnaise, salsa and a delicious 7-layer dip into a not-so-great smelling slurry to make some great slimy and disgusting sounds to use in our sound design.
Open World Ambience
One of the audio pillars for Diablo IV is “Living Audio”. What this means is that the soundscape is ever evolving and never static. This pillar is built deep into the sound design variations we create for all types sounds, including when it plays back real-time in the game— especially the ambience. Because of the importance of the massive open world, we wanted to give the ambience as much detail as possible and think of it on the same level of Hero sound design. Having the audio and the systems changing subtly over time is key to this pillar. We always want the subtle changes in ambience (that might not be very noticeable) less repeatable and feel more natural and immersive overall.
The World Building team has done an amazing job giving us huge amounts of inspiration in filling out the regions visually so that we can follow it up with immersive ambient audio.
Since the player might be in the open world for a large amount of time, we wanted to support each exterior region with unique-sounding environments that also include subtle changes to the audio mix over time. To help achieve this, we use audio systems like real-time occlusion, high-quality reverbs, and environment reactive delay/echoes.
We’ve provided some long-form recordings of in-game footage with a static shot where you can hear the ambience changing over time. Not only does this show some cool ambient sound design, but we also wanted to provide these to you for your tabletop RPG sessions, or even just to sit back and get lost in while working. The clips were recorded around 5-6 minutes and looped to be almost one hour.
Dungeon Ambience
When it comes to the Diablo dungeon crawling-style ambience, we take a special delight in creating various and unique sounding experiences to heighten your immersion. Our approach to the dungeon ambiences is a bit less intense compared to the new open world, as we don’t want to distract you away from a key part of what makes a Diablo game fun— dungeon crawling. This is one area where we can take more liberties in diving deeply into the hellish and creepy soundscapes while having the monsters onscreen to accompany the audible experience. For Diablo IV we are taking a more realistic approach to “what you hear is what you get” when within a dungeon. With long reverberation and sound occlusion, we want you to pay close attention to what might be just around the corner, mentally preparing you for the next pack of enemies.
Breakable Interactives
Scattered around the dungeons are a plethora of gratifyingly great breakables. The Interactives team have been creating hundreds of amazingly detailed breakable objects in Diablo IV. For the amazing amount of detail they put into objects as they break, we in turn wanted to fill in every sliver and chunk of destruction you see with believable physics audio. Destroying objects in Diablo should sound just as gratifying and believable as taking down monsters. We put a lot of effort into making sure that all objects have an extremely gratifying break sound, while supporting the debris with tiny bits of audio to accompany the pieces that break apart and fly across a room. I’m still amazed at the level of detail we have for the breakables in Diablo IV. One of my favorite things to do when I see a room full of them is to have at em’!
Game Mix
Finally, I wanted to talk a little about the isometric camera. It provides some interesting challenges when it comes to bringing all the elements of the game mix together. Because you can see battlefield at a certain angle out to a certain range, we have to make sure that the monsters existing on the screen are covered with audio, but have the overall mix not feel too cluttered, nor too empty. There’s a lot of real-time juggling of sound playback based on priorities and importance to you, the player.
For Diablo IV we are able drive the real-time audio mix more than ever before. Because of the isometric camera view, we must trigger sounds on just about everything you see but focus your ears on the most important sounds you should be paying attention to. We've been carving away at audio mix states and an audio importance system that will allow certain important monster sounds to poke out when they are needed. Clarity of game audio mix is hard to achieve in a game where you can have multiple heroes as well as various amounts of monsters on screen, while having detailed ambiences means we need to craft different audible mix states depending on the situation.
We hope you have enjoyed this brief look into the sound design of Diablo IV. There’s so much more to talk about, but alas, we will have to save it for another time. We welcome any feedback you might have about anything you have heard in the videos or learned about in this quarter’s blog. Thank you for taking the time to learn more about the soundscape of Diablo IV!
We wanted to provide an update on the Advanced Vector Extension (AVX) items from the dev team after some further developments from this past weekend. The team believes they have a potential fix, but this goes beyond just AVX specifically. The fix could potentially affect all users, even those outside of not having AVX support, so we are wanting to make sure we do proper testing.
While CPUs without AVX are below our minimum spec, we want to try to ensure as many people as possible can play. Setups without AVX were working in Beta. In optimizing the game, we inadvertently included the need for AVX for launch. QA is going to spend another few days of testing across all of these scenarios and setups in an effort to ensure we are not impacting existing players.
We will update this thread once we have an update from the testing over the next few days.
We apologize for the added delay and appreciate your patience.
For those that may have already purchased but would prefer to purchase post fix, we do recommend reaching out to Customer Service for a refund.
