Xbox Live Operations Center

This November will mark 12 years since the launch of Xbox Live, Microsoft’s online gaming service that currently boasts more than 48 million global users. Ensuring all of those Live members are constantly connected with each other and enjoying all of the service’s many features is paramount to delivering the Xbox experience gamers have come to expect. To pull it off, Microsoft has staffers working around the clock at its Xbox Live Operations Center (XOC) in Redmond, Washington, reports Total Xbox.

As part of their job function, XOC staffers respond to clear-cut problems. The majority of their time, however, is dedicated to ferreting out issues that no one knows about. Team members search for irregularities and investigate them to discover whether or not they are problems that need solving. They can’t guess at whether something is a threat to uninterrupted Xbox Live service or not; they have to know. They have to find out.

Microsoft Gaming Ninja (yes, that’s his real job title) Eric Neustadter informed Total Xbox that his team has to be ready for anything at all times. “It’s a bit like being in the fire department or police, where you have to think through what would I do in [a worst case] scenario. My boss was in the fire department for years, and he went from doing that to running services like this. In a lot of ways, it is a similar mindset.”

Redundancy is Neustadter’s go-to tool for preventing an Xbox Live doomsday scenario. “We have all the things you need to run 24/7/365, like multiple power grids connected into the campus,” he explained. “When that fails, we have diesel generators with huge tanks out there so that we can run for roughly 72 hours without power, and then we can bring in tankers with more diesel and refill the tanks while they’re running. We also have a disaster recovery site… somewhere else on a different power grid with almost all of this replicated so we can go over there and work instead.”

The 2007 holiday season was the closest the XOC ever came to facing down the worst event possible. It was during that time that both Halo 3 and Call of Duty: Modern Warfare launched. Neustadter and the rest of the center’s staffers were pushed to the brink as Xbox Live grew at an astounding rate. Several weeks during that time period became so overwhelming that some employees lost sleep as they worked even harder than normal.

But the team made it through. Xbox Live remained up and running, which meant gamers were free to frag and bag in their shooter of choice.

Today, though, Xbox Live is a service rich with features well beyond blockbuster first-person shooter matchmaking. It’s a multimedia entertainment platform, which means there are a plethora of different types of potential threats Microsoft must be prepared to handle.

“So things like the season premiere of Game of Thrones – there’s a noticeable change in our customers’ behavior when that happens,” Neustadter stated. “We keep a calendar of those kinds of events. If you didn’t know they were coming and your job was to monitor the Xbox Live service, you’d freak out because all of a sudden you’d see this drastic change in user behavior up or down and you wouldn’t know why and you’d spend a lot of time chasing your tail.”

Xbox Live’s user base can’t wait patiently for winter to arrive, and neither can Xbox Live Operations Center employees; they have to be ready for it before it comes. Luckily for Live subscribers, they are.

Source: Total Xbox