Red Ring of Death: Xbox pathologist explains the plague

Mere weeks after the launch of the hotly anticipated Xbox 360 in 2005, rumor spread like wildfire of a dreaded video-game Grim Reaper. We didn’t know why, how or when he would strike — all we knew was that he manifested himself with three red lights, cursing 360 consoles with a permanent GAME OVER. Microsoft ensured us that it was a minor problem that happened only to a select few, but time eventually revealed the ugly truth: everyone was at risk. After class-action lawsuits and bitter word of mouth, Microsoft finally put a three-year warranty in place, guaranteeing that all systems affected by the red lights would be fixed at no charge.

Chances are you have your own story about the Three Red Lights of Death. Perhaps your console was under warranty and returned after a few weeks. Maybe you bought a new one to skip the wait. You may have even cracked it open and somehow fixed the problem yourself. For me, my first system was killed by Overlord, one month before the launch of Halo 3. Four years later, my second system croaked after the first level of Gears of War 3, at which point in time the warranty had expired.

The Wizard of X

A replacement Xbox 360 Slim would have set me back at least $300, so I was somewhat relieved when my local GameStop referred me to David Peyregne, owner of Computers for Less. Peyregne is an experienced technician who has run his own business fixing computers and video game systems for over a decade. A former journalism student who turned to computer science, Peyregne sometimes lets his southern drawl come through his hollow voice that was scarred from polyps at a young age. With a husky explanation, he handed my system back to me, good as new, for $100, a sum much less daunting than the cost of a new console. Recently, I sat down with Peyregne to get the whole story on the Red Ring of Death: what causes it and how does Peyregne fix it? As it turns out, it took him a long time to figure it all out.

“I went through all of the different types of repair scenarios. There’s so much misinformation on the web,” Peyregne reflected. But when these problems began, rumor and hearsay was all there was. So Peyregne combined most of the popular repair techniques he came across into something he calls the “hybrid fix.”

“A lot of times, if it’s just overheating, it’s the thermal paste because they don’t use a good quality thermal paste between the CPU, GPU and the heat sinks, and it dries up and gets crusty and then it blocks the heat transfer,” he explained. “A lot of times you could just replace that [thermal paste], and it would work again! But, that wasn’t really the root of the problem.”

Peyregne would also replace the X-clamps that were supposedly getting loose and breaking the flow of heat from the GPU to the heat sink, causing the Xbox to overheat. “Now I know that this is not the case,” Peyregne reflected. Continuing, the Xbox repairman said that the “X-clamp is not a problem. Now whenever I do a repair, I put the original factory X-clamps back on. That’s not the problem.”

Eventually, these fixes culminated in what technicians call a re-flow. “We all knew that it was the chip breaking because of the lead-free solder,” said Peyregne. “The re-flow is when we take your motherboard, and we heat up the entire motherboard specifically around the GPU chip, which 95% of the time is the problem. We heat up that GPU chip to almost 217 degrees. You don’t want it to get liquid and melt; you want it to get soft, and you want that lead-free solder to re-flow back together with this point on the motherboard, and then you turn off the machine, let it cool off and you have a new connection.”

The re-flow is what many local repair shops can do to repair your dead Xbox. However, a hybrid fix and a re-flow typically last only six months to a year before the Three Red Lights strike again.

Lead-free = deadly

So Peyregne changed tactics. “I don’t do the hybrid fix anymore. Now, we go to the root of the problem, which is removing the chip from the motherboard and replacing the lead-free solder with lead solder that we can still use in this country,” he said of his new approach. “It’s not 100% lead. It’s a 60/40 mix that’s still legal in this country. If you’re the government, you can still request 100 percent lead products, but for consumers, it has to be [a] 60/40 mix, which is still a hundred times better than lead-free solder!”

With this lead solder, Peyregne does what is called a “re-ball”, where he removes the lead-free solder, cleans off the chip, and replaces it with the soft, long-lasting lead solder using a professional machine. After a re-ball repair, Peyregne can usually give you a lifetime guarantee because lead solder Does. Not. Break.

“The problem with lead-free solder is not only does it get brittle, but it actually degrades over time,” he explained. “It gets worse. That’s why when people come in and they don’t want to pay for the re-ball, they just want to pay for the re-flow, it’s half the cost because it’s half the work, and I give my customers that option. But I tell them, it’s not gonna last! You might get lucky, it might last you a year. You might get lucky and, it might last you two years. But eventually, this solder is going to get brittle and break again. It actually degrades over time. You can look at it under a microscope and it gets weaker and weaker and weaker.”

So why do all of our Xboxes have lead-free solder? Why did Microsoft sell machines that are destined to eventually break? Short answer: Europe. “In 2002, the European Union started a consortium, and what it says basically is that any electronic device that is sold or even marketed towards children is considered a toy and must be 100 percent lead-free,” Peyregne said of the issue. “People say, ‘Kids can’t really get in to the video game systems. It’s not like they’re gonna take them apart and chew on them!’ I think it’s because they don’t know what to do with it when it’s disposed of. It’s a smaller geographic region. They don’t have the landfills that we have in this country. They don’t know what to do with it.” As we all know, lead is very poisonous, and any improper disposal could cause serious problems.

This new consortium affected not just Microsoft, but also Sony and Nintendo. Game machines are naturally marketed towards kids, so each company has had to make do with lead-free solder. As a result, Peyregne’s business is not just fixing Xbox 360s, but also PS3s and Wiis because they have the same fatal problems killing their systems. Even the new Xbox 360 Slims are at risk. “We need lead in our electronics,” Peyregne said. “Lead-free solder doesn’t last. It’s 95 percent tin, about two percent zinc and about three percent nickel, and it’s brittle. It’s not a pure metal; it’s a hybrid. It’s something that scientists came up with to use that’s lead-free, and it doesn’t work. It doesn’t last. Our grandfathers and our great-grandfathers knew that lead was great! It’s a soft metal; it doesn’t get brittle; and it doesn’t break. It’s a pure metal. It’s not mixed with anything else.”

So, if the European Union is so concerned about lead in Europe’s electronics, why do the rest of us have to suffer from the Three Red Lights? “You have to understand the game manufacturers’ point of view,” Peyregne reasoned. “Seventeen countries in the European Union — that’s a huge market! So here’s what they had to decide. They could’ve done it. They could’ve said, ‘OK, if they want to have an inferior product, then we’re gonna dedicate this many factories for European sales, and we’re gonna make good ones for the people that can have lead solder in the United States.’ But that’s my guess, I don’t have any sort of verified proof on this. And it just makes sense that it would have cost way too much money to do that.” It didn’t happen, it would seem, because the draw of unified manufacturing allowing Microsoft to keep the cost of the Xbox 360 lower was too strong.

Still, with rampant misinformation all over the web, some people still dispute Peyregne’s findings. Many different methods involved tinkering with the cooling system, including Peyregne’s original hybrid fix. Even manufacturers like Nyko sell fitted fans to pump more air through the system (which, incidentally, only blow the hot air back in). But if it were the cooling, why did these problems not happen with previous systems? “The PS3 has a beautiful cooling system. The fan must be six inches around!” Peyregne said in praise of Sony’s machine. “And the entire bottom portion of the PS3 is a huge heat sink that the fan sits right in the middle of. It’s got great cooling! The cooling in the original Xbox was less than what’s in the Xbox 360, and it doesn’t get the Red Ring of Death!

Continuing, he recounted how no system made prior to 2002 had to deal with the lead issue. “This change came about in 2002. It’s not rocket science. It’s not hard to figure out. Everything before 2002 didn’t have this problem. The PS2, the PS1, the original Xbox, Nintendo 64…you can brush the dust off that old Atari 1600 you got in your attics, blow out the cartridge and put it in, and I betcha’ it works. OK? Because it’s got lead solder in it. It’s very obvious to me, and it should be to everyone.”

The Frankenfixes

Unfortunately, most of us don’t have the engineering background of David Peyregne. So, the community of gamers went to the source they always turn to for answers: the internet. “These games are so well-loved that people just want to do whatever they can to get it working again without paying a lot of money to do it, and the information wasn’t readily available.”

Naturally, this led to some confusion. Even GameStop tried its hand at fixing the Xbox 360, only it fell for the X-clamp fix. “It’s amazing the way that these ideas take off on the web and become gospel, and then everyone else does it.”

What are some of these repair ideas that propagated over the web? One of the most popular ones is the towel trick. By wrapping the Xbox 360 in several towels, the system heats up after just 20 minutes of running. “A poor man’s re-flow,” Peyregne calls it. “Think about what they’re trying to do. They’re trying to heat up that motherboard so that the solder reconnects. They’re doing this, attempting in a repair to fix a problem without actually knowing what the problem is, only that some people have had success with it. It’s not gonna last. In fact, it’s not even going to last as long as a good re-flow does [when] done by a service technician […] because it’s not getting hot enough.”

But the problems don’t stop there. “You’re not just heating up the GPU chip. You’re heating up everything inside of there,” said Peyregne. “All that plastic, all of those capacitors, all of those resistors, they could blow or even just get weak. I have seen Xboxes come in over the years that were so damaged on the inside that I was under the impression that maybe they had put it in their mother’s oven. They were that damaged. And if you look on the Internet, guess what? There are kids doing that: putting them in their mother’s oven to try to attempt a re-flow.”

The penny trick is another trick that was popularized thanks to the internet. For fear the memory modules were overheating, pennies would be placed on top of each memory module as a makeshift heat sink. “If they did it right, then it wouldn’t hurt anything,” Peyregne admitted. “It might not have really helped, but it wouldn’t necessarily hurt anything […] Problem number one is that pennies after the 1970s aren’t even made out of copper anymore. They’re made of zinc. Zinc is not a good heat conductor. Then, they’re wrapping the pennies in electrical tape and gluing it to the bottom of the memory module. Now what you’re doing is, not only are you not transferring heat with a heat sink, but now what little heat was dissipating if you had nothing on it at all is now being blocked by all the goop and electrical tape and zinc pennies that you’ve taped to the bottom of it. It’s making it hotter!”

The more Peyregne described these crimes against machinery, the more atrocities he remembered with a cringe. “Another thing that kids talk about on the internet is using a hairdryer or a heat gun… Um, heat guns are for removing paint. Off of houses. That’s what they’re good for. That’s what they’re made for. NOT heating up a motherboard, OK? Again, sometimes these kids have results with this, but most of the time they end up doing more damage than not.”

But wait, there’s more

Lead-free solder isn’t the only problem with the Xbox 360. “On a motherboard on a regular computer, you have what’s known as standoffs that lift the motherboard off the chassis so it doesn’t ground off these little brass standoffs that you glue onto the case.” After measuring the standoffs in an Xbox 360, Peyregne discovered something odd. “Two of the standoffs are about .7mm higher than all the rest of the standoffs, which was actually putting a bow in the motherboard, causing it to have a flex in it right from the factory. Right from the factory, the motherboard is under stress. By that motherboard being under stress, it was helping to weaken that BGA (ball grid array) connection of that chip on that motherboard so that it wouldn’t last as long with the lead-free solder.” For those not familiar with the finer points of soldering, BGA is a style of soldering commonly used in mass production that makes wider connection points in precisely measured spots. While it takes less space and generates less heat, any stress in those close connections can tear them apart. So, Peyregne would grind down the raised standoffs to even out the motherboard and take away the stress as part of his original hybrid fix.

However, this straining flex has remained with every iteration of the Xbox 360, even the Slim. “Why would they do that?” Peyregne asked rhetorically. “Why would they make two of the standoffs on the chassis higher than all the rest? And there’s all sorts of speculation. Some people say that they did that to raise for greater airflow in a specific area. Well, if they want a greater airflow, then why not raise all of the standoffs an equal amount to keep the motherboard flat and increase that airflow? It’s just…we don’t know why they did what they did. But the new 360 Slim has it too, so there must be a reason!”

Furthermore, while the Xbox 360 has been marketed from the get-go to operate vertically, many characteristics of its internals suggest that was not originally the plan. “The engineers that designed the Xbox 360, I know — I know — there’s no way they could have designed […] for the Xbox 360 to be stood upright. It just doesn’t make sense! I think that the marketing team got involved and said, ‘Is there any reason why we can’t put little rubber feet on the bottom of this, maybe make the bottom wider, change the look of it a little bit, so that it will stand up and look cool like the PS3 and the Wii?'”

So, what are the problems? “The heat sinks in the original 360 are heavy. When you have a BGA connection, especially that’s soldered on with lead-free solder, with the weight of the heat sink over time pulling on that chip, gravity pulling down…” said Peyregne before trailing off. That heavy heat sink pulls on the GPU and the motherboard, adding to the stress. “In a short term, big deal. It wouldn’t make a difference. But if you’re talking about a machine that you expect to last for years and years and the weight of those heat sinks pulling [is] on those chips, it’s gonna make a difference.”

The tray-loading drive is another giveaway. Remember how some people found their games getting viciously scratched? “When you have a DVD tray drive, gravity helps hold that disc down in the bottom of the drive,” said Peyregne. “When you stand it upright and it’s rotating in there, there’s about a quarter inch of space where gravity would normally take that space by holding it into the bottom of the drive, but now that’s no longer there, and that disc will wobble.” Microsoft claimed games only got scratched when the system was knocked over, but Peyregne’s customers say otherwise. Perhaps you do as well.

These disc problems don’t happen with the other systems. “The PS3 is designed so that it’s all held together when it’s standing up,” Peyregne said. “There’s no pressure on the heat sinks. The Wii? The same way. In the PS3 and the Wii, they put DVD slot drives that hold the disc in place, sucks it in, kind of like the drive that’s in your car, and holds it in place. Those are okay to stand upright.”

And then there is the cooling design. While horizontal, the Xbox 360 takes in cool air from the sides and blows the hot air out the back. Peyregne reasoned, “When you stand the Xbox upright, you’re blocking that intake where it’s sitting on the surface. [Then] when you put the hard drive on the top, that’s blocking more airflow so that now you only have a little bit of airflow from one corner going into the system and a whole lot of hot air blowing out the back, and it’s not getting enough cool air.”

So, for the sake of your motherboard, games and GPU, keep your system flat and in a place it can breathe.

Three steps forward, two steps back

Microsoft addressed quite a few of these design flaws in the Xbox 360 Slim. The internal hard drive opens up the side for more air intake, the heat sink is lighter and the fan sits right on top of a combined CPU/GPU chip to keep it cool. However, the new DVD drive brought with it new problems.

“The lasers are failing,” Peyregne complained. “It’s a different laser. It doesn’t last as long. I see it every day.” And even if it does last, units can still fall victim to an open-tray error. “The trays get jammed all the time. I have one out there right now where the tray is jammed… A DVD drive should always work freely. On a computer, you can see the little pinhole where you can manually — with your computer off — you can push in that tab and you can pull the tray in and out. These [Xboxes] that come in that are jammed can’t be budged without breaking them. They have to be taken apart, the tray actually removed, and then put back in the proper spot with the gears realigned and cleaned and then it works again. They’re not broken when they do that — they’re just jammed.” This is all, of course, in addition to the original problem of tilting a tray-loading drive on its side.

Do you want to be a billionaire?

The problem remains: we’re stuck with these systems using poorly made lead-free solder and that will eventually break. This lead-free policy isn’t likely to go anywhere, so we could be stuck even into the next generation. I asked Peyregne if there is any way to make a lead-free solder that can last. He confirmed that there is — if you’re an alchemist. “I tell all the kids that come in, you want to be a billionaire, like in the song?” he replied. “Think of a way to make a solder that has the characteristics of lead solder that doesn’t get brittle and break, that works as well as lead solder, and if it’s affordable […] market that. You’ll be a billionaire, because there are other soft metals they could use.”

You might be surprised to hear what those other soft metals are. “Gold and silver would work great…but it’s too expensive,” Peyregne explained. “My lead solder balls that I use for re-balling these chips are not made with gold in that format. The reason why I checked into it — and you’re kinda smiling — and actually, it wouldn’t increase the cost that much! We’re talking 24-carat gold. Jewelers use it in spools. You ever seen the spools of lead solder that maybe you saw your dad or your grandpa breaking off a piece? Jewelers have that in gold and silver. They make it that way. So I checked. I said, ‘Does anybody make gold or silver solder in this format, in little tiny .06mm, solder balls?’ Nobody did. Nobody makes it. And one company actually got back to me and said the reason why is because security would be too hard with their employees.” Still, that can’t stop us from one day fantasizing that we could have our Xbox 720 or whatever re-balled in gold.

The future

At this point, we’re only a short time away from learning exactly what Microsoft has been cooking up for the next generation. We’re all dying to know what the next Xbox will be called, what games it will have, how it will look and whether or not Illumiroom is part of it. But, will Microsoft continue to use lead-free solder? Unfortunately, Peyregne thinks so. “I think they will keep it the way it is because they’re having success. Why fix it if it isn’t broken? They can sell them in Europe, and they can sell them here. And people that get aggravated here will probably pay to get them fixed or buy another one. And so I really don’t expect them to make a change.”

But what if they could offer us a better model? Peyregne believes that Microsoft needs to consider whether it wants to continue making money off of customers buying replacement systems because the things are basically disposable, or whether it wants to build a machine that will last for a decade, a machine like the original Xbox and sell it at a higher price. “I would pay more money for that, and I think most people would too,” he said.

Perhaps Microsoft could look into whether or not it could legally market a “professional” Xbox SKU specifically for adults that contains lead. Or maybe it could use solder with gold or silver. Peyregne suspects the amount of precious metals required for solder is quite small and would only increase the price by about 50 bucks, maybe less. The question is, would you be willing to pay that?

When it breaks…

Computers for Less still operates in Grandville, Michigan where it started, but Peyregne personally works in Orem, Utah. If you don’t live around there, he knows plenty of other technicians who do the same re-ball repair. Some of them will even let you mail in your system so they can fix it up and ship it back. When you have a problem with your Xbox 360, or any other system, you can get in touch with these people at a community forum. There, you can get answers to your questions from Peyregne and other professional technicians who have been doing this for years.

In the meantime, be nice to your Xbox 360 and hope that Microsoft builds the next one with some healthier guts.