You build in redundancy for a reason, but in some cases it can backfire. Credit: Pete Linforth / The Digital Artist The Mercury retrograde kicked in big time on Wednesday as Facebook suffered an eight hour-outage that also affected Instagram and Facebook Messenger. No one was believed to be harmed; a few might have even had offline interactions with other human beings. Facebook said it wasn’t an attack, like a Denial of Service attack, and has since issued a statement attributing it to a configuration error. “Yesterday, we made a server configuration change that triggered a cascading series of issues. As a result, many people had difficulty accessing our apps and services,” said Travis Reed, a Facebook spokesman. “We have resolved the issues, and our systems have been recovering over the last few hours. We are very sorry for the inconvenience and we appreciate everyone’s patience.” The question for me is how could a company with redundant data centers around the U.S., not to mention internationally, be taken down like this? All told it has seven data centers in the U.S. Redundancy is supposed to help prevent this kind of problem. Well, not exactly. In the case of a bug or operating problem, redundancy doesn’t help. In fact, it can spread the problem quickly, notes analyst Rob Enderle. “Redundancy can help with certain things like a complete system failure, but it doesn’t help with a virus or software bug because it can replicate it, so redundancy can’t help here,” he said. A software bug shouldn’t have affected Instagram and Messenger, but Enderle figures that the problem was related to a shared-code issue, and whatever it was that failed used the same code or a derivative of that code, so it replicated across all services. “At the very least they should have firewalled the services to avoid something like this,” he said. Still, Enderle thinks something else going on here because in this day and age, an eight-hour outage shouldn’t last this long unless you are under attack, and Facebook said it was not under attack. “They should have rolled back whatever it was in minutes. It’s not like they are a novice company. This should not have happened,” he said. And given the trust issues Facebook has had, it’s in the best interests of the company to come clean, if only for the sake of its advertisers. Most of us just went on with our day. Related content news Data center provider razes 55 homes to make room for Illinois campus Stream Data Centers bought 55 homes in a 34-acre subdivision and plans to break ground on a 2 million square-foot data center campus in late 2024. By Andy Patrizio Mar 21, 2024 4 mins High-Performance Computing Data Center news Nvidia debuts massive Blackwell-powered systems The DGX SuperPOD features eight or more DGX GB200 systems and can scale to tens of thousands of Nvidia Superchips. By Andy Patrizio Mar 18, 2024 3 mins CPUs and Processors High-Performance Computing Data Center news Nvidia launches Blackwell GPU architecture The next-gen Blackwell architecture will offer a 4x performance boost over the current Hopper lineup, Nvidia claims. By Andy Patrizio Mar 18, 2024 4 mins CPUs and Processors High-Performance Computing Data Center news HPE, Pure Storage unveil enterprise storage products New servers and storage services are targeted at high performance workloads, which means AI. By Andy Patrizio Mar 14, 2024 4 mins Generative AI Enterprise Storage Data Center PODCASTS VIDEOS RESOURCES EVENTS NEWSLETTERS Newsletter Promo Module Test Description for newsletter promo module. Please enter a valid email address Subscribe