Microsoft said a software defect that sat unnoticed on its servers since July combined with two other problems to cause last week’s global Microsoft 365 outage. The company set out the findings in a preliminary post-incident report dated Nov. 15, and an executive described them publicly on Monday, Nov. 16.[1][3]
Related: Microsoft Restores Service After Day-Long Global Outage, Apologizes and Blames a Network Upgrade.
The three factors
The report dated the outage from 4:20 p.m. UTC on Thursday, Nov. 12, to 4:10 p.m. UTC on Friday, Nov. 13, just under 24 hours.[1][2] The failure hit the networking gateway, the set of servers that sits in front of many Microsoft 365 services.[1]
The first factor was a fix written specifically for those gateway servers, which began rolling out in July. Under certain load, an internal Windows process exhausted a server’s processor and memory, and the machine stopped responding. The change ran worldwide for more than three months without trouble.[1]
Microsoft’s load testing did not catch the defect. The problem is worse on newer machines with more processor cores, and the company’s tests focused on weaker hardware. Neither the defect nor the fix applies to broader Windows deployments.[1]
The second factor was a shift in traffic. About 15 hours before the outage, Teams traffic moved from an alternative gateway onto the primary gateway that also supports Exchange Online, the company’s email service. Microsoft described that kind of failover as part of normal operations.[1]
The third factor was capacity. Nearly 14% of the gateway’s server capacity was already out of service because of an unrelated issue from the previous day.[1]
“Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July,” Charles Lamanna, Microsoft’s executive vice president of Copilot, Agents and Platform, said in a post on X.[3]
How the failure spread
The servers began to fail during the peak window of U.S. traffic. Microsoft’s traffic systems automatically sent users to healthy servers, which then hit the same defect under the added load. The failure crossed the fleet in less than two hours. At the lowest point, more than 80% of servers were unresponsive.[1]
Partitions meant to keep a failure inside one region did not hold. The report attributed that to a separate issue and did not explain it further.[1][2]
Teams, OneDrive, SharePoint and the Office apps moved off the faulty gateway and were partly restored earlier. Exchange Online could not move, for what the report called “scale reasons,” which left Outlook mailboxes out of reach for most users.[1]
Why recovery took a day
Restarting the servers did not work. Waiting traffic overwhelmed each machine as it returned to service. Microsoft’s usual rollback tools could not quickly undo a change made months earlier.[1]
Engineers began applying an update that reverted the defective change at 7:20 a.m. UTC on Friday. They took large groups of servers offline, updated them and restarted them. Europe returned to healthy levels at 11:35 a.m. UTC, and global webmail availability recovered at 2:17 p.m. UTC.[1]
The report listed the incident as a technical issue unrelated to cyberattacks or to AI agents. Microsoft has since added capacity and new limits on how traffic is distributed.[2]
What comes next
Microsoft committed to a final report within five business days of the incident’s resolution, and it is expected on Nov. 20.[1][2] That document may explain why the regional barriers failed.[2]
The outage reached well beyond office email. Jails could not release prisoners, hospitals turned to paper records and schools lost contact with parents.[3]
“This is a prime example of the importance of not only cyber resilience, but business resilience,” Matt Hull, vice president of cyber intelligence and response at NCC Group, said.[3]
Earlier coverage: the outage’s second day and Monday’s bank app disruptions that Chase blamed on a software bug. More in technology news.
Sources
- Office Watch: Microsoft 365 outage: what went wrong, in plain English (with the text of Microsoft’s preliminary Post Incident Report, EX1550543/MO1550821)
- Ars Technica: Three small failures combined to take down Microsoft 365
- The Guardian: Microsoft identifies months-old bug as a root cause of global outage
Read next in Technology
TechnologyMicrosoft Restores Service After Day-Long Global Outage, Apologizes and Blames a Network Upgrade
TechnologyThe FAA, Microsoft and Big Banks All Failed in One Week. Officials Say None Were Cyberattacks. Here Is What Is Still Unexplained
TechnologyChase Blames a Software Bug as Customers at Chase, U.S. Bank and Wells Fargo Report Outages
Most read this week
- Finland Confirms Strike on Russian Warship in Gulf of Finland; Ukraine’s SBU Claims Drone Attack
- U.S. Envoys Witkoff and Kushner Hold Talks With Putin in Moscow, Then Head to Dublin to Brief Zelensky
- Israeli Strike on Gaza’s Nasser Hospital Kills Six, Including a Surgeon, Health Ministry Says
- Nvidia Board Hires Wachtell’s Joon Kim to Lead Huang Probe as Shares Fall 5.47% and an Investor Sues
- Drones Kill Workers at Zaporizhzhia Nuclear and Thermal Plants, IAEA Says, as Russian Strikes Hit 5 Substations
Latest
- French Public-Sector Workers Strike Over Pay as Saturday Turnout Falls
- El Salvador’s War on Haiti’s Gangs: The Stadium Stronghold, the Losses So Far and What Comes Next
- Musk’s SpaceX Denies Limiting Ukraine’s Starshield as Zelensky Blames the Cap for Russian Strikes; Trump Says He ‘Hasn’t Heard’
- Trump Lands in Manila, Signs a Thailand Trade Statement and Warns Iran Will ‘Pay a Price’ Over Any Plot
- Kataib Hezbollah Vows to Confront Iraqi Forces if They Enter Jurf al-Sakhar, Denying Oil Smuggling
Get The Boston Report Briefing
Our free weekly email with the most important stories from Boston and the world. Unsubscribe anytime.
Free. We never sell your address. Privacy policy