On reddit tons of people are like “hurr durr it was caused by a configuration error that propogated down causing failed health checks and remaining non-updated servers got overloaded durr” and that’s the “techy” explanation.
Maybe that’s enough for MS IT guys and no shade but, with e.g. crowdstrike we knew pretty quickly exactly what the cause was - an array out of bounds read because of an extra malware definitions file (or something to that effect i do not recall exactly).
It is suspect to me that this outage went on for this long, despite how much it affected, and how the recovery went along so haphazardly it seems. They are probably not done with RCA yet but i also wonder if it was a more negligent fuckup they want to keep under wraps to avoid bad rep and e.g. UK 999 system choosing a different tech corp to work with next time as some european govt agencies have already done.
What do you think?
Infrastructure of this size is just code. It’ll spin up instances, route traffic and balance nicely.
They probably added a trivial timeout to their health check. Say, if an instance doesn’t respond within 60 seconds it gets cycled out.
Probably worked fine on Monday morning when they rolled it out.
It probably took them until Saturday night to remember that it takes two minutes for an instance to launch, so every instance will be cycled out before it tags in.
Just a wild guess about the kind of miss that might’ve caused it. I haven’t read it heard anything.