Thread: Facebook outage
View Single Post
Old 10-04-2021, 02:03 PM   #13
Hack&Lube
Atomic Nerd
 
Join Date: Jul 2004
Location: Calgary
Exp:
Default

It sounds like BGP peering with the Facebook peering routers is down and this happened due to a configuration change via some kind of automation. This would break all DNS and routing. This may have taken down the peering routers completely and since they can no-longer be accessed remotely, they need to be accessed physically inside the Facebook datacenters and this is causing further issues since datacenter access is severely restricted and staffing is all remote due to Covid-19.

Basically if their BGP routers withdrew their DNS records or are no-longer advertising FB as a valid route, nobody can get to their DNS records to route traffic to facebook. This seems to have propagated everywhere instantly. What's worse is their internal systems were also using this so nobody can access their buildings or facilities to actually fix the problem directly.

I found this cached article where Facebook engineering is lauding their peering automation. This sounds like the candidate for what broke: https://webcache.googleusercontent.c...&ct=clnk&gl=ca

Some tech friends were like "haha!" until they were like "nooo my stonks!"

Last edited by Hack&Lube; 10-04-2021 at 02:20 PM.
Hack&Lube is offline   Reply With Quote