10-04-2021, 02:22 PM
|
#22
|
|
That Crazy Guy at the Bus Stop
Join Date: Jun 2010
Location: Springfield Penitentiary
|
Quote:
Originally Posted by burn_this_city
IT version of an inside job to bring down the company?
|
Live shot inside the FB headquarters:
|
|
|
|
The Following 9 Users Say Thank You to Cecil Terwilliger For This Useful Post:
|
|
|
10-04-2021, 02:23 PM
|
#23
|
|
Franchise Player
Join Date: Dec 2012
Location: On your last nerve...:D
|
Quote:
Originally Posted by Skaloper
Looks like everyone went to Twitter...
|
They had fun with it. A lot of the big brands joined in. Made me laugh a few times.
https://twitter.com/user/status/1445078208190291973
Last edited by Minnie; 10-04-2021 at 02:27 PM.
|
|
|
|
|
10-04-2021, 02:25 PM
|
#24
|
|
Atomic Nerd
Join Date: Jul 2004
Location: Calgary
|
ELI summary of what it sounds like to me:
1. There are Facebook datacenters all around the world with routers that tell internet on the traffic how to get to the facebook servers
2. Someone or some process automatically updated all these routers with the wrong config and now there is no way to connect to them
3. All these datacenters have security systems that are also tied to the internet and this broke all the authentication for keycard access so they can't actually get in to physically fix the problem
If Facebook were a fleet of 10,000 cars, this would be like all the drivers locking their keys in their car all at the same time and not being able to get back in.
|
|
|
|
|
The Following 2 Users Say Thank You to Hack&Lube For This Useful Post:
|
|
|
10-04-2021, 02:26 PM
|
#26
|
|
Atomic Nerd
Join Date: Jul 2004
Location: Calgary
|
Quote:
Originally Posted by Nyah
|
Facebook would have backups of the routing tables if they could only get to the physical hardware to restore them. If they didn't then WHOOHOO!
|
|
|
|
|
10-04-2021, 02:27 PM
|
#27
|
|
Truculent!
|
Quote:
Originally Posted by Hack&Lube
It sounds like BGP peering with the Facebook peering routers is down and this happened due to a configuration change via some kind of automation. This would break all DNS and routing. This may have taken down the peering routers completely and since they can no-longer be accessed remotely, they need to be accessed physically inside the Facebook datacenters and this is causing further issues since datacenter access is severely restricted and staffing is all remote due to Covid-19.
Basically if their BGP routers withdrew their DNS records or are no-longer advertising FB as a valid route, nobody can get to their DNS records to route traffic to facebook. This seems to have propagated everywhere instantly. What's worse is their internal systems were also using this so nobody can access their buildings or facilities to actually fix the problem directly.
I found this cached article where Facebook engineering is lauding their peering automation. This sounds like the candidate for what broke: https://webcache.googleusercontent.c...&ct=clnk&gl=ca
Some tech friends were like "haha!" until they were like "nooo my stonks!" |
You are an atomic nerd.
__________________
Quote:
Originally Posted by Poe969
It's the Law of E=NG. If there was an Edmonton on Mars, it would stink like Uranus.
|
|
|
|
|
|
10-04-2021, 02:32 PM
|
#28
|
|
Franchise Player
|
It's probably not quite as easy as just putting a new configuration in. With traffic demand as big as Facebook, if you, say, start with one BGP router, that route gets hammered, and will probably take down nodes as all Facebook traffic hits one target. So you would have to be careful about restoring it. Hopefully they have a disaster recovery plan for this scenario, or it might cause problems for days.
|
|
|
|
10-04-2021, 02:38 PM
|
#29
|
|
First Line Centre
Join Date: May 2012
Location: The Kilt & Caber
|
Quick! Everyone get your idiot relatives who get all their information from FB vaccinated before it's back up again!
|
|
|
|
|
The Following 6 Users Say Thank You to Nyah For This Useful Post:
|
|
|
10-04-2021, 02:39 PM
|
#30
|
|
Franchise Player
Join Date: Feb 2007
Location: A small painted room
|
|
|
|
|
|
The Following 2 Users Say Thank You to Ozy_Flame For This Useful Post:
|
|
|
10-04-2021, 02:42 PM
|
#32
|
|
Franchise Player
Join Date: Apr 2012
Location: Maryland State House, Annapolis
|
Zuckerberg kinda looks like a real life Lionel Hutz lol.
__________________
"Think I'm gonna be the scapegoat for the whole damn machine? Sheeee......."
|
|
|
|
|
10-04-2021, 03:13 PM
|
#33
|
|
Franchise Player
|
Quote:
Originally Posted by Hack&Lube
If Facebook were a fleet of 10,000 cars, this would be like all the drivers locking their keys in their car all at the same time and not being able to get back in.
|
Or you could just say it is like facebook was:
|
|
|
|
|
The Following 2 Users Say Thank You to Flabbibulin For This Useful Post:
|
|
|
10-04-2021, 03:15 PM
|
#34
|
|
Franchise Player
Join Date: Jul 2010
Location: Calgary - Centre West
|
Part of me is happy about this, well deserved for how destructive Facebook has been to society as a whole.
Another part of me is thinking "ah ####, there goes anything I used OAuth through Facebook for".
__________________
-James
GO FLAMES GO.
|
|
|
|
|
10-04-2021, 03:20 PM
|
#35
|
|
First Line Centre
|
I think life is actually better without Facebook
|
|
|
|
|
The Following User Says Thank You to Calgary14 For This Useful Post:
|
|
|
10-04-2021, 03:21 PM
|
#36
|
|
Franchise Player
|
Quote:
Originally Posted by Calgary14
I think life is actually better without Facebook
|
How the hell am I gonna wish my mother a happy birthday?
|
|
|
|
|
10-04-2021, 03:22 PM
|
#37
|
|
Franchise Player
|
Where am I going to get my medical advice though
__________________
GFG
|
|
|
|
|
The Following User Says Thank You to dino7c For This Useful Post:
|
|
|
10-04-2021, 03:22 PM
|
#38
|
|
Franchise Player
Join Date: May 2016
Location: ATCO Field, Section 201
|
Quote:
Originally Posted by Calgary14
I think life is actually better without Facebook
|
I haven't had facebook for 3-4 years and I agree completely.
|
|
|
|
|
The Following 2 Users Say Thank You to TheIronMaiden For This Useful Post:
|
|
|
10-04-2021, 03:24 PM
|
#39
|
|
Atomic Nerd
Join Date: Jul 2004
Location: Calgary
|
I also stopped using Facebook for the last 5-6 years but I heavily use messenger to stay in touch with people.
|
|
|
|
|
10-04-2021, 03:46 PM
|
#40
|
|
Franchise Player
|
Aww crap, it looks like ti might be coming back to life. Kill it! Kill it with fire!
|
|
|
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is Off
|
|
|
All times are GMT -6. The time now is 07:44 AM.
|
|
