Stack Exchange network consists of 183 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.
It's been somewhat mysteriously reported that FB employees couldn't fix some router (BGP) misconfiguration in a timely manner
because "the people trying to figure out what this problem was couldn't even physically get into the building" to work out what had gone wrong.
It's also mentioned that "The shutdown meant ads weren't served for over six hours across its platforms."
FB may not want to say more because of embarrassment or something, but it sounds like a rather odd story. Is there any corroborating evidence that lack of physical access was the culprit for the prolonged outage?
In a reddit thread during the outage, a user named ramenporn who claimed to be on the recovery/investigation team posted (among other things): "There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified." They later deleted the comments and their account, so it's hard to verify this. See: archive.is/Idsdl
I can't comment on how Facebook does their security... but I can tell you that, as a Google employee, if I showed up to a random data center and demanded access, security personnel would likely escort me from the premises (and I doubt my badge would work, either). Datacenters are extremely sensitive buildings with very different security to "regular" corporate offices.
...these facilities are designed with high levels of physical and system security in mind. They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them. So it took extra time to activate the secure access protocols needed to get people onsite and able to work on the servers.
This doesn't break down how long it took them to get into the building vs. how long it took them to be able to modify the hardware and routers. It's not too hard to imagine how this distinction could be lost and attributed solely to just being unable to enter the building.
We do know that getting into the building did prolong the outage to some degree, though for all we know it could have been just a few minutes.
Something I read (can't remember where...) stated that three different groups of people needed to be involved: one group who had physical access to the servers, another who had the credentials to log in and modify settings, and a third who had the technical expertise to make the necessary fixes. Remote access was not an option (because network issue...!) so they had to get someone from each of the three groups physically present in the server room.
@DarrelHoffman Not sure a cell phone really helps. Its not like they knew what was wrong, they had to diagnose the problem. Unless the cellphone camera was going to be pointed at the screen, which kindof works, but it a painful way to work
@DarrelHoffman I'd assume that high security datacenters are built as a faraday cage, rendering any radio communication (including cellphones) useless. I also imagine that there are some gates requiring biometric authentication. And that's not all ... so, in short, no matter which of those measures are actually implemented there: The information that a security system that is cut off from the internet requires multiple people to physically be there is completely believable to me.
@orithena: There's not really a need to build them as Faraday cages. They usually have false floors (i.e. floor plates on a regular grid of metal), false ceilings (i.e. ceiling plates hanging from a regular grid of metal), long rows of metal racks full of metal boxes with metal piping and metal cable trussing running between them. Even without purposefully designing them that way, they are essentially giant metal boxes filled with lots of small metal boxes installed in middle-sized metal boxes, all emitting EM radiation. i.redd.it/pu3odmbdpqm71.jpg
@DarrelHoffman Pretty sure the ability to gain physical access / enter credentials / fix the problem is distributed among three groups of people for security reasons, so that not a single person can alter things on the servers by himself. So, if a person from the "fix the problem" group can simply place two cellphone calls to also get physical access and log in, it defeats the whole purpose, don't you think so? Therefore, they all needed to be physically present.
That answer is the only one needed to address the immediate claim.
This answer looks at some proposed mechanisms for why this might have been the case, given the larger context of the outage. (Consider it a supplement--if the original question was "Why did JFK die?" and the strictly correct answer is "He was shot," this answer is explaining how that results in death.)
As of this writing, Facebook has not given more detail, however many different socialmedia posts have explored mechanisms for how a networking problem would impede building access--namely, that the electronic systems authorizing access were also caught up in the outage.
Outside parties like CloudFlare, a major network facilitation company unrelated to Facebook, originally became aware of the issue through missing DNS records. DNS is the lookup system that converts between memorable resource names--like websites--and the actual numeric addresses that are currently providing the resource. Early speculation suggested that with DNS down, Facebook also could not access its own systems, including the LDAP server directory system that would track which employees are allowed to access which facilities.
However, the Facebook writeup of the outage indicates that the order of events was a little different. A routine maintenance operation (gone awry) accidentally turned off the main internal networking connections ("backbone") between Facebook data centers. As a result, none of Facebook's internal systems could communicate. Facebook's internal DNS servers--the machines that tell traffic how to get to Facebook--also lost connectivity to the data centers. Now, those systems are designed to function only if they think they can provide reliable data: if they lose connection to the actual Facebook servers, they can't do their job of telling others where to find Facebook resources. So they tell the whole Internet to stop asking them, using something called the Border Gateway Protocol, or BGP (a system which helps networking machines map the best ways to send traffic back and forth).
Essentially, at that point, Facebook's DNS servers all called in sick at once, and nobody could find Facebook any more. But this wasn't strictly a DNS, or even strictly a BGP, problem, as careful observers realized soon after (though the BGP-to-DNS issue caused splash damage to the whole Internet in the form of elevated DNS traffic). Connections between Facebook services' load balancers (that direct traffic from outside to specific locations inside Facebook networks) and the broader Internet still worked in some cases. The root cause was that Facebook had nuked its own internal networking.
Regardless of the exact mechanism, the impact on physical access would be a breakdown of communication between the door lock readers--which get an ID code from an employee's badge--and the directory system that confirms which employee IDs are supposed to have access to which facility. I had originally stated this was due to the DNS problem (meaning that the door readers could no longer find the location of the LDAP server) but best practice is to make directory servers accessible only on private (or virtual private) networks, not the Internet (see also here and probably more other references than I have time to track down). It's more likely that the directory server that grants access was connected through the same internal backbone connection that went down to begin with.
In any event, there's a physical override for this, with an old-fashioned key. But you don't issue a copy of that key to everybody with access to the building--they might make copies, you'd have to get them back when their roles changed, etc. etc. Instead, there's a small security team with overrides for physical access. However, to the extent that the engineering team uses Facebook internal products (e.g. Messenger) for communication, those would also have been impaired by the outage; and there would have been delays in finding other contact information due to the directory being unreachable.
Again, this is a reconstruction of the mechanism through which physical access would have occurred. We won't know for-sure-for-sure until and unless Facebook releases a more specific post-mortem, but my aim is to demonstrate the plausibility of the reported claims based on the surrounding circumstances.
@CJR That's the "chain of events" in the third paragraph. I was trying not to get too into the weeds, but as I interpret 1 and 2 the original screwup caused Facebook's DNS servers to lose connectivity with the data centers; the DNS servers responded to the error by sending BGP updates that cut them off from the Internet.
According to this blog post on engineering.fb.com, the initial trigger was a routing problem on FB's internal backbone network; that led their facilities to stop advertising themselves to the outside world (via BGP) as ways to reach FB; this made the FB DNS servers unreachable, so DNS entries started expiring. So the chain of failures was internal routing -> external routing (BGP) -> DNS.
@GordonDavisson I don't think we have any real evidence that there was card access problems (LDAP or otherwise), just tweets. If not-normally-onsite engineers had to be sent onsite, I doubt they even had card access in the first place. The way their writeup is written, I interpret it as just being a slow process to allow a person into the building (potentially exacerbated if HR records/etc showing that this guy is a real employee were also offline)
In facilities that I'm familiar with, the card readers don't depend directly on the network. They have dedicated wiring to a battery-backed control box in each building. The control box maintains a local copy of the access control list in internal storage that can be updated in near real from servers on the network, but can function independently using the last known data in the not-so-unlikely event of a network or power outage to avoid exactly this type of situation.
By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.
