Grade D Telecom Infra : Vol. Sofla Internet Blackout

bump for interest

>>62467062
Interesting. I was doing some fundamental analysis on companies to possibly construct a a long/short pair for portfolio and came across information that during Katrina there was something similar, telco companies infrastructure buildings getting flooded for example, but more limited and nowhere near as severe as this.
So there are some major data centers hosted in Miami, where they down too?

File: 14821.png (21KB, 200x194px) Image search: [Google]

21KB, 200x194px

>>62467648
Yeah, if you search around the internet well enough you can find out where all the major telco data centers are as well as co-located data centers. Typically you'll find a lot of co-located centers near the main telco trunks. There was no major damage to south florida except some downed trees and power lines. The hurricane went underneath FL and road the west coast. The east coast didn't get a direct hit yet that's where all of the outages occurred : In all of the regions where the hurricane didn't go. Tampa got a direct hit and didn't face nearly the outages that south florida did. South florida (broward/dade) has a highly dense population.. With all of the exercises and scenario testing being done, I wouldn't be surprised it these jokers intentionally cut power and communication under the guise of the hurricane to test out experimental equipment, how people would respond, data collection, etc.

I'll be doing further investigation this weekend to track down the real reasons for the outages especially w.r.t to the telecom infra... Something hasn't sat write since a rep told me the shut down the switches of the main trunks.

>>62468386
Also, given the lack of investigation, reporting, care, and or concern on how major cities can be in literally blackout for over a week sets the precedence for this happening elsewhere and more often.... Knowing they can get away with this shit w/o answering any serious inquiries will allow them to pull the rug at will on other locations... All they'll need is a convenient excuse.. Voices can be silenced via yanking the telecom and claiming (were working on the problem w/ TBD solutions). Meanwhile, they can string along any other major infrastructure
in the communications blackout while executing on other main agendas.

>>62468386
Here's a few things I recently saw that I thought where interesting.

https://www.lowendtalk.com/discussion/124124/curious-about-what-ll-happen-to-servers-hosted-in-miami-or-latin-america-s-communications

http://status.quadranet.com/incidents/d094tq2wjs51

http://www.datacenterknowledge.com/manage/irma-heads-florida-one-miami-data-center-especially-critical

http://www.thewhir.com/web-hosting-news/as-irma-heads-for-florida-one-miami-data-center-is-especially-critical

http://www.thewhir.com/web-hosting-news/use-a-data-center-in-irmas-path-heres-how-not-to-lose-your-data

http://www.megatrndz.com/2017/09/will-latin-america-go-dark-after.html

http://liftrnews.com/data-center-in-path-of-hurricane-irma/

>>62468487
>http://www.datacenterknowledge.com/manage/irma-heads-florida-one-miami-data-center-especially-critical
Yeah there's several redundancies that should have prevented any major blackout. It's just more than likely that these redundancies weren't activated and these fuckers pulled something like literally shutting off large swaths of their network as opposed to letting it reroute through the redundant links. As stated, these builds sit far above sea level and have cat5 storm ratings. Furthermore, they all have independent and highly resilient power line links. That being said, even if their power was truly damaged, it would have been one of the first things they worked on restoring and would have gotten restored much earlier in the week. On top of this, the Hurricane didn't even hit miami, it hit the vacant west coast. The winds were equivalent to cat 1/2/3.

All of the solid hurricane models suggested it was headed towards the west coast yet the media kept pushing for and the mayor even mandated evacuations for miami. Nearly 8-9 million people evacuated cat 1/2 storm force winds and magically when people return, the whole telecom infrastructure in is blackout extending from miami-dade all the way up to ft.lauderdale. They pulled some massive shit amidst this storm. The only question is what...

> http://status.quadranet.com/incidents/d094tq2wjs51
Yep, power was out due to some downings but was restored for the most part by the 11th. Thus, power was not the excuse for the broad based (land line) telecom outage. Cell connectivity including internet (LTE) was stable throughout.
Note no one at these co-location centers saying they had disruption issues.

Meanwhile, millions of business/residential units saw no connect events on their modems indicating that major ISPs literally shut down connectivity.

>>62468701
https://www.reddit.com/r/weather/comments/6ycx7w/i_am_a_new_yorker_voluntarily_traveling_to_miami/
I work as a Critical Facilities Engineer for a nationwide Data Center. my company is sending me down to our Miami Location to man the fort of our Miami Facility. I work for CoreSite.

Edit 5: Day 4 is about to begin
The entire hurricane has been downgraded to a category 3 storm and headed towards the west coast of Florida. Miami has been getting some intermittent outer band weather for the last 24 hours. FPL has reported 110,000 outages in Miami-Dade County & 40,000 in Broward County. Our neighborhood of Miami seems to be fine so far, we're still on utility power. All mandatory evacuation points in Miami seem to be getting hit harder with outer band weather than we have. We're in Evacuation zone E so it seems like their predictions up to this point have been accurate.

Edit 6: Day 4 is complete, the Hurricane did touch down in Miami at roughly 5am EST. Where we are located we did not sustain much damage.
Over 2 million reported outages between Miami-Dade & Broward counties (nothing new I'm sure). Miami mainly saw outer band weather and was more of a category 2 where we were. We switched over to generator power early sunday morning due to excessive brown outs in a short period of time. We lost utility completely at 5PM and we're running solely on generator power. No damage in our neighborhood, no power lines down, no flooding

Edit 7: Day 5 Looks like Irma has came and gone from Miami. Utility power came back to the data center this morning at 9am.

I understand power redundancies, but how do you have Internet redundancy?

If a carrier loses service, that is outside of our Service Level Ageeement. We're a carrier neutral facilities where we have many ISP's in there.

>>62468885
Yea, so essentially what I said.. :
> cat 2/3 level winds
> No serious damage
> Power went out to swaths of people (Seems it was cut in advance of the storm tbqh)
> Solid data centers saw zero outtages even though they were near most of the more serious windws and outages
> All ISPs and serious data centers have multiple redundant links .. There are like 6/8 of them that wire through Florida... and yet telecom outages for a week+ for millions in south Florida.

I like how no one in this space and close to this equipment is commenting on how they managed to keep their data centers up throughout the whole storm while they were trunked into an ISP that somehow managed to loose connectivity to millions of others...

As I stated, smells like some shit went down.

>>62468885
> I understand power redundancies, but how do you have Internet redundancy?If a carrier loses service, that is outside of our Service Level Ageeement. We're a carrier neutral facilities where we have many ISP's in there.

These guys have multiple data centers across the country/world that they can migrate data and even live operational compute instances with the flick of a switch to ensure the uptime numbers they promise customers. Some of this is automatic and doesn't even require human intervention :
i.e :
https://www.vmware.com/products/vsphere/vmotion.html

http://www.miamiherald.com/news/weather/hurricane/article173160011.html
SEPTEMBER 13, 2017
Still no internet or cell service? You’re not alone
Sanchez can’t get a straight answer about his U-verse internet service. Like thousands of other South Floridians, the internet is vital to his livelihood.
In the aftermath of Hurricane Irma, Sanchez is among the many locals who have called to express their frustration and anger over outages. Sanchez is particularly disgusted, he said, by the lack of information provided by his provider. AT&T first told him there was no outage in his neighborhood, and that it must be his modem. Then the company said his area was experiencing an outage. When will service be restored? he asked. “We don’t know, we have no idea,” he said he was told.
“We are in the dark and we are a society that depends on the internet,” Sanchez said on Wednesday.
He called AT&T’s corporate offices. but said he couldn’t get through. “The companies are not providing any information to their customers, and I don’t think government is putting enough pressure on them.”
of South Floridians seem to agree, according to social media and calls Outages on Comcast Xfinity, AT&T U-verse and Atlantic Broadband are widespread. On top of that, customers of cellular carriers Verizon, AT&T, Sprint and T-Mobile are also experiencing service disruptions. Sens. Rubio and Nelson sent a letter to FEMA Tuesday requesting they coordinate with relevant federal, state and local officials to help communications providers restore Florida’s networks.
The four providers of local cellular services said they were unable to say how many cell sites are out of service in South Florida — even though they supply that exact information for every county in Florida to the Federal Communications Commission’s daily “snapshot” reports in the aftermath of the storm. The FCC reports provide the number breakdown by county but do not include providers’ names.

>>62469971
Here’s what the latest FCC report says:
▪ Eighty-two percent of cell sites in the Keys and other parts of Irma-ravaged Monroe County are not working, according to an FCC report issued Wednesday.
▪ In Miami-Dade and Broward counties, about 30 percent of all sites remain non-functioning — a decrease from 40 percent on Tuesday, the report said.
▪ After the Keys, Southwest Florida is experiencing the most outages.
▪ Statewide, 18 percent are without service, down from 24.6 percent Tuesday.
As for internet service restoration, Comcast spokeswoman Kramer said the company estimates service will be restored by Sept. 18, based on the significant degree of commercial power outages in South Florida.
All of Comcast’s Miami-Dade and Broward facilities remain on generator power, she said “Our teams are now deploying generators into neighborhoods so that we can bring our services back on line even when our equipment does not yet have commercial power,”
In addition, network maintenance teams are in the field repairing cut fiber lines. “We have also seen damage to our lines caused by fires when the power lines on the poles become re-energized with commercial power.
In Monroe County, Comcast has re-established connectivity to its network and is now restoring services in Key Largo. “We still have no access beyond Marathon, but we have crews standing by and barges ready to bring in additional crews and equipment as soon as these areas become accessible,”

https://arstechnica.com/information-technology/2017/09/hurricane-irma-took-7-million-cable-and-wireline-subscribers-offline/
Sep 13, 2017
Hurricane Irma took 7 million cable and wireline subscribers offline
Comcast, AT&T, other ISPs try to get customers online as power outages persist.
The FCC has been requiring providers to report outage figures to the commission each day during the storm, but the publicly available numbers do not include outage figures for each ISP. Comcast, AT&T, Charter, CenturyLink, and Frontier are among the providers trying to keep customers connected in the areas hit by Hurricane Irma.
As for wireless connectivity, 24.6 percent of cell sites in Florida were out of service yesterday, the FCC said. Five Florida counties (Collier, Hendry, Highlands, Monroe, and Union) had outages in more than 50 percent of cell sites.
Power outages are a major contributor to the cable and wireline disruptions, and those outages could persist for weeks.
The telecom providers also suffered damage to their networks.
"The overall impact is a combination of commercial power outages as well as damage to our system," a Comcast spokesperson told
Frontier’s FiOS customers take a hit on Monday, Frontier said that about 50 percent of its Florida customers did not have power. "Our biggest issue continues to be getting commercial power back in working order

>>62469971
>>62469985
>>62470054
And yet none of these faggots felt it necessary to disclose any of this until pressured by the govt. into doing so. Furthermore, treating vital infrastructure like telecom in this modern world like its some shit tier convenience you just get around to enabling once you've done all the major work. Businesses literally without the ability to function, people who work from home left w/o connectivity being fed bullshit by CSR reps. A literal fucking clown show brought on by Grade D infrastructure that no one had the wherewithal to upgrade/maintain before some wind came rolling through. Buried fiber should have been all over the place given how much money they've been collecting yet these fuckwits still have telecom lines running above ground across power line runs... Power cut to data centers and telco slabs? Where the fuck is the emergency protocol to get power restored to such a vital infrastructure..? But the shekels keep pouring in so lets let people be w/o connectivity including businesses for multiple weeks... Let feed them generic bullshit when they call in. Could you imagine wtf would have occured if a cat5 actually came barreling through miami? You'd literally have to pack your shit up and vacate for months (millions of peoples and businesses) until these assholes pulled their head out their asses.

1st world my ass.. America is one real tragedy away from being third world. Myopic fucks pissing away money w/ nothing saved up or provisioned for a rainy day.