Jump to content

Millions of websites offline after fire at French cloud services firm


clapclapclap

Recommended Posts

 

https://www.reuters.com/article/us-france-ovh-fire-maginot-firewall-fail

Spoiler

PARIS (Reuters) - A fire at a French cloud services firm has disrupted millions of websites, knocking out government agencies’ portals, banks, shops, news websites and taking out a chunk of the .FR web space, according to internet monitors.

Slideshow ( 4 images )

The fire, which broke out on Wednesday shortly after midnight at OVHcloud, destroyed one of four data centres in Strasbourg, in eastern France, and damaged another, the company said.

There was no immediate explanation provided for the blaze, which erupted just two days after the French cloud computing firm kicked off plans for an initial public offering.

Europe’s largest cloud services provider told clients including the French government, the Centre Pompidou and cryptocurrency exchange Deribit to activate their disaster recovery plans following the blaze.

“Firefighters were immediately on the scene but could not control the fire” in the affected data centre, founder and chairman Octave Klaba said on Twitter. He said the plan for the next couple of weeks would include rebuilding the centres’ equipment and checking their fiber optic connections.

Some 100 firemen fought the blaze which sent a thick plume of black smoke into the night sky. Video images showed firefighters dousing one smouldering, multi-storey building in the early morning as they cooled down the site.

Founded by Klaba in 1999, OVHcloud competes against U.S. giants Amazon Web Services, Microsoft Corp’s Azure and Alphabet Inc’s Google Cloud, which dominate the market.

“OVH is a pretty important hosting company on the internet,” said Mike Prettejohn, who directs UK-based network security company Netcraft. He said the affected servers hosted 3.6 million websites, including niche government platforms in France, Britain, Poland and the Ivory Coast.

Prettejohn estimated that just under 2% of the sites with the French .FR domain extension had been affected.

French politicians have championed OVHcloud as a possible alternative to U.S. cloud services providers, but it has so far lacked the scale and financial clout to dent their market share.

The company said on Monday it had started the process for a potential IPO, without giving details.

The Strasbourg centres were among OVHcloud’s 17 data centres in France, and 32 globally.

Several clients said their websites had gone offline or emails could not be accessed. There was no immediate news of any major data losses.

The Centre Pompidou, one of France’s best known art complexes, said its website was down. Cryptocurrency exchange Deribit said its blog was down but that trading had not been impacted, free chess server Lichess.org said it had lost 24 hours’ worth of games history and news outlet eeNews Europe said its websites were offline.

PR agency Caroline Charles Communication said its data was held at another site but that it had lost access to emails in the middle of Paris Fashion Week.

OVHcloud declined to comment on its fire safety protocols. The site did not pose a toxic risk, it added.

Reporting by Matthieu Protard and Mathieu Rosemain in Paris and Raphael Satter in Washington; Writing by Richard Lough; Editing by Mark Potter and Matthew Lewis

AVQ4fugp?format=jpg&name=small

ngxEJLIL?format=jpg&name=small

EwGnLy_XIAMfxjc?format=jpg&name=large

EwG-C84WYAcLpLj?format=jpg&name=small

Update:  Interpol has named Frank Gehry as an Architect of Interest.

  • Hook 'Em 1
  • Like 1
  • Haha 2
Link to comment
Share on other sites

5 minutes ago, AUS-97HORN said:

heck of a disaster backup plan they have there.

they should have been able to fail over somewhere within 10 mins. 

The dirty little secret in IT is that backup plans suck for the most part.  Most companies put the disaster in disaster planning. 

  • Hook 'Em 2
  • Like 1
Link to comment
Share on other sites

14 minutes ago, Hate said:

There is no way that wasn't arson.  With all of the protections modern data centers have, that was done purposefully. 

 

Quote

The company said on Monday it had started the process for a potential IPO, without giving details.

giphy.gif

Link to comment
Share on other sites

OVH is huge. They are the largest EMEA-based cloud hoster. They are in IDC's top 20 list of server consumers.  In the grand scale of their business, though, I think this is fairly small. I think they have like 30+ data centers. But I assume some big French customers were serviced from that data center, being as it's in France. Likely government. Not good.

To someone's point above, doesn't sound like disaster recovery went all that swimmingly.

Here's a look into the board room as they learned about the fire.

tv show smoking GIF

Link to comment
Share on other sites

2 minutes ago, Enchubben said:

I saw reports elsewhere that the fire department showed up to go to war with the fire, but saw how dangerous it was and just surrendered the facility. 

Fucking cheap, lame, cracks about the French, and their cowardly, surrendering ways.  I like it.

Link to comment
Share on other sites

11 hours ago, Hate said:

The dirty little secret in IT is that backup plans suck for the most part.  Most companies put the disaster in disaster planning. 

Several years ago, one of my customers did a complete remodel of the cool old building that hosted one of their DCs. We sold them a big server refresh to go into the newly expanded DC, and were on site the day they were going to light everything back up. We had a professional photographer there and everything because my company was going to do a big press release on how we helped company XYZ modernize their DC. Had a big glass wall, cool lights, all that shit. Anyway, they throw the switch, and the entire building goes dark. CIO is like "huh...well, the battery backups should come on any second now........huh....well, the generator should come on any second now........" The batteries in the battery backup were dead as doornails, and someone had stolen all the fuel (like, 1,000 gallons of diesel) from the generator tank. It was like a scene from a comedy watching these poor fuckers run around trying to figure out why the whole building was pitch black. 

  • Like 2
  • Haha 2
Link to comment
Share on other sites

My organization was building a new facility for a different business unit and they incorporated a bunch of money into the build to house our backup/DR datacenter. We were included in all the design and drawings discussions to make sure we had everything we needed included on the design.

Construction took about two years, and about a month prior to opening the facility for use, the contractors allowed us in to start putting in switching and networking equipment.

We walked into the proposed datacenter and someone had done a change order mid stream that no one bothered to tell IT about. The change order ran about six exposed 2" water lines directly over the top of, and down the full length of, the eight server racks.

Management couldn't understand why we refused to put several million dollars worth of equipment into the room.

 

 

Edited by Orange&White
Link to comment
Share on other sites

15 minutes ago, Orange&White said:

We walked into the proposed datacenter and someone had done a change order mid stream that no one bothered to tell IT about. The change order ran about six exposed 2" water lines directly over the top of, and down the full length of, the eight server racks.

Management couldn't understand why we refused to put several million dollars worth of equipment into the room.

sounds to me like you wasted a perfectly good opportunity to turn those racks into liquid cooled servers.

  • Haha 1
Link to comment
Share on other sites

19 minutes ago, Orange&White said:

My organization was building a new facility for a different business unit and they incorporated a bunch of money into the build to house our backup/DR datacenter. We were included in all the design and drawings discussions to make sure we had everything we needed included on the design.

Construction took about two years, and about a month prior to opening the facility for use, the contractors allowed us in to start putting in switching and networking equipment.

We walked into the proposed datacenter and someone had done a change order mid stream that no one bothered to tell IT about. The change order ran about six exposed 2" water lines directly over the top of, and down the full length of, the eight server racks.

Management couldn't understand why we refused to put several million dollars worth of equipment into the room.

 

 

no problem. you just gotta drip the faucets.

  • Like 2
  • Haha 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.



×
×
  • Create New...