Datacenter Issues

[bobby]

Hey Everyone,

We're aware of an issue at this time, this is with the datacenter itself, not our server. We've talked with other hosting companies that use the Caro.net facility, and they are experiencing issues as well.

Thanks!

[bobby]

At 12:41 a utility fault caused phase B of the main power feed of DC3 to disconnect which caused our ATS to transfer power to our generator until which time we were on our generator ’til 1:15. Our ATS read a fault and and went into EPO with respect to the initial issue which caused phase B to fail. Local service technicians are working on solving the problem as fast as possible.

[bobby]

Pingdom (our monitoring service) reported that this downtime occured at 1:14 PM. Which correlates to the time that we went off of generator.

We're awaiting for another update. I apologize for all inconveniences.

[bobby]

Hey all,

There's an update coming in the next couple of minutes. As soon as I get that update, I'll post it here.

Thanks!

[bobby]

Hi!

We believe that this is now close to resolution. Just to clarify as well, I know some of the datacenter jargon can be a little hard to read. Basically, at 12:41PM today, the power company had a problem. The Automatic Transfer Switch, switched over to generator. However, at 1:14 the ATS noted a fault on the Generator, and did an EPO (Emergency Power Off).

Once power is restored, we should be able to immediately bring the server back up.

Thanks,

[bobby]

Hello,

We've received an official update:

Technicians are nearly finished with the necessary work and are testing all phases. We will update again as soon as possible. Thank you for your patience.

Thanks!

[bobby]

Hello,

We believe that we are getting really close to resolution. I'm watching for a new update.

Thanks!

[bobby]

Hello,

We're now definitely nearing completion. Here's the latest update:

Power circuits are now being brought up individually. During this process, some of you will find your services come up before others. Please be patient as you have been. Services are returning.

So service should return here shortly.

Sorry about all the problems folks,

Thanks!

[bobby]

Hello all,

We're back in business! The server currently has high load, because it just started up. It will take a few minutes to settle down, but everything appears healthy, and there appears to be no damage.

Thanks!

[bobby]

Hey guys,

Last night we were down again. I do apologize again for the downtime:

As yet another unfortunate turn of events, the temporary generator we were using to supply power to DC3 caught fire at approximately 03:00 EST this morning. As a result of the fire we were forced to EPO DC3 by order of the Mecklenburg County Fire Marshall.

Shortly after the fire began we requested a “replacement, replacement” generator from our vendor knowing the likely outcome of these events. Once the fire department had completely put out the fire it was determined that the generator was all but a complete loss and was of no further use to us.

Due to delays outside of our control it took roughly four more hours for the “replacement, replacement” generator to be on site and we had it connected and supplying power in just under 30 minutes.

At this time all service should be restored and we’re vigilantly monitoring the situation to ensure there are no further interruptions.

Considering the unbelievable nature of the past 24 hours, pictures of some of the events will be forth coming and we will be providing links to these photo sets within this RFO.

I'll provide the pictures as soon as I receive them.

Thanks!

[bobby]

Hey everyone!

A small update,

We're still on generator backup. Everything is still running normally, and we should have normal power back soon.

Thanks!