I am putting the final report here for you: it is interesting because it explains (roughly) how a network room is managed energetically and what are the power problems that can arise in the event of an incident. It also shows that an accident is often a succession of minor incident and that a simple missing check (in this case the automatic world release) can lead to a "catastrophe".
Dear Customer,
After more than 20h on a war footing, we can finally give you some explanation of the major technical incident we met between May 10h 17 and 11 May 15h30 (time or all services are operational in 100% ).
This is an exceptional incident in its nature as in its consequences ...
The summary :
Yesterday to 16h15 the power supply EDF (GEG) of the building has tripped at the cell located outside the building. We do not know (and GEG does not know the precise reasons that the food has cut) ..
In such a situation, the power supply to the servers must not be cut, since the building is secured by 3 large inverters as well as a 400kva diesel generator. The system normally works well since we already had an EDF cut a week ago and it had no impact.
It did not happen yesterday as you have seen.
The Cogent staff (international group that operates this datacenter) intervened at the beginning of the week to carry out maintenance and tests on the diesel generator. The people who intervened have obviously not ironed the group in automatic start mode for it to start as soon as it detects that there is more current EDF.
The result was irrevocable: the group did not start, the inverters were completely emptied and the servers were no longer powered.
Moreover, the site is normally monitored from several noc located in Paris, New York and from Spain so that this type of trouble can be detected very quickly by Cogent and that they can intervene. This time it did not work since there was an incident on the monitoring system for a few days. (which must be settled this day or tomorrow).
PHPNET was present on the site less than 10 minutes after the electrical failure of servers to one hand to start the group electrogene manually to restore the power and secondly to restart all servers.
As a problem never happens alone ... When GEG restored the mains power, the automatic tilt system tried to reboot it and that's where we got our
second electric break ...
In fact, the normal switching procedure is: edf => inverters => eletrogenic group. Conversely, to switch back to edf, the circuit is electrogene goupe => inverters => edf.
The inverters did not have time to recharge enough, the power was cut again for the servers.
To top it off, the EDF power supply again tripped a few minutes after the rocker because the power consumption of the site was too important. The air conditioning units and servers consuming 3 has two times more electricity at startup, consumption is not made after the settings present on the outside of the building, creating a new cut for servers.
So we had to cut the air conditioning and restart each part of the building within a certain time, so as not to redisjtter.
These numerous power cuts caused the loss of many hard drives in the servers and (especially) the loss of several file systems. However, we successfully switched to our backup system last night for a few hours.
The 1 mail server (cluster1) had to be restored to our last backup because its data was not recoverable. The situation is now fixed.
Today the responsibility of this incident must be reported to Cogent who should have made sure that the group was in automatic start mode and especially monitor properly
the data center to intervene before the inverters are empty.
We will initiate the necessary negotiations to obtain financial compensation that will be reflected on your PHPNET subscription up to the cut you have suffered.
Various solutions concerning the layout of a datacenter are currently under study PHPNET because we no longer want to depend on the goodwill of providers like Cogent or Redbus.
We will keep you informed about this project which should end in the 12 months to come.
The entire PHPNET team joins me in thanking you for your understanding and for excusing us for the inconvenience this may have caused you.
Excellent weekend to you,
----
PHPNET
123 ter Course of the liberation
38100 GRENOBLE
) that the server is saturating.