On 10th late night (LKT), we found one HDD on DS3 RAID 10 showing a SMART failure, our DC staff replaced the HDD and put the RAID 10 on re-sync, but white the RAID was rebuilding we saw another HDD showing SMART fail error, DC staff added another to replace it. but after 10 min, the server totally failed. server didn't even come on to POST, its a dead giveaway for a bad motherboard. our RAID was on board think this was causing the HDD errors. at this time we lost all communication with the server. DC staff started putting back another identical server with the OLD HDD's but all the data is corrupted, so we added new HDD and started a fresh install. all new hardware and OS + Control cPanal installation took almost 3 hours.
Then we started the pain full and time consuming process of restoring all the data from our backup, this server had more then 1500 customer domains. it took almost 24 hours to full come back. but some site came up after 4 hours from the initial hardware failure. since data get restored alphabetically.
We had scheduled plans to migrate this server to new hardware during the end of April, 2013. but server fails on us before that. ORBITnet has a policy not to keep a servers for more then 18 months. even tho now the DS3 is back on new hardware, We still want to move with the hardware upgrade, as we always double the server hardware on each upgrade. such measures are taken to improve performance and reliability. We will update you on this and during this migration there will be 0 downtime.
We would like to apologize to our customers for letting them down! how ever we will learn from this and take action to prevent such from happening in the feature.
Thank you for all the support and understanding!
Dear Valued Customer,
One of the hosting servers [ds3.extentions.net - 22.214.171.124] has failed with multiple hardware failures last night, this server was schedule to be upgraded to new hardware during this month but the server has given up before that.
We are now working on having the server up and running, the new installation of the hardware is completed and now in the process of restoring data. there are no data loss.
It would take few more hours [3-4 hours] but depends on many other factores. each site will come online as soon as the data get restored on that given account. once all the accounts are up, we will schedule a date to move all the data to a new upgraded hardware but this process will not have any down time.
we thank you for the support and understanding your giving us during this time.