Our servers are hosted by Rackspace, which is a great company with excellent support and top notch datacenters. At 7:15pm CDT yesterday, a severe storm was coming through and Rackspace decided to switch power to generators. During the switch there was a mechanical failure that caused some servers to lose power unexpectedly.

When the servers came back online, we found that our database had become corrupted. Apparently, this is because the database was configured to write data to the filesystem, but the filesystem was configured to flush this to disk every 1 second. During that 1 second, that data was only stored in memory. So when the power went off, that data was lost. When the power came back on, the database freaked out because of that missing second. During this freakout, unknown bad stuff happened and the main database got corrupted beyond repair.

Glad to see they’re back online – a failure like this has taken many companies completely out of business in the past.