Another day, another online merchant suffers a data loss. Love My Shoes‘s web server had corrupt data, which then trickled down through the raid disks and made it all the way to the backup set before it was realized.
Last week we suffered a serious failure to our web server. Upon investigation, corruption at the file system level propagated to all the redundant raid disks in the server. This corruption was later backed up to tape. Unfortunately didn’t catch this corruption until it had overwritten a week’s worth of backups, by which time the operating system was so corrupted the machine failed to load completely.
The damage was limited, unfortunately and fortunately, to our product imagery only. We have been working diligently these last couple of days to restore whatever images we could not recover from our fall back data. It has been a lengthy and manually intensive process. All primary photography has been restored. We are now working on restoring the alternate color photography.
We’ve been saying time and again that having a backup plan isn’t enough, if you can’t restore the right data when you need it. Love My Shoes goes on to say, We have implemented a more staggered approach to our backups to avoid a repeat of the above scenario.
Luckily Love My Shoes has been able to piece back together the corrupt data, and their site is now back up & fully operational.
