News. Go figure.
So with all that traffic, I don't suppose I was the only one put out when Blogger had an aneurism last week. That's not a technical term, but I think it conveys the big picture. Time Magazine's Techland reported on Friday in Outage Silences Blogger Blogs that:
Bloggers at Google-owned blogger.com were locked out of their blogs for a number of hours overnight, apparently after a behind-the-scenes update that went awry.
The outage continued for more than twenty hours, during which posting and commenting was blocked. Content posted on Wednesday and Thursday disappeared, among other disconcerting rips in the fabric of the intertubes. Google promised the missing content would be restored, but, well ... you know ... one hoped for the best. As far as I can make out, much of the affected content does appear to be back on-line as of Sunday evening, though Blogger's status updates have not yet declared that data is fully recovered. I am aware of at least one blogger whose posts are back on-line but whose comments are still MIA.
Here's Google's not-quite-post mortem explanation, posted on Friday morning as Blogger is Back:We’re nearly back to normal -- you can publish again, and in the coming hours posts and comments that were temporarily removed should be restored. [...] Here’s what happened: during scheduled maintenance work Wednesday night, we experienced some data corruption that impacted Blogger’s behavior. Since then, bloggers and readers may have experienced a variety of anomalies including intermittent outages, disappearing posts, and arriving at unintended blogs or error pages.
A couple hours after that announcement this blog was still unrestored, and when it did come back my latest post was incorrectly timestamped and its labels had been concatenated into gibberish. No big deal. I fixed it Friday evening.
But, as Ed Bott argued on ZDNet, Google's Blogger outage makes the case against a cloud-only strategy. I've got to agree with him there. Here's the key concept as Bott put it:
If your data matters, you need a hybrid strategy, with local storage and local content creation and editing tools. If your local storage fails, you can grab what you need from the cloud. If your cloud service fails, you’ve still got it locally. But if you rely just on the cloud, you’re vulnerable to exactly this sort of failure.
In September of last year, I posted a blog titled Safeguarding cloud ephemera Part II: keeping your blog alive. In it I suggested several concrete strategies for backing up products of the many hours bloggers put into their posts. I summarized with this:
Whatever way you save your blog posts, backup matters. For your digital copies -- whether in blog export format, e-mail, word processing formats, etc. -- be sure you take the same kinds of precautions with the data that you would with any other file(s) you hope to keep beyond the life of your current computer's hardware. Back it up. Save it in a safe place, on a device that you will be able to read into the future. When technology changes, it's your responsibility to move data to a format or device that the new technology can read. If you delay this chore, it can become onerous or impossible, as my experience converting a pile of near-obsolete 5.25" floppy diskettes showed me earlier this year. There are no magical solutions to this problem ... letting a "cloud" provider safeguard your data works until it doesn't; and that nifty floppy / CD / DVD / Zip disk / external hard disk / flash drive will become obsolete in two or five or ten or fifteen years. Bank on it.
Did I get my knickers in a twist when Blogger went bye-bye last week? Nope. A friend and frequent commenter on this blog e-mailed on Friday, wondering what happened to Thursday morning's post. He wanted to comment on it, he said, but all of a sudden it had gone missing. "I'd say something here about 'I hope you have a copy,'" he wrote, "but I know you well enough to know that there at minimum there's one copy, possibly multiple versions!" He was right. I take my own advice.
By the time this post hits the intertubes, most bloggers hosted on Google infrastructure will have been made whole, or whole enough to repair the damage without too much trouble. I hope that doesn't leave too many feeling too complacent.
Technology breaks. It's always best to hedge one's bets...
Related posts on One Finger Typing:
Google yanks APIs, developers caught with pants around ankles
Moving one's life to the cloud
Safeguarding cloud ephemera Part I: the big picture
Safeguarding cloud ephemera Part II: keeping your blog alive
Posts
Theres always data insurance! But usually companies willing to invest in data insurance are also investing in data redundancy. Even when utilizing a cloud technology most do have backups. It doesn't seem terribly likely that google would leave critical code like this without at least single redundancy. I would venture to say that if indeed it was data corruption, it was not due to lack of redundancy. An amusing, yet slightly plausible explanation: a disgruntled employee damages the code, updates the repository, and somehow gets the repository to delete all previous versions. If the redundant repositories are updated in real time, or even slightly after the malicious deletions, the code is permanently lost. In this scenario google may fight tooth and nail to prevent people from knowing it was done intentionally by one of their employees. Of course, they'd also be trying to do some major damage control.
ReplyDelete@A.R: It's true, companies who have a cloud strategy often have strategies to protect their investments. It's bloggers without an IT staff who face a lot more risk of losing their work.
ReplyDeleteYour speculation about a disgruntled employee is interesting as much because it gives a peek behind the data farm's curtain as because it might actually be so. It's easy to forget that huge, faceless companies with multi-billion-dollar valuations on Wall Street are staffed by ... wait for it ... people! And every one of those people is subject to frailty, fallibility, and every flavor of pissed-off that might lead an employee to want to muck up her/his company.
One bad apple...