Richard Walker

Thankfully, these lessons haven’t been learned the hard way – not entirely anyway.

I’m not referring to my own web services – I don’t provide any of these. I’m referring to a particular web service I use to create and lodge invoices electronically.

Granted they are working hard to restore their services so I’ll refrain from outing them, but I’ll share a few things I’ve learned, not just with this latest incident but with web services in general.

1. Keep the customers informed. If you’re planning an outage period, or things aren’t going as expected, send a single email to your customer base explaining the planned/estimated outage times, or if applicable, an explanation of what went wrong and what you’re doing to fix it. While your customers might still be peeved, at least they’re in the loop.

2. Try and avoid using social networking tools to announce outages. Twitter is NOT a suitable means by which to communicate outages, as progressive and “nu-skool” as this approach may seem.

3. For the love of God, tread carefully when dealing with DNS changes. DNS, by nature and with regards to change, is a slow, unweildy beast. If you make a mistake and spot it too late (i.e. overnight) the mistake may have already propagated, and fixing it will take just as long (unless you set your TTLs ridiculously low).

4. If you can’t release a product or revision to the product without a prominent risk of failure, just don’t. The old mantra “real programmers ship” is as vague as it is foolish….. I’d rather wait another week for an upgrade I might not even notice, than go 24 hours without a crucial service because something went “bang”. You can still make ambitious release dates, but just put some real time, effort, and most of all, serious and careful thought, into setting up your development and release infrastructure. If you do things right, announcing an outage for purposes of maintenence/upgrade will be a thing of the past… your customers won’t even notice.

5. Prepping a release at 6pm on a Friday evening is a no-no. Your developers are probably burnt out from a week of frantic preparation, and this sets the stage for errors. I’ve been told on numerous occasions that shipping a particular feature/product on a particular date is “important to the business”. What’s more important is your image, and having exhausted developers cram something out the door in time for the upper echelons to nod their heads in approval, and then having said product blow up in your face because of an error that got missed, makes you look stupid in the eyes of your customers. Again: it is always better to ship late than ship broken. As a customer, I’d rather see a polished, functional product than a broken, rushed one. A poorly executed release reflects badly on you and your product. Take the time to do it properly.

That’s it for today’s rant!