Business

Providing Web Services – Lessons Learned.

Posted in Business on June 22nd, 2009 by Richard Walker – Be the first to comment

Thankfully, these lessons haven’t been learned the hard way – not entirely anyway.

I’m not referring to my own web services – I don’t provide any of these. I’m referring to a particular web service I use to create and lodge invoices electronically.

Granted they are working hard to restore their services so I’ll refrain from outing them, but I’ll share a few things I’ve learned, not just with this latest incident but with web services in general.

1. Keep the customers informed. If you’re planning an outage period, or things aren’t going as expected, send a single email to your customer base explaining the planned/estimated outage times, or if applicable, an explanation of what went wrong and what you’re doing to fix it. While your customers might still be peeved, at least they’re in the loop.

2. Try and avoid using social networking tools to announce outages. Twitter is NOT a suitable means by which to communicate outages, as progressive and “nu-skool” as this approach may seem.

3. For the love of God, tread carefully when dealing with DNS changes. DNS, by nature and with regards to change, is a slow, unweildy beast. If you make a mistake and spot it too late (i.e. overnight) the mistake may have already propagated, and fixing it will take just as long (unless you set your TTLs ridiculously low).

4. If you can’t release a product or revision to the product without a prominent risk of failure, just don’t. The old mantra “real programmers ship” is as vague as it is foolish….. I’d rather wait another week for an upgrade I might not even notice, than go 24 hours without a crucial service because something went “bang”. You can still make ambitious release dates, but just put some real time, effort, and most of all, serious and careful thought, into setting up your development and release infrastructure. If you do things right, announcing an outage for purposes of maintenence/upgrade will be a thing of the past… your customers won’t even notice.

5. Prepping a release at 6pm on a Friday evening is a no-no. Your developers are probably burnt out from a week of frantic preparation, and this sets the stage for errors. I’ve been told on numerous occasions that shipping a particular feature/product on a particular date is “important to the business”. What’s more important is your image, and having exhausted developers cram something out the door in time for the upper echelons to nod their heads in approval, and then having said product blow up in your face because of an error that got missed, makes you look stupid in the eyes of your customers. Again: it is always better to ship late than ship broken. As a customer, I’d rather see a polished, functional product than a broken, rushed one. A poorly executed release reflects badly on you and your product. Take the time to do it properly.

That’s it for today’s rant!

Tags: Staying Awake, Why HCG, My Provigil

Second-rate IT hardware and your business.

Posted in Business, Hardware on November 17th, 2008 by Richard Walker – 3 Comments

If there’s one valuable lesson I’ve had to learn and re-learn over and over again, it’s that a business, fledgling or otherwise, simply must not skimp on some pretty crucial IT components…. so I’ve made a list of what sort of things the new or established business should look at prioritising, tech-wise (in no particular order):

Power protection. This is simply a MUST. There is no excuse for second-rate hardware, as a failure here has the potential to do a lot of damage, most importantly to things you DON’T skimp on, or even worse, your data.

A decent business-grade-or-better UPS is a must, and if you can’t verify it’s good for at least a thousand-amp surge, get a surge protector that is. I have one plugged into the wall near my feet that cost me $30, and has a $25,000 connected equipment warranty. From memory, it’s good for about 1,300 amps of current.

Whichever UPS you choose, it is essential that it has the capability to connect to your hardware via RS-232 or USB, so that your equipment can get the notifications it needs to start a clean shutdown. A 30-minute battery life is utterly useless during a 35-minute power failure, if all that happens when the batteries run dry is that the equipment goes down hard anyway!

If you can cover every piece of computing equipment in the office, so nobody goes down in sheets of flame when the power goes out, even better. Expensive, but in my opinion, downtime is even more so.

Decent networking gear. A little desktop 8-port switch running all your crucial bits isn’t really that much of an unsafe bet, but if it fails (and a fair percentage do, even if it’s never happened to you) you’re left with nothing tying your network together, and a bunch of workers twiddling their thumbs while you feverishly run off to Harris Tech to buy a new switch.

Get something reasonable. Doesn’t have to be a monstrous 48-port managed beast from Cisco with fiber connections etc, but get something rackmountable, 24 ports to allow for future expansion, and a decent brand…. I’m looking an unmanaged solution from Cisco square in the eyes here, but if it’s beyond your price range, go for something like Asus or D-Link.

On the topic of networking gear, a business-grade ADSL modem (assuming you have ADSL) is absolutely crucial. Again, see the point about failure: workers twiddling their thumbs while you hike to the nearest computer shop for a new modem. Consumer-grade modems often have problems dealing with a “larger” (5 or more people, certainly not large) internal network, as NAT tables etc tend to get overrun, and they will simply fall over and twitch/sputter/whimper if you happen to get (D)DoS’d. I’ve recently stumbled upon Draytek as a pretty good brand (although it got fried thanks to point #1 not being observed previously), but Cisco would be the definite preference…. again, if it’s out of your price range, Draytek is a sure bet.

Don’t even think about bringing in something you pulled off the shelf at your local computer store for $99.

Redundancy. $1,000 for your entire server infrastructure might look like awesome savings on paper, but you won’t be thinking that when a month’s worth of data goes down the tube because a hard drive dies. Think about how long it took X number of people to do all that work, and how much it cost to pay them…. weigh that up against paying a few thousand extra for a business-grade server with redundant storage, along with a backup option (tape preferably). It’s an easy choice, really…. pay $5,000 now for a business-grade server stacked inside a rack cabinet with a business-grade UPS, and a RAID solution. RAID5 is my personal favourite, as alongside the data protection, you get downtime protection too… if a drive dies, the lights stay on and you just replace the drive. It’ll crunch for a few hours rebuilding the array, but no-one else in the office going about their daily duties needs to be any the wiser…. and that’s exactly the way it should be. Hardware RAID5 is better….. don’t put your data’s fate in the hands of a generic software-based solution.

Guys who know their stuff. I’ve heard of too many places that get a “consultant” in for a few days to build them a server and then bugger off and leave it to its own devices. If your company or business model doesn’t warrant in-house IT talent, think seriously about hosting everything (email, web etc) offsite and then striking up a good relationship with an IT consultant who’ll work by the hour, and have a support contract in place…… you might not need him/her for 6 months, but when you do, he/she is an hour or so away and can get you back up and running ASAP. And you’ll have a nice fileserver running onsite with the peace of mind in knowing that it can get fixed same-day by your IT consultant of choice.

If you are the sort of business who needs IT talent, then make sure you match up your infrastructure with their skillsets….. it can be tricky, but find yourself a jack-of-all-trades who’s savvy with all aspects of your infrastructure, whether you’re a Linux-based or Windows-based shop. Make sure they’ve got the right equipment, and that they’re happy to kick in a few outside hours (weekends, nights etc) to keep things running or to fix things when needed. A 9-5 clock-puncher who disappears off the face of the earth between 5pm and 9am isn’t going to fix a 24/7 system when it goes down out of hours. Get some sort of monitoring set up so that they get that irritating SMS at 4 in the morning telling them that something is broken. If they’re good at what they do and take their job seriously, they’ll haul their butts out of bed and fix it.

Equipment for everyone. You don’t need to buy a $4,000 gaming rig for the guy who works at spreadsheets all day or writes content…. but make sure the equipment you do buy is good quality. Spend a little extra cash on good quality monitors, slightly-better-than-average componentry and a comfy, ergonomic chairs, keyboards and mice. You’re not trying to run a sweatshop and boiler room, so treat your staff with respect and make sure they enjoy using the equipment that makes the business earn money! More importantly, you don’t want a graveyard of broken-down computers and computer parts filling up the server room.

So!

That was a (by no means exhaustive) list of some of the things you’ll be looking at purchasing/providing when setting up or expanding your business, and in my opinion, they are things that should not be skimped on. If you look at that list and think “well, chairs, servers, computers… what else IS there??” that’s kind of the point….. it’s all the equipment that makes your business function (and keeps your staff comfortable), and essentially is what brings money into the business…. ergo it’s entirely justifiable to spend a bit of extra cash making sure you get the goods. And not the junk.