HA in a Small Business Environment

I think this comment by deet is spot-on with respect to disaster recovery and it’s place in a small business. We’re talking 100 nodes – not 10,000. He gets it – it’s about keeping your executives working, keeping your sales team selling, and keeping every employee working (inevitably one person without a computer is going to distract the rest).

For us, client redundancy is essential. System goes down, swap it out for “Rerun” (our spare pc) and they’re back up, and running on all systems. Server Redundancy is still in process, but virtualization will be our next step. And don’t forget the ever-so-learned lesson: local backups are the key to quick recovery (online, offsite are essential for catastrophic failure, but restoration takes day(s)).

deet‘s comments below:

Allow me to offer some suggestions that might be more helpful within the context of this article.

First and foremost, in a small business, a client failure is absolutely a high-impact failure. In a large organization, maybe there are 50 accountants. In a small organization, there may be only one. Even if there are three, or five… for any one to be offline all day, or for the master spreadsheet she was working on to disappear, or for the one copy of Quickbooks to crash… any of this and much more can grind the whole business to a halt and do irreparable damage. Remember, they’re all doing more than they should be doing. A loss of one small business employee can be more like the loss of three from a large company. In a small business, users get very upset when things break. You *have* to make it better, or you’re not doing your job.

Moreover, your rapid recovery strategy must be able to handle an extremely heterogeneous bunch. Disk imaging alone doesn’t cut it. You’ll deploy an image, and then they’ll spend two days putting everything back the way they want it. Sure, you’re off to other things, but they’re still complaining to the owner about what a disaster their system failure was. You don’t want that. You want them forgetting to mention that their system crashed, because your recovery effort was so effective.

Laughable? Maybe, in an overmanaged IT department. But in a small business, you can afford to treat users like people. If not, maybe you’re just a jerk.

So here are my top five tips for building a strong small business infrastructure.

1. Put every workstation on a small UPS. SE,SO is one thing, but the kind of rickety accounting software small businesses like to use isn’t forgiving and doesn’t provide the option. Your choice: $35 for a Back-UPS 350, or your accounting manager physically weeping? The choice is clear. Remember to keep all the switches and WAPs on UPSes also, so they actually *can* save their work to the server if the lights blink.

2. Back up your stationary clients every night. Keep the data as long as you can. Use a backup strategy that allows you to restore their system completely… maybe a standard image of the OS, plus a complete backup of the user’s applications, registry, and files. The user must be able to pick up right where they left off after you swap a hard drive or reinstall a clean OS. Big organizations don’t bother backing up workstations, I know. But in small organizations, you’ll find most of the company’s IP and core data live in ~/Desktop, no matter what your systems or policies dictate. I’ve used various combinations of Arkeia, rsync, NTBackup, and good ol’ scheduled scripts to great effect. The first time a client hugs you for restoring a file they accidentally deleted last week, you’ll understand.

Side note: do NOT rely solely upon online backup. It takes days to restore a complete system over the Internet. Augment online backup with local snapshots so you can restore quickly. Putting an external HD at every desk is simple. Get a Time Capsule for your Macs.

3. Upgrade to gigabit Ethernet. Look, let’s be serious — blablabla collapsed core design (jesus christ wtfbbq L3?) — in a SMB, you’re probably dealing with people who still have 10/100 NICs. A Netgear 24-port switch can be had for a couple benjamins, and the NICs are cheap too. Your “cable plant” may not be up to full gigabit spec, but whatever… gigabit is a HUGE improvement even in suboptimal conditions, and you’ll be seen as a magician for cutting someone’s report generation times by half or more. And if your non-redundant switch dies, which maybe it will eventually, you can have a new one bought and installed within a couple hours.

4. Take the server seriously. Okay, if there’s anywhere I’m likely to overbuild, it’s here. But stop a second — go slowly. You don’t want to have to explain to the owner that your redundant-everything beast of a machine has shat all the company’s project data because Newegg sent you a bad card from 3ware. Get something ready to go out of the box from a vendor you trust, whether HP or Dell or Apple or whatever. Use RAID 5 for main storage and mirroring for the boot volume. Back up nightly snapshots to a local external HD — something like a WD MyBook Mirror Edition. Back up incrementally to tape if you can afford it. The server itself doesn’t have to be high-performance at all, but something bought new with full warranty support (don’t void it!) is a minimum. Redundant NICs and PSUs help you solve problems, too. But absolutely make sure the data is protected in multiple ways, on-site and off-site. THIS is what the server is for.

And put it on a UPS with plenty of capacity. I shoot for an hour in very small groups, and 4 hours is achievable if you have a bunch that doesn’t mind working on their laptops in the dark. Journalists, commodities traders, and even some retail/wholesalers all want to keep running during critical business hours. For them, a 4-hour UPS and even a generator are *not* ridiculous ideas. And remember that critical business hours aren’t necessarily 8-5.

5. Use the Internet. Google for Domains does a great job with e-mail, calendaring, and contacts for small organizations. Avoid Exchange for a small company, the licensing costs will have you pulling all kinds of chicanery to add new users or find more storage space, all at considerable expense for a company still trying to find a profit. Don’t try to host it all on the company’s T1 or DSL line, when third party services can do it so much better. Email, calendars, and contacts are available online for free or incredibly cheap. Use these where you can.

6. And one bonus point: Use a domain one day. It can be a challenge when you walk into an office full of machines running XP Home, but keep working toward it. Once you have all the machines on the domain, your ability to solve problems quickly will improve… I’m talking about remote desktop, portable user accounts (“Here, use this temporary machine while I fix yours”), things like printer setup, and of course access to the server. A directory domain seems a bit like overkill, but so few SMBs use them, and so many could stand to benefit.

Look. Running a small/medium business is not about managing policies and infrastructure, it’s about giving people tools to do their jobs. Small businesses are often startups or niche service providers, run by dynamic people who like to move quickly and solve problems with creativity. They don’t like administration, they don’t like being told what not to do, and they don’t like fucking around with computers. They want you there to help them through the rough spots, and they’d prefer the computer not get in their way at all. Know their names. Wear clean shirts and deodorant. Answer their questions plainly and frankly, and remember to show them fun things to do. Be a part of their team. It’s like they say… you’re doing your job well when they don’t know you’re doing it at all.

Source: http://jbickford.com/u/3j