| They said it wouldn't happen to them. Or couldn't. | | | | when disaster struck. |
| That they were prepared. That they had taken | | | | Another sad case comes to mind - this time, a |
| steps. | | | | twenty five year old professional services firm. They |
| They were wrong. | | | | did indeed have a rudimentary disaster plan - |
| Computer systems are, with the possible exception | | | | complete with offsite tape storage, and mirrored |
| of phone systems, the 'soft underbelly' of most small | | | | servers at the presidents house. However, when I |
| businesses. You would be amazed at how a fifty | | | | met with them, it turned out that they had never |
| person hedge fund that manages over a billion dollars | | | | actually run a full-scale test of the system; it had |
| in client assets are loathe to spend $25,000 on | | | | been tested by their network administrator in his 'lab', |
| disaster preparedness. Such a trifling sum to | | | | and the machines (which were running Doubletake for |
| guarantee their business in the event of a disaster! | | | | Solomon, Exchange, File/Print, and BES) were shipped |
| And yet I see this miserly conduct happen over, and | | | | to the owner's house and left in his basement |
| over again. | | | | connected to his inexpensive Linksys 'Compusa |
| As an Information Security professional for the past | | | | special' hub. |
| 25 years, with over fifteen years spent at large | | | | I advised a full-on test of the system, and |
| enterprises such as Merrill Lynch and Ernst and | | | | recommended relocating the servers to a hosted, |
| Young, and ten years as an independent business | | | | generator backed facility in a 1/2 rack (for about |
| continuity/high availability infrastructure consultant for | | | | $700/month). I cringed when I heard that the |
| small and mid-size businesses, I've seen a lot of | | | | owner's home was up in the country, never visited |
| solutions that worked - and a lot that haven't. There | | | | by their IT admin, and was subject to occassional |
| are a large range of available options between the | | | | power outages. I also recommended a full-on test of |
| mirrored trading floors that the big brokerage houses | | | | their backup media and creation of a detailed |
| maintain, to the cheap and flimsy usb hard drive or | | | | recovery plan (their 'plan' was to have their key 10 or |
| ancient DLT backup tapes that, sadly, are all that | | | | so employees remote in to their CEO's home |
| pass for 'business continuity' solutions at many firms. | | | | servers). |
| And most good solutions don't need to cost a heavy | | | | Well, the company decided not to spend the money |
| price. | | | | 'at present' and chose to stay with their existing |
| Over the past two years, I've seen six companies | | | | solution, which their NetAdmin arrogantly told me was |
| that I either consulted with, or was speaking about | | | | 'only for the suit's piece of mind anyway'. How right |
| consulting with, go out of business or were forced to | | | | he was! This was clearly demonstrated when, about |
| layoff more than 50% of staff because of bad | | | | six months later, the floor directly above their server |
| planning and bad luck. In all but one of these cases, | | | | room suffered a broken water pipe that quickly |
| they could have avoided such catastrophic losses | | | | flooded down into their server room (floods are the |
| through simple, yet too often overlooked | | | | #1 cause of disasters I have seen over the past 5 |
| precautions. | | | | years) and took out their entire server rack, their |
| The first case that springs to mind, is that of a 10 | | | | phone system, their UPS system, and the room AC. |
| year old investment firm located in Manhattan on | | | | They frantically kicked their 'Disaster Recovery' plan |
| 41st and Lexington Avenue. Two years ago, they (or | | | | into operation - only to find that in the intervening |
| their investors) decided to investigate a business | | | | two years since they set it up the CEO had changed |
| continuity strategy. They had a few bad experiences | | | | internet providers and the static IP addresses they |
| with 9/11 and the NYC blackout, and didn't want to | | | | had set up were no longer valid. So, their plan was |
| get caught short again. So I spent an hour discussing | | | | DOA - no one was remoting in anywhere. And since |
| their DR/BC planning with them - only to find that | | | | the IP's at his house had changed, Doubletake had |
| they not only did not have one, but the one thing | | | | not completed a successful replication in over ten |
| they did have - backup tapes - were not being taken | | | | months (the Network admin later told me he was |
| off site. When I inquired why, they said that the | | | | counting on 'alerts' to tell him if replication failed - only, |
| secretary tasked with this duty often 'forgot', but | | | | they were never setup). Furthermore, the girl who |
| that it wasn't a big deal. Their attitude was - that so | | | | took their backup tapes home every day reported in |
| long as the tapes existed (never mind they had | | | | sick that day, so there was a 1/2 day delay in |
| never tested any of them, nor even checked to see | | | | getting the tapes from her. And finally, when they |
| if their backups actually finished) they could | | | | were sent up to his house in Westchester County |
| 'somehow' recover from a disaster. When I probed | | | | (by this time - two full days later - they had gotten |
| deeper, and found that their actual tolerance to | | | | new static IP addresses) the tape drive at his home |
| complete system downtime was less than 24 hours, | | | | was dead. |
| I realized that these guys needed some help. | | | | So they wasted another day getting new tapes and |
| Well, they decided to put action 'on hold.' Too busy | | | | patching the servers up to present spec. Five full |
| with other projects, they said. Their shortsightedness | | | | days after the flood, they finally had data flowing |
| would cost them big, when less than three months | | | | again (of course, their terminal services weren't set |
| later, a water main burst in the street outside their | | | | up properly, the firewall at his home wasn't |
| building (for those living in New York, you must | | | | configured right, and the slow upload speeds couldn't |
| remember this one - Lexington and the west side of | | | | handle more than about 3 concurrent users of |
| 3rd avenue were closed to cars from 39th st to 41st | | | | Solomon at once). This comedy of errors didn't end |
| for over 2 weeks). Their safety net - their tapes - | | | | with the company going out of business, but it did |
| were trapped in a building now ruled completely | | | | end painfully for them - with almost 30% of their |
| unsafe for entry by Coned and the fire department. | | | | workforce gone since I was last there (they did |
| For 6 days, this company had no access to data, no | | | | eventually contract out for my services though!) |
| servers, no receivables, no plan. Needless to say, | | | | The moral of this story is simple - disaster recovery |
| they suffered - and quite badly. The firm is no longer | | | | and business continuity can be done - and should be |
| in business. All for want of simply taking a tape | | | | done - by all businesses that need their computer |
| offsite, and having some idea of what to do with it | | | | systems to conduct business. |