Disaster recovery. With 9-11 and Hurricane Katrina now part of our national psyche, it’s a term that makes most of us instantly uncomfortable.
But, increasingly, disaster recovery planning is getting its due. There’s even corporatespeak for it: DR.
“Education and awareness are higher than they’ve ever been. It’s getting a lot of buzz,” notes Ben Thornton, disaster recovery and business continuity planning director for Optimus Solutions, an information technology solutions provider in Norcross, Ga.
“You’re seeing a lot more people dealing with it. But are we doing a great job? I don’t think we are yet, although financial institutions because of government regulations are doing a better job than most industries. But they can do an even better job,” he adds. “And they need to. Because it’s not a question of if you have a disaster, it’s when.”
The key to disaster recovery, according to experts, is communication. Cell phones, buddy branches, calling trees, emergency 800 numbers, alternate recovery centers — everything loops back to keeping the communication lines open.
As David Schirm, vice president of business continuity planning for Raymond James & Associates, puts it: “What 9-11 crystallized for us is just how dependent everyone in this industry is, not only on each other, but on critical infrastructure: telecom and power. The absolute truth is that communication is by far the most important element of being able to recover.”
In a recent nine-page “Notice to Members,” the National Association of Securities Dealers released a series of recommendations based on lessons learned during the back-to-back hurricanes, Katrina and Rita in 2005.
Schirm got it right. The vast majority of NASD’s “guidance” to financial services firms deals with staying in touch — advisors with their home offices, their colleagues and their clients. One interesting heads-up: Across the board, firms surveyed noted that text messaging proved surprisingly reliable as compared to the use of cell phones or land lines. In some cases, text messaging was the only reliable way to communicate with colleagues for a period of weeks. [See sidebar.]
Jimmac Lofton was a branch manager for Raymond James & Associates in Boca Raton, Fla., when Hurricane Wilma pummeled his building in 2005. He and his colleagues set up shop in a buddy branch for 12 days in what turned out to be a pretty seamless operation. “I’m pretty proud of the fact that we had the capacity to get back online and not let our problem become our clients’ problems,” he says. “All you need is a decent Internet connection.” What Lofton learned is that he could have done an even better job had his home been equipped differently.
“Had I been better equipped at my own home, I would have been in business just hours after the storm. I’ve got it all now — electric generators, wireless Internet,” says Lofton. “I’ll be ready for it next time.”
In today’s 24/7/365 world, technology is often the problem — and the solution to swift disaster recovery. That’s why most firms have one or more alternate data centers that can kick in if the corporate home office is compromised. But recovery is also about the simple things — like a contact list.
More than 18 months after Hurricane Katrina struck, LPL advisor Shelby McIntosh says business still isn’t back to normal in his office in suburban New Orleans. He has recommended that all of his colleagues get a master list of emergency contact numbers from each client, and that advisors provide the same. “We were months making contact with people. Everybody was so upside down,” he says. “It was a nightmare trying to find everybody. People were everywhere, just not here.”
Disaster recovery planning isn’t new to the highly regulated financial services industry but it has taken on a new life since 9-11. When treated as a “best practice,” it’s also being continually tested.
Commonwealth Financial Network, for instance, will likely institutionalize this year a procedure that calls for corporate employees — folks on the advisor tech help desk, in new accounts, cashiering and trading, as examples — to work periodically from home.
“We want to make sure systems are up to snuff, that there is connectivity and flow,” says Darren Tedesco, the firm’s director of business systems and strategic development. “If an event happens, our home office employees will be prepared to accommodate.”
The business continuity team at Wachovia Securities, meanwhile, plans sometime soon to test its recovery plan — unannounced. The drill will include the firm’s technology capabilities, facilities and partners.
“The business is always changing; so is the environment,” notes Barbara Bower, Wachovia Securities’ business continuity manager. “Technology changes, the world changes. That’s why a recovery strategy is like a living document. You’re always trying to update it. As businesses change, their recovery strategy has to change, too.”
Planning For Disaster
Disaster, of course, is a relative term that can cover a lot of territory — everything from a computer system crash to a power outage caused by ice storms.
In terms of planning, experts recommend taking inventory of your critical processes, then identifying potential hazards. “Once you define the hazards, you list the things that could happen and what you would do,” says disaster recovery consultant Andre Sharp, who heads Sharp Information in Hermosa Beach, Calif. “The idea is not to be prepared for every eventuality. The idea is to be prepared for most of them. Once you have a plan in place, you are prepared for just about everything. Without a plan, you’re going to be lost.”
Thornton describes “disaster” as: any circumstance in which my business can’t perform its critical business function within an acceptable period of time. When consulting with clients, he begins the disaster recovery conversation with a single question: What is most critical to you? Next, he asks: What constitutes an “acceptable” period of time?
As part of the process, Thornton identifies “single points of failure” that could compromise a business, then moves to eliminate or mitigate them. “Basically, you want to harden the environment to make it less susceptible,” he says.
As an example, if a firm has several call centers, in the event of disaster it might consider collapsing them into one — probably saving money in the process. But, as Thornton puts it: “What risk have you created? Because now you have a single point of failure in a place where you’re touching a lot of clients. You may make the same decision to consolidate, but you may want to think through an alternative.”
Five months ago, Wachovia’s New Orleans branch office celebrated its grand reopening — recovered, finally, from Hurricane Katrina. The bright blue carpet and fuchsia counters are gone, updated with a more contemporary color scheme. The furniture has been upgraded. Walls have been moved to create nice, large window offices. Best, perhaps, the mold has been eradicated.
Seventy percent of the branch on the 24th floor of a New Orleans high-rise had been destroyed by Katrina.
“It’s like almost anything else in life, something happens, something tragic, and life stops for some period of time,” says Billy Blakeman, complex manager in New Orleans for Wachovia Securities’ private client group. “Then you say ‘OK’ and you slowly return to normal. In general, this was a stoppage of life as people knew it. Even today, some people are trying to figure out how to get parts of that life back.”
All these months later, Blakeman still ruminates about what went right, and what he would do differently if there is a next time.
To the good: With phones down, Wachovia within 24 hours was advertising a special toll-free number at its Richmond headquarters on radio stations so that clients and advisors could call in. The firm also set up a website to facilitate client contact. The branch didn’t lose a single client record because everything is imaged online. Corporate headquarters meanwhile acquired hotel rooms for employees and sent a supply truck with tarps, food, water and 54,000 pounds of ice to the Lafayette, La., branch, which housed many of the displaced New Orleans employees.
If it happens again, Blakeman says he would be sure to have cash to distribute early on to employees. “After the third day, people were coming in wearing the same T-shirt.” Wachovia did give $500 to each employee to help cover expenses. Another recommendation: Program Wachovia’s toll-free disaster assistance number into everyone’s phones. And, just to be on the safe side, Blakeman recently put protective film stripping on the downtown branch’s windows — just in case.
After Hurricanes Katrina and Rita in 2005, the National Association of Securities Dealers surveyed selected firms in Louisiana, Mississippi and Alabama about how their business continuity plans stood up.
NASD recently released its findings in Notice to Members 06-74. Experiences varied, depending on the firm’s size and preparedness. Notably, firms with well-tested business continuity plans faced minimal disruption, regardless of size or proximity to the event. Among the highlights:
o Some firms reported it was challenging to identify and “verify” clients following the hurricanes. These firms acknowledged that they had “underdeveloped” client identification procedures.
o Some small, medium and large firms experienced problems at their respective back-up/recovery sites as a result of untested servers, untested systems, inadequate access to systems or inadequate capacity.
o Some firms determined that portions of their business continuity plans were incomplete or out of date. Some plans, as an example, contained outdated employee and client contact information.
o Firms that relied heavily on paper records experienced the loss of irreplaceable documents and critical business information.
o Some firms recommended shipping in cell phones that have area codes outside the impacted regions as they proved more reliable than cell phones with local area codes during and after the storms. Others found that having pre-loaded laptops with wireless cards or laptops shipped in by a parent company or clearing firm provided significant assistance in re-establishing continuity of operations.
o Having a website with screens for check-in, updates and postings for employees aided communication and coordination.
o Firms that intended to rely on call forwarding through local switching stations found that stations impacted by flooding could not re-route calls. These firms suffered from the inability to contact, or be contacted by, clients and employees.
o It’s critical that employees understand how to remove hard drives from desktop computers so that valuable information can be preserved even though hardware is lost.
o Some firms encountered two issues that posed significant employee-related challenges: employees refusing to leave the impacted region and the failure to quickly secure long-term office space and employee housing in alternate locations.
Ellen Uzelac is aBaltimore-based freelance writer and contributing editor of Research.