What CUs Can Learn about Business Continuity and Disaster Recovery from Last Winter’s Fiserv Outage
Neither snow nor rain nor heat nor gloom of night should stay credit unions from the swift completion of their appointed rounds.
As I sit here awaiting the impending quintessential sudden MidAtlantic thunderstorm, my mind wonders to business continuity and disaster recovery – as we all often do.
Potential disasters come in many forms. The heat of summer through early December is prime hurricane season. June through July is also peak tornado season. We won’t soon forget the Colonial pipeline shutdown, thanks to a ransomware attack. New York Times recently reported that California actually has two peak wildfire seasons: June to September and October to April.
And then comes old man winter and the havoc it wrought last year. Many credit union leaders will not forget how – not only did a freak Texas storm bring down the power grid, but it also brought down parts of Fiserv, including mobile and online banking, cards and bill pay, according to Credit Union Times. I serve on the board of a credit union up here in Maryland, and even we were affected.
Credit unions are critically dependent upon their core processors, and your members don’t care why things are down; they care that they can’t access their money when they need it most. Of course, this also happened during the global COVID-19 pandemic, but regardless, your credit union is the one on the front lines.
According to one CEO interviewed by CU Times, she was told Fiserv servers lost power and backup generators ran out of fuel. CU Times cited media reports that Fiserv was physically moving servers out of Texas and into Georgia. “I have lost a lot of faith in what has happened with Fiserv. They obviously don’t have a robust system,” she said, adding, “We are at their mercy.”
Yes and No
James Green, a business continuity and disaster recovery specialist with Origami Risk, said what didn’t surprise him was that of the 241 power plants in Texas that had problems in the cold weather in 2011, 30 of those plants had the exact same problems in 2021, and 340 had problems in general.
Watch the video with James “BCP” Green on YouTube!
“Most organizations try to mitigate the last known risk,” Green explained. “What I saw in doing some research on Texas is they used the 2011 storms as a baseline, and the problem with that is, no one ever asked, ‘What if there’s something worse than that?’ The second thing is a lot of the guidelines and standards that came up from the state of Texas were ‘recommendations.’ And I guarantee what’s going to happen now, because we’ve had loss of life this time around, those won’t be recommendations. Those will be requirements.”
However, Green added, “What did surprise me were some massive gaps I feel Fiserv had in their business continuity and disaster recovery programs.” For example, Fiserv had generators, but it was not a surprise that they run on fuel.
“What did surprise me were some massive gaps I feel Fiserv had in their business continuity and disaster recovery programs.”
“How do you not know how long your generators run?” he questioned. “How are you going to refuel that generator? Most companies have dedicated fuel to run those generators.” He added that their failovers also had problems switching over from Dallas to Georgia. The plan to drive servers in trucks over several states opened concerns about privacy and security. “It was just a really bad disaster recovery plan.”
When disaster strikes, credit unions can take some steps to mitigate risk. In the short term, Green recommended communicating proactively early and often with your credit union’s members. Over the long term, be very aware of the risks your critical vendors present.
Green suggested that credit union leaders put it in their contracts that they have the right to see the vendor’s business continuity plans and disaster recovery programs. Large credit unions can pro ably even throw their weight around to sit in on tests. All credit union should negotiate when renewing their contracts, tighten up the language and, ideally, add financial penalties for outages.
Ncontracts, a well-known vendor management firm, acknowledged that while Super Storm Sandy was not predictable, institutions that were very diligent about business continuity planning might have avoided further problems. “An agreement between regulators and the [unnamed] company after the fact revealed ‘unsafe and unsound practices relating to the [telecommunications service priority] disaster recovery and business continuity planning and processes.’” Ask the right questions before getting into bed with a third-party, so your credit union can be better prepared.
On top of the standard due diligence necessary for every business partner, credit unions should also thoroughly examine critical vendors’ business continuity plans to understand how they align with the credit union’s planning, including:
Third-party capacity. If a vendor faces disaster, your institution probably won’t be the only one affected. That’s why it’s important to determine if the vendor has the capacity to restore every client within its recovery time objectives and recovery point objectives. In the event it cannot quickly restore services, the vendor should have a workable agreement lined up with an alternate provider—or else the institution must find its own backup vendor as part of its BCP.
Third-party management. Just like financial institutions, many vendors outsource activities to service providers. These subcontractors must also have effective BCPs. The prime vendor should regularly review them and conduct its own due diligence—otherwise, your institution will have to do it. Ultimately, regulators view it as the institution’s responsibility.
Cyberthreats. From malware to distributed denial of service attacks (DDoS) to insider threats, vendors must be able to respond to cyberattacks and have an actionable incident response plan. They also must stay on top of emerging threats. This is particularly important for vendors using the cloud.
Testing. Guidance strongly encourages regular testing of vendor’s business continuity plans and examining the results to identify potential problems. Before signing a contract, be sure to ask for the results of the vendor’s last business continuity test, especially for critical vendors.
Who Ya Gonna Call? Wescom Resources Group offers managed services and business continuity and disaster recovery for Corelation’s Keystone and Symitar clients.
Credit Union Business Partner Case Studies
Open Lending
Open Lending CTO Sarah Lackey said the company maintains a detailed disaster recovery and business continuity plan reviewed and updated annually. “Given the significance of these policies, we’ve built SOC 2 controls around the review of these policies and include semi-annual testing of our disaster recovery plan,” she said. “Additionally, we maintain an Emergency Action Plan for any incidents that occur in the office during business hours and an Incident Response Plan.”
The Austin, Texas-based company’s (NASDAQ: LPRO), business continuity plan designed to consider many different types of disasters, including natural disasters. The policy outlines communication plans, key stakeholder contact information and basic protocol for each type of incident.
“Communication is key during a disaster. We’ve taken a few simple steps to help ensure proper & timely communication,” Lackey said, including:
Skilled volunteers are identified annually to act as the “Emergency Response Team” when a disaster is declared, and it is their responsibility to execute protocol in our BCP
Internal stakeholder email distribution lists are created, maintained, and documented in the BCP
Our product, Lenders Protection, includes a feature that can generate an alert to all users of the system if ever needed.
Third-party contacts (including first responders) are reviewed, tested, and updated annually
Open Lending’s disaster recovery plan also considers multiple disasters occurring at one time, including catastrophic failures should they arise.
“The semi-annual disaster recovery testing is a critical factor to the success and quality of our disaster recovery plan,” Lackey added. “Over the years, our test results have inspired many efficiencies and automated enhancements to our disaster recovery process.” She emphasized the company’s business continuity plan is heavily focused on security, data integrity, availability, and decisioning response times. If Open Lending’s Lenders Protection is not affected by disaster directly, core business continues with minimal impact.
As a result, even though the Open Lending offices were without power and water for days, there were no storm-related disasters with our datacenter that affected the availability or performance of Lenders Protection. “Core business functionality was not directly affected by the storm,” Lackey reported. Given the lack of impact on the Lenders Protection platform, client communication was limited, but “absolutely” part of the plan.
Ser Tech
The Dallas-based Ser Tech was another company hit by the 2021 snowstorm in Texas, but its business operations were not affected. Shana Richardson, CEO and co-founder of Ser Tech, explained that Ser Tech is on a Tier 1 power grid – meaning it throws in some big bucks so the company can be on the same power grid as hospitals and other essential services in Texas.
“We understand the critical nature of credit data services to our credit union clients, and we’re not about to let them down,” Richardson said. Ser Tech’s business operations also were not affected during the Texas blizzard last year.
“Ser Tech was already operating remotely because of COVID-19,” Richardson explained. “While some of our employees were personally without power during the storm, that meant our company was still up and running to serve credit unions and their members in dire need. We ensured our employees were cared for and continued with business as usual.”
Credit Unions’ Responsibility
Ultimately, when the member is complaining about an outage of any sort, it falls to the credit union. And that presents a reputational risk, so force the issue with your business partners. No company is bulletproof, so know the risks involved – intimately – and have a plan.
“In the credit union industry, because of our ties to the community, we must consider, ‘What are our responsibilities to the community?’ In some of our members’ most critical time of need, we let them down.”
Green pointed out that Fiserv was Chick-Fil-A’s card processor, and that massive company made the decision to give away meals and offer coupons within an hour. “In the credit union industry, because of our ties to the community, we must consider, ‘What are our responsibilities to the community?’” he said. “In some of our members’ most critical time of need, we let them down.” No water or heat and no money to buy it or move out of the area.
Not to mention, it can be a public relations nightmare. Just ask these credit unions that were called out in the local, mainstream media:
New Bedford Credit Union (which struck back at Fiserv very publicly)
Green noted that some credit unions deployed pop-up branches and other extraordinary things to serve member, but we need to see more credit unions outside of areas regularly hit by storms ready for action should disaster strike. He asked credit unions, “How can we be more prepared to help members in worse case scenarios because we’re seeing those worse-case scenarios?”
They’re not just one at a time anymore, Green noted, but we had the pandemic, ransomware, civil unrest, winter storm. Credit unions need to be able to handle two to three disasters at a time. Wise words as COVID-19 has started spiking again in some areas.
