We have all heard of organizations, large and small, that have suffered cyber-attacks. Some of the highest-profile cases might even make the mainstream evening news. Investment management firms are no exception: advisers, hedge funds, private equity and venture funds face heightened cyber risk and regulatory expectations, and a single incident can disrupt trading, fund operations, and investor communications. The news cycle moves on, but what are the immediate consequences for companies that suffer from a cyber-attack?
Consider a situation where a ransomware attack has locked you entirely out of your systems. Has your IT team or service provider ever tested the process of recovering from ‘bare metal’? In such a case, the first challenge may be to find a ‘clean,’ uncompromised system that can be used to safely access the backups.
At the same time, there will be many decisions to be made that will affect the time, effort and expense required to restore operations.
- Should we just pay the ransom? The legality and ethics involved in this decision are quite complex and will depend on your specific circumstances. Consider instances where your business is impacted because of an attack on one of your vendors supporting key business processes. In such a case, the option to pay the ransom may not even exist.
- Are we required to notify law enforcement or other regulators? Such notifications can be mandatory but may also make the incident public, potentially leading to reputational and regulatory exposure.
- Should we engage with our cyber insurance carrier? Notifying your carrier will set in motion a specific chain of events. Legal counsel, which may include a ‘breach coach, and digital forensics firms will be engaged through the carrier. How will you support day-to-day business operations while this process plays out?
- What is our communications plan? Who is authorized to communicate on behalf of your organization to employees, customers, vendors, partners, regulators, law enforcement, or the media?
Tabletop exercises that allow team members to go through various scenarios in a controlled environment can provide valuable training and practice to support this decision-making process.
The technical and logistical challenges will become clearer as the IT team assesses the damage to the systems. Here once again, having a documented playbook that has been practiced and tested will pay dividends. This is not the time to ‘make it up as you go’. Some examples of critical technical decisions include the following.
- Can we verify that there are backups that were not affected by the attack? Backups need to be offline, or else it is quite possible that the ransomware will lock out the backups as well as the production systems.
- Are the backups encrypted, and do we have the necessary software and encryption keys we need to restore? More than once, we have seen companies that stored their backup keys on their production systems. The same ransomware that locked them out of their production systems is also denying them access to the keys they need to restore the backups.
- Do we have everyone with the right skills and knowledge to recover the operations? While it would be great if the attack took place at 10am on a Wednesday when none of the technical staff or leadership were on vacation, this is rarely the case. Eliminating or reducing ‘key man’ risk is critical. If ‘Bob’ is the subject matter expert on the accounting system, who is Bob’s backup if he is not available?
- Are the procedures for rebuilding and recovering systems documented and tested? Clear, well-defined procedures can allow a broader range of team members to support technical recovery efforts.
So, let’s say that you do have backups and they are available. The team is assembled. Now what? Where do you restore the backups to? What system do you use to access the backups if all of your systems have been encrypted by ransomware? This is where some of the decisions made in the first few hours may reduce or expand your options. For example, if you have engaged your cyber insurer, the forensics team that they may engage might require that systems be preserved for forensic review. That makes those systems unavailable for recovery, which places the acquisition of new replacement hardware on the critical path of the recovery process.
Let’s assume that we have the hardware available, but that hardware is infected by the ransomware. How do we put these systems into a ‘known clean’ state so that we can safely start restoring our backups? Typically, this begins with resetting the system that will support the recovery process to factory default before installing a clean copy of the operating system (Windows, MacOS, Linux, etc.). Options include reverting to an on-premises host or hosts that can be used to provision virtual systems as needed. The other option is to leverage one of the cloud service providers, such as Microsoft, AWS, or Google, to support the process. Both approaches have been proven to work in practice, and both require one or more ‘known clean’ laptop/desktop ‘workstations’ to work from.
In practice, whether the environment is on-premise or in the cloud, the critical point is that all systems used in the process must be ‘known clean’. This includes not only a new version of the operating system, but endpoint protection software that will be used to both protect these systems and to inspect the backups as they are restored. Think of it as building a ‘clean room’ or staging area where backups are restored into quarantine and then inspected for signs of ‘infection’ before being transitioned into the new production environment.
The new production environment must be established to also be ‘known clean’. The same process of installing a clean version of the operating system, security software is required for all systems that will attach to the ‘new’ production environment. Here is another potential decision point. If the original production environment that was compromised by the attack included old, unsupported hardware, operations systems or applications that contributed to the success of the attack, then now is the time to bring these systems up to date. Restoring outdated and therefore vulnerable systems will be self-defeating. Sometimes this is easier said than done due to many possible factors. This is where experience counts in assessing other possible ways of mitigating the risks posed by end-of-life, unsupported systems.
Once the new secure production environment has been established, the process of restoring the backups through the quarantine becomes a matter of repetition. This operation typically continues 24×7 with constant verification and oversight to make sure that the new ‘prod’ stays secure.
Closing Thoughts
Benjamin Franklin is often credited with the aphorism ‘Failing to plan is planning to fail’. This still rings especially true with respect to recovering from a cyber-attack. Teams that have never actually tested their ability to recover files, systems, networks and all of the other technical components that support day-to-day business operations are highly unlikely to succeed in a real-world scenario. Cyber resiliency is a business problem, the scope of which extends beyond IT into every aspect of business operations.
The average length of time to recover from a cyber-attack is 19 days, with costs around $5M. Consider also the impact of such a disruption to your business in terms of reputation and trust.
Governments and large organizations around the world have developed and implemented significant capabilities to monitor and detect malicious activity in their networks and systems. CBIZ Managed Security Services (MSS) makes world-class security monitoring accessible to organizations of all types and sizes. For investment advisers and fund managers, this model helps safeguard trading and valuation systems, protect LP and investor data, and demonstrate strong cyber governance to regulators and due diligence teams. This model leverages the investment in staff, experience and expertise of the CBIZ Security Operations Center (SOC) to deliver a solution within reach of a much wider range of companies.