By Tim Brown, CISO and Vice President of Security, SolarWinds
Saturday, December 12, 2020, is a day I’ll never forget. That was the day I learned nation-state threat actors had exploited our software in what would later be known as SUNBURST. Because it’s been written about thousands of times before, I won’t rehash the particulars of the event itself here. Instead, I’d like to share four lessons I learned about how to respond to a large-scale cyberattack.
1. The first days: Preparation helps control the chaos
I often refer to the days immediately following December 12, 2020, as “controlled chaos.” The chaos portion is self-explanatory, but what about the “controlled” part?
Simply put, we were in control the entire time, no matter how chaotic things seemed, because we’d prepared for such an incident. We ran tabletop exercises, planned for different scenarios, mapped out hypothetical intrusions, tested our response methods, and looked for and plugged potential security holes. We also built an incident response team comprised of representatives from across the company. It included members from our security, legal, marketing, IT, and engineering teams, and our board of directors.
As you plan your threat response, consider the following:
- Do you have a cybersecurity incident response playbook?
- Have you performed tabletop exercises and run various attack scenarios?
- Do you have the right people on the incident response team—a good mix of strategic and tactical expertise?
- Do you have ways to contact people, even on the weekend (or during a pandemic)?
- Do you have a list of backup contacts in case someone isn’t available?
- Do you have alternative communication methods established in case you cannot trust your existing ones?
2. The initial weeks: Separating teams creates an agile and efficient response
We quickly learned we needed to split our team into different groups for an agile and efficient response. Thus, one big team became multiple smaller teams, each overseen by leaders within their respective organizations (i.e., the legal team was led by our general counsel, the engineering team by our head of engineering, and so forth). These teams would work independently, then reconvene each evening to share what they learned, discuss solutions and ideas, and so on.
Having different teams allowed individuals to focus on each facet of the response. For example, engineering could focus on how the attack affected our build while IT investigated how the attackers got in. The communications team created responses for customers, partners, and the press, and what ultimately became the government affairs team devised a plan to contact various government agencies.
We also learned organizing these teams was impossible without a third-party “quarterback.” So, we brought in an external organization to coordinate our teams’ work. They set up meetings and ensured everyone was on the same page and information was being shared.
As you coordinate your teams, ask:
- Do we have a plan in place to get teams together?
- Do we have a third-party “security helper” on call or retainer? (This is often a good insurance policy)
- Do we have enough teams to cover every aspect of our business?
3. The following weeks and months: Unbiased partners help amplify the truth
At the time, there was a lot of misinformation floating around. We were being outnumbered, out-marketed, and out-communicated. And unfortunately, social media made misinformation spread like wildfire—and has helped it be equally hard to extinguish.
To help, we partnered with reputable and experienced organizations like the Cybersecurity and Infrastructure Agency (CISA), Krebs Stamos Group, and others. The organizations performed forensics while amplifying the truth about the attack, helping people understand this was not just an isolated incident.
Amplifying the truth was the only agenda our partners had. Sadly, that’s not the norm. I discovered many organizations out there want to promote their brand or have ulterior motives. Fortunately, the organizations we worked with had no such baggage.
Indeed, they allowed us to focus on ensuring our customers were in the right state. We wanted to be there to answer their questions, assure them, and, most of all, make sure they were secure and protected. Our partners helped us block out the noise so we could focus on helping our customers.
To summarize:
- Bring in the correct partners and add new partners as necessary
- Watch out for hidden agendas
- Prioritize what’s most important to you (For us, our customers were our top priority)
- Don’t spend time responding to every inaccuracy; it will only distract you from your priorities
- Stay focused
4. The final months: Going above and beyond leads to an exemplary outcome
As the months wore on, I remember a colleague telling me, “If you’re going to come out of this, you have to be special. It won’t be enough just to fix the issue. You need to really go above and beyond.”
As it turns out, we fixed the issue—but did much more than that. We found the source for SUNBURST and made it publicly available. We testified before the U.S. House and Senate. We implemented assistance programs to help our customers. We held briefings with the FBI and other global law enforcement agencies.
We ensured the world knew what we were doing and why we were doing it. In being transparent, we were helping others understand what we went through so they could better protect themselves. It’s not enough to be transparent, of course. To get through it, and come out stronger, we needed to have products and services people love and enjoy using, which leads me to three final recommendations:
- Be open and honest throughout the entire process
- Communicate early and often—not just to your customers, partners, and employees but to the world
- Make the type of products you would want them to use, and make them Secure by Design
The months have turned into years. The tenets of transparency and humility have served us well. The SUNBURST incident has turned into a catalyst for good. Supply chain security is now front of mind for many. Executive orders and cyber security strategies are leading us towards attestation for software security. Executive and boardroom conversations have security as a necessary topic, and the security defenders of the world are being looked upon for guidance in managing cyber risk.
The investigation into SUNBURST formally concluded in May 2021—six months after the attack was first uncovered. But I like to think our response to the attack will live on for much longer. Because what started as a dark day in December 2020 made us a stronger, more resilient, and better company. I hope the lessons I learned can help you do the same.
