Auburn High School in Rockford, Illinois. (Auburn High School)
The school districts of Rockford, Illinois and Rockingham County, North Carolina learned some very valuable lessons in transparency and communication, timely incident response, access management, data redundancy and disaster recovery after each experienced a debilitating malware attack years ago.
Information security leaders at these two districts shared their war stories last week at the K-12 Cybersecurity Leadership Symposium, hosted by the K12 Security Information Exchange (K12 SIX) – the first-ever ISAC specifically created with local school districts in mind.
Such lessons are vital, considering what’s at stake. As part of the symposium, Doug Levin, K12 SIX national director, and president of EdTech Strategies and the K-12 Cybersecurity Resource Center, revealed troubling findings from his newly published report, “The State of K-12 Cybersecurity: 2020 Year in Review.”
According to the report, there were 408 publicly disclosed cyber incidents affecting school districts last year – 18% more than in 2019. If you account for the unknown attacks that were never reported, the true number is likely 10 to 20 times greater, Levin estimated.
“2020 didn’t happen in a vacuum… There has been a steady and alarming uptick in not only the frequency of K-12 cyber incidents but in terms of their significance and impact on students and teachers and other school community members,” said Levin. Indeed, this past year, there were at least 15 school districts across 13 states that had to closes for weeks or months due to a ransomware attack.
And despite at least one report that school attacks are trending down so far in 2021, there will no doubt be more attacks to come. With that in mind, educational districts – and organizations in other industry sectors for that matter – could learn a thing or two from the presenters who already went through an attack scenario.
Rockingham County, North Carolina
Kacey Sensenich, chief technology officer at Rockingham County Schools (25 schools, 11,691 students in the 2019-2020 school year), ran up against an Emotet trojan infection in December 2017. Emotet, whose infrastructure was disrupted in a law enforcement operation earlier this year, is known for dropping the TrickBot banking trojan, and it can even deliver a secondary ransomware information.
On Dec. 11 of 2017, Sensenich began observing signs of abnormal network behavior. Google warned the district that its email accounts were sending out spam messages. A couple of days later, computers weren’t communicating properly with the internet.
Eventually, Sensenich’s team found the offending source file on a compromised machine that had been infected via an opened phishing email that used a fake invoice as a lure. The district thought it had mitigated the issue, but days later the same problems resurfaced – so on Dec. 19 the network was taken offline for a full-fledged remediation.
Fortunately, an attempted secondary ransomware infection failed to take hold due to firewall and AV protections. “So we did not lose any official data, but we [decided] as a district the best solution was to wipe clean everything and build from scratch,” said Sensenich.
The team took advantage of the fact that Christmas break was upon them, buying some time. “It was all back up on Jan. 2,” said Sensenich. “So before our students walked back in the door, we had internet connectivity and our voice over phone service back up.”
Still, the mitigation and repair required 42 consecutive days of work, including the Christmas and New Year’s Day holidays. Some members of the IT team even showed up in their pajamas. “We worked anywhere from 12 to 18-hour shifts – the entire staff – to bring it back so that when [the students] came back, all services were eventually restored,” Sensenich continued.
Looking back, Sensenich identified some key policy and incident response gaps that likely exposed the district to unnecessary risk.
For example, prior to the infection, district staff members acted as administrators of their own computers. In retrospect, this was too much privilege. “There were so many programs… that teachers needed to be able to manage that we just couldn’t support it all,” she said. “We let them be administrators of their machines. I will say that going forward, they will never be administrators of their machines, as long as I’m sitting here.”
Additionally, Sensenich regrets not shutting down the network sooner after the first signs of trouble. “We didn’t know what we had until we identified it,” she explained.
Also, Sensenich realized her district required more robust back-ups to provide better data redundancy. Under its new and improved set-up, Rockingham employs a primary backup server that backs itself up in network storage boxes at multiple offsite locations. “It holds our information two to four weeks, depending on the load, but we take that backup and send it to two different locations in Google,” said Sensenich. So now, “if we were attacked again, we can pick a day – a day before the attack, a week, a month, a year – and go back to that backup. We’re taking advantage of Google for Education’s unlimited backups.”
Sensenich said Rockingham now has tens of thousands of available back-up, “Because after the malware attack we said we’d never delete again. And so as long as Google wants to hold it, why not?”
As the attack happened, the district also made some savvy decisions that helped the schools survive the crisis and better fortify their systems against future digital assaults.
For starters, transparency and communication with students and parents was key. When the network was pulled down on Dec. 19, the superintendent recorded a video message, district residents received a recorded phone call with key details, and the schools held a press conference too.
“We didn’t feel that hiding behind anything was the right thing to do,” said Sensenich. “We stepped out and said, ‘Here’s what we were the victim of, here’s what it did to us and here’s what we’re going to do to get us back.’”
Another decision Sensenich said was the right call: rebuilding the network from scratch during the disaster recovery process. Essentially, her team saw the attack as a way to fix some flaws that had long existed.
“Very rarely do you say, ‘I get to turn my network off for two weeks,’” said Senenich. “And with that two weeks, we updated everything. If it was a piece of software that wasn’t current, it became current. If it was a server that needed to have a new install or a new connection, we built all of that.”
Budgeting for cyber is never easy in the public sector, but the attack provided the local board of education with a clear-cut motivator to increase the cyber budget and hire a network security engineer.
“We did end up putting quite an investment monetarily into the recovery, but we’re better for it; we had that opportunity to bring us back up to where we needed to be,” said Sensenich. “And…our long-term goal is to ensure that we continue to have this new funding line that we didn’t have prior to this event.”
Finally, Sensenich said the incident demonstrated the criticality of teamwork during a crisis event. And that starts with leading by example to gain the respect of your employees.
“When I told them they needed to work their Christmas break, and they weren’t going on their vacations and we needed to do this, everybody just came and did it,” said Sensenich, including herself. “It was all about ‘all hands on deck.’ It’s important you already have that established, so when the crisis hits you know who your people are.”
Rockford Public Schools, Illinois
While Rockingham was spared the brunt of a ransomware encryption attack, Rockford was not.
Jason Barthel, chief information officer of Rockford Public Schools (42 schools, roughly 27,000 students), described to symposium attendees what happened after the district was hit with a two-stage infection, featuring a combination punch of the TrickBot banking trojan and Ryuk ransomware. The latter struck on the evening of Sept. 5, 2019, shortly before the new school year was set to begin.
The attack knocked the schools’ virtual servers offline. “And if we back up a day prior to that event, we actually had a core switch hit max utilization and CPU utilization,” said Barthel. “We came to find out that the threat actor was actually mapping our network to plan to proliferate this virus.”
The initial infection stemmed from a succesful email phishing campaign that “allowed the threat actor to gather our credentials and download that information and [gain] some additional command and control,” Barthel continued. Due to the ransomware infection, “we lost access to about 85 of our 400 servers across the network,” and both file and certain back-ups were encrypted.
The IT staff rushed in that evening to disconnect the internet connection, and isolate and assess the encryption damage. Key decision-makers across the district concluded that the school year was safe to start, but some work would have to be pen-and-paper-based. It ultimately took weeks to brings systems back up online and several months to achieve full restoration.
Like Sensenich, Barthel recounted lessons learned from the experience.
Among the biggest setbacks from the attack was the encryption of the back-ups, and one reason this occurred was that they were not air-gapped. “They were actually using domain credentials for access to those backups, so that’s one thing we really focused on: having those air-gapped backups located at our disaster recovery site,” said Barthel.
Barthel said the district even “went a little old school” and further protected itself by bringing back the use of tape-based back-ups that go to a safe deposit box each month.
Looking back, Barthel also wishes the staff had been better trained to identify and avoid threats such as phishing emails. Following the incident, Rockford implemented security awareness training software to help educate its staff of roughly 5,000.
It appears the training has been effective. Shortly after the ransomware attack occurred, the district ran a phishing simulation exercise that resulted in a 48% click rate among staffers. But after implementing the training, the district ran another phishing test that resulted in just a 2% click rate.
Barthel’s team also implemented multi-factor authentication as another layer of defense. “It was challenging because it does add some complexity, a little bit of extra time for the staff members to log in and get to their get to their class materials and things like that,” he said. “But that has been a lifesaver for us.”
As for the mitigation efforts following the attack, Barthel praised the district’s response. Once his team was able to ascertain that students could safely attend class, the next step was to get functional technology back in the hands of the pupils. So the district relied heavily on Chromebooks, which would not be affected by the Windows-based malware.
As part of its more long-term response, the district also took steps to ensure that its cybersecurity framework better aligned with the NIST Cybersecurity Framework and its five functions: identify, protect, detect, respond and recover.
“We actually just finished an assessment… and it’s just pretty impressive to see how far we’ve come,” said Barthel. Moreover, the district developed a business continuity plan and focused on “really strengthening our protection around detection and perimeter preventative resources and tools to keep us safe going forward.”