The Committee on Oversight and Government Reform released a fascinating 231-page report detailing the how and why behind the epic breach at the United States Office of Personnel Management.
Richard Spires, the former CIO of the IRS and DHS, remarked on OPM’s failure to take a data-centric approach to information security:
“[I]f I had walked in there [OPM] as the CIO—and, you know, again, I’m speculating a bit, but—and I saw the kinds of lack of protections on very sensitive data, the first thing we would have been working on is how do we protect that data? OK? Not even talking about necessarily the systems. How is it we get better protections and then control access to that data better?”
What data was taken?
A picture of the damage inflicted by the OPM breach is painted through a series of powerful quotes, like this one from James Comey, Director of the FBI:
“My SF-86 lists every place I’ve ever lived since I was 18, every foreign travel I’ve ever taken, all of my family, their addresses. So it’s not just my identity that’s affected. I’ve got siblings. I’ve got five kids. All of that is in there.”
It’s hard to refute the argument that this is the most devastating breach of all time given the scale and sensitivity of the data that was stolen:
- 4.2 million personnel files of former and current government employees
- 21.5 million security clearance background investigation files
- 5.6 million fingerprints
The background investigation files include things like mental health history, alcohol abuse, gambling issues, and other deeply personal information.
How OPM happened
The landmark event that everyone thinks of when they hear “OPM breach” is the theft of 21.5 million background investigation files from the Personnel Investigations Processing System (PIPS) – a legacy mainframe that stores the organization’s crown jewels. This breach was disclosed in 2015.
However, a file share breach disclosed back in 2014 appears to have played an instrumental role in the eventual PIPS breach. In fact, investigations showed that hackers had access to OPM’s network since July of 2012 and were discovered only after advanced monitoring was enabled in March of 2014.
Regrettably, we’ll never know the extent of documents exfiltrated prior to March 2014.
On March 20, 2014, the Department of Homeland Security’s Computer Emergency Response Team (US-CERT) informed OPM’s own response team that a hacker had exfiltrated OPM data from the network.
To “better understand” the threat posed by the hacker (referred to as Hacker X1), OPM monitored the adversary’s movements for two months until they discovered a second hacker (Hacker X2) who gained initial access using a contractor’s stolen credentials.
Brendan Saulsbury, an OPM contractor with OPM’s IT Security Operations, says:
“So we would sort of observe the attacker every day or, you know, every couple of days get on the network and perform various commands. And so we could sort of see what they were looking for. They might take some documentation, come back, and then access, you know, somebody else’s file share that might be a little bit closer or have more access into the system.”
Hikit and SMB
Hacker X2 dropped Hikit malware to establish a backdoor, escalate privileges, and perform keylogging. Hikit was found on numerous systems and was beaconing back to a C2 server. OPM sniffed the hacker’s traffic to determine what was being exfiltrated.
Activity logs showed that the hackers would logon between 10 p.m. and 10 a.m. ET using a compromised Windows domain administrator account and search for PII on file shares using SMB commands.
OPM watched a hacker exfiltrate documents from a file share which contained information that described the PIPS system and how it is architected.
Appendix D of US-CERT’s June 2014 incident report describes the stolen file-share data:
OPM’s Director of IT Security Operations, Jeff Wagner, testified:
“In 2014, the adversary was utilizing a Visual Basic script to scan all of our unstructured data. So the data comes in two forms. It’s either structured, i.e., a database, or unstructured, like file shares or the home drive of your computer, things of that nature. All the data that is listed here, all came out of personal file shares that were stored in the domain storage network.”
The value of the data known to be exfiltrated was initially dismissed as being fairly inconsequential, but the US-CERT investigation report makes it clear that the hackers were doing reconnaissance on OPM’s file-sharing infrastructure in order to get closer to PIPS:
“The attackers primarily focused on utilizing SMB [Server Message Block] commands to map network file shares of OPM users who had administrator access or were knowledgeable of OPM’s PIPS system. The attacker would create a shopping list of the available documents contained on the network file shares. After reviewing the shopping list of available documents, the attacker would return to copy, compress and exfiltrate the documents of interest from a compromised OPM system to a C2 server.”
When asked if the documents exfiltrated from the file shares would yield an advantage in future attacks, Wagner replied:
“It gives them more familiarity with how the systems are architected. Potentially some of these documents may contain accounts, account names, or machine names, or IP addresses that are relevant to these critical systems.”
Not so trivial after all.
After conceding that the hackers were getting “too close” to PIPS, security ops decided to “boot” the hacker in an operation called the “Big Bang.”
They successfully booted Hacker X1 in late May 2014, but Hacker X2 maintained a foothold, traversing the cyber kill chain en route to the famous PIPS breach:
“Beginning in July through August 2014, the Hacker X2 exfiltrated the security clearance background investigation files. Then in December 2014, personnel records were exfiltrated, and in early 2015, fingerprint data was exfiltrated.”
A stunning lack of visibility
US-CERT identified numerous gaps in the OPM’s centralized logging strategy:
“Gaps in OPM’s audit logging capability likely limited OPM’s ability to answer important forensic and threat assessment questions related to the incident discovered in 2014. This limited capability also undermined OPM’s ability to timely detect the data breaches that were eventually announced in June and July 2015.”
The big takeaway from US-CERT’s gap analysis is that traditional security strategies have a severe vulnerability when it comes to insider threats. By Jeff Wagner’s own admission, OPM had focused heavily on perimeter security, but lacked the technology necessary to detect and stop attackers who were already inside.
The report outlines OPM’s history of inadequate security controls and failed audits:
- 2005 – the Inspector General (IG) gives OPM a bad security grade, says they’re vulnerable to hackers
- FY 2013-2015 – OPM’s IT spending is at the bottom of all federal agencies
- 2014 – the IG says “material weaknesses” have become “significant deficiencies”
- 2015 – despite a mandate, only one percent of OPM employee and contractor accounts were required to use multi-factor authentication
- 2015 (post-breach) – IG still sees an “overall lack of compliance that seems to permeate the agency’s IT security program.”
Why all CISOs need to pay attention to what happened at OPM
OPM isn’t exceptional. Many of the breaches that grab headlines are eerily similar.
First, they start with someone who is already an insider, like Edward Snowden, or an attacker hijacks the credentials of an insider, as was the case with Target and OPM. The explosion of ransomware has proven just how easy it is to get inside, and every vector seems to be working at scale – phishing, hijacked websites, cloud file-sharing.
Second, what do they take? Files and emails — unstructured data. In the Wikileaks and Snowden incidents, an insider took confidential cables, or emails. What was taken in the Sony Pictures breach? Emails, video files, files containing passwords. All unstructured data. Ransomware also shows how vulnerable this data is – a single infected user account can encrypt thousands of files without being noticed, many of which that user probably shouldn’t have access to in the first place.
There are of course other kinds of data we need to worry about, but unstructured data is what most organizations have the most of and know the least about. And so much of it contains sensitive information like that taken in OPM: social security or credit card numbers, health records, or detailed roadmaps describing how to infiltrate a massive database of PII.
Employees and contractors have access to all this data just by showing up to work—usually to much more than they need to do their jobs. Outsiders only need to steal an employee’s or contractor’s credentials through phishing or some other means, and then they have access to it, too.
It’s just too easy for data to be stolen, and we have to make it harder.
SIEM by itself is not enough
“Currently, OPM utilizes Arcsight as their SIEM [security information and event management] solution of choice, but there are numerous gaps in auditable events being forwarded to Arcsight for analysis, correlation, and retention.”
Many organizations don’t forward file access events to their SIEM because native auditing is performance intensive, the raw audit logs are too noisy and voluminous, and SIEM vendors often charge by data volume. In order to protect file-share data from insider threats and outside attackers that find their way inside, security technologies like SIEM and UBA must have credible telemetry from the file shares, including access activity and content awareness.
A data-centric approach
As Richard Spires points out, we need a new approach that focuses more on the data itself than the infrastructure that allows us to access that information. It’s one thing to lose a server; it’s another to lose millions of files containing employees’ deepest personal secrets.
Organizations need to get a grip on where their information assets are, who is using them, and who is responsible for them. There are just too many unknowns right now. They need to put all that data lying around in the right place, restrict access to it and monitor and analyze who is using it.
One thing organizations have started to realize is that they can jump light years ahead of where they are today very quickly just by installing technology to watch and analyze how employees use data. Smart AI and machine learning can be used to look for patterns of abuse and help you spot breaches before they happen. Think of it like the fraud detection that your credit card company uses – it’s very effective in stopping thieves from stealing money. The same analytics can help prevent insiders and outside attackers from stealing data.
There is no security silver bullet. But if you’re not watching what is going on with your unstructured data, which is growing exponentially, you have an intolerably dangerous blind spot – it’s almost impossible to detect an attack and very difficult to assess the scope of the damage, making recovery arduous and expensive. Organizations have overlooked this for a long time because the notion of organizing, categorizing and sorting out this metadata has been daunting. But that doesn’t need to be the case anymore.
I’ll close with the bold statement that the report opens with – one that is directed to federal CIOs, but that all CIOs and CSOs should take to heart:
“Federal CIOs matter. In fact, your work has never been more important, and the margin for error has never been smaller.
As we continue to confront the ongoing challenges of modernizing antiquated systems, CIOs must remain constantly vigilant to protect the information of hundreds of millions of Americans in an environment where a single vulnerability is all a sophisticated actor needs to steal information, identities, and profoundly damage our national security.”
Caveats & Notes
The report, which is titled “The OPM Data Breach: How the Government Jeopardized Our National Security for More than a Generation,” was authored by Republican Congressional staffers. You can read the Democratic response to the report here.
Regardless of partisan politics, the report contains important information about attack vectors, timelines, files stolen, and exfiltration methods. That’s what we’ll stick to here.
If you want to read the whole report, OPM released it as a rasterized image (so you can’t CTRL-F to search). Luckily, Dan Nguyen made an OCR’d PDF and plain-text versions for us:
If you’re feeling ultra-ambitious, OPM itself released a doc shortly after the breach explaining how they plan to improve their security posture. You’ll find that here.