Download full report (PDF)
Back to the Future – SAS 2016
As Thomas Rid left the SAS 2016 stage, he left us with a claim that turned the heads of the elite researchers who filled the detective-themed Tenerife conference hall. His investigation had turned up multiple sources involved in the original investigation into the historic Moonlight Maze cyberespionage campaign who claimed that the threat actor had evolved into the modern day Turla. What would this all mean?
The Titans of Old
Moonlight Maze is the stuff of cyberespionage legend. In 1996, in the infancy of the Internet, someone was rummaging through military, research, and university networks primarily in the United States, stealing sensitive information on a massive scale. Victims included the Pentagon, NASA, and the Department of Energy, to name a very limited few. The scale of the theft was literally monumental, as investigators claimed that a printout of the stolen materials would stand three times taller than the Washington Monument.
To say that this historic threat actor is directly related to the modern day Turla would elevate an already formidable modern day attacker to another league altogether. Turla is a prolific Russian-speaking group known for its covert exfiltration tactics such as the use of hijacked satellite connections, waterholing of government websites, covert channel backdoors, rootkits, and deception tactics. Its presumed origins track back to the famous Agent.BTZ, a campaign to spread through military networks through the use of USB keys that took formidable cooperation to purge (in the form of an interagency operation codenamed Buckshot Yankee in 2008). Though mitigating the threat got the most attention at the time, further research down the line saw this toolkit connecting directly to the modern Turla.
Further confirmation came through our own Kurt Baumgartner’s research for Virus Bulletin 2014 when he discovered Agent.BTZ samples that contacted a hijacked satellite IP jumping point, the same that was used by Turla later on. This advanced exfiltration technique is classic Turla and cemented the belief that the Agent.BTZ actor and Turla were one and the same. This would place Turla back as early as 2006-2007. But that’s still a decade ahead of the Moonlight Maze attack.
By 2016 the Internet was over-crowded with well-resourced cyberespionage crews. But twenty years ago there were few players in this game. Few paid attention to cyberespionage. In retrospect, we know that the Equation Group was probably active at this time. A command-and-control registration places Equation in the mid-1990s. That makes Equation the longest running cyberespionage group/toolkit in history. To then claim that Turla, in one form or another, was active for nearly as long, places them in a greater league than their pre-historic counterpart in pioneering state-sponsored cyberespionage.
A Working Hypothesis
By the time of the SAS 2016 presentation, we had already discussed at length how one might go about proving this link. The revelation that the Moonlight Maze attacks were dependent on a Solaris/*NIX toolkit and not a Windows one as is the case with most of Turla, actually revived our hopes. We would not have to look for older Windows samples where so far there were none, but could instead focus on another discovery. In 2014, Kaspersky announced the discovery of Penquin Turla, a Linux backdoor leveraged by Turla in specific attacks. We turned our attention once again to the rare Penquin samples and noticed something interesting: the code was compiled for the Linux Kernel versions 2.2.0 and 2.2.5, released in 1999. Moreover, the statically linked binaries libpcap and OpenSSL corresponded to versions released in the early 2000s. Finally, despite the original assessment incorrectly surmising that Penquin Turla was based on cd00r (an open-source backdoor by fx), it was actually based on LOKI2, another open-source backdoor for covert exfiltration written by Alhambra and daemon9 and released in Phrack in the late 1990s. This all added up to an extremely unusual set of circumstances for malware that was leveraged in attacks in from 2011-2016, with the latest Penquin Sample discovered just a month ago being submitted from a system in Germany.
Kurt Baumgartner’s prescient observation upon the discovery of the first Penquin Turla samples
Our working hypothesis became this: “The Turla developers decided to dust down old code and recompile it for current Windows victims in the hope of getting a stealthier beachhead on systems that are less likely to be monitored.” Were that to be the case, Penquin Turla could be the modern link that tied Turla to Moonlight Maze. But in order to prove our hypothesis and this historic evolution, we’d need a glimpse at the original artefacts, something we had no access to.
The Cupboard Samples
Our last hope was that someone somewhere had kept a set of backups collecting dust in a cupboard that they might be willing to share. Thomas took to the road to follow up his sources and eventually stumbled upon something remarkable. The Moonlight Maze operators were early adopters of a certain degree of operational security, using a series of hacked servers as relays to mask their original location. During the later stages of their campaign, they hacked a Solaris box in the U.K. to use as a relay. Unbeknown to them, the system administrator—in cooperation with the Metropolitan Police in London and the FBI—turned the server against the malicious operators. The machine known as ‘HRTest’ would proceed to log everything the attackers did keystroke-by-keystroke and save each and every binary and archive that transited through it. This was a huge win for the original investigators and provided something close to a six-month window of visibility before the attackers ditched this relay site (curiously, as a result of the campaign’s first publicity in early March 1999). Finding these samples was hard and fortuitous—due to a redaction error in an FBI FOIA release, we were able to ultimately track down David Hedges after about a year of sleuthing. “I hear you’re looking for HRTest,” David said when he finally called Thomas for the first time. Then, the now-retired administrator kicked a machine under his desk, chuckling as he said “well it’s sitting right here, and it’s still working.”
Thomas Rid, David Hedges, Daniel Moore, and Juan Andres Guerrero-Saade at King’s College London
Paydirt but not the Motherlode
What we had in our hands allowed us to recreate a portion of the constellation of attacks that constitutes Moonlight Maze. The samples and logs became our obsession for months. While Juan Andres and Costin at GReAT reversed the binaries (most compiled in SPARC for Solaris and MIPS for IRIX, ancient assembly languages), Daniel Moore went so far as to create an entire UI to parse and load the logs onto, so as to be able to visualize the extent of the networks and nodes under attack. We set out to profile our attackers and understand their methods. Among these, some salient features emerged:
Moore’s Rapyd Graph Data Analyzer tracking the victims of Moonlight Maze linked to HRTest
- The attackers were prolific Unix users. They used their skills to script their attack phases, which allowed a sort of old school automation. Rather than have the malware communicate to command-and-control servers and carry out functions and exfiltration of their own accord, the attackers would manually log in to victim nodes and leverage scripts and tasking files (usually located in the /var/tmp/ directory) to instruct all of these nodes on what they should do, what information to collect, and finally on where to send it. This allowed them to orchestrate large swaths of infected machines despite being an ‘operator-at-keyboard’ style of attack.
- The operators were learning as they went. Our analysis of the binaries shows a trial and error approach to malware development. Many binaries were simply open-source exploits leveraged as needed. Others were open-source backdoors and sniffers. However, despite not having exact compilation timestamps (as would happen in Windows executables), it’s possible to trace a binary evolution of sorts. The devs would test out new capabilities, then recompile binaries to fix issues and expand functionality as needed. This allowed us to graph a sort of binary tree of development to see how the attacks functionalities developed throughout this campaign.
- Despite their early interest in OpSec, and use of tools specifically designed for this effect, the operators made a huge mistake. It was their standard behavior to use infected machines to look for further victims on the same network or to relay onto other networks altogether. In more than a dozen cases, the attackers had infected a machine with a sniffer that collected any activity on the victim machine and then proceeded to use these machines to connect to other victims. That meant that the attackers actually created near complete logs of everything they themselves did on these systems—and once they did their routine exfiltration, those self-logs were saved on the HRTest node for posterity. The attackers created their own digital footprint for perpetuity.
So what’s the verdict?
A complete analysis of the attack artefacts is provided in the whitepaper, for those interested in a look under the hood of a portion of the Moonlight Maze attacks. For those who would like to jump straight to the conclusion: our parallel investigation into the connection between Moonlight Maze and Turla yielded a more nuanced answer predicated upon the limitations in our visibility.
An objective view of the investigation would have to admit that a conclusion is simply premature. The unprecedented public visibility into the Moonlight Maze attack provided by David Hedges is fascinating, but far from complete. It spans a window between 1998-1999 as well as samples apparently compiled as far back as late 1996. On the other hand, the Penquin Turla codebase appears to have been primarily developed from 1999-2004 before being leveraged in more modern attacks. What we are left with is a circumstantial argument that takes into account the binary evolution witnessed from 1998-1999 as well as the functionality and tools leveraged at that time, both of which point us to a development trend that could lead directly to what is now known as Penquin Turla. This includes the use of tasking files, LOKI2 for covert channel communications, and promiscuous sniffers – all of which made it into the modern Penquin Turla variants.
The next step in our ongoing parallel investigation would have to focus on a little known operation codenamed ‘Storm Cloud’. This codename represents the evolved toolkit leveraged by the same Moonlight Maze operators once the initial intrusions became public in 1999. In 2003, the story of Storm Cloud leaked with little fanfare, but a few prescient details led us to believe a more definitive answer may be found in this intrusion set:
Storm Cloud reference in a 2003 Wall Street Journal Article mentions further use of LOKI2
Just as the SAS 2016 talk enabled us to find David and his time capsule of Moonlight Maze artefacts, we hope this glimpse into our ongoing research will bring another dedicated sysadmin out of the woodwork who may still have access to Storm Cloud artefacts, allowing us to settle this question once and for all. Beyond the historical value of this understanding, it would afford greater perspective into a tool being leveraged in cyberespionage attacks to this day.
The epic Moonlight Maze hunt continues…
If you have information or artefacts you’d like to share with the researchers, please contact penquin[at]kaspersky.com
Penquin’s Moonlit Maze
From the same authors
In the same category
Daniel J Furlin
I just read the article: HACK DISARMS SIMPLISAFE’S HOME WIRELESS SECURITY SYSTEMS….by Tom Springs. I was contemplating getting Simplisafe. The article mentions similar concerns for ADT, Vivint, etc. So, my question is…..what home security system IS safe? I’m guessing only systems that are SUPER expensive. The article doesn’t elaborate in that.