Full Analysis of Flame’s Command & Control servers

Our previous analysis of the Flame malware, the advanced cyber-espionage tool that’s linked to the Stuxnet operation, was initially published at the end of May 2012 and revealed a large scale campaign targeting several countries in the Middle East.

The Flame malware, including all of its components, was very large and our ongoing investigation revealed more and more details since that time. The news about this threat peaked on 4th June 2012, when Microsoft released an out-of-band patch to block three fraudulent digital certificates used by Flame. On the same day, we confirmed the existence of this in Flame and published our technical analysis of this sophisticated attack. This new side of Flame was so advanced that only the world’s top cryptographers could be able to implement it. Since then, skeptical jokes about Flame have disappeared.

Later in June, we definitively confirmed that Flame developers communicated with the Stuxnet development team, which was another convincing fact that Flame was developed with nation-state backing.

We also published our analysis of the Flame command-and-Control (C&C) servers based on external observations and publicly available information. That helped our understanding of where the C&C servers were located and how they were registered.

With this blog post, we are releasing new information that was collected during forensic analysis of the Flame C&C servers. This investigation was done in partnership with Symantec, ITU-IMPACT and CERT-Bund/BSI.

Brief C&C Server Facts

Operating system: 64-bit Debian 6.0.x

Virtualization: In most of cases running under OpenVZ

Programming languages used: PHP (most of code), Python, bash

Database: MySQL with InnoDB tables

Web server: Apache 2.x with self-signed certificates

Server Analysis

One of the C&C servers we analysed was owned by a European company with data centers in another European Union country. We managed to get a server image which was an OpenVZ file-system container.

Working with the OpenVZ file-system containers added some constraints which made forensic analysis difficult. For instance, having only OpenVZ containers doesn’t let you look into the slack space of the hard drive to recover some deleted files.

This server configuration was a typical LAMP (Linux, Apache, MySQL, PHP) setup. It was used to host a web-based control panel as well as to run some scheduled fully automated scripts in the background.

It was accessible over the HTTPS protocol, ports 443 and 8080. The document root directory was /var/www/htdocs/ which has sub-directories and PHP scripts. While the systems had PHP5 installed, the code was made to run on PHP4 as well. For example, /var/www/htdocs/newsforyou/Utils.php has the “str_split” function defined which implements the “str_split” function logics from PHP5, that was not available in PHP4. The developers of the C&C code most likely implemented compatibility with PHP4 because they were not sure which one of two major PHP versions would be installed on the C&Cs.

Figure 1 – Contents of the “newsforyou” directory

The C&C control panel code was discovered in “newsforyou/CP/CP.php”. Opening it with a web browser displayed a login prompt:

Figure 2 – Control panel login

The username and password hash were later found in local MySQL database in the “settings” table.

Login: username
Password hash (MD5): 27934e96d90d06818674b98bec7230fa

(ok, cracked: 900gage!@#)

We reset the password hash and logged in to see how the panel looked from the attacker’s side. We were more than surprised when we logged in.

Figure 3 – Control panel interface

Our first impression was that the control panel appeared to be implemented by script-kiddies. It looked like a very early alpha version of a botnet C&C control panel. However, revisiting this picture one more time made everything clear – the attackers deliberately chose this interface. Unlike traditional cyber-criminals who implement eye-candy web interfaces which the average user can easily recognize as a botnet control panel, the developers of the Flame C&C made it very generic and unpretentious.

The C&C developers didn’t use professional terms such as bot, botnet, infection, malware-command or anything related in their control panel. Instead they used common words like data, upload, download, client, news, blog, ads, backup etc. We believe this was deliberately done to deceive hosting company sys-admins who might run unexpected checks.

There is no way to send commands to infected systems using only the C&C panel web interface and this is another difference from traditional botnets. An infected machine was controlled using a message-exchange mechanism based on files (the developers called the files “data containers”).

To send a command or set of commands to a victim, the attacker uploaded a specially crafted tar.gz archive, which was processed on the server. A special server script extracted the archive contents and looked for *.news and *.ad files. These files were put into corresponding directories “news” and “ads”. The C&C allows an attacker to push an update to a specific victim, or all victims at a time. It is possible to prioritize a command which allows to organize an order of commands (i.e. collect all data and only after self-removal). The priority and target client ID was transferred in an unconventional way. They were stored in the filename that the attacker uploaded to a C&C. Below is a template of the news filename expected by the C&C:

<random_number>_<user_type>_<user_id>_<priority>.<file extension>

Source files analysis show that the C&C can understand several communication protocols to talk to different clients:

• OldProtocol
• OldProtocolE
• SignupProtocol
• RedProtocol (mentioned but not implemented)

A close look at these protocol handlers revealed four different types of clients codenamed SP, SPE, FL and IP.

We can confirm that the Flame malware was identified as client type FL. Obviously, this means there are at least three other undiscovered cyber-espionage or cyber-sabotage tools created by the same authors: SP, SPE and IP.

Figure 4 – Clients and Protocols relations found in this C&C>

A typical client session handled by the C&C started from recognition of the protocol version, then logging of connection information, followed by decoding client request and saving it to the local file storage in encrypted form. All metadata about files received from the client was kept in a MySQL database.

The C&C script encrypts all files received from the client. The C&C uses a PGP-like mechanism to encrypt files. First, the file data is encrypted using the Blowfish algorithm in CBC mode (with static IV). The Blowfish key is generated randomly for each file. After file encryption, the Blowfish key is encrypted with a public key using asymmetric encryption algorithm from the openssl_public_encrypt PHP function.

The encryption parameters:

IV: 12345678

ОpenSSL public key:

—–BEGIN PUBLIC KEY—–
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtZslxFiR9KJE05Nhh7Xk
+lVVpD9F6AQnvZeknDiwL3SBjZB/dB/LLXtwiet8LUS6JYCXnaIq4NxW1PymwGFZ
zuc/B3p+ZAFPt06veOHOfaMAI0KDMb+laNPINvn/jJ8TfvCaUMUuMEY4sayh0xwD
MwSAazMYI8rvaaS/BqhI/6vPN6D02UIpwT1TSBVeRRoPBHuYE7A93b8vJw9sBGIp
KXZ90sgP1CjdAmCbhYelelninKdeTKCGvd5YXt86grWgEVf5WXzxXi3ZK1T4w0Yt
mNhUEAwS7zCdtZ+Ak8b0M83wAirASvPZiBl6qF8hqCT5pKkwgBG//kk8JicboLsM
VQIDAQAB
—–END PUBLIC KEY—–

This guarantees that no one other than the attacker can read the files obtained from the server file storage, because only the attacker has the private key which is essential to decrypt the files from the clients.

In terms of functionality, infected clients support very few commands:

• GET_NEWS: Gets file(s) from ./news sub-directory that are assigned to current client ID. The news files contain updates and extra modules of Flame, as well as special commands, such as changing registry key values.
• ADD_ENTRY: Stores information collected by the client.
• ADD_SUB_ENTRY: Same as ADD_ENTRY.
• GET_AD: Gets files from ./ad_path directory. This functionality seems to be broken in the C&C, because there was no ./ad_path but ./ads directory. However the attacker can upload files to ./ads directory via the control panel.

We have noticed that the naming convention of files and classes in the scripts is not common for Unix developers because every word in the variable, function and file name started with a capital letter. Also, there are special classes that are defined but never instanced, such as IProtocol (Protocol.php) and IRequestHandler (RequestHandler.php). Classes that are never instanced are used for inheritance mechanism in Object Oriented Programming. These classes are called interfaces in programming languages such as C++, C# and Java. Putting capital “I” to prefix such classes is a common habit for many Windows C++/C# developers.

One of the most valuable traces left by the developers in the scripts were their nicknames and internal timestamps:

• @author [O] censored in 12/3/2006
• @author [D] censored on 12/4/2006
• @author [D] censored on 01/23/2007
• @author [H] censored on 09/02/2007
• @author [O] censored, [D] censored
• @author [D] censored (modifications)
• @author modified heavily by [D] censored
• @author [R] censored @copyright 2011

Here is what is known regarding the activity of the developers:

• [D] censored: worked on at least 31 files
• [H] censored: worked on at least 10 files
• [O] censored: worked on at least 4 files
• [R] censored: worked on at least 1 file (duplicate files remover)

Figure 5 – Timeline of Flame C2 programmers’ activity

From this, we can conclude that the first C&C files were created on 03 December 2006, which means that the Flame platform is much older than we originally estimated.

There were four people responsible for this C&C development. It is obvious to us that one of them, [H] censored, was more experienced than the others. He coded some very smart patches and implemented complex logics; in addition, he seems to be a master of encryption algorithms. We think [H] censored was most likely a team lead.

The operators of the C&C used automation to prepare the server environment. We believe this is because they had so many servers that it was unreasonable to do everything manually. Below is a part of an interesting script (LogWiper.sh) from the filesystem root directory that remained on the current server:

#! /bin/bash
apt-get install -y chkconfig
#stop history
echo “unset HISTFILE” >> /etc/profile
history -c
find ~/.bash_history -exec shred -fvzu -n 3 {} \;
service rsyslog stop
chkconfig rsyslog off
service sysklogd stop
chkconfig sysklogd off
service msyslog stop
chkconfigm syslog off
service syslog-ng stop
chkconfig syslog-ng off
shred -fvzu -n 3 /var/log/wtmp
shred -fvzu -n 3 /var/log/lastlog
shred -fvzu -n 3 /var/run/utmp
shred -fvzu -n 3 /var/log/mail.*
shred -fvzu -n 3 /var/log/syslog*
shred -fvzu -n 3 /var/log/messages*
#stop logging ssh
cp /etc/ssh/aa
sed -i ‘s/LogLevel.*/LogLevel QUIET/’ /etc/ssh/sshd_config
shred -fvzu -n 3 /var/log/auth.log*
services sh restart
#delete hidden files
find / -type f -name “.*” | grep -v “.bash_profile” | grep -v “.bashrc” | grep “home” | xargs shred -fvzu -n 3
find / -type f -name “.*” | grep -v “.bash_profile” | grep -v “.bashrc” | grep “root” | xargs shred -fvzu -n 3 #stop apache2 logging
sed -i ‘s|ErrorLog [$/a-zA-Z0-9{}_.]*|ErrorLog /dev/null|g’ /etc/apache2/sites-available/default
sed -i ‘s|CustomLog [$/a-zA-Z0-9{}_.]*|CustomLog /dev/null|g’ /etc/apache2/sites-available/default
sed -i ‘s|LogLevel [$/a-zA-Z0-9{}_.]*|LogLevel emerg|g’ /etc/apache2/sites-available/default
sed -i ‘s|ErrorLog [$/a-zA-Z0-9{}_.]*|ErrorLog /dev/null|g’ /etc/apache2/sites-available/default-ssl
sed -i ‘s|CustomLog [$/a-zA-Z0-9{}_.]*|CustomLog /dev/null|g’
/etc/apache2/sites-available/default-ssl
sed -i ‘s|LogLevel [$/a-zA-Z0-9{}_.]*|LogLevel emerg|g’ /etc/apache2/sites-available/default-ssl
…

shred -fvzu -n 3 /var/log/apache2/*
service apache2 restart
#self delete
find ./ -type f | grep logging.sh | xargs -I {} shred -fvzu -n 3 {} \;

The script above is used to prepare server environments to work as a C&C server. They wipe existing log files and disable further logging. This file gives an overview of the C&C operators’ skills and preferences. First of all, we see that the operators installed and used chkconfig. Here is an official description of chkconfig package on Debian systems:

Chkconfig is a system tool to enable or disable system services. Chkconfig is a utility to update and query runlevel information for system services. Chkconfig manipulates the numerous symbolic links in /etc/init.d/, to relieve system administrators of some of the drudgery of manually editing the symbolic links. In Debian, there are several tools with similar functionality, but users coming from other Linux distributions will find the tools in this package more familiar.

This is rather interesting as Debian has a lot of other popular tools to handle the start-up scripts manipulation: rcconf, update-rc.d, sysv-rc-conf, etc. The chkconfig tool on Debian is an implementation of a popular RedHat tool with the same name. It appears that the people who managed the C&Cs are more familiar with RedHat systems. This reminds us of the Duqu C&Cs which were all based on RedHat CentOS.

The script generically stops all known system logging daemons such as rsyslog, sysklogd, syslog-ng, msyslog. While the first three are common on Debian systems, msyslog isn’t present in official public repositories. It appears that that msyslog is supported by Core Security company and is able to run on Linux, BSD, Irix, Solaris and AIX. The latest version of msyslog daemon was released on Apr 15, 2003. None of the analyzed systems had msyslog daemon installed. It is not clear why the operators had special lines for msyslog daemon.

The C&C operators used the shred tool to wipe information. This was also used in the latest server wipes by the Duqu team.

At the end of the script, there is a line to remove the original script. The is done with the shred command as well but this is a mistake because the name of the shredded file is “logging.sh”, while the current script was called LogWiper.sh which is why it was not deleted from the system.

The system cron daemon was configured to run several periodic tasks:

• Every 30 minutes: php /var/www/htdocs/newsforyou/UnloadChecker.php
• Every 6 hours: python /home/…/pycleaner/Eraser.py
• At midnight: php /home/…/delete.php

In addition to the webserver root directory, there was a home directory for a user with a three-character name. According to analysis of the other servers, these names were always three characters long and picked randomly. This revealed several other files and directories used in the project:

Figure 6 – non-root user home directory

LogWiper_fixed.txt is the same file as LogWiper.sh described above.

delete.php is a PHP script that was run on schedule to delete news files that are older than 30 days. File removal was done in a secure manner using an external call to system shred tool. It also handled deleting files meta-information from the database.

The directory pycleanscrhad four files in it used for automated cleanup procedures:

Figure 7 – pycleanscr directory

This script was designed to remove temporary files from the webserver tmp directory and was intended to run as a scheduled job. However a mistake made by the authors didn’t let it run at all. The cronjob was set to execute python /home/…/pycleaner/Eraser.py, while the right path was python /home/…/pycleanscr/Eraser.py.

Apparently the operators had several variants of scripts to control disk usage and prevent overuse of the disk space.

We also see a RequestHandler.php file in the home folder, indicating they were working on that file. It was put there on 14th May 2012. This is the same file as in “newsforyou” webserver directory. Later on, when we analyzed other server images, we realized that the file was different from other servers. The difference in this file was in pushing connecting clients a module internally called “SHREDER” (mentioned above on the timeline) when no other modules were in the queue of “news”. The “SHREDER” was a well described (Symantec blogpost) Flame-related malware self-removal Windows binary file.

The Simulator directory contains special scripts to do stress testing of the C&C by creating queries identical to original malware HTTP requests. The operator can specify the number of working threads and the frequency of incoming requests. This indicates that the developers used real “production” servers for testing purposes as well.

Following our previous publications, we received many questions regarding the amount of data that was exfiltrated by Flame. This is very difficult to estimate if you only have access to the malware executable. It would be even impossible to estimate that data having access to the Flame C&C, as the operators downloaded and removed the stolen data from the C&C every 30 minutes. If the download scripts weren’t working for some reason, that was not a problem as there were other scripts on the server side controlling the amount of data preserved on the server. If the threshold was reached, the older data was deleted to free disk space.

So, even if you take down and analyze the server contents you will see only a small fraction of real data that was stolen. However, due to a mistake of the attackers they got locked out of it and left behind a collection of files that were not deleted. That helped to measure the real amount of stolen files uploaded to this C&Cs weekly: 5.5 GB (compressed).

On one of the servers, the attackers forgot to delete the HTTP logs; this allowed us to get an idea of how many victims connected to the server by POST requests to the Flame C2 code:

Figure 8 – Web server log stats

During a period of just one week (25 March – 2 April), 5377 unique IPs were seen connecting to the server, the vast majority in Iran: 3702. What is also surprising is the large number of IPs from Sudan: 1280. Our previous statistics did not show a large number of infections in Sudan, so this must have been a dedicated campaign targeting systems in Iran and Sudan.

If just one server handled 5000+ victims during a one-week period and given several servers were available, we can estimate the total number of victims for Flame is probably higher than previously estimated, exceeding 10,000.

SP, SPE, FL, IP, Gauss and Stuxnet

The recovered C&C code handles four main clients: SP, SPE, FL and IP. From this list, the most interesting malware is IP, which is also the most recent based on the assigned client codes:

/// Types of clients
define(‘CLIENT_TYPE_UNKNOWN’, 0);
define(‘CLIENT_TYPE_SP’, 1);
define(‘CLIENT_TYPE_SPE’, 2);
define(‘CLIENT_TYPE_FL‘, 3);
define(‘CLIENT_TYPE_IP’, 6);

The IP client uses the “Signup Protocol” and it’s the only client in the list which uses it. To detect the “Signup Protocol”, the code looks for a specific “uid=number&action=number” line in the HTTP request:

if (preg_match(‘/^uid=d+&action=d+/’, $data) === 1)
{
return array(RC_SUCCESS, PROTOCOL_SIGNUP);
}

If you are wondering, none of the malware we have analyzed uses this scheme. For instance, Gauss uses “POST /userhome.php?sid=string==&uid=string=”. Similarly, Stuxnet uses “index.php?data=…”. So we are pretty sure this malware codenamed “IP” is neither Gauss nor Stuxnet.

Additionally, the current C&C code doesn’t appear to be able to handle Gauss or Stuxnet connections.

Sinkhole hits from “SPE”

With help from our partners, Kaspersky Lab set up a sinkhole for Flame. We previously published statistics from the sinkhole data in our previous posts.

Perhaps the most interesting thing is that based on the C&C code of Flame, we were able to catalog the sinkhole connections into two main categories:

• OldProtocol connections, coming from Flame
• OldProtocolE connections, used by SPE

Therefore, we can confirm the malware known as “SPE” exists and is currently in-the-wild.

In addition, there are no hits from either the mysterious SP or IP malware.

Conclusions and summary findings:

During our joint investigation with industry partners – Symantec, ITU-IMPACT and CERT-Bund/BSI – we uncovered a number of things previously not known about Flame and its associated platform:

• The development of the platform C&C code started as early as December 2006.
• The source code comments indicate at least four programmers: [D] censored, [O] censored, [H] censored and [R] censored.
• The most recent source code change was made on May 18, 2012 (LogWiper_fixed.txt, Constants.py).
• The C&C code handles four different malware – named SP, SPE, FL and IP by the authors. It also handles three communication protocols (OldProtocol, OldProtocolE, SignupProtocol).
• Of the four malware, only Flame is known; the other three are currently unknown.
• For communication, Flame uses “OldProtocol”.
• An unknown, Flame-related malware named SPE exists and is in-the-wild.
• Flame is not the most “modern” of the malware known by the C&C code.
• The most recent malware is called “IP” and it is yet unknown.
• The code is/was still in development; a new protocol called “Red Protocol” is not yet fully implemented.

The details disclosed in this blog post indicate several important facts. First of all, the work on these cyber-espionage projects started earlier than previously estimated – as early as 2006. Secondly, the code which handles the requests is complex and uses encryption heavily. The programmers who worked on it tried to make it seem like a legal CMS platform. Last but not least, the stolen data is encrypted on the server in such a way that only the attackers can read it, through strong public key cryptography. These features are not normally found in malware created by everyday cyber-criminals, reaffirming our initial conclusions that Flame is a nation-state sponsored attack.

Based on the code from the server, we know Flame was a project from a list of at least four. The purpose and nature of the other three remain unknown.