Attacking Diffie-Hellman protocol implementation in the Angler Exploit Kit

Exploit kit creators have been inventing increasingly interesting methods of masking their exploits, shellcodes, and payloads so that it is harder for analysts to define the type of the exploit and know what actions they may perform. Several days ago analysts found the usage of the Diffie-Hellman cryptographic protocol in the Angler Exploit Kit, which is one of the most popular exploit kits at the moment.

In Angler, threat actors used the Diffie-Hellman protocol to get a structure with the shellcode of one of the recent exploits for the CVE-2015-2419 vulnerability for the Internet Explorer 11 browser and then for the CVE-2015-5560 vulnerability exploit for Adobe Flash. Most likely, the goal of the threat actors was creating difficulties in firewall detection of the exploit (as firewalls cannot decipher a shellcode and exploit by the means of the intercepted traffic analysis) and also making it harder for the analysts to get the exploit code. However, the experts from Kaspersky Lab managed to perform a successful attack against Diffie-Hellman protocol implementation and decipher the shellcode.

Angler vs. Analysts

To make matters worse for analysts, JavaScript code and ActionScript code multiple obfuscation and a user IP ban upon sending the encrypted structure with a shellcode to the user were used in addition to the Diffie-Hellman protocol. After getting the structure with the shellcode by that means (encrypted with a one-time key by using the Diffie-Hellman protocol), the exploit kit sample becomes unusable after one processing: the analyst is unable to understand what a specific file does, reproduce the attack, and, quite often, identify the exploit and vulnerability at all.


A key exchange and getting a shellcode for the CVE-2015-2419 exploit

There is a key exchange request in the picture above. As a response, a browser gets from the threat actors’ server an encrypted array that contains a shellcode to exploit the vulnerability. The same traffic request has been used to download the Flash vulnerability exploit.

As the secret for key generation is new each time, an analyst is unable to send it to the browser once more, reproduce the attack, and identify the vulnerability, even if he has the recorded traffic.

Diffie-Hellman Protocol Implementation Features

The used implementation of the Diffie-Hellman protocol includes the following:

  1. The server generates a random number g (16 bytes) and sends the HTML page with the number g and JavaScript implementation of the Diffie-Hellman algorithm to the user’s browser.

  2. JavaScript generates a random modulo p (16 bytes) and a random private key Ka (16 bytes) in the user’s browser, and then JavaScript calculates the public key A = gKa mod p and sends the three numbers (g, A, p) to the server as a JSON object along with the Internet browser version.


  3. The server generates its own random private key Kb and its random encryption key Kx (16 bytes) and finds the Diffie-Hellman shared secret Kdh = AKb mod p. After that, the server encrypts the shellcode by using the XTEA algorithm and the key Kx, then base64_encode and urlencode, getting the string b as a result. Then, the key Kx is also encrypted by XTEA with the key Kdh, base64_encode, and urlencode, getting the string k as a result. And finally, the server calculates its public key B = gKb mod p and sends Base64-encrypted JSON object that contains B, k, and b to the browser:


    After Base64 decoding:


  4. A user’s browser calculates the Diffie-Hellman shared secret Kdh = BKa mod p, decrypts k urldecode, base64_decode, and XTEA by using the key Kdh, getting the key Kx, and eventually decrypts the urldecode, base64_decode, and XTEA shellcode by using the key Kx.

It is safe to assume that the aim of using the given sophisticated cryptographic system is shellcode interception prevention by listening to the Internet traffic between the server with the exploit kit and the user’s browser. We managed to perform a successful attack against the implementation of the encryption protocol and decrypt the shellcode. We used the modified Pohlig-Hellman algorithm for the attack (a deterministic algorithm of discrete logarithm-finding in the residue ring modulo a prime number).

According to the original algorithm, for the case when the Euler function expansion of the modulo p into prime factors qi is known (coprime factors Qi)

Attacking Diffie-Hellman protocol implementation in the Angler Exploit Kit

the complexity of finding the private key Ka and the Diffie-Hellman shared secret Kdh by using intercepted public keys A and B is

Attacking Diffie-Hellman protocol implementation in the Angler Exploit Kit

We used an optimized algorithm of finding the discrete logarithm in the residue ring modulo a prime number, taking into account the infinitesimality of logp with respect to qi, and low probability of occurrence of large prime factors raised to the power of greater than one in the Euler function φ(p); i.e., αi will equal one for large qi with a high probability. Owing to that, the complexity of the modified algorithm is

Attacking Diffie-Hellman protocol implementation in the Angler Exploit Kit

which allows us to perform a successful attack in case if all qi < 1018. The experiment has shown that the given condition is observed in more than a half of cases of using the aforementioned Diffie-Hellman protocol implementation (the case of randomly generated g, p, Ka, and Kb without their extra security checks).

Description of the Modified Pohlig-Hellman Algorithm

  1. Let us find the expansion of the number p into prime factors (the factorization can be easily done with Cryptool):

    p = 0x1b0b5c6e6b0b5b7e6c6d0b1b0a8c3c7e = 35948145881546650497425055363061529726 = 2 * 101 * 521 * 195197 * 7138079603 * 245150552958961933

  2. Let us find the Euler function for the number p:

    φ(p) = (2-1) * (101-1) * (521-1) * (195197-1) * (7138079603-1) * (245150552958961933-1) = 17761863220777184249809368812124288000

  3. Let us find the expansion of the Euler function into prime factors:

    φ(p) = 2^10 * 3^2 * 5^3 * 13 * 19 * 79 * 167 * 383 * 48799 * 45177719 * 5603527793

  4. In order to find the browser’s private key Ka, it is necessary to find a discrete logarithm:

    A = gKa mod p
    A = 0x5eff90f1c48342f5d519cd02b5dfd8b = 7892150445281019518426774740123123083
    g = 0x40a262b1360a6a16612ca8251161a9a5 = 14017453774474660607531272629759062185 (mod p)

    As immediately finding Ka modulo φ(p) is quite time-consuming, let us find Ka by turns for each of the coprime factors Qi of the Euler function φ(p)

    [1024, 9, 125, 13, 19, 79, 167, 383, 48799, 45177719, 5603527793],

    and, by using the obtained results and the Chinese remainder theorem, let us immediately find Ka modulo φ(p).

  5. In order to find Ka modulo Qi, it is necessary to find a discrete logarithm

    Attacking Diffie-Hellman protocol implementation in the Angler Exploit Kit

    To do that, we shall

    5.1. take the number H=⌊√(Qi)⌋+1;
    5.2. calculate Dc=DaH mod p;
    5.3. make a sorted table of values Dcu mod p for 1 ≤ uH;
    5.4. find such a value of 0 ≤ v ≤H, that the element Db ∙ Dav mod p is in the table;
    5.5. The value of Ka modulo Qi equals Hu-v.

    The implementation of the described algorithm in Java is given in the Appendix A. As in the reviewed example the maximum value of Qi is only several billions, the program execution time did not exceed several seconds.

    For some of the Qi factors of the Euler function φ(p), there are several possible Ka values (there are possible Ka modulo Qi values in the row number i):

  6. By going over all of the possible combinations of obtained Ka values by using the Chinese remainder theorem, we find several tens of possible Ka modulo φ(p) values:



  7. All of the obtained values of the private key Ka lead to the same value of the Diffie-Hellman shared secret Kdh = BKa mod p:


  8. By knowing Kdh, it is possible to decrypt the encryption key Kx from k and the shellcode by using Kx. The PHP script for decrypting the intercepted shellcode by using the known Diffie-Hellman shared secret is given in the Appendix B. The decrypted shellcode is given in the Appendix C.

Testing of the Diffie-Hellman Protocol Implementation Attack in the Angler Exploit Kit

To test the effectiveness and functionality of the attack, several tests were conducted.

  1. A test with a traffic dump from with the exploit for CVE-2015-2419.



    p = 146455792068641286704746413745292278846 = 2 * 2269 * 1057223 * 1292823547 * 23612186462182360807

    φ(p) = 73195553541542938096767116236244889696 = 2^5 * 3^6 * 7^3 * 17 * 617 * 7127 * 528611 * 231492024139042753

    Owing to a significantly large factor φ(p) (about 1018), finding the Diffie-Hellman shared secret took several hours:


    The decrypted shellcode is given in the Appendix D.

  2. A test with a traffic dump from with the exploit for CVE-2015-2419 and CVE-2015-5560.

    The new version of the Angler Exploit Kit has minor changes in the server-to-script communication protocol:



    As compared with the previous version, indices “g”, “A”, “p”, “B”, “b”, and “k” were replaced by the parts of the number g, and the order of the numbers sent to the server was changed (now, it is g, p, A not g, A, p as it was before). Besides that, the XTEA algorithm had two constant values and used when decrypting the shellcode bit operation modified:

    Before (the original XTEA implementation) After
    for(var h=g[0],k=g[1],l=84941944608;0!=l;)
    for(var h=g[0],k=g[1],l=433284421593;0!=l;)

    For the given traffic, we managed to factorize the Euler function φ(p)

    p = 123758666691284322087508686576379854395 = 5 * 11 * 47 * 73 * 83 * 173 * 1055371 * 43277569507162384847671

    φ(p) = 85339058418474058501009217357034700800 = 2^14 * 3^6 * 5^2 * 23 * 41 * 43 * 127 * 277 * 1949 * 102798053917762603

    find the Diffie-Hellman shared secret


    and decrypt the shellcode for CVE-2015-2419 (given in the Appendix E).

    In addition to that, threat actors started to use the Diffie-Hellman key exchange pattern also for Flash exploits in the new version of the Angler Exploit Kit (i.e., the creators of the exploit kit programmed the same algorithms in PHP, JavaScript, and ActionScript). The protocol exploit and shellcode download format for the Flash vulnerability is the same as the one for the shellcode vulnerability for Internet Explorer:

    {"4256":"425667992b18942d377eff0218961ce7","6799":"3d0d6c3b4a5b5e2c2d5d6d6e1a5a2e1a","942d":"18,0,0,209","377e":"false","2b18":"0339b845ae35e9d7af629fa2d0d0fed3"} {"4256":"014b170e00b46fd3fc35ce8766293c69","6799":"YZfySNTEMcSWl8QqrgSuGA%3D%3D","2b18":"ZEQNbP…zH5Uk%3D"}

    Modulo p and the Euler function φ(p) factors:

    p = 81152602799751951422044316006212054554 = 2 * 3 * 36329424479 * 10983441260369 * 33896452871009

    φ(p) = 27050867599169456821145398677392574464 = 2^11 * 7 * 13 * 199 * 91279961 * 11640265409 * 686465078773

    the Diffie-Hellman shared secret:


    The decrypted exploit and shellcode for CVE-2015-5560 is given in the Appendix F.

    Appendix A. The Diffie-Hellman Protocol Attack Implementation in Java

    Appendix B. Intercepted Encrypted Shellcode Decryption PHP Script by Using the Known Diffie-Hellman Shared Secret

    Appendix C. Decrypted Shellcode

    Appendix D. Decrypted Shellcode for the CVE-2015-2419 Vulnerability from the Traffic Dump of the Older Angler Version

    Appendix E. Decrypted Shellcode for the CVE-2015-2419 Vulnerability from the Traffic Dump of the New Angler Version

    Appendix F. Decrypted Exploit and the Shellcode for the CVE-2015-5560 Vulnerability from the Traffic Dump of the New Angler Version

Attacking Diffie-Hellman protocol implementation in the Angler Exploit Kit

Your email address will not be published. Required fields are marked *


  1. dev

    Please provide us MD5 of above sample

  2. merge


    please publish md5 sample

  3. Calle Svensson

    Nice article, I have a few questions.

    1. Usually when calculating the group order we use the Euler totient function, phi, defined as here: You are linking to what looks like a more general Euler function. Is there a reason for this?

    2. According to Wikipedia, the Pohlig–Hellman worst case complexity when the factorization is known is O(sum_i a_i(log(n)+sqrt(q_i))) but you have skipped the sqrt (which is technically still true but a looser bound. How come?

    3. How do you go from simplifying the first complexity to the second? Is it possible that you missed the sqrt in the first and then accidentally put Q_i=q_i^a_i instead of just q_i in the second or am I missing something?


APT trends report Q1 2024

The report features the most significant developments relating to APT groups in Q1 2024, including the new malware campaigns DuneQuixote and Durian, and hacktivist activity.

Subscribe to our weekly e-mails

The hottest research right in your inbox