A race against the spammers

A few days ago, the latest VBSpam results were published. The testing, conducted by Virus Bulletin in August, saw Kaspersky Linux Mail Security 8.0 detect 99.93% of all the spam messages used in the test. This is a new record for Kaspersky of which we are very proud (if the number of congratulatory emails flying back and forth between us is anything to go by). Eugene Kaspersky also mentioned the result in his blog (http://eugene.kaspersky.com/2012/09/27/kaspersky-server-anti-spam-no-longer-the-underdog-more-top-dog/)  – he’s proud of us too 🙂

A few days ago, the latest VBSpam results were published. The testing, conducted by Virus Bulletin in August, saw Kaspersky Linux Mail Security 8.0 detect 99.93% of all the spam messages used in the test. This is a new record for Kaspersky of which we are very proud (if the number of congratulatory emails flying back and forth between us is anything to go by). Eugene Kaspersky also mentioned the result in his blog (http://eugene.kaspersky.com/2012/09/27/kaspersky-server-anti-spam-no-longer-the-underdog-more-top-dog/)  – he’s proud of us too 🙂

This high detection rate did not impair the quality of detection either – there was just one false positive out of the more than 10,000 messages that constituted the non-spam collection used by VB during testing.

Although we truly are happy with this excellent result, we did kind of see it coming: KLMS 8.0 is an aggregate of all the new features and technologies recently developed at Kaspersky Lab. All those new features and technologies were designed to help our anti-spam solution detect spam faster.

About a year ago, spammers actively began using a technique that had already been tried and tested. In a race to stay ahead of the spam analysts and anti-spam database updates, they started sending millions of messages to their mailing lists within very short periods of time – several minutes to half an hour. As a result, part of the spam mailing successfully made it into users’ inboxes before security products could receive updates and start blocking the new spam messages. These rapid-fire spam mailings have until recently been the main problem for anti-spam solutions.

It was in response to this that Kaspersky Lab developed the new technologies that would become part of KLMS. In fact, Kaspersky Lab’s response was threefold.

Early in 2012, our technology of enforced anti-spam database updates in real time became available. This technology made the delivery of the most critical updates to the product much faster. In the Spam Lab, we call it Mobius; in the product, it’s called Enforced Anti-Spam Update Service. With this technology, it takes less than a minute from the creation of an anti-spam record in the Spam Lab to it being delivered to the product!

The Mobius server backend receives the text and image signatures from the Spam Lab as soon as the analysts add them to the databases. These signatures are the most critical in terms of delivery time, so these signatures are the ones our new service works with. At this stage, the databases are automatically tested for errors. Once tested, the update is delivered to the front end. The front end immediately sends the update to the Mobius client and into the product, using the TCP connection which is maintained continuously between these two components.

Then, the updates are released virtually as fast as the spam messages are sent. But do you remember the story about the tortoise and the hare? Instead of trying to compete with the hare the cunning tortoise seeks the help of a friend. One tortoise stood at the start while the other appears at the finish line. Meanwhile, the silly hare kept running to and fro between them. Naturally, Mobius gave good engines to our “tortoise”; however, even before Enforced Anti-Spam Update Service became available, an idea emerged that meant there was no need to try and keep up with the hare, but simply sit in the bushes – in our case, in the cloud – and await his arrival.

This idea was implemented in UDS (Urgent Detection System) and, over time, evolved into UDS 2. UDS2 is a cloud-based security system than provides two-way communication between Kaspersky Lab’s users and the company experts. When a spam message is processed, at the same time a signature is added to the database, the hash value is calculated for the message and posted in the cloud. Earlier, we used to take 16-byte fuzzy hashes; after UDS2 was launched, we started to take 4-byte shingles, or micro-signatures. Shingles work much more effectively when it comes to junk added into text and when text fragments are simply swapped within spam messages. On the user side, shingles are also created for suspect messages and sent to the same cloud; the micro-signatures that have been created in the Spam Lab based on the spam messages are already available in the cloud. The signatures received from the user are compared against those stored in the cloud.

The first generation UDS simply responded to requests, whether or not a specific signature existed in the database. UDS2 clusters signatures together based on their similarity, and the Content Reputation technology based on UDS2 calculates a spam reputation for them. The first version of this technology is called Content Reputation v1 (Rescan); basically, it is a quarantine that delays for some time (currently 50 minutes) mail that is not recognized as spam, but is considered suspicious as the signatures created by it coincide with clusters with high spam reputations.

Normally, such messages are few in number: most mail traffic is either detected as spam or recognized as reliable. The 50-minute delay gives the analysts in the anti-spam lab time to make a final decision on whether the suspect messages are in fact spam or not. Thus, not only do we anticipate spam deliveries – we delay them as well!

Autoblock is yet another technology based on UDS 2. If a message’s signature falls into a cluster with a high spam reputation, it is automatically blocked in the cloud. With this technology, an analyst can check a blocked message later; if it is not considered to be spam, the block will be immediately rolled back automatically.

So, in the next stage of the race we have every chance of outrunning our opponents!