Fully tested

As you may have read the AMTSO had another meeting a couple of weeks ago. AMTSO is strongly committed to improving the overall relevance of anti-malware testing.

During our latest meeting we accepted two new papers. The first paper is on whole product testing and the second is on performance testing. As pointed out by the vast majority of people in the AV space the tests of old have never truly accurately reflected real life performance. With the changes that the threat landscape has seen over the past few years this has become truer than ever.

So rather than having tests which focus on individual components to test detection of, or rather protection against, threats an entire product should be tested. Just think about a scenario where an email-borne threat is not detected by the file scanner, but the anti-spam component is able to flag the message it comes with as spam.

The other document talks about how to more accurately test the performance – or speed (impact) – of AV solutions. One scenario where this will be useful could be determining the amount of RAM a certain product may occupy. Many people will try to establish this by looking at the amount of (virtual) memory taken by the processes belonging to the product. However, certain products may also inject some of their DLLs into other processes, therefore unintentionally masking some of their footprint. It’s therefore the best practice to compare the entire RAM usage.

The bad news, I say jokingly, comes from one of the new documents we continued working on in Helsinki. The False Positive testing document has proven to be quite the challenge and sparked a lot of debate. Especially the area of testing false positives on web resources – such as domains and web scripts – an interest of mine, proved to be particularly challenging.

It definitely looks like testers are continuing to improve their tests to more accurately reflect real life scenarios. And that’s great for two main reasons. Most importantly, it gives users better information. Secondly, it gives vendors the opportunity to spend their resources focusing on things that protect the user. So it’s great to see the progress that we’re making in AMTSO.

If you haven’t had a chance to read the documents go to the AMTSO web site and have a look!