Testing and Accountability

AMTSO (the Anti-Malware Testing Standards Organization) is a coalition of security professionals, including many antivirus product vendors, product testing organizations and publishers, and some interested individuals. Given the highly technical nature of its activities, it is inevitable that the organization owes some of its authority to the expertise of the security specialists within its ranks, but that doesn’t make it a vendor lobby group. As Kurt Wismer (not himself a member) points out here “many of them are employed by vendors precisely because that’s one of the primary places where one with expertise in this field would find employment.” Given some recent negative publicity aimed at AMTSO ( example), we want to collectively clarify the following points on behalf the anti-malware industry, where we come from, and indirectly on behalf of AMTSO.

We find it strange that expertise in the testing field is somehow seen as a disqualification, given the specialist expertise that characterizes the group.

While some distrust anything a vendor says and accept uncritically anything a tester says, others are puzzled that different tests can vary so dramatically in their evaluation of the same product. While this may sometimes be simply due to poor testing practice, there are other, deep-seated reasons, one being the high volume of malware and new attacks seen every day. Vendors work hard to close the gap between the ideal 100% detection and what is actually achievable, by developing a range of technologies, both proactive and reactive. The capabilities of products can change, while tests using broadly similar methodology can generate dramatically ‘conflicting’ results due to different approaches to the selection, classification and validation of samples and URLs, among other factors.

AMTSO aims to promote precisely the kinds of tests that clearly show up these variations, and its members were flying the flag for real world testing before AMTSO ever formally existed, believing that sound testing benefits vendors and customers as well as testers. As an industry, we are all too aware that we cannot currently offer detection of all known and unknown malware. The relatively high scores achieved in established tests by major vendors do not necessarily reflect real world performance, but real-world detection cannot be measured in terms of product comparison with no checks on selection, classification and validation of malicious samples and URLs.

Another misconception is that AMTSO members simply don’t like tests done by non AMTSO members. This is not the case: none of the undersigned have a problem with labs that intend to provide objective, real-world testing. (Though other testers are entitled to object vehemently when one company claims to be the only one doing live, internet-connected testing, and that all other testers are doing static testing based on the WildList).

However, charging consultancy fees for the release of any information relating to a test (even to participants) is very different to the transparency that AMTSO advocates, though we recognize that full-time testers generate revenue like any other business. However, when a tester claims to have shared information about methodology in advance, and fails to provide methodological and sample data subsequently, even to vendors prepared to pay the escalating consultancy fees required for such information, this suggests that the tester is not prepared to expose its methodology to informed scrutiny and validation, and that compromises its aspirations to be taken seriously as a testing organization in the same league as the mainstream testing organizations committed to working with AMTSO.

No-one believes that AMTSO has all the answers and can “fix” testing all by itself, but it has compiled and generated resources that have made good testing practice far more practicable and understandable. The way for testers (and others) to improve those resources is by talking to and working with AMTSO in a spirit of co-operation: the need for transparency is not going to go away.

Roel Schouwenberg, Kaspersky Lab

Luis Corrons, Panda Security

David Harley, ESET

Mark Kennedy, Symantec Corporation

Igor Muttik, McAfee