Secunia tests

By now most people have seen the Secunia test results and all the ensuing discussions. Frankly, I was a bit surprised by the vehemently negative reaction from a number of AV vendors.

And it doesn’t seem to be about the 20% difference between the ‘winner’ and the rest. Criticism has focused on the testing methodology, which many people thought was dubious. Some of the suggestions were useful – mostly those from Andreas Marx, the well-known AV solutions tester from Germany. The general tone, though, seems to be that many AV vendors thought their results would have been a lot better if the test methodology had been different. And maybe they’re right.

But I think people are too focused on looking for mistakes in the tests and/or attempting to explain their poor PoC detection rates. Sure, criticizing Secunia’s testing methods is justified, but only if we’re discussing testing methodology, and nothing else.

As I see it, Secunia wasn’t trying to highlight the weaknesses of AV solutions – I think they were trying to make a different point…

At Kaspersky, we’ve taken a decision not to detect PoC vulnerabilities – it’s far more sensible to focus on protecting users from the real threats and exploits that are being used by malware authors in the real world. That’s what our antivirus databases are for. The point isn’t so much that detecting PoCs is a pretty difficult task (although the test results clearly show that even Microsoft and Symantec, with all of their resources, didn’t fare all that well) but that detecting PoC s is a dead end, and doesn’t address the fundamental problem.

So what is the problem?An abundance of vulnerable applications. And the solution for this problem doesn’t lie in detecting 65% or even 99% of PoCs. Nor does it lie in good or bad AV testing methodology. The only real solution is proper patch management. In the context of the post test discussion, I get the feeling that a lot of people are conveniently forgetting or ignoring Secunia’s “What to do” list:

Fortunately, the AV industry is taking steps to tackle the patching issue. Our product, Kaspersky Internet Security 2009 is so far the first and only product to contain a vulnerability scanner. It identifies applications that have unpatched vulnerabilities – a log gives details of the vulnerability, including a name, threat level and what needs to be done to install the necessary patches.

This is just a first step towards a fully-functional system for managing risk on personal computers, and we’re going to continue active work in this area.
We need to treat the disease, not the symptoms. In this case, the disease is all the vulnerable applications which pose a potential risk that is exacerbated by users’ lack of knowledge. And this is not something the AV can, or should, tackle alone – it’s a matter of security in general.

Moreover, no AV vendor, no matter how well they do on such tests, has the right to say ‘Great, we protect you against all exploits, so you needn’t patch’. No company would dare say this, and everyone agrees patching is necessary. This fact alone leaves those who are hotly discussing Secunia’s test results and methodology without a leg to stand on.

We’re happy about the increased awareness of vulnerabilities and the responsibilities of AV vendors that we’re seeing. The AV industry can’t begin solving the problem of patching soon enough for me. We need both new technologies and user education – we need to talk about patch management until home users understand that it plays just as big a role in security as AV software does.

Patch management begins with the head, and not with the software.