Good guys doing bad things, part 2

A few days ago David wrote about ConsumerReports, which created around 5,500 new virus variants in order to test antivirus solutions. Like most antivirus companies, we weren’t particularly impressed by this.

Recently a writer for heise.de, probably the best known German IT website, picked up on the topic, criticizing the reaction of antivirus companies: “[they] fail to notice that they sound like Mercedes dealers complaining about the ‘elk test’ – arguing that there are enough real accidents to analyze the safety measures of their cars.”

This comparison is specious: in the context of antivirus testing, the ‘real accident’ is a computer or network infected by in the wild malware, and the ‘elk test’ is controlled testing under laboratory conditions. We’ve got nothing against controlled testing, as long as it uses malware which exists in the same form in the wild. We’re also in favour of testing solutions which have deliberately not been updated – old signatures mean that heuristics and proactive protection technologies can be fully tested.

I can’t see any benefit in using newly created variants of existing malware in tests. And the argument that these new creations won’t be made publicly available is irrelevant here. At the end of the day, such tests could lead to an atmosphere of open competition, with the testers attempting to trick as many antivirus solutions as possible by using more new and different malware. Of course, this would all be in the name of security… but it could decrease the amount of effort virus writers have to put in, with the burden ultimately being borne by end users.