Publications

Cascading false positives

Security researchers work together and share information in many ways and in many contexts that aren’t constrained by company boundaries, but it’s unusual for security researchers working for different vendors to join forces in a company blog.

However, John Leyden of The Register contacted us both when he was writing an article on the controversy following Kaspersky Lab’s dramatic demonstration of the way in which false positives can cascade from one vendor to another. This is a major issue, because it can and does introduce a serious bias into comparative detection testing and analysis. After responding to John’s questions, we continued the discussion subsequently by email and found that we (along with most of the AV industry) were in agreement on all major points, and decided that it was more important to clarify those points, than to continue debating the detail of the demonstration.

The fact that the demonstration used Virus Total as a channel for cascading the “artificial” false positives to other vendors should not be seen as in any way detrimental to Virus Total. Hispasec have never endorsed the use of the service as a substitute for comparative testing or for sample validation, either of which are very likely to generate misleading results.

Multiple scanners are not in themselves the problem, whether they’re hosted on public sites, specialist resources, or used by testers or anti-malware companies in-house. As tools for comparative analysis or precursors to more detailed analysis, they have a great deal of value. However, that value depends on the user’s knowledge and understanding of how to make the most appropriate use of them.

Mainstream testers and security vendors have extensive understanding of these issues: however, many tests do not take them sufficiently into account. The Kaspersky Lab experiment did at least bring the issue to the attention of some of the press and publishers who most need to be aware of it, and who would probably have taken far less notice of a less controversial presentation.

As supporters of AMTSO, the Anti-Malware Testing Standards Organization, we are in emphatic agreement that away from static testing and toward dynamic testing is a positive direction. We hope that more reviewers now appreciate that dynamic testing with small but properly validated sample sets offers more realistic assessment of detection capability with less risk of unintended bias. If more people realized this, it would allow vendors to spend more time on real threats and less on making sure they detect samples that shouldn’t be included in a test set.

Magnus Kalkuhl, Senior Virus Analyst, Kaspersky Lab
David Harley, ESET Research Fellow & Director of Malware Intelligence

Cascading false positives

Your email address will not be published. Required fields are marked *

 

Reports

APT trends report Q2 2021

This is our latest summary of advanced persistent threat (APT) activity, focusing on significant events that we observed during Q2 2021: attacks against Microsoft Exchange servers, APT29 and APT31 activities, targeting campaigns, etc.

LuminousMoth APT: Sweeping attacks for the chosen few

We recently came across unusual APT activity that was detected in high volumes, albeit most likely aimed at a few targets of interest. Further analysis revealed that the actor, which we dubbed LuminousMoth, shows an affinity to the HoneyMyte group, otherwise known as Mustang Panda.

WildPressure targets the macOS platform

We found new malware samples used in WildPressure campaigns: newer version of the C++ Milum Trojan, a corresponding VBScript variant with the same version number, and a Python script working on both Windows and macOS.

Ferocious Kitten: 6 years of covert surveillance in Iran

Ferocious Kitten is an APT group that has been targeting Persian-speaking individuals in Iran. Some of the TTPs used by this threat actor are reminiscent of other groups, such as Domestic Kitten and Rampant Kitten. In this report we aim to provide more details on these findings.

Subscribe to our weekly e-mails

The hottest research right in your inbox