The latest issue of Consumer Reports has a review of antispyware, antivirus and antispam programs. It has some people in the industry a bit confused.
Igor Muttik at McAfee has the first antivirus company public response to the review.
There are several things here that do not seem right:
- It is claimed that created viruses were “the kind you’d most likely encounter in real life” which is, of course, something the testers cannot know.
- Creating new viruses for the purpose of testing and education is generally not considered a good idea – viruses can leak and cause real trouble (you can read an open letter on the AVIEN site about that).
- There is a more scientific way of measuring real proactive detection of AV products on future malware – it is called “proactive testing” or “retrospective testing”. The idea is to measure, say, 3-month old AV product against real field viruses that appeared within these last 3 months. The discussion of the methodology of such tests can be found here and some real test results with common AV products are on the AV-comparatives.org site.
(Minor side note: He expresses some confusion about Consumer Reports reporting the results as from September 2006, but this normal procedure in the magazine business).
Creating viruses for the sake of testing is a bad idea. Our very own Joe Wells and many other luminaries in the antivirus space wrote a letter to CNET on this very issue quite a while back. It’s pretty surprising that a magazine like Consumer Reports would make such an error. There are some in the antivirus community that are appalled at what they believe to be shoddy work.
Publications need to use industry-standardized methods for testing. Organizations like Virus Bulletin have been doing this for years. Why can’t publications follow their lead?
Remember, though, that antispyware testing is quite a bit different than antivirus testing, a subject Eric Howes is taking on.
(Hat tip to Andreas Marx)
UPDATE: TechWorld article here.