What’s the deal with security product testing anyway? | Webroot

It’s common for savvy online shoppers to check third-party
reviews before making an online purchasing decision. That’s smart, but testing
the efficacy of security software can be a bit more difficult than determining
if a restaurant had decent service or if clothing brand’s products are true to
size.

So, with the arguably more significant consequences of
antimalware testing, how can shoppers be sure that the product they choose is
up to the task of protecting their family from malware? Which reviews are
worthy of trust and which are just fluff?

Red flags in antimalware testing

Grayson Milbourne is the security intelligence director at
Webroot and actively involved in improving the fairness and reliability of
antimalware testing. While acknowledging that determining the trustworthiness
of any single test is difficult, some factors should sound alarm bells when looking
to honestly evaluate antimalware products.

These include:

The pay-to-perform model

In any test, the humans behind the product being evaluated
have a vested interest in the performance. How far they go to influence those
results, however, varies. One extreme way to positively influence results is to
fund a test designed for your product to succeed. Often, the platform on which
a test appears can be a sign of whether this is the case.

“YouTube tests are almost always commissioned,” warns
Milbourne. “So, if you see things on YouTube, know that there is almost always
someone paying for the test who’s working the way the test comes out. I try to
avoid those.”

If only one product aces a test, that’s another sign that it
may have been designed unfairly, maybe with the undisputed winner’s strengths
in mind.

Every vendor acing a test

Tests in which all participants receive high scores can be
useless in evaluating product efficacy. Because we know catching malware is
difficult, and no single product is capable of doing it effectively 100 percent
of the time, tests where every product excels are cause for concern.

“If every product aces the test, maybe that test is a little
too easy,” says Milbourne. No product is perfect, so be wary of results that
suggest so.

Failing to test in “the big picture”

No one piece of software can stop all the threats a user may
face all of the time. But many vendors layer their preventative
technologies—like network, endpoint and user-level protection—to most
effectively protect against cyberthreats.

“Testers are still very worried about what happens when you
encounter a specific piece of malware,” says Milbourne. “But there’s a lot of
technology focused on preventing that encounter, and reactive technology that
can limit what malware can do, if it’s still unknown, to prevent a compromise.”

In addition to how well a product protects an endpoint from
malware, it’s also important to test preventative layers of protection which is
lacking in 3^rd party testing today.

The problem with the antimalware testing ecosystem

For Milbourne, the fact that so few organizations dedicated
to efficacy testing exist, while the number of vendors continues to grow, is
problematic.

“There are about five well-established third-party testers
and another five emerging players,” he says. But there are well over a hundred
endpoint security players and that number is growing.”

These lopsided numbers can mean that innovation in testing
is unable to keep up with both innovation in security products as well as the
everchanging tactics used by hackers and malware authors to distribute their
threats. Testing organizations are simply unable to match the realities of
actual conditions “out in the wild.”

 “When security
testing was first being developed in the early 2000s, many of the security
products were almost identical to one another,” says Milbourne. “So, testers
were able to create and define a methodology that fit almost every product. But
today, products are very different from each other in terms of the strategies
they take to protect endpoints, so it’s more difficult to create a single
methodology for testing every endpoint product.”

Maintaining relationships in such a small circle was also
problematic. Personal relationships could easily be endangered by a bad test
score, and a shortage of talent meant that vendors and testers could bounce
between these different “sides of the aisle” with some frequency.

Recognizing this problem in 2008, antimalware vendors and
testing companies came together to create an organization dedicated to
standardizing testing criteria, so no vendor is taken off guard by the
performance metrics tested.

The Anti-Malware Testing
Standards Organization (AMTSO) describes itself as “an international
non-profit association that focuses on addressing the global need for
improvement in the objectivity, quality and relevance of anti-malware testing
methodologies.”

Today, its members include a number of antivirus and
endpoint security vendors and testers, normally in competition against one
another, but here collaborating in the interest of developing more transparent
and reliable testing standards to further the fair evaluation of security
products.

“Basically, the organization was founded to answer questions
about how you test a product fairly,” says Milbourne.

Cutting through the antimalware testing hype

Reputation within the industry may be the single most
important determinant of a performance test’s trustworthiness. The AMTSO, which
has been working towards its mission for more than a decade now, is a prime
example. Its members include some of the most trusted names in internet
security and its board of directors and advisory board are made up of seasoned
industry professionals who have spent entire careers building their
reputations.

While none of this is to say there can’t be new and
innovative testing organizations hitting the scene, there’s simply no
substitute for paying dues.

“There are definitely some new and emerging testers which
I’ve been engaging with and am happy to see new methodologies and creativity
come into play, says Milbourne, “but it does take some time to build up a
reputation within the industry.”

For vendors, testing criteria should be clearly
communicated, and performance expectations plainly laid out in advance. Being
asked to hit an invisible target is neither reasonable nor fair.

“Every organization should have the chance to look at and
provide feedback on a tests’ methodology because malware is not a trivial thing
to test and no two security products are exactly alike. Careful review of how a
test is meant to take place is crucial for understanding the results.”

Ultimately, the most accurate evaluation of any antimalware
product will be informed by multiple sources. Like reviews are considered in
aggregate for almost any other product, customers should take a mental average
of all the trustworthy reviews they’re able to find when making a purchasing
decision.

“Any one test is just one test,” reminds Milbourne. “We know
testing is far from perfect and we also know products are far from perfect. So,
my advice would be not to put too much stock into any one test, but to look at
a couple of different opinions and determine which solution or set of solutions
will strengthen your overall cyber resilience.”