Web Vulnerability Scanners Comparison –
Invicti (formerly Netsparker) Confirmed a Market Leader

The latest independent web application security scanners benchmark results have been published. How did Invicti (formerly Netsparker) fare when compared to the other web vulnerability scanners? In short, Invicti was:

None of the other web vulnerability scanners in the comparison, including the open source ones, performed as well as Invicti. For more detailed information about these comparisons, including results of the vulnerability detection rates, read on. This post also explains how the vulnerability scanner tests were conducted and displays the results of each individual test.

Please note: Netsparker is now Invicti. While the results of the benchmark test remain valid, this post may not reflect the latest product names and features.

It is a test that compares the features, coverage, vulnerability detection rate and accuracy of automated web application security scanners, also known as web vulnerability scanners or dynamic application security testing (DAST) solutions.

Individual tests were conducted by the independent information Security Researcher and Analyst, Shay Chen. He compared both commercial and open source vulnerability scanners, but in these results we are only focusing on the commercial solutions.

Shay has been conducting benchmark tests and improving the platform since 2010. So far he has released six (2010, 2011, 2012, 2013/2014, 2015, 2017/2018). His work is considered the de facto comparisons results by the application security industry.

How Are Tests Performed?

Shay Chen and his team built The Web Application Vulnerability Scanner Evaluation Project (WAVSEP), a testbed that they scan to see how every scanner performs. In it the scanners are tested against realistic setups, with crawling the most basic HTML website to identifying security vulnerabilities typically found in modern Single Page Applications (SPA). The WAVSEP is an open source project and new tests are incorporated every year. You can download it from the WAVSEP GitHub repository.

Testing Vulnerability Management & Assessment Solutions

This year Shay and his team went a step further. They have been installing and integrating DAST solutions in real-life enterprise SSDLC (Secure Software Development Lifecycle) processes to get a better understanding of how they can expand the WAVSEP testbed and test the scanners. So the have implemented automated vulnerability scanners in financial, hi-tech and telecom organizations.

They wanted to test more than security vulnerability detection rates. They wanted to see how these tools can really help business improve their vulnerability management and triaging processes, and their information security programme. As Shay himself explains:

Some of these experiences led us to develop test cases aimed to inspect issues in proclaimed features that we noticed didn’t work as expected in actual implementations, and some to the creation of comparison categories that are apparently crucial for real-world implementations.

The Negative Impact of False Positives

Shay and his team also talked about the importance of accurate scan results in the report, after their first-hand experience with scanners in real-life environments. Quoting from the official benchmark results:

Weeding out a reasonable amount of false positives during a pentest is not ideal, but could be performed with relative ease. However, thousands upon thousands of false positives in enterprise SSDLC periodic scan scenarios can take their toll.

False positives occur in scan results to the detriment of the web application security industry. So much so, that large organizations, that have hundreds or even thousands of web applications, limit their efforts to a handful of mission-critical websites and ignore the rest. I was quite shocked to learn this, though it is unsurprising because many hacks and data leaks that happen every year.

False Positives Make Scaling Up Web Security Impossible

If a solution reports false positives, it is impossible – unless you have an army of people – to scale up your efforts and secure all your web applications. Even if you have the budget for such an undertaking, there is still the troublesome problem of human error.

This is why we developed Invicti’s proprietary Proof-Based Scanning, technology that automatically verifies detected vulnerabilities – proving they are real flaws, and not false positives. The benefits of such technology are plentiful, and since the scan results are accurate, you can easily scale up your efforts. In a real-life environment, with thousands of web applications, you can start the vulnerability triage process and fix them within a matter of hours.

Evaluation Criteria

In the 2017/2018 benchmark tests, Shay and his team included several previously uncovered aspects of scanners and new tests to check the detection capabilities of previously uncovered vulnerabilities. This included OS command injection, and repurposing XSS via RFI tests that can also be used for server-side request forgery (SSRF) evaluation.

You might notice that vendors such as Qualys, Tenable Nessus, Retina and Nexpose are not mentioned in these comparisons. We have checked with Shay and he confirmed that he contacted all vendors but not all of them wanted to contribute and participate towards these benchmarks.

Graph with Global Detection & False Positives Rates

This graph is a visual representation of the global results, illustrating both the vulnerability detection and false positives rates side by side for each vendor.

OS Command Injection Detection

The OS command injection vulnerability tests is one of the new tests. Invicti was the only scanner to detect all the vulnerability instances in the test.

Remote File Inclusion / SSRF

This was also one of the new tests included in the WAVSEP benchmarking tests. Invicti and WebInspect were the only two scanners that detected all the vulnerabilities in this test. AppSpider followed with 82.67%, and then Burp Suite with 74.67%. Though Burp Suite also had 16.67% false positives.

Path Traversal

This time Invicti and Appscan led the field, both of which detecting all the path traversal/directory traversal vulnerabilities. HP Webinspect came in second, followed by Appspider. Burp Suite was the scanner that detected the least at 78.31% and also reported 12.5% false positives.

SQL Injection

This is one of the classic tests; the SQL injection vulnerability. In this test, Invicti and Appscan detected all the vulnerabilities. HP Webinspect followed with 98.46%. None of the scanners reported any false positives in this test.

Reflective Cross-site Scripting (XSS)

All scanners but Burp Suite detected all the cross-site scripting vulnerabilities.

Unvalidated Redirect

In the unvalidated redirect vulnerability tests two of the scanners, WebInspect and AppScan, reported vulnerabilities. Appscan also performed very poorly with a detection rate of only 36.67%. On the other hand, Invicti and AppSpider detected all the vulnerabilities.

As a rule of thumb, nothing beats a live environment test. Though it impossible to test all the web security scanners available on the market. So, these comparisons are incredibly useful because they highlight who the market leaders are – those vulnerability scanners that can detect the most vulnerabilities and generate accurate results.

Once you determine which two or three solutions you’d like to test, request a trial from the vendor to test the vulnerability scanner. In fact, at Invicti we always encourage prospects to test our web security solution by scanning a staging copy of their web applications, as explained in How to Evaluate Web Application Security Scanners.

To do such test it is really easy – register for a trial of Invicti Enterprise or install Invicti Standard on a Microsoft Windows virtual machine.

Which is the Best Web Application Security Scanner?

The best web vulnerability scanner is the one that detects the most vulnerabilities in your web applications, is easiest to use and can help you automate most of your work. Finding vulnerabilities in a web application is not just about the duration of the scan, but how long it takes to setup the scan (pre-scan) and verify the results (post scan). How long it takes you to complete the whole process including the triaging of vulnerabilities and testing of fixes. Therefore, when you evaluated solutions, you should ensure that automated vulnerability confirmation is part of the equation.

Read Shay Chen’s full report: Evaluation of Web Application Vulnerability Scanners in Modern Pentest/SSDLC Usage Scenarios.

Can Invicti Identify Security Flaws in Your Web Applications and APIs?

Invicti can scan any type of web application, regardless of the technology it was built with. It uses a Chrome based crawling engine and can identify vulnerabilities in legacy, and custom built, modern HTML5, Web 2.0 applications and Single Page Applications (SPA). It also has vulnerability checks for popular frameworks, libraries and popular open source software such as WordPress, Joomla! and Drupal.

The Invicti vulnerability scanner is very easy to use and most of the pre-scan configuration can be automated. It is an all in one vulnerability management solution, with multi user support and integration capabilities. Though to test it all you need to do is specify the URL and credentials (to scan password protected websites), and launch a vulnerability can.

Past Comparisons Between Automated Web Application Security Scanners

See the previous results for the comparisons between the 2015 web application security scanners and 2013-2014 web application security scanners.