2013/2014 Web Vulnerability Scanners Comparison – Netsparker Confirmed as a Market Leader

NOTE: Read the article 2015 – How Does Netsparker Compare with other Automated Web Application Security Scanners? for more recent information about Netsparker scanners’ scanning capabilities.

Earlier on this month, information security researcher and analyst Shay Chen released the 2013/2014 Web Application Vulnerability Scanners Benchmark, where he compared 63 different web vulnerability scanners, or as they are also known web application security scanners.

The comparison contains a good wealth of information and for those who have time, it is worth to dive into and analyze all of the results. We of course already did our homework; we analyzed the results and are more than happy with results; Netsparker Web Application Security Scanner smoked the competition and is only second to IBM AppScan with only 4% difference; a scanner that costs much more than Netsparker.

Hence when you also include the price in the equation, Netsparker is the best web vulnerability scanner with the best return on investment; while IBM Appscan has an expensive price tag users still have to spend a lot of time to verify its findings as opposed to Netsparker, which automatically checks its own findings to report no false positives.

How did Netsparker Perform When Compared to Other Scanners?

There are several different angles on how you can look at the results to determine which is the best web vulnerability scanner for you. To start off with, below are the graphs for each web vulnerability class tested in this benchmark:

SQL Injection Vulnerabilities Detection

Netsparker detected all of the 136 SQL injection vulnerabilities like most of the other web vulnerability scanners. Only NTOSpider and N-Stalker did not detect all SQL injection vulnerabilities.

Cross-Site Scripting Vulnerabilities Detection

Netsparker detected all of the 66 cross-site scripting vulnerabilities like most of the other web vulnerability scanners. Only BurpSuite and N-Stalker failed to detect all XSS vulnerabilities.

Path Traversal / Local File Inclusion Vulnerabilities Detection

Here is where things start becoming interesting; IBM AppScan and Netpsarker are on a league of their own when it comes to detecting directory traversal and local file inclusion vulnerabilities.

According to the benchmark, IBM AppScan detected all vulnerabilities while Netsparker missed 30. However, when these vulnerabilities were scanned individually, Netsparker identified them all. This leads us to discover a very rare bug that only happens when a particular custom 404 configuration happens (it’s quite hard to see this bug in the real world, hence we didn’t know about it before this benchmark). We are looking into this issue and addressing it. So if you scan for these vulnerabilities individually, or if they were on a different website Netsparker would have identified them all. Feel free to download the benchmark and test it yourself to see this.

Third placed NTOSpider missed 154, HP WebInspect missed 228, Acunetix WVS missed 348 and so on.

XSS via RFI Vulnerabilities Detection

In this case, only Netsparker, IBM Appscan and HP WebInspect detected all 108 XSS via remote file inclusion vulnerabilities. Next in line is NTOSpider which detected 86 instances, and then Acunetix which detected 84 instances.

Unvalidated and Open Redirects Vulnerabilities Detection

HP WebInspect detected the most invalidated redirect vulnerabilities by detecting 15 out of 60, followed by Netsparker and IBM AppScan with 11 detections.

Old Backup Files Detected

Acunetix WVS leads the pack in this test, by detecting 60 out of 184 backup files. Then followed by Burpsuite with 46 detections, SyHunt with 34 detections and the rest follow.

Total Number of Identified Vulnerabilities

All Web Application Vulnerability Classes

After going through each individual vulnerability class chart, now it’s time to add up all vulnerabilities together and see how the scanners performed over all. As per the chart below, Netsparker and IBM AppScan were the only two automated web vulnerability scanners to identify more than 1,000 web application vulnerabilities. Both scanners lead thanks to excellent detection of critical path traversal and LFI vulnerabilities.

Netsparker detected 1,112 vulnerabilities and is only second to IBM AppScan, which detected 1,147 vulnerabilities. Next in line is NTOSpider with 958 vulnerabilities, then  HP Webinspect with 917 vulnerabilities followed by Acunetix, which detected 819 vulnerabilities. BurpSuite, Syhunt and N-stalker follow with 791, 716 and 484 identified vulnerabilities respectively.

Direct Impact Web Application Vulnerabilities

Below is another chart showing how many direct impact vulnerabilities each web vulnerability scanner detected. By direct impact we mean critical vulnerabilities that if exploited could affect the operations of the web application and the business itself, hence excluding the “Old backup files” and “Unvalidated / Open redirects” vulnerabilities from this chart.

As we can see after excluded non-direct impact vulnerabilities the performance of the 2 major players was unaffected. The performance of all other scanners, especially the last 4 in the group was drastically affected, in a negative way. This shows that both IBM AppScan and Netsparker are more focused on identifying critical vulnerabilities.

False Positives and Web Security Scans Time Consumption

When compared to comparisons of previous years, all web vulnerability scanners improved their detection rate and all of them managed to reduce the number of reported false positives. Funnily enough Netsparker, the only false positive free web vulnerability scanner reported 3 false positive SQL Injection vulnerabilities. How did this happen?

To start off with, Netsparker is shipped with an exploitation engine that is automatically triggered once a vulnerability is detected. If the vulnerability is exploited it is not a false positive.

During these tests, Netsparker detected all of the 136 SQL injections and reported 3 additional ones. Netsparker exploitation engine confirmed all of the 136 valid SQL injections but was unable to confirm the 3 additional false positive ones, which were specifically marked as unconfirmed.

The Time Efficiency Factor – Netsparker Still Leads the Way

Even though Netsparker reported 3 false positive SQL injection vulnerabilities, it still leads the pack. When using Netsparker, the user only has to verify the 3 unconfirmed vulnerabilities.

On the other hand, all other web vulnerability scanners do not have an exploitation engine, hence the user has to confirm all vulnerabilities. Therefore in the case, a normal user would have had to confirm 136 SQL injection vulnerabilities, which might take quite a bit of time!

Which is the Best Web Vulnerability Scanner?

The best web vulnerability scanner is the one which detected most vulnerabilities, is the easiest to use and can automate most of your work.  As we all know, users have to verify a scanners findings, therefore automated vulnerability confirmation is also something that should be considered in the equation. Verifying findings is a time-consuming process and for sure you are better off spending time remediating the issues rather than verifying the findings.

How to Choose the Best Web Vulnerability Scanner for You?

Although the above statistics are a good indication of who are the web application security market leaders, don’t base your judgement just on these facts. There is no better way to determine which is the best tool for you than to get your hands dirty and scan some of your own test websites with a number of different web vulnerability scanners.

If you are new to this geeky world of automated scanning, the article how to evaluate web vulnerability scanners will give you a better insight of how to choose the right web scanner for you. And if you’d like to learn more, read this Getting Started with Web Application Security.

What is Next for Netsparker Web Application Security Scanner?

Of course, we are very happy that even though we are the youngest contender in this industry, we are already up there with the major players such as IBM AppScan, although we have to admit that it would have been even more awesome if we beat them as well.

We have done very well in identifying almost all critical vulnerabilities and can see that our lowest point is detecting old backup files on websites. We never really focused on these type of issues since the cost of identification and the worthy of finding is not of a great value. However, we will ship these checks as an option in the upcoming releases, so users will have the option to enable them during a web application security scan. We will continue working hard to ensure that Netsparker is easy to use and can detect as much web application vulnerabilities as possible automatically.

Last but not least we would like to thank Shay Chen for all his professional work and dedication.