How to evaluate web application security scanners

Ask 20 penetration testers which web application security scanner they prefer to use and you will get 20 different answers, if not more. Every web vulnerability scanner has its own pros and cons and what works for Mr. X does not necessarily work for Mr. Y, as with everything else. Therefore, you shouldn’t base your purchasing decision or build a web application security program only on what your colleagues say or think or on single factors such as whether it’s an open-source application, whether it’s available for Windows or Linux, how nice the user interface is, and so on. The only sensible way is to do some testing and make your own decision.

This guide will explain how to evaluate vulnerability scanning tools and choose the right vulnerability detection tool that fits your requirements. Note that this list focuses on tools for dynamic application security testing (DAST), not products for static application security testing (SAST) that check the application source code.

Make a list of requirements

Before getting your hands dirty with web application scanning, compile a list of your current web security requirements. The most typical ones are:

• Automating tasks: There are several advantages to automating the identification of web application security vulnerabilities. The most common reason from a management point of view is to save time, but it is not just about that. Read more in the following article: Why web vulnerability testing needs to be automated.
• Lowering the costs of web application security: Doing in-house scanning can be far cheaper than hiring a seasoned expensive penetration tester or consultant every time.
• Increasing the coverage: As opposed to penetration testers, an automated web application security scanner has an extensive set of heuristic web vulnerability checks that are frequently updated by a dedicated team of researchers and security experts. This allows users to identify all types of web application vulnerabilities in custom-made web applications. Some scanners also have a vulnerability database for known web applications such as WordPress and Joomla, which is useful if your business or customer uses such web applications.

In short, automated web application security scanners are mostly expected to save you time during application security testing and ensure that all technical web vulnerabilities are identified. Once you’ve listed your requirements, proceed to gather as much information as you can about the websites and web applications that you will be scanning.

Document all the web applications to be scanned

The next step in the selection process is to document the web applications that you will be scanning using the automated web application vulnerability scanner. During this stage, it is important to identify the most common factors of web applications. For example, if something is implemented in 2% of the web applications that you are scanning, then you can test such parts manually. But if something is implemented in 60% of web applications or more, then its testing should definitely be automated. The following list of questions will help you build a list of common factors for your web applications:

1. Which development frameworks are used to build web applications?
2. Is there any type of authentication mechanism (access control) used in web applications?
3. What type of database backend server is used?
4. Are URL rewrite rules being used?
5. Do you need a client-side certificate to access web applications?
6. Are there any client-side scripts that need to executed and tested?
7. Do you use custom 404 error pages?
8. Are there any types of protection mechanisms implemented in web applications, such as Anti-CSRF mechanisms?

Once you have the list of common factors, you can shortlist the web application security tools that you would like to evaluate. For example:

1. If all your web applications are developed with PHP and .NET, you do not need to consider automated tools that cannot scan and identify vulnerabilities in such web technologies. For example, some web application security scanners cannot automatically crawl, scan and identify vulnerabilities in web applications built with .NET, hence there is no need to test such scanners.
2. If authentication is implemented, automated web security scanners that you want to evaluate must support the type of authentication mechanism that is being used and you must be able to configure them to automatically log into the website during a web security scan.
3. If URL rewrite rules are being used for search engine-friendly URLs, the scanner should be able to identify such URL structures and scan the website properly without getting stuck in some sort of infinite loop and without requiring extensive configuration changes.
4. If the web applications you want to scan have anti-CSRF protection or a similar mechanism, the scanner should still be able to automatically scan the web application without missing any parts of it. Note that such safeguards are common barriers for most automated tools.

Testing web application security scanners

Once you have completed all the paperwork, it is time to get hands-on and start scanning websites. Contact the relevant vendors to get a trial or demo version of each web vulnerability scanner that you would like to evaluate.

How web application security scanners work

If you are new to automated tools such as web application security scanners, before doing anything else, you need to understand how such tools work. First, the scanner crawls the target website or web application and identifies all possible web application attack entry points and parameters. During this stage, the crawler accesses every link that it discovers, including links in client-side scripts and similar sources. During the scanning stage, the scanner sends specially crafted HTTP requests, which include a payload simulating a known vulnerability that is used to test if a website is vulnerable or not.

Although automated scanners are designed not to be intrusive, there is still a small probability that such attacks might affect your web application or database, depending on how secure your web application already is. Keeping this in mind, we move to the next point.

Always scan realistic test and staging web applications

First and foremost, it is important to scan realistic web applications, i.e. applications that are similar or ideally identical to the real live ones that you will be scanning in production. Unfortunately, many people evaluate web application security scanners against vulnerable web applications such as DVWA (Damn Vulnerable Web Application) and OWASP WebGoat developed by the Open Web Application Security Project. Such web applications have been built for educational purposes only and might be completely different to real-life web applications that your business or customers use.

Therefore, do not base your decisions on scan results from such demo sites. Don’t forget that a web app security scanner can easily cover thousands of different vulnerabilities, yet these simple test systems will only show how the scanner behaves for maybe ten of these thousands of vulnerability cases. To evaluate the true effectiveness of a web application security scanner, you should always run your tests on real websites and application in test or staging environments.

It might take some time until you get used to the automated tools that you are using, so start by playing it safe and only scanning test websites. With time, you will get used to the tools that you are using and soon you should be able to scan live sites. In fact, it is recommended to scan both staging and live sites and applications because some vulnerabilities (such as issues caused my misconfigurations) might only be introduced when switching from the staging server to the live server.

Evaluating web application security scanners and their scan results

Now that you know what you need and how to evaluate the software, it is time to fire up the scanners. Here are some evaluation guidelines that should help you determine the best automated web application security scanner that fits your requirements.

Web application coverage

During a scan, check the list of all crawled objects. Ensure that the sitemap representation of your web application includes all the files and their variations, scripts, client scripts, input parameters, directories, and other resources on your website. If not all objects are listed, it means that the crawler is not able to automatically crawl all of the web application, so it might not identify all vulnerabilities.

Web vulnerability reporting

Once the automated website security scans are finished, start comparing the scanner findings.

• Which web application security scanner detected the most vulnerabilities?
• Which web application security scanner reported fewer false positives?
• Are there any vulnerabilities that you know exist but were not detected by a scanner (false negatives)?

By finding the optimal balance between these three criteria, you should identify the best automated web security scanner. To start with, you do not want a scanner that reports hundreds or thousands of vulnerabilities but most are false positives. Of course, you could simply ignore false positives or deselect them manually, but such tools and methods cannot scale when you actually start using your scanner regularly on large web applications. At scale, problems with false positives are amplified and as your security engineers and developers spend more and more time verifying false positives, they will start losing faith in the scanner. To learn more about false positives and the impact they have on a penetration test, refer to the article The Problem of False Positives in Web Application Security and How to Tackle Them.

On the other hand, you also do not want a scanner that cannot detect many of the most commonly exploited vulnerabilities. The best automated scanners are typically those that can report all of the commonly exploitable web application vulnerabilities (such as SQL injection, cross-site scripting (XSS), command injection, directory traversal, and so on), generate the fewest false positives, and require the least configuration.

Each automated scan should always be accompanied by manual penetration testing to manually identify one-off or completely new vulnerabilities that would rarely be exploited in a real-life attack. Community white-hat hackers can also help in that with the right bounty program. But at least the bulk of the work is done even before the manual testers get to work, so your developers can already start remediating the reported vulnerabilities.

Integration with other web security and development tools

Another important aspect of a web application security scanner is how easy its is to integrate with external systems and workflows. For example, does it output scan results in standard formats that can be parsed by other tools, such as XML? Can an automated web security scan be launched via the command line or batch files? The more integration features a scanner has, the easier it is to integrate web application security scanning into the SDLC for web applications.

And for those penetration testers who always like to dig deeper into web applications, can the scanner import the output of other web security tools typically used during manual security audits, such as other proxy servers, or HTTP analyzers? Support for such tools is another important factor to consider, since the more integration a scanner offers, the more you can automate the entire process.

Choosing your web application security scanner

As we have seen, there are several criteria that you should look into when choosing a web application security scanner. What is important is that you should focus on your needs rather than relying on the opinions of other people. Ensure that the automated tool you are about to choose can scan web applications that you want to audit and allows you to automate as much as possible, to save time, money, and other resources.

For a systematic, multi-step approach to selecting a dynamic application security testing solution that fits your needs, see our Web Application Security Buyer’s Guide.