Behind the jargon: False negatives in application security

What are false negatives and false positives?

When an automated test gives a wrong result, two types of error are possible: a false negative, when the test wrongly says that nothing was found, or a false positive, when the test sounds an alarm for something that isn’t there. Applied to application security testing, a false positive happens when your tool reports a vulnerability that doesn’t exist in the application. When you get a false negative result, the solution has failed to detect an issue that you know exists.

If you read that last sentence again, you may wonder how anyone is supposed to know about a vulnerability if it hasn’t been found. The whole point of security testing is to detect issues that you don’t know about yet – if you knew about them, you would fix them. This is why the concept of false negatives only makes sense in artificial test environments such as you would use for benchmarking or evaluation – scenarios where you know the expected results before you start the test.

False positives, on the other hand, are extremely important in real-world security testing. Security solutions that don’t have the technology to decide whether some application behavior indicates a vulnerability often err on the side of caution and report everything they find, even if it means many false alarms. This makes results inherently uncertain and makes it impossible to automate security workflows – we have a whole white paper about the harm that false positives can bring.

Why scanners might miss vulnerabilities

In a vulnerability scan, a false negative is simply a fancy way of saying that your security solution missed a vulnerability. While vulnerability scanners don’t have the creativity and intuition of experienced penetration testers, any worthwhile solution should find the vast majority of typical weaknesses, including cross-site scripting (XSS), SQL injection, and command execution vulnerabilities. If it doesn’t, this is often due to insufficient test coverage, inadequate security checks, or incorrect setup – or any combination of these.

Challenge 1: Test coverage

Knowing what to test is as important as the testing itself, so accuracy in the crawling phase is crucial. Modern websites often use custom error pages to provide user-friendly error messages, URL rewriting to aid SEO, anti-CSRF tokens to improve security, and authentication to restrict access to sensitive data. JavaScript-heavy single-page applications (SPAs) generate most of their HTML dynamically, so to test them, you need a way to simulate user interactions. A less sophisticated scanner may struggle to find all the places it needs to test – and if it stumbles on authentication, it might leave whole sections of the application unscanned. 

An advanced application security solution like Invicti will account for all these to crawl every corner of the application and identify as many attack surfaces as possible. Authenticated scanning in particular is a core capability of Invicti, not a bonus feature, with broad support for modern authentication methods. The scan engine uses a full embedded browser to simulate the entire DOM of the target page, run tests, and examine how the application responds to payloads. The pool of URLs to be tested is populated from multiple sources, not only from easily accessible hyperlinks. All this ensures that the scanner can probe every part of the application, just like an attacker would.

Challenge 2: Security checks

The meat of an application security test are the security checks themselves. Vulnerability scanners started life as scripts to automate routine tasks during manual penetration testing, but a modern application security solution has to go much further. The goal is no longer to speed up manual testing – the challenge now is to do the tests automatically with little or no human intervention. Every missed vulnerability is a security risk and every false alarm creates additional work, so an inaccurate scanner can be worse than no scanner at all.

Invicti comes with several thousand security checks, from common weaknesses such as cross-site scripting (XSS) and SQL injection to advanced out-of-band and second-order vulnerabilities. When the additional IAST module is used, it provides extra depth for the core scanner to find even more issues and deliver server-side insights, including information about misconfigurations. This ensures that all the potential attack surfaces are subjected to a battery of tests that incorporate over a decade of security research and development. With Proof-Based Scanning, you get automatic confirmations that are 99.98% accurate – and you get them for over 94% of direct-impact vulnerabilities.

Challenge 3: Configuration

A vulnerability scanner simulates the actions of real-life attackers by manipulating user-accessible page elements and other exposed endpoints. Every web application is different, so doing this accurately requires careful setup and a deep technical understanding. If the tool is not advanced enough or you don’t have the resources to manually set up all the scan parameters, the scanner might miss many vulnerabilities simply because it’s not testing the right places and in the right way.

With Invicti, you get an advanced vulnerability testing solution that is easy to set up and optimize at any scale, from dozens to thousands of websites. The scanner itself does a lot of setup work for you, while the intuitive user interface makes it easy to configure application-specific details such as authentication and scan parameters. You also get detailed documentation and world-class technical support with onboarding guidance to quickly get you using the product to its full potential.

Look for trustworthy results in real-life scenarios

To sum up, false negatives are an academic concept with little bearing on application security testing in the real world. What you need from a vulnerability scanner are reliable and actionable results. You want it to find all the vulnerabilities that attackers could exploit at any time to steal data or execute malicious code. You want it to send accurate notifications without false alarms that would distract your security team and developers. You want it to do its job automatically at any scale with your modern applications and workflows.

When evaluating web application security products, real-life effectiveness is the key. The best way to see if a solution is right for your organization is to run it on your actual applications with vendor assistance to make sure the setup is right. Invicti provides a zero-obligation proof-of-concept test period, so give it a try. See the difference that actionable results make.