Vulnerability Assessments and Penetration Tests – What’s the Difference?

Confusion between the terms 'penetration testing' and 'vulnerability assessments' often begins at the level of language. Those who are not full-time professionals in web security, such as journalists reporting on a big story that affects consumers, use the terms interchangeably, as if referring to the same process.

Experienced professionals in the industry know the difference, but those new to it can be easily confused. Why? Even professionals sometimes use terms in fuzzy or inexact ways, when they should distinguish between things that differ. Let's be clear on the difference between the two.

What Are Vulnerability Assessments?

A vulnerability assessment involves running a series of multiple tests, against defined websites, web applications, IP addresses and ranges, using a known list of vulnerabilities, such as the OWASP Top 10 list. Assessors may also run tests against systems they know to be incorrectly configured or unpatched. Often, automated security scanning tools are used. Commercially licensed, subscription-based tools are regarded as coming with less risk – regular updates, release notes bring less chance of the inclusion of malicious code. (Their open source equivalents, however, have the significant advantage of being the exact same tools that malicious hackers prefer.)

Vulnerability assessments tend to include the following stages:

• Identifying all resources, and connected resources, within an organisation's IT systems
• Assigning a value or priority to each one
• Conducting an assessment of lists of known vulnerabilities across a large number of attack surfaces (from login screens to URL parameters to mail servers)
• Fixing the most critical vulnerabilities and making decisions about how to the deal with the rest

What is Penetration Testing?

Penetration testing (pen testing), on the other hand – while it may be considered to be a type of vulnerability assessment – involves replicating a specific type of attack that might be carried out by a hacker. A pen tester will often explore the systems until they find a vulnerability. They may even employ a vulnerability assessment tool to uncover a vulnerability. Once they find something, they will then try to exploit it, to determine whether it would be possible for a hacker to achieve a certain objective (access, change or delete data, for example). Often, while doing this, they may accidentally encounter other vulnerabilities, and follow where they lead. The pen tester may use an automated tool at this point to run a series of exploits against the vulnerability.

Some penetration tests are referred to as 'white box' to indicate that the penetration tester has been given detailed information about the environment, such as a list of assets belonging to the organization, source codes, employee names and email addresses etc. When they are referred to as 'black box', this indicates tests that are conducted without any prior information about the internal structure, access to source code etc.  This kind of pen test of course, can more closely resemble the activities of a malicious hacker, but may also lead to less thorough coverage of the companies potentially vulnerable assets.

What Results Can I Expect From Each Approach?

The answer to this question might best be asked by thinking backwards: What results do you want?

Vulnerability Assessments Report Across All Vulnerabilities

The results are collated in an automated, lengthy report, with a comprehensive list of detected vulnerabilities arranged by priority, determined by how by severe and business-critical they are. As time goes on, this list can reveal changes since the last report. One of the criticisms of the results achieved is that, unlike in penetration testing, they can contain false positives or false negatives. Naturally, this is not the case if you use Netsparker web application vulnerability scanner to conduct your vulnerability testing. It is one of our key features – automatically verifying identified vulnerabilities with Proof-Based Scanning™.

Reports should include guidance on how to remediate the detected vulnerabilities, and tools sometimes come with patches subscribers can use. In most cases, results are then allocated to dedicated development teams who conduct fixes, remove the most serious vulnerabilities, and otherwise address the less serious ones. In an ideal world, this activity is ongoing, scheduled regularly, and built into the organisation's SDLC.

Penetration Testing Reports Deep Into Each Vulnerability

With pen testing, there is no lengthy public report, though some record and publish their actions and anonymized findings, blog about their experiments, or live hack at conferences. If you hire a pen tester, however, they should deliver a (pen test) report, but it tends to be focused on the attack method or exploit, and exactly what data can be compromised. It will generally be accompanied by suggestions on what a hacker might be able to do to, or with, it. This helps business analysts and non-technical professionals, who may not understand all of the technology behind such tests, grasp business process impacts quickly.

Sometimes reports also incorporate remediation advice. However, not all pen tests incorporate exploitation of vulnerabilities in the way that Netsparker does. It may be sufficient simply to illustrate that an attack is possible. In some cases the pen test report may simply report theoretical vulnerabilities because attempting to exploit them may result in a catastrophic denial of service (DoS). And, finally, there is no assessment of vulnerabilities, since the goal is simply to do one thing, or least to determine whether it can be done.

Which Approach Should My Organisation Adopt?

The main question to ask is: What is your current security posture?

Vulnerability Assessment Builds Security Continuous Improvement Into your Enterprise SDLC

Vulnerability assessments are a highly systemised way for established organisations to gain a comprehensive picture of their security posture, and then maintain and continuously improve on it. When new devices, ports, websites, web applications, or services are added, they are included in regular scans. A vulnerability assessment is a great way to identify, and eventually fix, common vulnerabilities in your applications and servers.

Most security professionals recommend that vulnerability testing is conducted at least quarterly. Our recommendation, however, given that Netsparker allows you to configure scheduled scans, is to scan much more frequently. In any case, you should conduct vulnerability tests following any significant change or addition to your web applications or web APIs. With Netsparker, if you want to, you can run scans daily, with notifications drawing your attention to detected vulnerabilities as they arise. Resources can then be deployed rapidly to deal with critical and important threats.

Penetration Testing Exposes Fragile Cracks In Your Security Architecture

Since penetration testing is so specific, it is best suited to environments where an organisation's web and network security is considered to be already robust. Organisations may ask a tester to attempt to do something specific, such as gain access to a transactions or bank details database, or alter or delete a single record. The purpose is to reduce exposure to certain risks. Penetration testers check for weak points in the architecture. While vulnerability assessments mostly take care of software vulnerabilities, penetration testers may often use phishing, social engineering and onsite engagements in order to reach their goal. Therefore they can give a much more accurate depiction of a company's security level. They act exactly as malicious hackers, without producing any devastating loss or alteration of data, of course! For example, a penetration tester might try to establish a connection to a remote server without being detected, in order to exfiltrate sensible data from a system. It is a useful way to demonstrate if attackers with particular objectives in mind stand a healthy chance of success. Ostensibly, though, a pen tester would conduct an endless series of attempted hacks.

The recommendation is that penetration testing is conducted at least once per year.

What Scenarios Can Help Determine the Choice of Approach?

Both vulnerability assessments and penetration tests should be run against network devices, and internal and external servers. It's crucial to determine whether an attack is possible from the outside (for example, by a malicious attacker targeting publicly-available target surfaces on the internet) or the inside (for example, by a disgruntled employee or contractor, a user with permissions they should not have, or a compromised machine within the internal network).

Vulnerability Assessments Help Enterprises Maintain Consistent Compliance With Standards

Sometimes organisations need to work within certain parameters: they have PCI DSS or other forms of compliance to adhere to and want to test if the current architecture, systems and devices would pass the test. They may want to run a port scan or check against everything on the OWASP Top 10 List. In such scenarios, a vulnerability assessment will provide a more realistic and systematic approach. Even a very large team of developers could never comprehensively reach the end of such tests.

Penetration Testing Helps All Organisations Keep Ahead of the Hackers

Penetration testing comes at security from a different angle. Testers will uncover security risks in the same way that hackers do – by conducting attacks with a single purpose in mind, to gain access to certain data or to change something on an organisation's website, for example. Pen testers are best commissioned with an open mind, leaving them free to conduct both requested attacks and anything else that occurs to them, depending on their professional experience.

What About the Testers?

One of the most important questions to consider, to help distinguish between vulnerability assessment and pen testing, is: Who's conducting the testing?

Information Security Professionals Establish Internal Procedures for Continuous Improvement

Contrary to some articles on the subject, vulnerability testing is not a fully-automated process in the sense that all it takes is to push a button. The person who manages regular, automated vulnerability assessments must be already skilled and experienced in information security procedures. They must know what environments and attack surfaces to assess and what to assess them for, as automated security scanners will still require some configuration. And they must be able to interpret the resulting reports and make recommendations on what needs to be done next.

In-house security professionals responsible for vulnerability assessment continuously add value to the security status of organisations and their resources. First, they can establish a baseline. They are likely to want to establish some systems, particularly an assessment schedule and reporting. They can help raise awareness within, while facilitating a continuous reduction in security risks. Meanwhile, they will most certainly expand their own knowledge and skills. Arguably, they are much more likely to feel loyal to an organisation in which they already work.

Penetration Testers Tell It Like It Is

Penetration testers, likewise, must also be knowledgeable professionals who are confident in their abilities.

Most professionals in the field recommend that penetration testers should be independent, external professionals. They must maintain enough distance from your company or systems that they are not hampered by concerns about personal financial security, loyalty or politics. This enables then to state the blunt truth about your security status, however much it hurts!

What About Cost?

How much vulnerability testing costs depends on the scope of the engagement. For small organizations the price will be significantly lower than for a big corporation with thousands of potentially vulnerable machines, IPs and internet facing hosts.

Regardless of the cost, vulnerability assessments produce a better yield on investment. While a pen test may be a deep slice of how secure your system is, it only reveals one thing in one direction. Vulnerability assessments take the long view, investing time and resources in developing systems and procedures that will yield a solid level of security on which to further develop your systems and integrate new components.

So, Which Approach Do We Select?

Simply put, do both. Both approaches have the capability to uncover gaping holes in your security and reveal other less obvious vulnerabilities, ones you weren't even looking for. One thing is certain, if you're not scanning or testing, you will encounter a loss of data. The only question is when. Whether it's a known vulnerability that you've not addressed, or the result of a bored hacker's Sunday afternoon adventures (yes, it's true, they're not all malicious!), the result is the same.

The mature, preventative approach is to establish vulnerability testing and scanning as part of your regular SDLC, and additionally employ some unusual types to do what a hacker might do, but on friendly terms ('white box' pen testing). Then you can read all the reports and results, examine the recommendations and make smart decisions on how to keep your organisation's security posture ahead of the bad guys.