In short, a vulnerability scan (VulScan) finds possible problems while penetration tests (pentests) validate if problems can be exploited.
While an organization can use one or the other as an effective tool to improve security, the most mature organizations use vulnerability scans in combination with penetration tests to maximize their security benefits. The table below summarizes the main points of this article. Each issue will be explored in depth before we arrive at recommendations based on our analysis. Use the headings in the table below to jump ahead to a section of interest.
| Vulnerability Scans vs Penetration Tests | ||
|---|---|---|
| Vulnerability Scans | Penetration Tests | |
| When to Use | To scan infrastructure and discover known vulnerabilities | To explore discovered vulnerabilities to verify if they can be exploited and what damage could result from the exploitation or to discover non-vulnerability exposures in key systems |
| How to Use | Primarily tool-based, can be automated | Primarily hacker-based, with tools used when needed |
| Popularity | Less popular with IT professionals, but more popular with organizations due to less invasive and easier to execute nature | Less popular with organizations due to cost and difficulty to scope, but more popular with IT pros because hacking is seen as cooler |
| Frequency | Most perform quarterly vulscans, with additional scans after significant infrastructure changes | Most perform annual pentests for external penetration tests |
| Internal or External | Tends to be Internal | Tends to be External |
| Time to Conduct | Usually takes hours, but can be days for larger infrastructures | Usually takes weeks, but could take months for comprehensive testing |
| False Positives | Occur regularly; may also find true positives with negligible associated risk | Essentially zero false positives since the penetration testing results verify exploitation risk |
| Comprehensiveness | Scans all applicable infrastructure, only limited by scanning tool capabilities | Tends to be limited in scope due to budgets, time constraints, and capabilities |
| Minimum Requirement | Patching, but depends upon compliance regulations | External tests, but depends upon compliance regulations |
| Operations Disruptions | Possible network bandwidth issues or system instabilities can be caused by vulscans | Possible data corruption or operations disruptions can occur from certain types of pentests |
| How to Handle Overload | Known vulnerability exclusions; CSV Ranking or Risk Ranking | Rolling or Partial Testing |
| Costs | Moderate to low; cost of tools and IT Security time to install, configure, maintain, use, and analyze | High, typically requires outside vendors with highly trained penetration testing professionals |
| Benefits | Identifies vulnerabilities to be validated, categorized, prioritized, and mitigated | Verifies if vulnerabilities can be exploited and risk of damage before malicious attackers can cause damage |
To ensure a common frame of reference, let’s start with a recap of the basic definition for both concepts critical to network security.
What is a Vulnerability Scan?
A vulnerability scan, or vulscan, performs a systematic check of IT systems to look for known security holes. There are two types of vulnerability scans.
IT infrastructure vulnerability scans will be performed by IT or cybersecurity teams to inspect internal IT systems using admin-level permissions for known vulnerabilities in:
- Internal networking equipment, such as switches and routers
- File servers
- Network access storage (NAS) devices
- Individual computers
- Peripheral devices like printers and scanners
- Internet of Things (IoT) devices connected to the network, such as security cameras, TVs, etc.
- Critical applications and internal processes, such as Active Directory (AD); Domain Name System (DNS); and accounting, banking, or operations management software
Application or website vulnerability scans will be performed by development operations (DevOps) or development security operations (DevSecOps) programmers to scan software libraries, application programming interfaces (APIs), and supply chain components for known vulnerabilities.
Also read: 12 Types of Vulnerability Scans & When to Run Each
What is a Penetration Test?
Penetration tests, or pentests, probe systems for weaknesses to determine if they can be exploited. The common black box penetration test scans the external IT infrastructure of the organization such as firewalls, web servers, web applications, gateways, and VPN servers. These tests are performed without any advanced knowledge of its systems, but different penetration tests use credentials, test physical security, involve social engineering, phishing attacks, dropped USB drive attacks, test system volume capacity limits, and more.
Bounties can be considered a form of penetration testing that incentivizes non-contracted ethical hackers with payment after finding a flaw instead of through a contract agreed upon in advance. However, bounties can be unpredictable and, while very useful, bounties should not be relied upon to replace formal penetration tests.
Also read: 7 Types of Penetration Testing: Guide to Pentest Methods & Types
When to Use Vulnerability Scans and Pentests
The simplified justification to pick between vulscans and pentests can be expressed as:
- Vulnerability Scans: To scan infrastructure and discover known vulnerabilities
- Penetration Tests: To explore discovered vulnerabilities to verify if they can be exploited and what damage could result from the exploitation or to discover non-vulnerability exposures in key systems
To elaborate, vulnerability scans use commercial or open source tools that can be used by less experienced IT or security personnel to perform periodic, automated or on-demand testing. These tests can be performed quickly and inexpensively to detect known vulnerabilities.
However, vulnerability scanning tools cannot determine how easily the detected vulnerability can be exploited or how much damage can result from the exploitation of the vulnerability. Human expertise, often in the form of penetration testing, needs to be performed to verify the exploitability of the detected vulnerability and the extent of the potential damage.
Penetration testing can also discover additional flaws that might expose the organization to risk that cannot be detected by vulnerability scans. For example, pentests can discover weak passwords, gaps in security, or other security weaknesses that come from inadequate security architecture.
How to Use Vulnerability Scans and Pentests
Vulnerability scans are primarily tool-based, so a security team will often acquire and configure the tool for internal use. For organizations with more limited resources, vulnerability scanning can be included in the offerings of managed IT service providers (MSPs) and managed IT security service providers (MSSPs) or even through a vulnerability management as a service (VMaaS).
Penetration tests can be performed by internal teams who pick up breach and attack simulation (BAS) tools. However, internal teams can lack the expertise to thoroughly test systems and will be tempted by conflicts of interest to hide potential flaws in IT setup from management. Instead, organizations will typically engage third-party penetration testing vendors or MSSPs with pentesting expertise for the relevant assets to be tested (applications, websites, firewalls, containers, etc.).
Pentest vendors can retain experienced ethical hackers that the average organization cannot afford. These hackers use their experience to rapidly assess systems, target the most likely vulnerabilities to be exploited, and use hacking tools when needed.
Popularity of Vulnerability Scans and Pentests
The relative popularity of vulnerability scans and penetration tests is quite interesting because the popularity among professionals is inverse to the actual usage and demand. Among IT and security professionals, vulnerability scans tend to be less popular than penetration tests because they are not as technologically interesting as hacking — it seems more like homework. However, vulnerability scans tend to be less invasive, less expensive, and easier to execute which makes them much more frequently used among organizations.
The popularity for penetration testing, also known as red team exercises, can be measured by the higher demand and number of classes and certifications devoted to red team techniques when compared to defensive blue team techniques. This popularity also is skewed by the reality that a penetration tester only needs to find one way in to be successful and a blue team defender needs to be skilled against every possible red team tactic. IT teams just don’t feel like they can win on defense, so hacking just seems like more fun.
However, penetration tests cost much more money and can be hard to scope and understand for an organization. These factors lead some organizations to focus on vulnerability scans for everyday needs and to avoid penetration testing unless required by a compliance standard.
Frequency of Vulnerability Scans and Pentests
Most organizations perform vulnerability scans at least quarterly, with additional scans performed after significant changes to IT infrastructure or application updates. More aggressive organizations can scan monthly, weekly, or even continuously. After mitigation of discovered vulnerabilities, a rescan or penetration test will be required to validate the fix.
Compliance standards often define the frequency of vulscans and pentests. The PCI Data Security Standard (PCI DSS) requires vulnerability testing to be conducted at least quarterly and after every significant change to IT infrastructure, security tools, or applications. Recent research by Veracode finds that more frequent vulnerability scanning has reduced the typical number of vulnerabilities by two-thirds and decreased the time to fix vulnerabilities by more than 30%.
Many organizations never perform a penetration test, and of those that do, many only perform annual external penetration tests. More sophisticated organizations perform tests more frequently, such as 2-4 tests per year and also add pentests after significant infrastructure changes (especially for internet-accessible systems or after fixing previously discovered issues).
Are Vulnerability Scans or Pentests Internal or External?
Vulnerability scanning tools can be applied to external systems or even to systems that an organization does not control. However, vulnerability tools do not attempt to be stealthy and tend to be used primarily on internal systems.
Penetration tests tend to be performed on externally accessible assets from the point of view of a cyberattacker. However, penetration tests can also be performed internally without credentials to simulate attackers obtaining compromised low-access credentials (gray-box pentest) or with full credentials for more thorough testing (white-box pentest).
Also read: 7 Types of Penetration Testing: Guide to Pentest Methods & Types
Time Needed to Conduct Vulnerability Scans and Pentests
Vulnerability scans usually take a few hours, but the total time will vary greatly on the number of systems scanned and the type of systems scanned. Large systems can take days to complete a scan and vulscans can take only a few minutes for simple scans of a small office.
As with vulscans, penetration test durations will be highly variable and dependent on the number of systems tested and the type of pentests required. A typical test usually takes weeks to complete, but could take several months for comprehensive testing, especially for multi-national, multi-office organizations seeking social engineering, physical security tests, and other time-consuming pentests.
Do Vulnerability Scans and Pentests Generate False Positives?
Vulnerability scanning often will detect vulnerabilities that will be determined to be false positives. The scans may also locate true positives with negligible associated risk. Regardless, each vulnerability finding must be verified and either mitigated or flagged to be ignored.
Low-end pentests that only use off-the-shelf software will produce the same false positives that plague vulscans. Effective penetration tests usually generate zero false positives since the penetration testing results verifies exploitation risk by proving the hacker can access protected data or create operational disruption.
Naturally, after considering false positives, one must consider if false negatives are possible. Both forms of testing will only report on vulnerabilities found, but will not report that other assets pass without vulnerabilities. In the strict sense of the definition, no false negatives are generally produced, because neither testing procedure creates “all-clear” reports on systems on which no flaws were detected.
However, both forms of testing can overlook vulnerabilities. Vulnerability scans will always miss vulnerabilities not yet entered into their databases or on assets that are not scanned. Assets can be missed either because the tool is not capable of scanning an asset class (such as Kubernetes containers) or because the asset list simply was not updated prior to the scan.
Penetration tests tend to be narrow in scope and generally cannot test all systems or for all possible vulnerabilities. Even the most experienced pentester cannot know all possible vulnerabilities, and most organizations do not have the budget or time to test all possible systems. Pentesters make choices to focus on the most likely vulnerable systems and the most likely vulnerabilities.
Comprehensiveness of Vulnerability Scans and Pentests
Vulnerability scans target all applicable infrastructure and tend to only be limited by the scanning tool’s capabilities. Most can scan for tens of thousands of vulnerabilities, but a security team must also recognize that vulnerability tools can only scan for vulnerabilities that are known and already programmed into the scanner and on a device detected for scanning.
This limitation is neither unusual nor problematic as long as the security team recognizes that even scanned systems cannot be assumed to be vulnerability-free. The IT and security teams must also be sure to update the tool to scan all of the available assets so that no scannable asset is simply overlooked.
Many vulnerability scanners are also specialized by asset class. For example, some can only scan applications or websites and others can only scan Windows and macOS operating systems. Less common asset classes such as networking equipment, peripheral devices (printers, external hard drives, etc.), internet of things (IoT), industrial control systems (ICS), and industrial or operating technology (OT) often require specialized vulnerability scanning tools or manual investigation.
Penetration testing tends to be limited in scope due to budgets, time constraints, and tester capabilities. Pentests not only test known vulnerabilities, but can also discover unknown vulnerabilities, security gaps, and security weaknesses that technically are not vulnerabilities (just poor security architecture). However, while they may probe deeper than vulscans, pentests tend not to be as comprehensive and will be focused around the pentester’s expertise or the systems the organization can afford to test.
Minimum Requirement Vulnerability Scans and Pentests
The minimum level of vulnerability scanning is equivalent to patch management and examines traditional IT systems (servers, workstations, and laptops) for unpatched systems. For penetration tests, the minimum level tends to be external pentests on internet-accessible systems.
However, in both minimal cases, organizations must recognize that these minimum-requirement tests will likely leave the organization exposed to risk. Most organizations pursue minimum testing to minimize expenses, but even these minimums depend upon the specific compliance regulations.
However, this will likely be an evolving condition over the next few years. New York State already requires both penetration testing and vulnerability assessments for its financial institutions, and both PCI DSS and the National Institute of Standards and Technology (NIST) cybersecurity framework (800-115) have vulnerability scans as basic requirements.
Some vendors and consultants offer low-cost automated pentesting; however, these tests often prove ineffectual and produce similar results to vulnerability scans. These minimal approaches can catch easy-to-find and well-known vulnerabilities, but will not typically be rigorous enough to satisfy most compliance requirements.
Also read: What Is a Pentest Framework? Top 7 Frameworks Explained
Operations Disruptions from Vulnerability Scans and Pentests
Vulnerability scans probe each device port by port to test for known vulnerabilities, which can cause possible network bandwidth issues and even system instability. More sophisticated tests probe network equipment and check for misconfigurations, which will place even more demands on systems.
Most organizations schedule vulnerability scans for off-hours to minimize operations disruptions and may even exclude certain systems known to be sensitive to scanning. An organization with automated vulnerability scanning capabilities can also perform scans every time a new device is connected to the network; that also allows future vulnerability scans to be incremental and focus on specific vulnerabilities or devices that have not been scanned within the last month or quarter.
Penetration testing can also cause operations disruptions with certain types of tests. However, penetration testers will typically recognize how much disruption might be caused and can prepare the organization in advance for recovery. For disastrously disruptive tests that might cause data loss or full operations failure, an organization might create a test environment for hackers to probe so that a successful hack does not affect operations.
Penetration testing will often be thought of as an exercise to break into the systems to steal data. While valid, this definition ignores other forms of penetration testing such as volume capacity tests for infrastructure, such as when testing for distributed denial of service (DDoS) resilience.
How to Handle Overload From Vulnerability Scans and Pentests
Many organizations that need to perform vulnerability scans or penetration tests can become overwhelmed by the results because of resource constraints (budget, staffing, etc.).
With vulnerability scanning, overwhelm can often be controlled by ignoring known vulnerabilities or through filtering vulnerabilities using CVSS or risk-based ranking. Likewise, penetration testing overwhelm may be controlled by rolling or partial pentesting.
For example, vulscans of healthcare facilities notoriously find many issues, since many medical imaging devices run on operating systems that no longer receive updates (see Three Ways to Protect Unfixable Security Risks). Instead of producing the same list of vulnerabilities with each scan, the organization can exclude the devices with known vulnerabilities from regular scans to focus on fresh issues.
Scans also often overwhelm an organization with vulnerabilities with low Common Vulnerability Scoring System (CVSS) ratings that either cannot cause significant damage or are difficult to exploit. To avoid overload, an organization might choose to ignore scans below a selected ranking or to filter vulnerabilities in vulnerability management software to focus on high-priority vulnerabilities.
Risk-based filtering is similar to CVSS filtering, but instead of focusing on the vulnerability, the organization focuses on the importance of the asset and the likelihood of exploitation. For example, an organization might ignore a high-ranking data exfiltration vulnerability in a server that contains publicly available marketing materials in favor of focusing on a lower-ranking privilege escalation vulnerability in the credit card payment database due to the potential damage to the organization.
Penetration testing limitations will often focus on specific types of tests or specific systems in a given time period and leave other testing for later.
While organizations can use all of these techniques to limit scope and scale, they also need to recognize that such limitations might leave the organization exposed to potential attack via ignored systems or vulnerabilities.
Of course, service providers can provide another option for organizations that can obtain additional resources through outsourcing. A wide variety of MSPs, MSSPs, or even Vulnerability Management as a Service (VMaaS) providers can enable organizations to deal with overwhelm without remaining exposed to higher risk of attack.
Costs of Vulnerability Scans and Penetration Tests
Vulnerability scans tend to be moderate to low-cost options, with the expenditure primarily based on the cost of tools. However, organizations should also account for the time for IT or security teams to install, configure, maintain and use the tool as well as to analyze the results.
Penetration tests tend to be higher in cost because they typically require outside vendors with highly trained penetration testing professionals. Fortunately, pentest costs can be controlled by organizations through preparation and scope control.
Benefits of Vulnerability Scans and Penetration Tests
Vulnerability scans and penetration tests both provide high value to any organization.
Vulscans identify vulnerabilities to be validated, categorized, prioritized, and mitigated. Automated tools can allow for quick and continuous identification of systems and services at risk to help organizations reduce opportunities for attacker exploitation.
Penetration testing verifies if vulnerabilities can be exploited and checks for other security gaps that might risk data exploitation or system damage. Effective pentests help organizations further harden systems and minimize opportunities for malicious attackers to cause damage.
Bottom Line: For Best Results, Use Both Vulscans and Pentests
When an IT team struggles to keep up with its workload, vulnerability scans and penetration tests seem like more trouble than they’re worth, especially if there’s robust perimeter security in place. However, with a single bad click, an attacker could be inside the organization and exploiting any available weaknesses.
By using both vulscans and pentesting, an organization can catch more oversights, mistakes, and security gaps. Organizations that really want to avoid the headaches, risks, and consequences of a breach will run both penetration testing and vulnerability scanning on a regular basis. Given the high cost of a data breach, prevention is money well spent.
Further reading:
