Automated Technologies vs. Manual Penetration Testing

Automated technologies vs. manual penetration testing: what are the differences, and what should your organization invest in?

Let's explore:

The Differences Between Automated and Manual Pentesting

The process of penetration testing typically consists, in a corporate setting, of a security professional attempting to evaluate the security effectiveness of a client’s network infrastructure or web/mobile application by trying to compromise a specified system. 

The security professional may be labelled as an ethical hacker because they often utilize tools and techniques commonly used by black-hat hackers to discover and exploit vulnerabilities in applications and services.  Once a potential client understands the definition of a penetration test, and are defining their requirements, we often find them asking common questions.

• Should we conduct a manual penetration test or use an automatic vulnerability scanner such as Nessus or Qualys?

• Do automated tools exploit vulnerabilities to validate the level of risk?

• Which approach results in the most vulnerabilities found and how accurate are the findings?

• How do vulnerability scanners handle false positives and negatives?

• What are the costs associated with each method?

In using both automated and manual testing approaches, it is important to identify all possible attack surfaces, as a malicious attacker may only need one vulnerability to obtain unauthorized access to your sensitive information.  Penetration testing companies often rely on a variety of automated and manual testing approaches, but it is best to understand each to achieve the greatest coverage.

Automated Tools

• Speed: Automated tools work at a much faster rate by order of magnitude. It is much more difficult to manually test each component, service, and protocol manually with the same speed that a machine or script can.

• Coverage: Capable of covering larger attack surfaces with more ease by implementing crawling of web applications to identify potential attack inputs especially “low hanging fruit” and technical related vulnerabilities. Manual testing would require a large amount of time and skill to guarantee the same coverage and comparison to known vulnerabilities.  Difficult for automated tools to accurately test in-house web applications and services which can result in missed logical vulnerabilities.

• Efficiency: The processing capabilities of a machine are excellent. Automated tools can initialize and execute a large number of payloads for each test, but may not choose to execute the payloads correctly for each scenario.  Usually, fuzz the application with multiple payloads and then wait for a reaction.

• Qualifications: Automated tools have gone through intensive product testing for reliability and validity especially for professional versions. Manual testing skills is solely based on the individual pen tester’s expert skill set and experience.

• Reporting: Reports can be created easily and quickly. Usually, have graphical features such as charts for effective visual data comprehension.  Can be generic output that may not be capable of describing how the finding was validated.

• Investment: Open source tools and vulnerability scanners are usually free, but lack support or warranty. Professional licensing for vulnerability scanners and other automated tools can range dramatically in costs.

 Manual Approach

• Effectiveness: Automation alone is not capable to ensure that an application is thoroughly tested from a security perspective. Automated tools are poor at testing for logical vulnerabilities. Logical vulnerabilities require an understanding of the scope and flow of the application to identify any security issues.  Certain findings, for example, CSRF (Cross-Site Request Forgery) and business logic vulnerabilities need an experienced certified security professional to be capable to exploit and validate all potential security scenarios.

• Validity: Automated tool results usually contain a large number of false positives and negatives (30% to 90% depending on methodology and product) that can create a false sense of security or lack of security. These inaccuracies exist due to the lack of tool capabilities.  It is the responsibility and expertise of the manual tester initializing the automated tool to validate the results and identify the true security findings.

• Accuracy: Automated tools are only as reliable as their updates. If a new vulnerability or exploit has been introduced into the environment without a known category (i.e. zero-day), it is impossible for the automated tools to discover and identify the security threat.  In manual testing, it is possible for the tester to create their own exploit depending on the situation and vulnerability.  This allows the execution of comprehensive testing methodology that automated tools will overlook and fail to detect.

• Custom Reporting: Once the penetration test is complete, the tester is capable of creating a comprehensive report that is as individual as the test results.  At its most basic level, it will describe the vulnerabilities found, exploits used, data collected, risk rating, supportive evidence, affected assets, and mitigation recommendations.  These reports are fine-tuned to the needs of the client so they gain the greatest security understanding of their infrastructure, application, or device.

• Investment: The costs of manual testing depends on the scope and size of the engagement. In most penetration testing engagements, the cost and licensing of additional automated tools are covered under the negotiated penetration test contract unless special requirements call for installation of additional devices. In comparison, the cost of a data breach is growing exponentially as shown in current studies.

Security Testing Done Right

In conclusion, the use of automated tools and scanners are used as a starting point to save the manual effort and time for the more comprehensive testing, which is exactly why at Packetlabs, all of our penetration testers, at a minimum, have been extensively trained in hands-on manual testing by achieving the OSCP (Offensive Security Certified Professional) certification.

There is the implementation of automated tools into our testing methodology, but we do not solely rely on them.  An experienced security professional has the advantage to take on the perspective of the attacker, is capable of analyzing and contemplating custom attack scenarios.  Our certified security professionals can adjust their methodology and plan dynamically as the attack surface expands and exploit vulnerable components presented to them, resulting in comprehensive testing.