External penetration testers are used to demonstrate that the organization’s software needs help. Finding critical vulnerabilities in high-profile applications provides the evidence that executives often require. Over time, the focus of penetration testing moves from trying to determine if the code is broken in some areas to a sanity check done before shipping or on a periodic basis. In addition to breaking code, this sanity check can also be an effective way to ensure that vulnerability prevention techniques are both used and effective. External penetration testers who bring a new set of experiences and skills to the problem are the most useful.

All penetration testing results are fed back to engineering through established defect management or mitigation channels, with development and operations responding via a defect management and release process. In addition to application vulnerabilities, also track results from testing other software such as containers and infrastructure configuration. Properly done, this exercise demonstrates the organization’s ability to improve the state of security and emphasizes the importance of not just identifying but actually fixing security problems. One way to ensure attention is to add a security flag to the bug-tracking and defect management system. The organization might leverage developer workflow or social tooling (e.g., JIRA or Slack) to communicate change requests, but these requests are still tracked explicitly as part of a vulnerability management process.

The organization creates an internal penetration testing capability that uses tools as part of an established process. Execution can rest with the SSG or be part of a specialized team elsewhere in the organization, with the tools complementing manual efforts to improve the efficiency and repeatability of the testing process. The tools used will usually include off-the-shelf products built specifically for application penetration testing, network penetration tools that specifically understand the application layer, container and cloud configuration testing tools, and custom scripts. Free-time or crisis-driven efforts aren’t the same as an internal capability.

Penetration testers, whether internal or external, routinely make use of artifacts created throughout the SSDL to do more comprehensive analysis and find more problems. Example artifacts include design documents, architecture analysis results, misuse and abuse cases, code review results, cloud environment and other deployment configurations, and source code if applicable. Focusing on high-risk applications is a good way to start. Note that having access to SSDL artifacts is not the same as using them.

All applications are tested periodically, which could be tied to a calendar or a release cycle. High-risk applications could get a penetration test at least once per year, for example, even if there have not been substantive code changes, while other applications might receive different kinds of security testing on a similar schedule. Any security testing performed must focus on discovering vulnerabilities, not just checking a process or compliance box. This testing serves as a sanity check and helps ensure that yesterday’s software isn’t vulnerable to today’s attacks. The testing can also help maintain the security of software configurations and environments, especially for containers and components in the cloud. One important aspect of periodic security testing across the portfolio is to make sure that the problems identified are actually fixed. Software that isn’t an application, such as automation created for CI/CD, infrastructure-as-code, etc., deserves some security testing as well.

The SSG uses external penetration testers to do a deep-dive analysis on critical software systems or technologies and to introduce fresh thinking. One way to do this is to simulate persistent attackers using goal-oriented red team exercises. These testers are domain experts and specialists who keep the organization up to speed with the latest version of the attacker’s perspective and have a track record for breaking the type of software being tested. When attacking the organization’s software, these testers demonstrate

a creative approach that provides useful knowledge to the people designing, implementing, and hardening new systems. Creating new types of attacks from threat intelligence and abuse cases typically requires extended timelines, which is essential when it comes to new technologies, and prevents checklist-driven approaches that look only for known types of problems.

Build a capability to create penetration testing tools, or to adapt publicly available ones, to attack the organization’s software more efficiently and comprehensively. Creating penetration testing tools requires a deep understanding of attacks (see [AM2.1], [AM2.8]) and technology stacks (see [AM3.4]). Customizing existing tools goes beyond configuration changes and extends tool functionality to find new issues. Tools will improve the efficiency of the penetration testing process without sacrificing the depth of problems that

the SSG can identify. Automation can be particularly valuable in organizations using Agile methodologies because it helps teams go faster. Tools that can be tailored are always preferable to generic tools. Success here is often dependent on both the depth and scope of tests enabled through customized tools.