Why Do We Need External Reviews?

External security reviews serve a number of different purposes besides compliance:

Independence

Outsiders don’t have a stake in making your team or organization look good. In fact, their interest is opposed to this: they want to find serious problems so that you will bring them in again, or so that you will ask for their help in fixing these problems. From a penetration tester’s perspective, the worst engagement imaginable is one where they don’t find any meaningful issues, and they are incentivized to work hard to avoid that situation!

Expertise

Hiring specialist security testers and auditors is not only challenging but expensive. There is a global skills shortage, and many tests need specific or niche skills that suit the technology or context. Keeping these on the team may not be an option for your organization, and therefore seeking such expertise from outside parties can make the most practical and financial sense.

Management support and escalation

Spending real money on external reviews can be an effective tool for ensuring that management and the executive team understand the risks faced by the organization. Problems identified by an outside party will often be given more visibility than issues found during internal reviews, and can be easier to escalate. Your organization’s leadership has an obligation to steer the company safely and reduce risk—external reviews can be a valuable tool to help them meet this obligation as well as helping them demonstrate that they have taken that responsibility seriously.

Learning and improvement

In order to understand if your approach to security is working, you need some way to measure its effectiveness. External reviews cannot provide a perfectly complete assessment of all aspects of your security program, but do provide both quantitative and qualitative data that you can feed back into your program to identify strengths and weaknesses and use to improve over time.

Objectivity

Outsiders will never understand your environment as well as you do. While this lack of contextual knowledge can sometimes provide challenges when engaging external parties, the objectivity they provide can be invaluable. They will not have the same biases, assumptions, and predispositions as an internal team, and can be considered to be in the truest sense fresh eyes. These eyes may see issues that you would not consider or expect, despite the outside party’s lack of familiarity with your business and systems.

Customer expectations

Finally, if you are building a product or service for others to use (and probably pay you for), your customers may well have the expectation that you have sought outside expertise in assessing the security of the thing itself and the environment it operates in. It is increasingly common during a procurement process for customers to ask for access to the full or executive summary of third-party assessment reports, along with the remediation actions you took to any findings. If you are not able to provide customers with such evidence, there is a very real chance you will lose out on business due to a diminished trust in your solution.

There are a range of security assurance services available, and it’s important to choose the services that will meet your needs. This means balancing the aim of the exercise, the cost, and the depth of the evaluation.

Let’s look at the common types of assessment and reviews available and their aims, as well as their limitations and common misunderstandings of what they can provide you.

Vulnerability Assessment

Vulnerability assessment is usually the cheapest of the external assessment options at your disposal, and as such tends to give you a broad view of your environment’s security, but one that lacks the depth of other approaches. In these assessments you hire an outside party to run a series of tools such as vulnerability scanners that look for common mistakes in configuration, unpatched software with known problems, default credentials, and other known security risks. Once the scans are complete, they will summarize the results in a report that you can give to your auditors, which should include an overall assessment of risk and a high-level remediation plan.

Even if you do have access to a set of tools that can perform automated security scans and you are running them on your own, there can still be value in having someone with broader experience running scans and evaluating the results. Vulnerability assessment tools in themselves are rarely complicated to run, but someone who does this for a living can make sure that they are configured correctly for a given environment, and help your organization understand the results and how best to remediate them.

A professional who has used these tools many times before should know how to make sure that the tools and scans are set up and configured to run properly, and what policies and plug-ins will provide the most value, instead of relying on defaults.

A common downside of automated security scanning tools is the volume of results they produce, the lack of context surrounding the results, and the number of false positives. With this in mind, having someone familiar with vulnerability scanning and their tooling can pay dividends in terms of having the results analyzed to reduce excess volume, remove false positives, and provide environment-specific context that the tooling is just unable to.

One of your key expectations when employing an outside expert to run a vulnerability assessment for you is that the expert will filter and prioritize the results correctly, so you can maximize your ROI when you remediate the findings. Your own system administrators and operations engineers may be too dismissive of findings (“that will never happen”), but an objective outsider can help you to understand what risks actually need to be addressed.

Furthermore, some vulnerabilities on their own may appear to be low risk and easily dismissed. But a skilled professional who is familiar with chaining vulnerabilities together to achieve complex compromises might recognize when low severity vulnerabilities, aggregated with other risks in your environment, could pose a higher risk to your organization.

It’s important to make a distinction between vulnerability assessments and other forms of more involved assessment. By its very design, a vulnerability assessment will only be able to provide you with a list of (potential) vulnerabilities and misconfigurations. It is a great way to identify low-hanging fruit and get a general perspective as to the overall security health of your environment, but it is not going to be either exhaustive or imaginative in the way in which an environment’s security posture is exercised. Getting a clean bill of health from a vulnerability assessment is certainly a positive, but don’t mistake this for a declaration that the environment is therefore free of defects and is secure.

Penetration Testing

We looked briefly at penetration testing and what’s involved in having a skilled white-hat security tester try to hack his way into your network or your application. Penetration testing is one of the most common security assessments, and unfortunately one of the least understood.

Good penetration testing combines a range of factors:

• A disciplined and structured approach to scoping and managing the testing itself.

• Testers with access to appropriate tooling that supports efficient and deep assessment.

• Experience and expertise with the systems and languages being assessed.

• The ability to build on first principles and adapt to novel environments quickly and creatively.

It may sound clichéd, but you get what you pay for with penetration testing, and cutting corners or going with inexperienced or low-skilled testers may result in an assessment that is little more than a vulnerability assessment.

Penetration testing is most often done as a final security check before pushing a major new release of a system to the final production environment. Most organizations hire a recognized penetration testing firm, let them hack for a couple of weeks, get the report, and because they left the test to so late in their project, try to fix whatever they can before they run out of time. In many cases the shipping date has already been set in stone, so fixes and mitigations may not even make it into the initial release itself.

These organizations miss the real value of a penetration test: to learn about how attackers think and work and use this information to improve their systems and the way that they design, build, and test their systems. A good term to keep in mind that captures this value well is attack-driven defense, the use of an attacker’s perspective on your environment or application to drive a continual prioritization and improvement of the defense you choose to put in place.

Your goal shouldn’t be to “pass the test.” Organizations that look at penetration testing this way may stack the deck against the pen testers, containing their work and feeding them limited information so that the number of problems they can find is minimized. They often justify this by stating that black-box tests are more realistic, because the pen tester and an outside attacker both start with the same limited information.

But the more information that you provide to pen testers, the better job they can do, and the more that you can learn. Gray-box testing, where the tester knows how the system works, just not as well as your team does, beats black-box testing hands down. White-box testing is the next step in terms of making information available to the pen testers by giving them full source code access and allowing them to combine code analysis, architectural review, and dynamic hands-on testing during their assessment.

A number of different attack scenarios can be built around the constraints and allowances provided to an external pen testing team. The reality is that it takes a fairly mature security organization to be able to actually make real use of those differing perspectives. Far more common is the situation where the external assessment is the first time the solution in question will have been through an offensive security assessment, and it will be conducted on a limited budget. In these cases, one of your primary motivations should be to get out of the way of the testers and to allow them to put their experience to best use in identifying the parts of the solution that raise the most flags for them so that they can dig in and demonstrate successful attack scenarios and their impact.

In this regard, a white-box assessment will allow the testers to avoid wasting time understanding the limitations of the system and any security controls it may have, and go directly to the source to answer those questions. Typically for a one-to-two week assessment, a white-box test will return you far more coverage, as well as more fully exercised attack pathways, than a gray- or black-box test. It will often also provide more specific guidance in terms of remediation, as the testers will have seen the code and may well be able to identify the exact deficiency behind any finding. This makes it much easier for the team responsible for working with the results to get remediations in place in a timely fashion.

As we looked at in detail in Chapter 11, manual testing, including penetration testing, can’t keep up with rapid Agile development or continuous delivery in DevOps. To meet compliance requirements (like PCI DSS) and governance, you may still need to schedule penetration testing activities on a regular basis, typically at least once per year. But instead of treating pen testing as a gate, think of it more as a validation and valuable learning experience for the entire team.

Red Teaming

Red Teaming is running an active attack against your operations to test not only the security of your environment as it actually works together in the day-to-day, but more importantly your incident detection and response and other defensive capabilities.

The goals and approach are different than a pen test. Instead of trying to uncover and prioritize vulnerabilities for you to fix, Red Teams actively exploit a vulnerability (or a chain of vulnerabilities) or use social engineering techniques to infiltrate the network and see how long it takes for your operations or security teams to discover what they are doing and react to them. Such an approach where you invite an external team to target your environment as a real group of attackers is also known as an attack simulation or a goal-oriented attack. The scope of Red Team engagements is very broad by definition or even no holds barred, with everything on the table.

Red Teams will find important vulnerabilities in systems. It is not uncommon for such engagements to produce zero-day vulnerabilities in both internal applications as well as commercial applications in use at an organization.

But their real value is helping your security and operations teams to learn and improve how to deal with real-world attack scenarios, conducted by highly motivated and skilled adversaries.

The deliverables that result from Red Team exercises are generally quite different from what you get from a pen test. Red Team reports often present findings in the form of attacker diaries, attack trees containing the pathways taken through an environment that led to a realized goal, and detailed descriptions of custom tools and exploits written specifically for the engagement. This offers more insight into the attacker mindset, with discussions covering the why as much as the what.

Failed attack pathways, things the testers tried and didn’t succeed, are also incredibly important, as they provide a measure of effectiveness of security controls that may be in place and offer some qualitative measures of the defensive investments you have made.

Red Team engagements are not cheap, and to get the most from them, the security posture of an organization needs to be fairly mature. Early on in an organization’s security journey, money may be better spent on a larger or more frequent set of penetration tests rather than opting for the much sexier goal-oriented-attack approach.

However, when you do feel you are in a position to benefit from working with a Red Team, then it can be one of the most enlightening and humbling experiences you, and your security program, can undergo. It can provide you some very insightful qualitative data that can answers questions that the more quantitative pen test and vulnerability assessment approaches cannot.

Again, the focus on learning should be paramount. Gaining a deep and nuanced insight into how a group of skilled and motivated adversaries traversed your environment should be used to inform your approaches to both defensive and detection measures.

Some organizations, especially large financial services and cloud services providers, have standing Red Teams that continuously run tests and attack simulations as a feedback loop back to security and incident response. We’ll look at this approach some more in Chapter 13, Operations and OpSec.

Bug Bounties

If there’s one type of external assessment that has ridden the hype-train over the 18–24 months that preceded the writing of this book, it’s bug bounties. Inevitably, and unfortunately, with the hype have come misconceptions, misunderstandings, as well as plain-and-simple misinformation. Rest assured that despite what the internet may tell you, you’re not grossly negligent if you don’t currently run a bug bounty. In fact, you’re more likely to be considered negligent if you enter into the world of bug bounties and your security program overall isn’t in a mature enough place to be able to handle it. If you don’t have a solid AppSec program in place, then you should invest in getting that up to scratch before even thinking bug bounties, if you don’t, you will find things accelerating off ahead of you and quickly be in a place where it will be hard to cope.

With all of that said, bug bounties are definitely an option as your security program progresses. What follows is an effort to help you cut through the hype and view the value you may get back from a bug bounty.

Configuration Review

A configuration review is a manual or scanning-based review of a system configuration against hardening best practices, or one of the standards like the CIS Benchmarks that we will look at in Chapter 13, Operations and OpSec.

This can be a good health check for the runtime platform, the OS or VMs or containers, and runtime software like Apache or Oracle that can be difficult to set up correctly.

As much as possible, this kind of checking should be replaced by running automated compliance scans on your own, on a regular basis.

Secure Code Audit

Even if your team is trained in secure coding and consistently using automated code review tools, there are cases where you should consider bringing in a security specialist to audit your code. This is especially valuable in the early stages of a project, when you are selecting and building out your frameworks, and want to validate your approach. You may also need to bring in outside reviewers in response to a security breach, or to meet compliance obligations.

Most auditors start by running an automated code review tool to look for obvious security mistakes and other low-hanging fruit. Then they will focus in on security features and security controls, and data privacy protection. But good reviewers will also find and report other kinds of logic bugs, runtime configuration problems, and design concerns as they go.

Code audits can be expensive, because they require a lot of manual attention and specialized skills. You need to find someone with security knowledge and strong coding skills who understands your development environment and can come up to speed quickly with the design and the team’s coding conventions.

Code auditors face the same challenges as other code reviewers. They can get hung up on style issues and misled by code that is poorly structured or too complex. And code auditing is exhausting work. It can take at least a couple of days for the reviewers to understand the code well enough to find important problems, and within a few more days they will reach a plateau.

So it makes sense to aggressively time-box these reviews, and you should be prepared to help the auditors as much as possible during this time, explaining the code and design, and answering any questions that come up. Assigning experienced developers to work with the auditors will help to ensure that you get good value, and it should provide a valuable opportunity for your team to learn more about how to do its own security code reviews.

Crypto Audit

Crypto audits have been broken out as a separate class of assessment from the more generic code audits for one simple reason: they are really hard and need a distinctly specialist skillset to do. Of all the areas of external assessment, crypto audits have the fewest experienced practitioners available and will be the hardest to source expertise on. It’s not uncommon to have to go on a waiting list and have your test scheduled far out into the future due to the lack of available people qualified in the art. Don’t, however, let this this tempt you to cut corners and substitute a true crypto auditor for someone who has more availability.

The reasons behind needing a specialist crypto audit fall into two main categories:

1. You want to validate your use of cryptography in a wider system. The questions you’re asking here center around Am I getting the properties I think I am from my use of cryptography? and Have I inadvertently undermined the guarantees a particular set of crypto primitives offers somehow?

2. You are designing and/or implementing your own cryptography primitives, and you want to validate that the design and implementation of your algorithms actually do what you want them to.

If you fall into the second category, stop. Go for a walk, probably a drink, and then revisit whether designing and implementing new or novel crypto primitives is actually the best solution to the problem you are trying to solve. Crypto is hard, like mind-bogglingly hard, and the most seemingly innocuous slip-up can bring everything tumbling down around you.

For most situations you will find yourself in, there are well-established, off-the-shelf components and patterns that will solve your problem, and there is no need to get inventive and roll your own. If, despite the walking and drinking, you’re convinced that you need to design some new algorithm, or alter an existing one, or that there is no implementation of an existing algorithm in the form or language you need, then the best advice we can give is ensure you have in-house expertise from the inception of the project, or reach out to an external specialist for help. Even after doing that, you will want a separate set of eyes to review whatever is produced to perform additional validation.

If your needs fall into the first category, then still realize crypto is hard and that gaining an external perspective on both the use and application of cryptography is something that can carry huge value. The findings and results that come back from your first crypto review will likely be surprising to you. Even for usages you thought were simple, there can be level of nuance, right down to the compiler being used to generate the binaries, so that you may start to question your own understanding of computers and numbers in general. This is good. This is what you’re paying for.

If your solution is making heavy use of crypto, then hopefully this helps stress the importance of getting some specialist eyes on it. The only remaining thing to say would be that once your crypto has undergone validation, it should be a part of your code base that isn’t touched outside of the rarest of circumstances, and should hopefully only be interfaced through well-defined APIs.

Choosing an External Firm

There are a number of factors that go in to choosing an external partner or consultant.

Why are you getting help?

• To meet a compliance mandate?

• As a proactive health check?

• Reactive response to a failed audit or a breach?

How much commitment will it require from your team to ensure that it is successful?

A configuration review or a standardized vulnerability assessment can be done without too much involvement from your team, at least until it comes time to review the report. You care about cost and availability and evidence of capability.

But a code audit or a Red Team exercise requires both parties to work much closer together, which means that you need to know more about the people involved, and how well you will work together.

What kind of security assessor are you working with?

There are different types of external providers that offer different value:

Security testing factories

IT security firms with highly structured offerings designed to meet compliance requirements. Often highly automated, methodology-driven, scalable, and repeatable.

Expert consultants and boutique shops

Individuals or small specialist teams relying heavily on expert skills and experience rather than methodologies, customized offerings, and custom-built tools. You get what you pay for in these cases: the people who present at Black Hat or Defcon or OWASP conferences, or who teach security classes, can be expensive (depending on their reputation and availability), and there aren’t that many of them around.

Enterprise service providers

Security services provided as part of a larger, more comprehensive IT services program that includes other consulting, training, and tools across multiple programs and projects. Highly scalable, repeatable, and expensive.

Auditor or assessor

Services from a recognized auditing firm, such as a CA firm, or from another qualified assessor (like a PCI QSA). Checklist-driven, assesses systems against specific regulatory requirements or some other standard.

It’s relatively straightforward to evaluate a security testing factory, by looking at its methodology. It can be harder to choose a boutique or an individual consultant, because you need to have a high level of confidence in their skills and expertise and working approach. And you must be able to trust them to maintain confidentiality and to work carefully, especially if you are allowing them inside your production network.

Getting Your Money’s Worth

Make sure that you understand what you are getting for your money.

Are you paying for someone with a certificate to run a scanning tool and provide an “official” report? Or are you paying for high-touch experts, who may take extra time to build custom tools and custom exploits to assess your system?

Key Takeaways

External reviews can be daunting, even for the most experienced teams. Choosing the right vendors and the right services, and learning how to get the most from these events, can really increase their value:

• There are a range of services that an external firm can offer: choose the right one on a per system and project basis.

• External reviews can provide objectivity and specialist skills that may not be naturally available inside your organization.

• It takes effort to get the most of your testing results. Work with your testing firm to make sure you understand the results and that they are fed into pipelines and systems.

• Choose your firm well, and don’t forget this is a big marketplace: you can ask questions and shop around.

• Take advantage first of simpler, cheaper assessments like vulnerability assessments to catch obvious problems before you go on to pay for more expensive engagements. Don’t pay expensive consultants to find problems that you can find for less, or find on your own.

• Don’t try to narrow the scope and set up tests or audits so you that you know you will “pass.” Give pen testers and other reviewers access and information so that they can do a thorough job, giving them a better chance to find real problems for you.

• Bug bounties can be an effective way to find security holes in your systems. But keep in mind that setting up and running a bug bounty program is an expensive and serious commitment. If you go in thinking that this is a way to get pen testing done on the cheap, you’re going to be unpleasantly surprised. An effective bug bounty program won’t be cheap to run, and it’s hard to back out of once you’ve started.

  Bug bounty programs are about community management more than about testing. Be prepared to treat all security researchers patiently and politely, respond to their findings in a meaningful way, and reward them fairly. Otherwise you can damage your organization’s brand, disrupt operations, and even encourage malicious behavior. Someone in your organization will have to spend a lot of time reviewing and understanding what they find, and often a lot more time politely explaining to someone who put a lot of their own time into finding, duplicating, and writing up a problem, why what he found (and expects to be paid for) was not, in fact, a bug, or at least not a bug that you are prepared to fix.

• If you are not taking advantage of external reviews to learn as much as you can from these experts, you are wasting their time and your money.