Architecture for incident control and isolation of blast radius

A well-architected system is probably the best defense against a potential attack. Even if such a system is compromised, a well-designed system has controls in place that ensures that access to one compromised resource of the system does not compromise the entire system, but instead contains the unauthorized access to the very specific region. To use security-related terminology, the blast radius of a security incident is isolated to a specific module. Systems implement architectural techniques such as the principle of least privilege, need to know, microsegmentation, zero trust, request-response strategies, and strong multifactor authentication for all of its users.

As an example, consider a microservice system that has implemented good segmentation at the network, application, and transport layers. Figure 9-1 shows a typical microservice application with multiple bounded contexts. There are four bounded contexts in this example: Marketing, Analytics, Finance, and User profile (which displays the personal information of each customer). Now assume that one of the microservices inside the User profile service context is compromised by an attacker.

Figure 9-1. One of the services within a particular bounded context that represents a business domain is affected.

As soon as such an attack is detected, security professionals can jump on the incident by taking remedial measures by isolating the affected bounded context (User profile services) and all the services within that context.

While the services inside the affected bounded context will be isolated, and hence taken down for a short duration of time, the rest of the application can possibly still continue to function. How much of your business will be affected will of course depend on the criticality of the service in question.

Figure 9-2 shows how security professionals can quickly isolate the infected part of the network.

Figure 9-2. When one domain is infected, security professionals can quickly isolate the domain while the rest of the application can continue to function.

This ability to isolate based on functionality is possible only due to the ability of microservices to be grouped into bounded contexts based on the business domain. With proper alignment of business domains with microservices, the security professionals can immediately communicate the effects of the outage to the stakeholders and possibly the end users of the application.

Since microservice contexts are aligned with business domains, the effect of the degradation of one particular service may generally be contained within its business domain. Figure 9-3 shows how service status dashboards can show an isolated incident on the status page.

Figure 9-3. Due to a cleanly segmented design, it may be possible to contain the effects of a security incident within one business unit while the rest of the application can continue to function without disruption. The business can also set up a status page that can provide customers with updates on the status of the outage.

Throughout this book, I have covered all of these different ways in which a well-architected system can block unauthorized access at various levels of the network and application layers. Hence, I will not go into the details of these in this chapter. However, it is worth knowing that the probability of attacks can be significantly reduced through the adoption of such design philosophies. Although incident response teams are generally not responsible for securing resources, they can be advocates of sound security practices.

Activity logging

One of the common places to find unauthorized activity is inside activity logs. Once a malicious user gains unauthorized access to your cloud infrastructure, they may attempt to use your resources to perform other unauthorized activities. This may include spinning up new resources or possibly changing IAM permissions on other users and allowing wider access to other cloud resources.

AWS CloudTrail is a service that allows you to monitor all API requests and activities of all users. Enabling CloudTrail on all of your accounts will enable you to monitor for any suspicious activities when it comes to resources. You can also find out what resources were used, when events occurred, and other details to track and assess your account’s activity.

CloudTrail logging

One way of accessing the CloudTrail events is by storing them into log files. These files can contain multiple events in a JavaScript Object Notation (JSON) format logged together.

On CloudTrail, this set of events that constitutes the state of your application is called a trail. You can enable this trail to log per region, per account, or per AWS Organizations. An AWS Organizations-wide trail is called an organization trail.

In order to maintain durability and high availability, all CloudTrail logfiles can be logged into an AWS S3 bucket that you specify. After a trail is created, CloudTrail automatically begins logging API calls to your S3 bucket. Stopping logging on the trail is as easy as turning it off or deleting it.

Figure 9-4 shows how you can create a CloudTrail trail using the AWS Management Console, then and going to the CloudTrail page and enabling the logging of management events within the trail.

Figure 9-4. CloudTrail can be enabled to log management events by specifying a trail name and an AWS S3 bucket to store the logs.

While creating the trail, you also get the choice of choosing which types of events you would want to log in the bucket. Figure 9-5 illustrates the process of choosing the events.

Figure 9-5. You can select the events you wish to log in the AWS Management Console.

Composable monitoring

Apart from logs, a different way of monitoring activity is through the use of metrics. These metrics are preaggregated data points that indicate the state of your infrastructure. This may include CPU utilization for compute services, storage utilization for storage services, or many other such statistics. In contrast to monoliths, microservices include multiple servers running in multiple locations, which generate a variety of metrics associated with each microservice. As a result, unlike monoliths where monitoring the health of one application can identify the health of the entire system, microservices require aggregation of multiple data points across the system.

A common term you may hear in most organizations is a single pane of glass. Many marketplace observability solutions offer the promise of being able to observe all the metrics that your organization needs and being able to display them to your stakeholders in one place. Over the years, I have become a huge cynic of such an approach to monitoring. A one-size-fits-all approach to monitoring just does not seem to work well in my opinion. In an environment such as a microservice environment where flexibility and autonomy is valued, different tools are better at capturing different aspects of a runtime microservice. Hence, in my opinion, it is best to use the right tool for collecting the right sets of metrics based on the task that the microservice performs. For example, one tool may be better at capturing metrics for Java-based microservices while another tool might be better for Python. Just because you want stakeholders to be able to view the metrics in the same place, does not mean the microservice developers should have to compromise on the type of tools they use to capture the data.

A strategy known as composable monitoring is utilized for aggregating such metrics across various microservices and storing them in a single place. Composable monitoring prescribes the use of multiple specialized tools which are then coupled loosely together, forming a monitoring platform. For those interested in specialized knowledge on monitoring, Practical Monitoring by Mike Julian (O’Reilly) goes into the details of how you can set up a composable monitoring strategy on your microservice organization.

Knowing the importance of composable monitoring, AWS decided to provide us with a fully managed service in the form of AWS CloudWatch in order to aggregate, compose, and visualize all your application metrics in one place.

Synthetic monitoring

A third way of monitoring an application is through the use of a technique called synthetic monitoring. In synthetic monitoring, behavioral scripts are created to mimic an action or path a customer or an end user might take on a site or a system. It is often used to monitor paths that are frequently used by users and critical business processes. Amazon CloudWatch Synthetics lets you monitor your endpoints and APIs using configurable scripts that are triggered based on a predefined schedule. Canaries are scripts written in Node.js or Python. AWS creates AWS Lambda functions within your account to run and execute these scripts.

Canaries can be created in the Synthetics tab on the AWS CloudWatch page within the AWS console, as seen in Figure 9-6.

Figure 9-6. Canaries are generally used as passive monitoring tools for ensuring the smooth functioning of an application.

Other AWS monitoring and security services

Now that I have introduced you to the monitoring tools, let me discuss a few other services that AWS offers that can help you secure your application. It is important to remember that there is no single best solution for monitoring, and you may have to utilize all the services that AWS has to offer in order to minimize the probability of an incident.

Precursors to an incident

In most cases, a threat may not have any detectable or identifiable precursors. However, in the rare event when a pre-cursor (an indicator of a legitimate, forthcoming security breach) is detected, there is a possibility to prevent the incident by adjusting one’s security posture to save a target from attack. At a minimum, the organization can monitor activity more closely that involves the target. Precursors to an incident may include but are not limited to:

• Common vulnerability scanners alerting on the presence of common vulnerabilities inside the code

• VPC flow logs showing the use of port scanners

• A sudden increase in denied authentication requests

• A sudden change in the traffic patterns of incoming traffic for a web application.

For certain sensitive applications, the presence of such a precursor may be enough to trigger the entire incident response plan, without waiting for the actual incident to occur. Unfortunately, not all indicators or precursors are 100% accurate, reducing the likelihood of detection and analysis. As an example, user-provided indications such as a complaint of a server being unavailable may often be false. As a consequence, a great number of security precursors act as very noisy indicators of security events.

AWS EventBridge

AWS provides users with the ability to interact with almost all the events that take place on their account. These events are streamed into a centralized fully managed service called AWS EventBridge. In a secure system, the baseline behavior of a cloud system that is normal can be identified proactively. When this event stream deviates significantly from baseline behavior, security architects can place alerts.

In this section, I will talk about how the AWS EventBridge can be used to better identify security breaches. I will start by introducing some of the components of AWS EventBridge.

EventBridge rules

Once you have the event bus configured to centrally stream all of the events in the account, you want to be able to distinguish between malicious and normal events. AWS EventBridge allows you to specify rules that will be evaluated against each event that is streamed to the event bus. If a rule matches the event data (and its associated metadata), you can specify any automatic action to take. Actions may include alerting the right people to a security incident or perhaps addressing the issue itself automatically. Specifically, a rule can invoke any other AWS service to take further action, such as the AWS Simple Notification Service (SNS) or AWS Lambda.

Figure 9-7 shows how you can specify a rule using an Event pattern that is evaluated against the events that are observed on AWS EventBridge. AWS provides extreme flexibility in specifying a pattern for events. Most AWS services send events to AWS EventBridge. Events contain metadata along with the event that includes data such as the name of the service, type of the event, time of occurrence, AWS region, and more. Patterns can be specified to include or filter based on any of these values.

Figure 9-7. In this example, I am defining a rule for anytime there is a finding as reported by AWS access analyzer.

EventBridge targets

Upon matching a rule to an event on the event bus, you can specify a target AWS service that AWS can automatically call for you. The target can be another AWS service you would like to invoke automatically or perhaps an HTTP API endpoint. AWS is extremely flexible in allowing you to configure event targets. The targets are called asynchronously outside of your application’s critical path; therefore, you don’t need to worry about performance implications. Figure 9-8 shows how you can define a target in the AWS Management Console.

Figure 9-8. AWS SNS for alerting and AWS Lambda for automatically attempting to remediate incidents are two popular targets for EventBridge events.

The input to the target services can also be tweaked to match the expectations of the target service. AWS EventBridge also allows integrations with other third-party alerting and monitoring services such as Zendesk, Splunk, Datadog, and others. More information can be found from AWS.

Possibility 1: Compromised infrastructure

The problem of compromised infrastructure happens when the microservices are deployed in environments where the customer bears the responsibility of securing the infrastructure as part of the AWS SRM.

In our example, it is possible for malware to be deployed on an EC2 instance that hosts Kubernetes nodes in a Kubernetes setup. An attacker may also gain access to your cloud infrastructure using a known exploit on a self-hosted service.

If the EC2 instance on which a microservice runs is compromised, taking down the service may not contain the threat effectively. The vulnerability will continue to exist in your infrastructure as long as the EC2 instance is running. In such situations, the main objective is to isolate the underlying cloud resource and not just the microservice that you identified to be the problem.

The AWS incident response framework recommends starting from the network layer by blocking and completely isolating any affected hardware in order to perform further analysis on it. In its guide, AWS prescribes a step-by-step process for performing this isolation task in a responsible way so that you and your team can then perform analysis on the affected pieces of infrastructure.

• Take a snapshot. Capture the metadata from any of your affected cloud infrastructure. This means somehow registering the state of the infrastructure at the time of the attack to ensure that any forensic activity will not result in losing precious information about the infrastructure. If your application runs on EC2 instances, this will mean taking a snapshot of the EBS volume that backs this instance. If you have RDS instances that were compromised, it will mean taking a snapshot of the database.

• Freeze the instance. Once the initial state of the application has been registered in the snapshot, you may want to disable any automated processes that may interfere with the state of the infrastructure. So, if you have any autotermination enabled on instances or any other policy that alters the state of the instance, it might be best to make sure the forensic analysis is not affected by such rules.

• Isolate. After steps 1 and 2, you are ready to isolate the instance. Recall from Chapter 5, that NACLs are best suited for specifying policies that deny access to network resources at a subnet layer. They are applied at a subnet level, so you may want to move your affected resource into its own subnet and isolate and contain any access to this resource using NACLs (and possibly security groups as a supplemental layer of protection). The NACLs should make sure that any access apart from forensic analysis will be explicitly denied in the access policy of this subnet. This isolation will ensure that only the security engineers will have access to or from the infected resource, and the rest of your application can continue to function as expected.

• Mitigate. Once isolated, you should also remove the resource from any autoscaling or load-balancing groups that it may be a part of. You should also deregister this instance from any load-balancing groups that it may be a part of.

• Maintain records. Finally for recordkeeping, it is important to set a predetermined AWS tag on such resources to indicate that these resources are kept isolated for the purposes of digital forensics and may be destroyed after the analysis is complete.

Figure 9-10 shows the result of a successful containment of an incident when the underlying infrastructure was compromised.

Figure 9-10. After containment and isolation, the original infrastructure, as described in Figure 9-9, will have a similar structure with a new node being spun up to replace the infected node.

Possibility 2: Compromised application

In the event that there is an application layer bug or a loophole that allowed an unauthorized user to gain elevated access, isolating the underlying compute instances may be a waste of time. This is mainly because restarting the same application on another piece of infrastructure will result in resurfacing the breach on another resource.

When only the infrastructure is compromised, your best business continuity plan may involve spinning up all services on new cloud infrastructure. The original breach will simply be replicated if a new service is spun up on new resources with compromised application logic. As a consequence, businesses must be prepared to encounter downtime of critical services while the root cause is resolved. A microsegmented system is significantly easier to contain attacks since microsegmentation helps you isolate segments within your application at the infrastructure level.

In addition, because it is the application logic that was affected, even services that are hosted on a fully managed AWS environment such as AWS Lambda or AWS ECS/EKS Fargate will likely continue to face the same problems.

In such a situation, the remaining microservices would typically continue to run on the existing infrastructure. The affected service is moved into a separate, isolated environment where security analysts can carefully assess its security vulnerabilities in a controlled environment. Figure 9-11 highlights this process.

Figure 9-11. Upon isolating the infected service and the instance that it runs on in a subnet using NACLs, the security professionals can perform forensic analysis in a safe and isolated environment while the rest of the application can continue to run on regular instances.

AWS Athena

In the process of forensic analysis, security professionals may require parsing and sifting through log files to find evidence related to security incidents. I have already highlighted various ways in which you can enable logging for different types of AWS activities (CloudTrail logs, CloudWatch logs, VPC flow logs). AWS Athena is a multipurpose tool that security professionals may want to use in going through these files if they are stored on AWS S3.

Amazon Athena is an interactive query service that makes it easy to analyze data directly in AWS S3 using standard SQL. In AWS Athena, you can specify the format of your logs, and AWS Athena allows security professionals to perform complex analysis directly on log files as if it were part of a queryable database. Sample scripts from AWS for Athena setup for all logging datasets are available from Amazon.

Live-box forensics

For organizations that are confident of their containment measures from Step 3, probably the best way of analyzing the incident may be to keep the affected resource running and analyzing the live environment under which the incident took place. This analysis is called live-box forensic analysis since it is performed on the same resources that were affected.

This means, if an EC2 instance was affected, security professionals can log into the box and take memory dumps to analyze for patterns of malicious code. Live-box forensics may be the most efficient form of analysis since it involves the preservation of most of the original evidence. It also allows security professionals to run common exploit scanners on the live environment where the breach occurred with a lower risk of having to deal with dead evidence. Live-box technique preserves and harvests vital evidence from an instance’s memory, cache, and other runtime processes.

Dead-box forensics

While live-box forensics are great for performing analysis, security analysts may not always feel comfortable allowing an infected machine to run regardless of how isolated it may be. Furthermore, in Step 2, security professionals may have taken steps while isolating the machine that may have already tampered with the existing evidence. As a result, an alternative method of digital forensics may also be used by security analysts to perform their root cause analysis.

This method uses any snapshots that were created during Step 3, just before containment, to recreate the machine and an environment that may be identical to what it was, when the attack actually took place. This method of analyzing based on a recreated environment (out of snapshots and events) is called dead-box forensic analysis.

Dead-box forensics allows for an extended and detailed investigation process to be performed in parallel on the same resource. It also allows for security professionals to revert back to the original state of the instance after performing each analysis since it relies on a recreated instance.

Multiple analysts can recreate the environment under which a resource was compromised by recreating the resource from the snapshot image, allowing for parallel analysis. Having said that, the loss of the vital in-memory information may result in the dead-box forensics becoming useless against attacks that only rely on in-memory attack vectors.

Tools for performing digital forensic analysis

In this section, I will introduce you to some tools that security professionals use in order to perform forensic analysis on AWS.

Cleanup

A good rule of thumb is to assume that anything an attacker may have come in contact with could have been compromised. Some common activities that cloud administrators can perform as part of this step include, but are not limited to:

• Re-encrypt all the sensitive data and snapshots using a new customer master key.

• Force users to change their passwords and enforce a stronger password policy across the organization.

• Tag data and resources that may have been compromised and set up added monitoring on these resources.

Security posturing

The root cause of the attack might be due to a leaked security key, a weak password, a weak encryption cipher, or just plain brute force. However, activities that occurred after the attack also have to be taken into consideration when performing the eradication step. Security professionals generally take this as an opportunity to improve the security posture of the organization to better prevent similar attacks from occurring. These include:

• Enforce multifactor authentication (MFA) for all principals who are trying to access the compromised resources (if not all the resources).

• Enable new firewall rules that block any access patterns that may be unnecessary and could have been exploited by the attacker.

• Review access control across the organization and narrow access to roles that may be too wide and that may have allowed attackers to gain unauthorized access.

Recovery

While many microservices may be resource agnostic, cloud administrators may still want to reuse some of the infrastructure that may have been altered or modified in the incident response process. If you were running a Kubernetes node on EC2 and suspected a malware infected it, you would isolate it as part of Step 2. After successful eradication of the malware, you may want to reuse the instance. Resource recovery is the step where you would start reverting back to the original state in a very careful and controlled manner.

This does not have to only apply to resources. Taking down your application may have been necessary if the application logic of any microservice was thought to have been compromised in Step 2. There may be a patch available from your development team to address this previously exploited vulnerability. Once this vulnerability has been patched, you may use this step to resume the service that was shut down in Step 2.

Recovery may involve such actions as restoring systems from clean snapshots, rebuilding systems from scratch, replacing compromised files with clean versions, and so forth. Constant monitoring during this phase is crucial to identifying any issues in the recovery process.

Simulate and iterate

Recovery is, however, a dangerous process. The vulnerability that existed in the first place may not always be fully patched. In the real world, it is very common for security professionals to believe that a loophole that existed originally has been patched and business can resume, only to find out that that is not the case. Hence, it is important for security professionals to prepare for the possibility of going back to Step 2 and repeating the entire process. In its whitepaper, AWS recommends users simulate various security incidents and continuously iterate upon the security measures that may have been put into place before closing the incident.

Encrypting a trail

Although CloudTrail uses AWS S3 for storing logs, just like any other AWS S3 bucket, the logs can be encrypted using AWS-managed encryption. AWS CloudTrail logs are ultimately AWS S3 objects, so the process of enabling and tweaking encryption is the same as it would be for any other AWS S3 object.

Apart from securing the infrastructure, encrypting logs also helps in maintaining compliance. Despite the fact that it is almost never acceptable for sensitive data to be kept inside logs, the fact that these logs are encrypted makes it less concerning if an application unintentionally logs more data than it is supposed to and keeps companies from violating compliance under such situations.

Log validation

From a security perspective, a principle to remember is that of nonrepudiation. Having a log trail that cannot be tampered with is an excellent way to prove compliance. AWS provides administrators with the ability to demonstrate the integrity of CloudTrail logs through a digital signature mechanism known as log validation.

Log validation can be enabled on individual trails through the AWS Console while enabling the trail, as seen in Figure 9-12.

Figure 9-12. Log file validation can be enabled through the AWS Management Console.

Through log file validation, AWS will start hashing and digitally signing the trails on behalf of the account. So, if regulators needed proof of log trail authenticity, AWS CloudTrail can provide administrators with the confidence they need.