2.3.1 More about metrics

Metrics always appear to be the most straightforward part of any monitoring architecture. As a result we sometimes don’t invest quite enough time in understanding what we’re collecting, why we’re collecting it, and what we’re doing with our metrics.

Indeed, in a lot of monitoring frameworks, the focus is on fault detection: detecting if a specific system event or state has occurred (more on this below). When we receive a notification about a specific system event, usually we go look at whatever metrics we’re collecting, if any, to find out what exactly has happened and why. In this world, metrics are seen as a by-product of or a supplement to our fault detection.

We’re going to change this idea of metrics-as-supplement. Metrics are going to be the most important part of your monitoring workflow. We’re going to turn the fault-detection-centric model on its head. Metrics will provide the state and availability of your environment and its performance.

Our framework avoids duplicating Boolean status checks when a metric can provide information on both state and performance. Harnessed correctly, metrics provide a dynamic, real-time picture of the state of your infrastructure that will help you manage and make good decisions about your environment.

Additionally, through anomaly detection and pattern analysis, metrics have the potential to identify faults or issues before they occur or before the specific system event that indicates an outage is generated.

2.3.2 So what’s a metric?

As metrics and measurement are so critical to our monitoring framework, we’re going to help you understand what metrics are and how to work with them. This is intended to be a simplified background that will allow you to understand what different types of metrics, data, and visualizations will contribute to our monitoring framework.

Metrics are measures of properties in pieces of software or hardware. To make a metric useful we keep track of its state, generally recording data points or observations over time. An observation is a value, a timestamp, and sometimes a series of properties that describe the observation, such as a source or tags. The combination of these data point observations is called a time series.

A classic example of a metric we might collect as a time series is website visits, or hits. We periodically collect observations about our website hits, recording the number of hits and the times of the observations. We might also collect properties such as the source of a hit, which server was hit, or a variety of other information.

We generally collect observations at a fixed time interval—we call this the granularity or resolution. This could range from one second to five minutes to 60 minutes or more. Choosing the right granularity at which to record a metric is critical. Choose too coarse a granularity and you can easily miss the detail. For example, sampling CPU or memory usage at five-minute intervals is highly unlikely to identify anomalies in your data. Alternatively, choosing fine granularity can result in the need to store and interpret large amounts of data. We’ll talk more in Chapter 4 about this.

So, a time-series metric is generally a chronologically ordered list of these observations. Time-series metrics are often visualized, sometimes with a mathematical function applied, as a two-dimensional plot with data values on the y-axis and time on the x-axis. Often you’ll see multiple data values plotted on the y-axis—for example, the CPU usage values from multiple hosts or successful and unsuccessful transactions.

These plots can be incredibly useful to us. They provide us with a visual representation of critical data that is (relatively) easy to interpret, certainly with more facility than perusing the same data in the form of a list of values. They also present us with a historical view of whatever we’re monitoring—they show us what has changed and when. We can use both of these capabilities to understand what’s happening in our environment and when it happened.

2.3.3 Types of metrics

There are a variety of different types of metrics we’ll see in our environment.

2.3.3.1 Gauges

The first, and most common, type of metric we’ll see is a gauge. Gauges are numbers that are expected to change over time. A gauge is essentially a snapshot of a specific measurement. The classic metrics of CPU, memory, and disk usage are usually articulated as gauges. For business metrics, a gauge might be the number of customers present on a site.

2.3.3.2 Counters

The second type of metric we’ll see frequently is a counter. Counters are numbers that increase over time and never decrease. Although they never decrease, counters can sometimes reset to zero and start incrementing again. Good examples of application and infrastructure counters are system uptime, the number of bytes sent and received by a device, or the number of logins. Examples of business counters might be the number of sales in a month or cost of sales for a time period.

In this figure we have a counter incrementing over a period of time.

A useful thing about counters is that they let you calculate rates of change. Each observed value is a moment in time: t. You can subtract the value at t from the value at t+1 to get the rate of range between the two values. A lot of useful information can be understood by understanding the rate of change between two values. For example, the number of logins is marginally interesting, but create a rate from it and you can see the number of logins per second, which should help identify periods of site popularity.

2.3.3.3 Timers

We’ll also see a small selection of timers. Timers track how long something took. They are commonly used for application monitoring—for example, you might embed a timer at the start of a specific method and stop it at the end of the method. Each invocation of the method would result in the measurement of the time the method took to execute.

Here we have a timer measuring the average time in milliseconds of the execution of a payment method.

2.3.4 Metric summaries

Often the value of a single metric isn’t useful to us. Instead, visualization of a metric requires applying mathematical transformations to it. For example, we might apply statistical functions to our metric or to groups of metrics. Some common functions we might apply include:

• Count or n — We count the number of observations in a specific time interval.

• Sum — We sum (add together) values from all observations in a specific time interval.

• Average — The mean of all values in a specific time interval.

• Median — The median is the dead center of our values: exactly 50% of values are below it, and 50% are above it.

• Percentiles — Measure the values below which a given percentage of observations in a group of observations fall.

• Standard deviation — Standard deviation from the mean in the distribution of our metrics. This measures the variation in a data set. A standard deviation of 0 means the distribution is equal to the mean of the data. Higher deviations mean the data is spread out over a range of values.

• Rates of change — Rates of change representations show the degree of change between data in a time series.

• Frequency distribution and histograms - This is a frequency distribution of a data set. You group data together—a process which is called “binning”—and present the groups in such a way that their relative sizes are visualized. The most common visualization of a frequency distribution is a histogram.

Here we see a sample histogram for the frequency distribution of heights. On the y-axis we have the frequency and on the x-axis we have the distribution of heights. We see that for the height 160–165 cm tall there is a distribution of two.

2.3.5 Metric aggregation

In addition to summaries of specific metrics, you often want to show aggregated views of metrics from multiple sources, such as disk space usage of all your application servers. The most typical example of this results in multiple metrics being displayed on a single plot. This is useful in identifying broad trends over your environment. For example, an intermittent fault in a load balancer might result in web traffic dropping off for multiple servers. This is often easier to see in aggregate than by reviewing each individual metric.

In this plot we see disk usage from numerous hosts over a 30-day period. It gives us a quick way to ascertain the current state (and rate of change) of a group of hosts.

Ultimately you’ll find a combination of single and aggregate metrics—the former to drill down into specific issues, the latter to see the high-level state—provide the most representative view of the health of your environment.

2.6.1 Static configuration

The checks generally have static configuration. Your check probably needs to be updated every time your system grows, evolves, or changes. In virtual and cloud environments, a host or service being monitored may be highly ephemeral: appearing, disappearing, or migrating locations or hosts multiple times during its lifespan. Statically defined checks just don’t handle this changing landscape, resulting in checks (and faults) on resources that do not exist or that have changed.

Further, many monitoring systems require you to duplicate configuration on both a server and the object being monitored. This lack of a single source of truth leads to increased risk of inconsistency and difficulty in managing checks. It also generally means that the monitoring server needs to know about resources being monitored before they can be monitored. This is clearly problematic in dynamic or changing landscapes.

Additionally, updates to monitoring are often considered secondary to scaling or evolving the systems themselves. Many faults are thus the result of incorrect configuration or orphaned checks. These false positives take time and effort to diagnose and resolve. They clutter your monitoring environment, hiding actual issues and concerns. Many teams do not realize they can change or delete the existing checks to remove these false positives—they take the monitoring as gospel.

2.6.2 Inflexible logic and thresholds

The checks are often inflexible Boolean logic or arbitrary static in time thresholds. They generally rely on a specific result or range being matched. The checks again don’t consider the dynamism of most complex systems. A match or a breach in a threshold may be important or could have been triggered by an exceptional event—or it could even be a natural consequence of growth.

It’s our view that arbitrary static thresholds are always wrong. Database performance analysis vendor VividCortex’s CEO Baron Schwartz put it well:

They’re worse than a broken clock, which is at least right twice a day. A threshold is wrong for any given system, because all systems are slightly different, and it’s wrong for any given moment during the day, because systems experience constantly changing load and other circumstances.

Arbitrary static thresholds set up a point-in-time boundary. During that period everything beneath that boundary is judged normal and everything above is abnormal. That boundary is not only inflexible but it’s entirely artificial. One system’s abnormality may be another’s normal operation. That means notifications wired to arbitrary thresholds will frequently fire off false positives, unrelated to any actual problems.

Boolean checks suffer similar issues to arbitrary thresholds. They are usually singletons, and often can’t take advantage of trends or prior event history. Is this really a failure? Is it a critical failure? Is it merely flapping? Could one or more failures of this check (or even across a series of checks) actually be survivable, especially in the context of a resilient and well-architected application?

2.6.3 Object-centric

The checks are object-centric, usually centric to single hosts or services. They require you to define a check on an object or objects. This breaks down quickly because each of those objects is usually part of a much larger, often much more complex system. These single object checks frequently lack any context and limit your ability to understand what the check’s output means to the broader system. As a consequence it is often hard to determine the criticality of the object’s failure.

Of course, some monitoring systems do attempt to provide contextual layers above object checks, usually via grouping, but rarely manage to model beyond basic constructs. They also lack the ability to process the dynamism in most modern environments.

2.6.4 An interlude into pets and cattle

By the end of this book a lot of folks are probably going to be surprised by how few fault detection checks we actually build. Traditional monitoring environments are often marked by thousands of checks. So why aren’t we going to replicate those environments? Well, as we’ve discovered prior, those sorts of environments aren’t easy to manage, scale, or massively duplicate, and often aren’t actually helpful in doing fault diagnosis. There’s another factor, though, related to how the process of fault resolution is changing.

Bill Baker, a former Distinguished Engineer at Microsoft, once quipped that hosts are either pets or cattle. Pets have sweet names like Fido and Boots. They are lovingly raised and looked after. If something goes wrong with them you take them to the vet and nurse them back to health. Cattle have numbers. They are raised in herds and are basically identical. If something goes wrong with one of them, you put it down and replace it with another.

In the past hosts were pets. If they broke you fixed them, often nursing a host—named for a Simpson’s character–back to life multiple times, tweaking configuration, fiddling with settings, and generally investing time in resolving the issue.

In modern environments, hosts are cattle. They should be configured automatically and rebuilt automatically. If a server fails then you kill it and restart another, automatically building it back to a functioning state. Or if you need more capacity you can add additional hosts. In these environments you don’t need hundreds or thousands of checks on individual components because the default fix for significant numbers of those components is to rebuild the host or scale the service.

2.6.5 So what do we do differently?

We’ve identified issues with traditional fault detection checks, and we’re advocating replacing these traditional status checks with events and metrics, but what does that mean? Rather than infrastructure-centric checks like pinging a host to return its availability or monitoring a process to confirm if a service is running, we configure our hosts, services, and applications to emit events and metrics. We get two benefits from events and metrics, firstly:

If a metric is measuring, an event is reporting, or a log is spooling, then the service is available. If it stops measuring or reporting then it’s likely the service is not available.

How will this work? The event router in our monitoring framework is responsible for tracking our events and metrics. It can potentially do a lot of useful things with those events and metrics including storing them, sending them to visualization tools, or using them and their values to notify us of performance issues. But most importantly it knows about the existence of those events and metrics. Let’s look at an example. We configure a web server to emit metrics showing the current workload. We then configure our event router to detect:

• If the metric stops being reported.
• If the value of a metric matches some criteria we’ve developed.

In the former case, if the metric disappears from our event router, we can be fairly certain that something has gone wrong. Either the web server has stopped working or something has happened between us and the server to prevent data from reaching our event router. In either case we’ve identified a fault that we may wish to investigate.

In the latter case, we get useful data from the payload of the event or metric. Not only is this data useful for long-term analysis of trends, performance, and capacity but it presents an opportunity to build a new paradigm for checking state. In our traditional monitoring model we rely on arbitrary thresholds to determine if we have an issue—for example, polling our CPU usage and reporting a warning if it is above a certain percentage. Now, instead of checking those arbitrary thresholds, we use a smarter approach. We can’t totally eliminate the need to set thresholds, but we can make our analysis a lot smarter by making the inputs to our thresholds more intelligent.

2.6.6 Smarter threshold inputs

In our new model we still use thresholds but the data we feed into those thresholds is considerably more sophisticated. We will generate better data and analysis and get a better understanding of the experience of our users from our collected metrics. All of this leads to the identification of valid issues and problems. In our new monitoring framework we will:

• Collect frequent and high-resolution data.
• Look at windows of data not static points in time.
• Calculate smarter input data.

Using this methodology we’re more likely to identify if a state is an actual issue instead of an anomalous spike or transitory state.

We’ll look at collection of high-frequency data and techniques for viewing windows of data in the forthcoming chapters. But calculating smarter input data for our thresholds and checks requires some explanation of some of the possible techniques we could choose and some we shouldn’t use. Let’s take a look at why, why not, and how we might use averages, the median, standard deviation, percentiles, and other statistical choices.

2.6.6.1 Average

Averages are the de facto metric analysis method. Indeed, pretty much everyone who has ever monitored or analyzed a website or application has used averages. In the web operations world, for example, many companies live and die by the average response time of their site or API. Averages are attractive because they are easy to calculate. Let’s say we have a list of seven time-series values: 12, 22, 15, 3, 7, 94, and 39. To calculate the average we sum the list of values and divide the total by the number of values in the list.

(12 + 22 + 15 + 3 + 7 + 94 + 39) / 7 = 27.428571428571

We first sum the seven values to get the total of 192. We then divide the sum by the number of values, here 7, to return the average: 27.428571428571. Seems pretty simple huh? The devil, as they say, is in the details.

Averages assume there is a normal event or that your data is a normal distribution— for example, in our average response time it’s assumed all events run at equal speed or that response time distribution is roughly bell curved. But this is rarely the case with applications. There’s an old statistics joke about a statistician who jumps in a lake with an average depth of only 10 inches and nearly drowns.

So why did he nearly drown? The lake contained large areas of shallow water and some areas of deep water. Because there were larger areas of shallow water the average was lower overall. In the monitoring world the same principal applies: lots of low values in our average distort or hide high values and vice versa. These hidden outliers can mean that while we think most of our users are experiencing a quality service, there are potentially a significant number who are not.

Let’s look at an example, using response times and requests for a website.

Here we have a plot showing response time for a number of requests. Calculating the average response time would give us 4.46 seconds. The vast majority of our users would experience a (potentially) healthy 4.46 second response time. But many of our users are experiencing response times of up to 12 seconds, perhaps considerably less acceptable.

Let’s look at another example with a wider distribution of values.

Here our average would be a less stellar 6.8 seconds. But worse this average is considerably better than the response time received by the majority of our users with a heavy distribution of request times around 9, 10, and 11 seconds long. If, we were relying on the average alone, we’d probably think our application was performing a lot better than our users are experiencing it.

2.6.6.2 Median

At this point you might be wondering about using the median. The median is the dead center of our values: exactly 50% of values are below it, and 50% are above it. If, there are an odd number of values, then the median will be the value in the middle. For the first data set we looked at—3, 7, 12, 15, 22, 39, and 94—the median is 15. If, there were an even number of values, the median would be the mean of the two values in the middle. So, if we were to remove 39 from our data set to make it even, the median would become 13.5.

Let’s apply this to our two plots.

We see in our first example figure that the median is 3, which provides an even rosier picture of our data.

In the second example the median is 8, a bit better but close enough to the average to render it ineffective.

You can probably already see that the problem again here is that, like the mean, the median works best when the data is on a bell curve… And in the real world that’s not realistic.

Another commonly used technique to identify performance issues is to calculate the standard deviation of a metric from the mean.

2.6.6.3 Standard deviation

As we learned earlier in the chapter, standard deviation measures the variation or spread in a data set. A standard deviation of 0 means most of the data is close to the mean. Higher deviations mean the data is more distributed. Standard deviations are represented by positive or negative numbers suffixed with the sigma symbol—for example, 1 sigma is one standard deviation from the mean.

Like the mean and the median, however, standard deviation works best when the data is a normal distribution. Indeed, in a normal distribution there’s a simple way of articulating the distribution: the empirical rule. Within the rule, one standard deviation or 1 to -1 will represent 68.27% of all transactions on either side of the mean, two standard deviations or 2 to -2 would be 95.45%, and three standard deviations will represent 99.73% of all transactions.

Many monitoring approaches take advantage of the empirical rule and trigger on transactions or events that are more than two standard deviations from the mean, potentially catching performance outliers. In instances like our two previous examples, however, the standard deviation isn’t overly helpful either. And without a normal distribution of data, the resulting standard deviation can be highly misleading.

Thus far our methods for identifying anomalous data in our metrics haven’t been overly promising. But all is not lost! Our next method, percentiles, will change that.

2.6.6.4 Percentiles

Percentiles measure the values below which a given percentage of observations in a group of observations fall. Essentially they look at the distribution of values across your data set. For example, the median we looked at above is the 50th percentile (or p50). In the median, 50% of values fall below and 50% above. For metrics, percentiles make a lot of sense because they make the distribution of values easy to grasp. For example, the 99th-percentile value of 10 milliseconds for a transaction is easy to interpret: 99% of transactions were completed in 10 milliseconds or less, and 1% of transactions took more than 10 milliseconds.

Percentiles are ideal for identifying outliers. If a great experience on your site is a response time of less than 10 milliseconds then 99% of your users are having a great experience—but 1% of them are not. You can then focus on addressing the performance issue that’s causing a problem for that 1%.

Let’s apply this to our previous request and response time graphs and see what appears. We’ll apply two percentiles, the 75th and 99th percentiles, to our first example data set.

We see that the 75th percentile is 6 seconds. That indicates that 75% completed in 6 seconds, and 25% of them were slower. Still pretty much in line with the earlier analysis we’ve examined for the data set. The 99th percentile, on the other hand, shows 11.73 seconds. This means 99% of users had request times of less than 11.73 seconds, and 1% had more than 11.73 seconds. This gives us a real picture of how our application is performing. We can also use the distribution between p75 and p99. If we’re comfortable with 99% of users getting 11.73 second response times or better and 1% being slower then we don’t need to consider any further tuning. Alternatively, if we want a uniform response, or if we want to lower that 11.73 seconds across our distribution, we’ve now identified a pool of transactions we can trace, profile, and improve. As we adjust the performance we’ll also be able to see the p99 response time improve.

The second data set is even more clear.

The 75th percentile is 10 seconds and the 99th percentile is 12 seconds. Here the 99th percentile provides a clear picture of the broader distribution of our transactions. This is a far more accurate reflection of the outlying transactions from our site. We now know that—as opposed to what the mean response times would imply—not all users are enjoying an adequate experience. We can use this data to identify elements of our application we can potentially improve.

Percentiles, however, aren’t perfect all the time. Our recommendation is to graph several combinations of metrics to get a clear picture of the data. For example, when measuring latency it’s often best to display a graph that shows:

• The 50th percentile (or median).
• The 95th and 99th percentiles.
• The max value.

The addition of the max value helps visualize the upward bounds of the metric you are measuring. It’s again not perfect though—a high max value can dwarf other values in a graph.

We’re going to apply percentiles and other calculations later in the book as we start to build checks and collect metrics.

2.7.1 Overhead and the observer effect

One thing to consider when thinking about data collection is that the process of collecting the data can also impact the values being collected. In normal operation many of the methods we use to collect data will consume some of the resources we’re monitoring. Sometimes this overhead becomes excessive and can actually influence the state of what you’re monitoring, or worse, trigger notifications and outages. This is often called the observer effect, derived from the related physics concept. The methods we’re going to use will focus on making that overhead as small as possible, but you should remain conscious of the effect. Granular collections—for example, hitting an HTTP site or probing an API endpoint aggressively—could result in your monitoring check being a measurable percentage of the service’s capacity.