Spring Cloud Sleuth

Spring Cloud Sleuth implements a distributed tracing solution for Spring microservices². Sleuth borrows many concepts and terminology from Dapper³, which is Google’s production distributed systems tracing infrastructure. The basic unit of work in Sleuth is called a span. A span represents the work carried out between two points in a communication network. For example, the Order Processing microservice (see Figure 13-1) receives an order from a client and processes the order. Then it synchronously talks to the Inventory microservice and, once it receives the response, publishes the ORDER_PROCESSING_COMPLETED event to the messaging system. Figure 13-2 shows how spans are identified between different points in the complete communication network. The span gets its initial value once the corresponding request hits the Order Processing microservice. In fact, the value of span is a 64-bit identifier (even though we use alphabetical letters to denote a span in Figure 13-2). Any message logged inside the Order Processing microservice will carry the span ID A. The request sent from Order Processing microservice and received by the Inventory microservice will carry the span ID B, while the request is inside the Inventory microservice, so it will hold the span ID C.

Each span has a parent span. For example, the parent span for span B is span A, while the parent span of span A is null. Similarly, span A is also a parent of span E. Figure 13-3 arranges spans by the parent-child relationship. A set of spans forming a tree-like structure is known as a trace. The value of the trace remains the same through out all the spans for a given request. As per Figure 13-2, the value of the trace is A, and it carries the same value across all the spans. The trace ID helps correlate messages between microservices. Once all the logs from different microservices are published into a centralized tracing system, given the trace ID, we can trace the message across different systems.

Let’s get our hands wet! Let’s see how to use Spring Cloud Sleuth to do distributed tracing with a set of example Spring microservices.

Engaging Spring Cloud Sleuth with Spring Boot Microservices

Engaging Sleuth with Spring Boot is quite straightforward. Once you download all the examples from the Git repository, you can find the source code related to this example available in the ch13/sample01 directory.

The spring.sleuth.sampler.percentage property in the application.properties file indicates what percentage of the requests must be traced. By default it is set to 0.1, which means that only 10% of all the requests will be published to Zipkin. By setting it to 1.0, all the requests will be published.

Well, we are not surprised if you do not find the example by its own that helpful. It does nothing more than logging—no tracing at all. The next section will clear your doubts and probably will convince you of the value of tracing.

Tracing Messages Between Multiple Microservices with Spring Cloud Sleuth

In both logs printed from the two microservices, the trace ID (76f19c035e8e1ddb) is the same, while each has its own span ID (76f19c035e8e1ddb and be46d1595ef606a0). In the next section, we see how to publish these logs to Zipkin and visualize the complete path of a request across multiple microservices.

Event-Driven Log Aggregation Architecture

Let’s revisit Figure 13-4, which is the high-level design of what we are going to discuss in this section. The design here is bit different from what we proposed in Chapter 2, “Designing Microservices” (see Figure 2-17). Figure 13-8 depicts the redone design as per the recommendation for a log aggregator architecture in Chapter 2.

Unlike in Figure 13-4 , here there is no direct coupling between the Zipkin server and the microservices. Each microservice publishes the logs to a messaging system, which can be either RabbitMQ or Kafka, and Zipkin picks the logs from the messaging system. The advantage of this model is that even if the Zipkin server is down for some time, microservices can independently keep on publishing logs and Zipkin picks them all from the messaging system when it reboots.

Distributed Tracing with Open Tracing Using Spring Boot Microservices and Zipkin

Zipkin has support for open tracing, but not the Sleuth. You can think about Sleuth as the tracing client, while Zipkin is the server that collects all the traced data. In the examples we discussed in previous sections, Sleuth was used as a tracing client for Zipkin, and it used a Zipkin specific format, which won’t work with open tracing. In this section, we see how to publish traced data from a Spring Boot microservice, which is compatible with open tracing. The source code related to this example is available in the ch13/sample03 directory.

Distributed Tracing with Open Tracing Using Spring Boot Microservices and Jaeger

In the previous section we explained how to publish open tracing compatible traces to Zipkin. Since it’s open tracing, not just Zipkin, any other product supporting open tracing should accept it. In this section, we see how to publish traced data from a Spring Boot microservice to Jaeger, which is compatible with open tracing (see Figure 13-9). Jaeger is another open source distributed tracing system inspired by Zipkin and Dapper and developed by Uber. The source code related to this example is available in the ch13/sample04 directory. We can spin up a Jaeger Docker instance with the following command, which will start running on HTTP port 16686 and UDP port 5775. After it starts, we can access its web console via http://localhost:16686/.

> docker run -d -p 5775:5775/udp -p 16686:16686 jaegertracing/all-in-one:latest

Exposing Metrics from the Spring Boot Microservice

Setting Up Prometheus

Once the Prometheus node is up, go to the URL http://localhost:9090/targets, and it will show all the services, Prometheus monitors, and the status of individual endpoints (see Figure 13-11).

Building Graphs with Prometheus

Now we’ve have all the metrics from our Spring Boot microservice, published to Prometheus. Let’s see how to build graphs to monitor those published stats.

First, go to http://localhost:9090/graph and pick the metric that you want to monitor. For example, we picked heap_used. Then when you click on the Graph tab, we should be able to see the used heap graph against time (see Figure 13-12). That way, you can add any number of graphs.

Building a Dashboard with Grafana

Once the server is started, you can log in to the Grafana management console via http://localhost:3000 with the credentials admin/admin. The first thing we need to do is introduce a new data source. Grafana uses data sources to build graphs. Click on Add Data Source and pick Prometheus as the data source type. Give it a name, let’s say prometheus_ds. The rest we can keep as it is, except for the HTTP URL property. This URL should point to the Prometheus server, which must be accessible from the node that runs Grafana. The data source configuration looks like Figure 13-13.

Now we can create a new dashboard via http://localhost:3000/dashboard/new. Choose the Graph, and then click on the Panel Title and click on Edit. Under Metrics, you choose any metrics made available via the selected data source. As shown in Figure 13-14 , we can define multiple queries. Here we have picked heap and heap_used. A query defines which information we need to pick from the metrics pulled from the Prometheus endpoint, and the graphs are rendered accordingly.

Once we complete this process, we can find the dashboard that we just created, listed on Grafana home page under Recently viewed dashboards. You can click on it to view the dashboard, which will appear as shown in Figure 13-15.

Creating Alerts with Grafana

Grafana lets you create alerts and associate them with a graph (or a dashboard panel). To create an alert corresponding to the dashboard we created in the previous section, first we need to edit the graph (see Figure 13-16) by clicking on its title and choosing Edit. Under Alerts, we can create a rule, which states under which conditions the system should raise an alert (see Figure 13-17). The only type of condition that Grafana supports at the moment is a query and for an alert rule, you can add multiple queries linked to each other (with an AND or an OR).

In addition to the max() function, Grafana also supports avg(), min(), sum(), last(), count(), median(), diff(), percent_diff(), and count_non_null(). See Figure 13-17.

Once the alert rules are set, we can configure whom to send the notifications along with a message, under the Notifications menu, when an alert is raised (see Figure 13-18). Grafana supports multiple notification channels, including Email, PagerDuty, Telegram, Slack, and many more.

Starting Fluentd as a Docker Container

The easiest and the most straightforward way to spin up Fluentd is using a Docker container. In practice, this is the most common approach too. In a production setup, where all your microservices are running in a Kubernetes environment (which we discussed in Chapter 8), the Fluentd node (which acts as a daemon) will run in the same pod, along with the corresponding microservice. In fact, you can treat the container running the Fluentd as a sidecar to the microservice (see Figure 13-20). The microservice, by default, will publish the logs to localhost:24224, which is the port Fluentd is listening to over TCP.

As we learned in Chapter 8, containers are immutable. In other words, when a container goes down it will not save any of the changes we have done to its filesystem while running. By default all the logs published to Fluentd by the Order Processing microservice will be stored in the /fluent/log directory in the container filesystem. To persist that data permanently, we need to use a Docker volume. Using the –v option in the previous command, we map the ~/data directory in the host filesystem to the /fluent/log directory in the container filesystem. Even when the container goes down, we should be able to find the log files in the ~/data directory. The –p option in the previous command maps the port 24224 from the Docker container (the port Fluentd listens by default) to port 24224 of the host machine. fluent/fluentd is the name of the container image, which will be pulled from the Docker Hub.

Once we get the Fluentd up and running, we can start a Docker container with the Order Processing microservice.

Publishing Logs to Fluentd from a Microservice Running in a Docker Container

In this command, we use the log-driver argument with the value fluentd⁸. Docker uses this driver to publish logs from the stdout⁹ (by default) to the Fluentd daemon (or the container, which runs Fluentd). The microservices developer does not need to make any changes here or do anything specific to Fluentd. By default, the fluentd log driver connects to localhost:24224 over TCP. If we run Fluentd on a different port, we need to pass the fluentd-address argument to the docker run command, with a value pointing to the Fluentd container (e.g., fluentd-address=localhost:28444).

If it all works fine, once the Order Processing service is started, we should see some logs available in the ~/data directory in the host filesystem. This is the directory we used to create a Docker volume before.

How It Works

Fluentd uses a config file, which defines the input sources and the output targets. By default it is in the /fluentd/etc directory of the container filesystem where Fluentd is running. The file is called fluent.conf. Let’s look at the default content of fluent.conf. The source tag defines where the data comes from. Under that we have forward and port elements, which make Fluentd accepts messages on port 24224 over TCP. The responsibility of the source tag is to accept messages and hand them over to the Fluentd routing engine as events. Each event has three elements: tag, time, and record. The sender (in our case the fluentd driver) defines the value of the tag.

The match element will tell Fluentd what to do with the matching messages. It matches the value of the tag element in each event with the criteria defined in the match element. In our case, it checks whether the tag starts with the word docker. The most common use case of the match element is to define the output targets. In the previous configuration, the output is written to a file in the /fluentd/log directory (in the container). If we want to send the output to different other systems, we can use any of the available Fluentd plugins¹⁰. Finally, the label element defined under the source element acts as a reference. For example, when the match element gets executed, if there is a label defined under the source, then only the match element under the corresponding label will get executed. The objective of the label element is to reduce the configuration file complexity.

Using Fluentd in a Microservices Deployment

In this section, we see how to extend this Fluentd example architecturally to fit into a production microservices deployment. As we discussed (and as shown in Figure 13-21), each Kubernetes pod (which we discussed in Chapter 8) will have an instance of Fluentd running in a container. This container can be treated as a sidecar. Each microservice in our deployment will have a similar setup. Each Fluentd node can filter out the logs they need and publish them to another Fluentd node, which can do the log aggregation. This Fluentd node also can decide what other target systems it has to publish logs.