Benchmarks are a black-box kind of measurement. Benchmarks assess a whole system's performance by submitting various kinds of load as input and measuring latency and throughput as system outputs. As an example, imagine that we are working on a typical shopping cart web application. To benchmark this application, we can write a simple HTTP client to query our service and record the time taken to complete a request. This client can be used to send an increasing number of requests per second and output a summary of the recorded response times.

Multiple kinds of benchmark exist to answer different questions about your system. You can replay historical production data to make sure that your application is meeting the expected performance goals when handling realistic load. Load and stress test benchmarks identify the breaking points of your application, and they exercise its robustness when receiving exceptionally high load for an extended period of time.

Benchmarks are also a great tool to compare different iterations of the same application and either detect performance regression or confirm improvements. By executing the same benchmark against two versions of your code, you can actually prove that your recent changes yielded better performance.

For all their usefulness, benchmarks do not provide any insight into how each part of the software performs; hence, they are black-box tests. Benchmarks do not help us identify bottlenecks or determine which part of the system should be improved to yield better overall performance. To look into the black box, we turn to profiling.

As opposed to benchmarking, profiling is intended to be used to analyze the internal characteristics of your application. A profiler enables white-box testing to help you identify bottlenecks by capturing the execution time and resource consumption of each part of your program. By examining your application at runtime, a profiler provides you with great details about the behavior of your code, including the following:

Most profilers instrument the code under observation, either at compile time or runtime, to inject counters and profiling components. This instrumentation imposes a runtime cost that degrades system throughput and latency. For this reason, profilers should not be used to evaluate the expected throughput and latency of a system in production (as a reminder, this is a use case for a benchmark).

In general, you should always profile your application before deciding to do any performance-driven improvement. You should make sure that the part of the code you are planning to improve actually is a bottleneck.

Profilers and benchmarks have different purposes, and they help us answer different questions. A typical workflow to improve performance should take advantage of both these techniques and leverage their strengths to optimize the code improvement process. In practice, this workflow looks like the following:

Keep in mind, it is important to run all benchmarking and profiling sessions in the same environment. Consult your resource checklist to ensure that your environment remains constant across tests. Any change in your resources invalidates your test results. Just like a science experiment, you must be careful to change only one part of the experiment at a time.