10

Evaluating and Benchmarking the Performance of Minimal APIs

The purpose of this chapter is to understand one of the motivations for which the minimal APIs framework was created.

This chapter will provide some obvious data and examples of how you can measure the performance of an ASP.NET 6 application using the traditional approach as well as how you can measure the performance of an ASP.NET application using the minimal API approach.

Performance is key to any functioning application; however, very often it takes a back seat.

A performant and scalable application depends not only on our code but also on the development stack. Today, we have moved on from the .NET full framework and .NET Core to .NET and can start to appreciate the performance that the new .NET has achieved, version after version – not only with the introduction of new features and the clarity of the framework but also primarily because the framework has been completely rewritten and improved with many features that have made it fast and very competitive compared to other languages.

In this chapter, we will evaluate the performance of the minimal API by comparing its code with identical code that has been developed traditionally. We’ll understand how to evaluate the performance of a web application, taking advantage of the BenchmarkDotNet framework, which can be useful in other application scenarios.

With minimal APIs, we have a new simplified framework that helps improve performance by leaving out some components that we take for granted with ASP.NET.

The themes we will touch on in this chapter are as follows:

• Improvements with minimal APIs
• Exploring performance with load tests
• Benchmarking minimal APIs with BenchmarkDotNet

Technical requirements

Many systems can help us test the performance of a framework.

We can measure how many requests per second one application can handle compared to another, assuming equal application load. In this case, we are talking about load testing.

To put the minimal APIs on the test bench, we need to install k6, the framework we will use for conducting our tests.

We will launch load testing on a Windows machine with only .NET applications running.

To install k6, you can do either one of the following:

• If you’re using the Chocolatey package manager (https://chocolatey.org/), you can install the unofficial k6 package with the following command:

  choco install k6

• If you’re using Windows Package Manager (https://github.com/microsoft/winget-cli), you can install the official package from the k6 manifests with this command:

  winget install k6

• You can also test your application published on the internet with Docker:

  docker pull loadimpact/k6

• Or as we did, we installed k6 on the Windows machine and launched everything from the command line. You can download k6 from this link: https://dl.k6.io/msi/k6-latest-amd64.msi.

In the final part of the chapter, we’ll measure the duration of the HTTP method for making calls to the API.

We’ll stand at the end of the system as if the API were a black box and measure the reaction time. BenchmarkDotNet is the tool we’ll be using – to include it in our project, we need to reference its NuGet package:

dotnet add package BenchmarkDotNet

All the code samples in this chapter can be found in the GitHub repository for this book at the following link:

https://github.com/PacktPublishing/Minimal-APIs-in-ASP.NET-Core-6/tree/main/Chapter10

Improvements with minimal APIs

Minimal APIs were designed not only to improve the performance of APIs but also for better code convenience and similarity to other languages to bring developers from other platforms closer. Performance has increased both from the point of view of the .NET framework, as each version has incredible improvements, as well as from the point of view of the simplification of the application pipeline. Let’s see in detail what has not been ported and what improves the performance of this framework.

The minimal APIs execution pipeline omits the following features, which makes the framework lighter:

• Filters, such as IAsyncAuthorizationFilter, IAsyncActionFilter, IAsyncExceptionFilter, IAsyncResultFilter, and IasyncResourceFilter
• Model binding
• Binding for forms, such as IFormFile
• Built-in validation
• Formatters
• Content negotiations
• Some middleware
• View rendering
• JsonPatch
• OData
• API versioning

Performance Improvements in .NET 6

Version after version, .NET improves its performance. In the latest version of the framework, improvements made over previous versions have been reported. Here’s where you can find a complete summary of what’s new in .NET 6:

https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-6/

Exploring performance with load tests

How to estimate the performance of minimal APIs? There are many points of view to consider and in this chapter, we will try to address them from the point of view of the load they can support. We decided to adopt a tool – k6 – that performs load tests on a web application and tells us how many requests per second can a minimal API handle.

As described by its creators, k6 is an open source load testing tool that makes performance testing easy and productive for engineering teams. The tool is free, developer-centric, and extensible. Using k6, you can test the reliability and performance of your systems and catch performance regressions and problems earlier. This tool will help you to build resilient and performant applications that scale.

In our case, we would like to use the tool for performance evaluation and not for load testing. Many parameters should be considered during load testing, but we will only focus on the http_reqs index, which indicates how many requests have been handled correctly by the system.

We agree with the creators of k6 about the purpose of our test, namely performance and synthetic monitoring.

Use cases

k6 users are typically developers, QA engineers, SDETs, and SREs. They use k6 for testing the performance and reliability of APIs, microservices, and websites. Common k6 use cases include the following:

• Load testing: k6 is optimized for minimal resource consumption and designed for running high load tests (spike, stress, and soak tests).
• Performance and synthetic monitoring: With k6, you can run tests with a small load to continuously validate the performance and availability of your production environment.
• Chaos and reliability testing: k6 provides an extensible architecture. You can use k6 to simulate traffic as part of your chaos experiments or trigger them from your k6 tests.

However, we have to make several assumptions if we want to evaluate the application from the point of view just described. When a load test is performed, it is usually much more complex than the ones we will perform in this section. When an application is bombarded with requests, not all of them will be successful. We can say that the test passed successfully if a very small percentage of the responses failed. In particular, we usually consider 95 or 98 percentiles of outcomes as the statistic on which to derive the test numbers.

With this background, we can perform stepwise load testing as follows: in ramp up, the system will be concerned with running the virtual user (VU) load from 0 to 50 for about 15 seconds. Then, we will keep the number of users stable for 60 seconds, and finally, ramp down the load to zero virtual users for another 15 seconds.

Each newly written stage of the test is expressed in the JavaScript file in the stages section. Testing is therefore conducted under a simple empirical evaluation.

First, we create three types of responses, both for the ASP.NET Web API and minimal API:

• Plain-text.
• Very small JSON data against a call – the data is static and always the same.
• In the third response, we send JSON data with an HTTP POST method to the API. For the Web API, we check the validation of the object, and for the minimal API, since there is no validation, we return the object as received.

The following code will be used to compare the performance between the minimal API and the traditional approach:

Minimal API

app.MapGet("text-plain",() => Results.Content("response"))
.WithName("GetTextPlain");
app.MapPost("validations",(ValidationData validation) => Results.Ok(validation)).WithName("PostValidationData");
app.MapGet("jsons", () =>
     {
           var response = new[]
           {
                new PersonData { Name = "Andrea", Surname = 
                "Tosato", BirthDate = new DateTime
                (2022, 01, 01) },
                new PersonData { Name = "Emanuele", 
                Surname = "Bartolesi", BirthDate = new 
                DateTime(2022, 01, 01) },
                new PersonData { Name = "Marco", Surname = 
                "Minerva", BirthDate = new DateTime
                (2022, 01, 01) }
           };
           return Results.Ok(response);
     })
.WithName("GetJsonData");

Traditional Approach

For the traditional approach, three distinct controllers have been designed as shown here:

[Route("text-plain")]
     [ApiController]
     public class TextPlainController : ControllerBase
     {
           [HttpGet]
           public IActionResult Get()
           {
                 return Content("response");
           }
     }
[Route("validations")]
     [ApiController]
     public class ValidationsController : ControllerBase
     {
           [HttpPost]
           public IActionResult Post(ValidationData data)
           {
                 return Ok(data);
           }
     }
     public class ValidationData
     {
           [Required]
           public int Id { get; set; }
           [Required]
           [StringLength(100)]
           public string Description { get; set; }
     }
[Route("jsons")]
[ApiController]
public class JsonsController : ControllerBase
{
     [HttpGet]
     public IActionResult Get()
     {
           var response = new[]
           {
              new PersonData { Name = "Andrea", Surname = 
              "Tosato", BirthDate = new 
              DateTime(2022, 01, 01) },
              new PersonData { Name = "Emanuele", Surname = 
              "Bartolesi", BirthDate = new 
              DateTime(2022, 01, 01) },
              new PersonData { Name = "Marco", Surname = 
              "Minerva", BirthDate = new 
              DateTime(2022, 01, 01) }
            };
            return Ok(response);
     }
}
     public class PersonData
     {
           public string Name { get; set; }
           public string Surname { get; set; }
           public DateTime BirthDate { get; set; }
     }

In the next section, we will define an options object, where we are going to define the execution ramp described here. We define all clauses to consider the test satisfied. As the last step, we write the real test, which does nothing but call the HTTP endpoint using GET or POST, depending on the test.

Writing k6 tests

Let’s create a test for each case scenario that we described in the previous section:

import http from "k6/http";
import { check } from "k6";
export let options = {
     summaryTrendStats: ["avg", "p(95)"],
     stages: [
           // Linearly ramp up from 1 to 50 VUs during 10 
              seconds
              { target: 50, duration: "10s" },
           // Hold at 50 VUs for the next 1 minute
              { target: 50, duration: "1m" },
           // Linearly ramp down from 50 to 0 VUs over the 
              last 15 seconds
              { target: 0, duration: "15s" }
     ],
     thresholds: {
           // We want the 95th percentile of all HTTP 
              request durations to be less than 500ms
              "http_req_duration": ["p(95)<500"],
           // Thresholds based on the custom metric we 
              defined and use to track application failures
              "check_failure_rate": [
          // Global failure rate should be less than 1%
             "rate<0.01",
          // Abort the test early if it climbs over 5%
             { threshold: "rate<=0.05", abortOnFail: true },
           ],
     },
};
export default function () {
    // execute http get call
    let response = http.get("http://localhost:7060/jsons");
    // check() returns false if any of the specified 
       conditions fail
    check(response, {
           "status is 200": (r) => r.status === 200,
    });
}

In the preceding JavaScript file, we wrote the test using k6 syntax. We have defined the options, such as the evaluation threshold of the test, the parameters to be measured, and the stages that the test should simulate. Once we have defined the options of the test, we just have to write the code to call the APIs that interest us – in our case, we have defined three tests to call the three endpoints that we want to evaluate.

Running a k6 performance test

Now that we have written the code to test the performance, let’s run the test and generate the statistics of the tests.

We will report all the general statistics of the collected tests:

1. First, we need to start the web applications to run the load test. Let’s start with both the ASP.NET Web API application and the minimal API application. We expose the URLs, both the HTTPS and HTTP protocols.
2. Move the shell to the root folder and run the following two commands in two different shells:

   dotnet .MinimalAPI.SampleinRelease et6.0MinimalAPI.Sample.dll --urls=https://localhost:7059/;http://localhost:7060/

   dotnet .ControllerAPI.SampleinRelease et6.0ControllerAPI.Sample.dll --urls="https://localhost:7149/;http://localhost:7150/"

3. Now, we just have to run the three test files for each project.
   • This one is for the controller-based Web API:

     k6 run .K6Controllersjson.js --summary-export=.K6 esultscontroller-json.json

   • This one is for the minimal API:

     k6 run .K6Minimaljson.js --summary-export=.K6 esultsminimal-json.json

Here are the results.

For the test in traditional development mode with a plain-text content type, the number of requests served per second is 1,547:

Figure 10.1 – The load test for a controller-based API and plain text

For the test in traditional development mode with a json content type, the number of requests served per second is 1,614:

Figure 10.2 – The load test for a controller-based API and JSON result

For the test in traditional development mode with a json content type and model validation, the number of requests served per second is 1,602:

Figure 10.3 – The load test for a controller-based API and validation payload

For the test in minimal API development mode with a plain-text content type, the number of requests served per second is 2,285:

Figure 10.4 – The load test for a minimal API and plain text

For the test in minimal API development mode with a json content type, the number of requests served per second is 2,030:

Figure 10.5 – The load test for a minimal API and JSON result

For the test in minimal API development mode with a json content type with model validation, the number of requests served per second is 2,070:

Figure 10.6 – The load test for a minimal API and no validation payload

In the following image, we show a comparison of the three tested functionalities, reporting the number of requests served with the same functionality:

Figure 10.7 – The performance results

As we might have expected, minimal APIs are much faster than controller-based web APIs.

The difference is approximately 30%, and that’s no small feat.

Obviously, as previously mentioned, minimal APIs have features missing in order to optimize performance, the most striking being data validation.

In the example, the payload is very small, and the differences are not very noticeable.

As the payload and validation rules grow, the difference in speed between the two frameworks will only increase.

We have seen how to measure performance with a load testing tool and then evaluate how many requests it can serve per second with the same number of machines and users connected.

We can also use other tools to understand how minimal APIs have had a strong positive impact on performance.

Benchmarking minimal APIs with BenchmarkDotNet

BenchmarkDotNet is a framework that allows you to measure written code and compare performance between libraries written in different versions or compiled with different .NET frameworks.

This tool is used for calculating the time taken for the execution of a task, the memory used, and many other parameters.

Our case is a very simple scenario. We want to compare the response times of two applications written to the same version of the .NET Framework.

How do we perform this comparison? We take an HttpClient object and start calling the methods that we have also defined for the load testing case.

We will therefore obtain a comparison between two methods that exploit the same HttpClient object and recall methods with the same functionality, but one is written with the ASP.NET Web API and the traditional controllers, while the other is written using minimal APIs.

BenchmarkDotNet helps you to transform methods into benchmarks, track their performance, and share reproducible measurement experiments.

Under the hood, it performs a lot of magic that guarantees reliable and precise results thanks to the perfolizer statistical engine. BenchmarkDotNet protects you from popular benchmarking mistakes and warns you if something is wrong with your benchmark design or obtained measurements. The library has been adopted by over 6,800 projects, including .NET Runtime, and is supported by the .NET Foundation (https://benchmarkdotnet.org/).

Running BenchmarkDotNet

We will write a class that represents all the methods for calling the APIs of the two web applications. Let’s make the most of the startup feature and prepare the objects we will send via POST. The function marked as [GlobalSetup] is not computed during runtime, and this helps us calculate exactly how long it takes between the call and the response from the web application:

1. Register all the classes in Program.cs that implement BenchmarkDotNet:

   BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);

In the preceding snippet, we have registered the current assembly that implements all the functions that will be needed to be evaluated in the performance calculation. The methods marked with [Benchmark] will be executed over and over again to establish the average execution time.

2. The application must be compiled on release and possibly within the production environment:

   namespace DotNetBenchmarkRunners

   {

        [SimpleJob(RuntimeMoniker.Net60, baseline: true)]

        [JsonExporter]

        public class Performances

        {

              private readonly HttpClient clientMinimal =

              new HttpClient();

              private readonly HttpClient

              clientControllers = new HttpClient();

              private readonly ValidationData data = new

              ValidationData()

              {

                    Id = 1,

                    Description = "Performance"

              };

              [GlobalSetup]

              public void Setup()

              {

                    clientMinimal.BaseAddress = new

                    Uri("https://localhost:7059");

                    clientControllers.BaseAddress = new

                    Uri("https://localhost:7149");

              }

              [Benchmark]

              public async Task Minimal_Json_Get() =>

              await clientMinimal.GetAsync("/jsons");

              [Benchmark]

              public async Task Controller_Json_Get() =>

              await clientControllers.GetAsync("/jsons");

              [Benchmark]

              public async Task Minimal_TextPlain_Get()

              => await clientMinimal.

              GetAsync("/text-plain");

              [Benchmark]

              public async Task

              Controller_TextPlain_Get() => await

              clientControllers.GetAsync("/text-plain");

              

              [Benchmark]

              public async Task Minimal_Validation_Post()

              => await clientMinimal.

              PostAsJsonAsync("/validations", data);

              

              [Benchmark]

              public async Task

              Controller_Validation_Post() => await

              clientControllers.

              PostAsJsonAsync("/validations", data);

        }

        public class ValidationData

        {

              public int Id { get; set; }

              public string Description { get; set; }

        }

   }

3. Before launching the benchmark application, launch the web applications:

Minimal API application

dotnet .MinimalAPI.SampleinRelease et6.0MinimalAPI.Sample.dll --urls="https://localhost:7059/;http://localhost:7060/"

Controller-based application

dotnet .ControllerAPI.SampleinRelease et6.0ControllerAPI.Sample.dll --urls=https://localhost:7149/;http://localhost:7150/

By launching these applications, various steps will be performed and a summary report will be extracted with the timelines that we report here:

dotnet .DotNetBenchmarkRunnersinRelease et6.0DotNetBenchmarkRunners.dll --filter *

For each method performed, the average value or the average execution time is reported.

Table 10.1 – Benchmark HTTP requests for minimal APIs and controllers

In the following table, Error denotes how much the average value may vary due to a measurement error. Finally, the standard deviation (StdDev) indicates the deviation from the mean value. The times are given in μs and are therefore very small to measure empirically if not with instruments with that just exposed.

Summary

In the chapter, we compared the performance of minimal APIs with that of the traditional approach by using two very different methods.

Minimal APIs were not designed for performance alone and evaluating them solely on that basis is a poor starting point.

Table 10.1 indicates that there are a lot of differences between the responses of minimal APIs and that of traditional ASP.NET Web API applications.

The tests were conducted on the same machine with the same resources. We found that minimal APIs performed about 30% better than the traditional framework.

We have learned about how to measure the speed of our applications – this can be useful for understanding whether the application will hold the load and what response time it can offer. We can also leverage this on small portions of critical code.

As a final note, the applications tested were practically bare bones. The validation part that should be evaluated in the ASP.NET Web API application is almost irrelevant since there are only two fields to consider. The gap between the two frameworks increases as the number of components that have been eliminated in the minimal APIs that we have already described increases.