Chapter 1. Laying the Performance Analysis Groundwork
Now that Microsoft’s .NET strategy has made it possible for us to connect information, devices, and people in a unified way, Web-enabling traditional desktop applications are no longer an option but a requirement. Customers will expect to have access to their office productivity applications from any Web-capable device, be it the browser on their desktop, a Personal Digital Assistant (PDA) device, or a Wireless Application Protocol (WAP) enabled phone. This drive to bring applications to the Internet will push development teams and the software they build to their limits. The Web-enabled application will have to perform as well and be as reliable and extensible as its desktop counterpart. This paradigm shift away from stand-alone desktop applications will focus increased attention on performance testing and tuning throughout the application development life cycle as many more variables that could affect application performance are introduced.
Why Is Performance Testing and Tuning Important?
When designing an application that will be accessed over the Internet, development teams must keep in mind that hundreds of thousands of users could concurrently access the application. This concurrent user load could place tremendous stress upon a system, cause unexpected delays, and result in a poor user experience. However, if the application has been adequately tested and tuned for optimal performance under stress, managers and software developers will be confident that their code will perform at optimal levels and have the necessary data to plan site capacity needs. The following real world example demonstrates what can happen when adequate performance testing has not occurred.
Effects of Current and Emerging Architecture Technologies
While the focus of this book will be on performance testing applications built with the Microsoft .NET Framework, the methodology presented also offers a backward compatible approach for Windows Distributed interNet applications Architecture (DNA). Given that .NET will still be relatively early in the market adoption cycle by the time this book goes to print, many Microsoft customers will still be running traditional ASP-based Web applications. Therefore, we present the performance analysis methodology that can be effectively utilized by Microsoft customers running both .NET Web services as well as Windows DNA Web applications. Where appropriate, we have included specific “how to” methodology for Windows DNA applications. The methodology laid out in this book focuses on identifying performance metrics that are business critical, such as response times, throughput, and scalability. As illustrated below, setting performance goals in the planning phase drives critical performance testing breakpoints.
When performance planning is completed, our methodology focus shifts to utilizing software load simulation tools for pushing Web applications to their performance breakpoints. The goal during the testing phase is to define the real-world performance limits for the Web application. Finally, the focus shifts to the methods of drilling down into each application layer through effective analysis techniques with the goal of identifying the bottlenecks and formulating performance-enhancing solutions.
What Is .NET?
As the purpose of this book is to explain how to performance test a .NET application, and many of the application samples used throughout this book were built using the .NET Framework, it is necessary to first understand what .NET is. In fact, experience has shown us that effective performance testing is difficult, if not impossible, without an in-depth understanding of the underlying technologies involved. The material presented here serves only as an introduction to Microsoft’s .NET initiative. For a more detailed perspective of the .NET initiative please visit http://www.microsoft.com/net.
The .NET Platform
What is .NET? Or alternatively, what are the services that make up the .NET platform? The .NET platform is a set of developmental tools and operational systems used to build, expose, and consume XML Web services thereby enabling a personal, integrated Web delivered through smart devices while using open standards. The main .NET platform components are as follows:
The .NET Framework is an environment for building, deploying, and running XML Web services and other applications. The .NET Framework has two parts: the common language runtime (CLR) and the class libraries, which include ASP.NET, Enterprise Services, ADO.NET, and Windows Forms.
Visual Studio .NET provides a complete development environment for building on the Microsoft .NET platform.
The Mobile Internet Toolkit is a set of programming interfaces that enable developers to target mobile devices like smart phones, PDAs, and server infrastructure.
The .NET Enterprise Servers include Application Center 2000, BizTalk Server 2000, Commerce Server 2000, Exchange Server 2000, Internet Security and Acceleration Server, Host Integration Server 2000, Mobile Information 2001 Server, and SQL Server 2000.
.NET services is the set of core XML Web services that will be supplied by Microsoft. However, XML Web services can be built by anyone.
Figure 1-1 illustrates how the core components of the .NET platform fit together to provide user experiences anywhere and on any device. Developers can use the .NET Framework to build XML Web services. The CLR is important because it is the engine at the core of managed code execution. With the runtime engine, developers creating Web services can integrate and execute their code in different languages. Web services that are created with the .NET Framework run on the .NET Enterprise servers and can be accessed anytime, anywhere, and on any device.
Standard .NET Protocols
The fact that .NET is built on open Internet-based standards makes the .NET Framework extensible and enables it to easily communicate with other Web and non-Web-based solutions. The predominant standards that .NET employs to make Web services possible are:
Extensible Markup Language (XML)
Simple Object Access Protocol (SOAP)
Hypertext Transfer Protocol (HTTP)
XML, a text-based language much like the ubiquitous HTML, is a specification that allows for custom HTML-like tags that describe both the document (metadata) and the content (data). The .NET vision needs XML because of the inherent problems with HTML. There are conflicting standards with HTML, which cause different browsers to handle standard tags in different ways. This means that Web designers need to create different versions of the same HTML document for different browsers. To date, efforts to create international HTML standards have not materialized. In addition, HTML has an inadequate linking system. HTML links are hard coded into documents and must be searched and changed for each link that changes. XML allows you to associate links to any element and to link to multiple locations, effectively solving these HTML limitations.
Web Service Description Language (WSDL)
.NET also employs a newer XML standard formerly known as the Service Description Language (SDL), and now known as Web Services Description Language (WSDL). As the name implies, WSDL is a language designed for describing a Web service. It is used to create or generate a .wsdl file that other services can use to determine the exposed functionality a Web service offers. Simple Object Access Protocol (SOAP) is used as the basic wire format for communicating between objects and solutions. Very much like Remote Procedure Calls (RPC) in functionality, SOAP uses XML to describe its contents. XML makes it simple, humanly interpretable, open, and extensible.
Universal Description, Discovery, and Integration (UDDI)
.NET also takes advantage of the new Universal Description, Discovery, and Integration (UDDI) specification that is aimed at creating Web service registries. These registries will enable companies to register information about their Web service solutions, as well as other data, so that those who wish to use them can easily find them. Internet Native Integration Methodology (INIM) allows for XML interaction between systems and an open set of standards. INIM works with any operating system, programming model, or network and can expose existing code as XML Web services, giving different systems the ability to communicate. UDDI specifications define a standard to publish and discover information about Web services.
What Is an XML Web Service?
Today’s Internet services are mostly portals that offer services that cannot be used anywhere else. One inconvenient result of this service is that companies cannot easily share information. For example, even contact and other personal information has to be entered for each site. XML Web services are units of application logic providing data and services to other applications and users. An example of a Web service is the authentication functionality provided by Microsoft .NET Passport service. Applications access XML Web services via standard Web protocols and data formats (i.e. HTTP, XML, and SOAP) independent of how each XML Web service is implemented. XML Web services combine the benefits of component-based development and the Web, and are a cornerstone of the Microsoft .NET programming model. XML Web services transform read-only Web sites into computing sites that can both expose methods and read other XML Web services. The use of XML allows for the sharing of data between any operating system and network via XML and SOAP services which act to connect formerly incompatible or disparate systems (i.e. accessing an application running on a Macintosh, a Unix, or a Linux via a Windows CE device).
Web Services are software solutions delivered via the Internet to any Web-enabled device. Today, this device is the Web browser on your computers, but the device-agnostic design of .NET will eliminate this limitation. XML is an industry standard, and XML support is currently offered in all Microsoft products, including the latest generation of servers. The .NET Framework is built into all the .NET products such as Microsoft Visual Basic .NET, Microsoft Visual C++, and Microsoft Visual C#.
Devices Drive Demand for Web Services
Web-enabled devices are driving the need for services provided by the .NET platform. There is a plethora of new devices available today. These devices are mobile, small, and smart, and all require different interconnectivity depending on the desires of their users. Meanwhile, businesses want information from a multitude of sources, and they want it immediately. Applications that supply information any time and from any location need to run across multiple client platforms with widely diverse capabilities. The following example illustrates what is possible with the .NET XML-centric programming model.
Web Services Will Increase Importance of Web Performance Testing
This paradigm shift in the way information is delivered to devices will require new ways to test and tune applications for performance. Just as traditional platforms cannot keep pace with this shift in technology, traditional functional testing methodologies cannot adequately define application performance and identify bottlenecks associated with these new .NET applications. A new way of approaching the traditional software development life cycle, which includes effective performance testing throughout, is required for the .NET world. Presenting these new requirements is the driving force behind this book.
Performance Goals
Given this general understanding of Web services based on .NET, what becomes obvious is that the expeditious delivery of these services to devices any time and anywhere is critical. There are many aspects to performance but the most important from the ACE team perspective is increasing performance by reducing end user response time bottlenecks. Optimizing user response times is our top goal because the response time is the only performance metric to which the users are directly exposed. Scalability and availability are important inputs to optimal response times and these will be discussed in detail. Just because the user does not “see” them, does not diminish their importance. Without adequate scalability and availability, the application response times will obviously suffer. Ultimately though, application responsiveness is the critical component that will determine success or failure with the customer in the performance game.
Computer Interaction Behavioral Patterns
Despite the relatively recent rise in Internet usage during the last decade, computer interaction response time studies have been around since the advent of the computer. Research indicates that users interact with a computer to perform a task. According to an article titled "Response Time In Man-Computer Conversational Transactions,” written in 1968 by R.B. Miller for the AFIPS Fall Joint Computer Conference, Vol. 33, 267-277, the computer user will react to response time performance in predictable behavioral patterns similar to those listed here:
0.1 second
is the limit for having the user feel that the system is reacting instantaneously, meaning that no special feedback is necessary except to display the result.
1.0 second
is the limit for the user’s flow of thought to stay uninterrupted, even though the user will notice the delay. Normally, no special feedback is necessary during delays of more than 0.1 but less than 1.0 second, but the user does lose the feeling of operating directly on the data.
10 seconds
is the limit for keeping user attention focused on the screen dialogue. For longer delays, users will want to perform other tasks while waiting for the computer to finish, so they should be given feedback indicating when the computer is likely to be done. Feedback during the delay is especially important if the response time is likely to be highly variable, since users will then not know what to expect.
When measuring response times, it might become apparent that the root cause for the high response times could be beyond your control, making it impossible to attain one-tenth of a second times. For example, there really is not a lot that can be done about Internet congestion or a slow client dial-up connection. Nevertheless, it is still necessary to understand the root cause of the high response times, to make an impact wherever possible. The goal of this book is to lay out a reliable methodology for identifying this cause, otherwise known as the bottleneck, and subsequently addressing that bottleneck in such a manner that it results in improved application performance. This book will hopefully serve as proof that proper performance testing and tuning methodologies integrated into the software development life cycle can result in sub-second response times for the resultant .NET application.
Performance Testing Your Application
With an increased movement toward the use of Web services driving an ever-increasing amount of traffic, the key challenge that emerges is one of ensuring optimal application performance. This book will offer the solutions to meet this challenge by presenting an in-depth approach for determining the following key performance-related factors:
Calculating maximum scalability
Quantifying average client response times under load
Identifying bottlenecks that prevent performance gains
Addressing these bottlenecks to tune for optimum performance
Additionally, this book will present alternative approaches for estimating Web application capacity with a methodology developed by Microsoft. This methodology has been dubbed Transaction Cost Analysis (TCA). The TCA methodology assists with estimating capacity planning needs for Web applications by associating server resource costs such as CPU to typical user operational costs. In this manner, one can estimate and prepare for site capacity needs prior to large traffic spikes that can occur as a direct result of large marketing or news events.
At the 10,000-foot level, the performance testing life cycle presented in this book consist of:
Planning Performance Analysis
Creating Effective Stress Scripts
Executing Stress Tests
Analyzing performance data to identify and address performance bottlenecks
Each of these steps will be discussed in the chapters that follow. To reiterate, performance analysis requires an extremely in-depth approach, coupled with experience and knowledge regarding the technologies utilized.
Figure 1-2 shows the performance analysis methodology this book will discuss.
Planning Performance Analysis
This step involves gathering key preliminary information that will structure and focus the testing approach. Data collected in the planning phase should, at a minimum, provide two things: 1) the details necessary to duplicate the production application environment as closely as possible, and 2) an understanding of how the application is used, including indicators of critical performance issues. Useful performance information sources can include marketing forecasts, production IIS logs, production performance logs, and functional specifications for the application. The quality of the performance data collected in advance of the actual performance testing is critical. It will help determine the requirements for the test environment and will be used in all phases of the analysis from staging the environment to deciphering performance test results. We present a detailed approach to planning in Chapter 2.
Creating Effective Stress Scripts
After gathering the required information and preparing your test environment, the next step is to create stress scripts that accurately simulate the expected production traffic. This is most effectively accomplished using historical data from the production site that is combined with expected data from the marketing or business analysts. Creating bulletproof stress scripts using Microsoft’s Application Center Test (ACT) tool will be detailed in Chapter 3.
Executing Stress Tests
After bulletproof scripts have been created to simulate peak client load, stress testing begins. At this point, it is critical to have verified script functionality to ensure the scripts simulate production site traffic as closely as possible as the quality of the stress test is directly tied to the quality of the scripts. In addition, using a methodology dubbed “smoke testing,” the optimal load should be identified prior to running the actual stress tests, which generate the performance data that will aid in pinpointing the bottlenecks. Details for executing stress tests, including key focal elements while smoke testing, are presented in Chapter 3.
Analyzing Performance Results
After the stress tests have been run and data generated, the analysis phase begins. The first concern is to verify that the stress test ran through the simulation successfully because the quality of the data is only as good as the quality of the test. The analysis phase is the most technically in-depth step in effective performance analysis methodology, so starting with high quality data is critical to deriving high quality results and conclusions. The majority of time budgeted for performance analysis should be concentrated in this phase. It is for this reason that three chapters of this book are dedicated to analysis concepts. Chapter 6 offers an in-depth approach to analyzing the Web tier. Chapter 7 offers an in-depth approach to profiling managed code. And finally, Chapter 8 offers an in-depth approach to identifying bottlenecks on the data or SQL tier. Experience has shown that the SQL tier can be a common place for bottlenecks if the code at this tier is not designed and tuned properly. Bottlenecks on the SQL tier are pivotal because it is more difficult to scale out databases through clustering versus the available options for scaling out the Web tier. Of course, there are entire books written on these technologies alone. The testing methodology in this book will focus on efficiently identifying performance bottlenecks and strategically offering tuning approaches aimed at achieving better performance.
Identifying Performance Bottlenecks
Bottlenecks that can affect end user response times include application and server throughput, end-to-end Internet connection speed, and Internet congestion. Server throughput (the rate at which the server can process client requests) should not be a problem given that high performance hardware is a readily available commodity that is relatively inexpensive when compared with site development costs. Like server hardware, network bandwidth is a readily available commodity, and with adequate network saturation monitoring, network capacity can easily be purchased prior to growth in traffic. In the same sense, the user connection to the Internet is a commodity; however, average connection speeds remain very low for the majority of the user community and will stay that way until prices for broadband connections become more affordable.
Despite the fact that bandwidth, servers and Internet connectivity are commodities, it only makes good business sense to apply these commodities efficiently and effectively after improving application code performance not as a prelude to or substitute for performance testing and tuning. Only after the application has been fully tuned to utilize these existing commodity resources in the most optimal fashion, does it make sense to invest more money in hardware and bandwidth.
Given the criteria of optimizing prior to expanding resource usage, where in the application can the greatest performance gains be achieved using proper performance testing and tuning techniques? The greatest impact is derived from improving the performance of the application code itself. Why? The initial costs of the development team, managers, and testers are significant enough to mandate the most efficient usage of this resource time. Creating optimal, efficient code is most effectively achieved through budgeting time and resources for performance testing and tuning during the traditional development life cycle rather than upgrading hardware and software after release to react to production problems. Building the code correctly the first time can save hard dollars in support costs and soft dollars in user acceptance given an unexpected spike in traffic. In the .NET world, the focus centers on the ability of an application to quickly process a client request and return the results while simultaneously processing millions of other requests. Here is the key area where attention to adequate application performance can make the most impact on end-user response times, regardless of the hardware platform and available bandwidth. With adequate performance testing techniques, application throughput can be accurately predicted allowing site administrators to prepare for the worst-case scenarios.
Verifying Performance Tuning Results
The final step to a performance/stress analysis is to clearly communicate your stress results to the application stakeholders. This should be done in a manner that effectively enables them to understand and improve performance based on the information presented in the analysis. Proof-of-concept testing (re-running the stress tests after analysis and tuning and comparing the performance results side by side) is the most effective way to communicate performance improvement results. It is not sufficient to speculate that your tuning efforts have improved performance; it has to be objectively and conclusively proven! Have response times decreased, and scalability increased? Have server resources required to serve the same level of client requests decreased significantly? These are the questions that a thorough performance analysis will address.
Conclusion
Now that you understand the groundwork for .NET and our analysis methodology, we will present an in depth approach to performance analysis planning in Chapter 2. We will detail how to analyze site traffic in order to characterize the site average and peak loads. This will offer the foundation for setting reasonable and attainable performance targets in the analysis phase.