An enterprise application's performance, scalability, and reliability should be among the foremost concerns in a real deployment environment. The underlying messaging middleware is critical to that environment.
Every JMS vendor claims to be the fastest in the marketplace. Some claim to be fast and scalable to thousands of clients. It is hard to take any one vendor's word for having the fastest and most scalable product. Different application architectures have different demands. The industry has no accepted messaging benchmark. Thus, the proof of any vendor's claims lies in your own testing and measurement.
Performance and scalability are terms commonly used together, but they are not interchangeable. Performance refers to the speed at which the JMS provider can process a message through the system from the producer to the consumer. Scalability refers to the number of concurrently connected clients that a JMS provider can support. When used together, the terms refer to the effective rate at which a JMS provider can concurrently process a large volume of messages on behalf of a large number of simultaneously connected producers and consumers. The distinction between performance and scalability, as well as the implications of what it means to combine them, is very important, as you will soon see. A simple test using one or two clients will differ drastically from a test using hundreds or thousands of clients. The following section is intended to be used as a guide to help with performance and scalability testing.
Before you embark on your performance and scalability testing effort, consider what you are trying to accomplish. Since any particular vendor may do well with one scenario and not so well in others, the makeup of your application is important to define. Here are some key things to consider:
The potential size of the user community for your application. While this may be hard to project, it is important to try to predict how it will grow over time.
The average load required by the application. Given a total size of the user community for your application, how many are going to be actively using it at any given time?
The peak load required by the application. Are there certain times of the day, or certain days in a month, when the number of concurrent users will surge?
The number of JMS client connections used by the application. In some cases, the number of JMS clients does not correspond to the number of application users. Middleware products, such as EJB servers, share JMS connections across application clients, requiring far fewer JMS client connections than other applications. On the other hand, some applications use multiple JMS connections per client application, requiring more JMS client connections than users. Knowing the ratio of users to JMS clients helps you determine the number of messages being processed per client.
The amount of data to be processed through the messaging system over a given period of time. This can be measured in messages per second, bytes per second, messages per month, etc.
The typical size of the messages being processed. Performance data will vary depending on the message size.
Any atypical message sizes being produced. If 90% of the messages being processed through the system are 100 bytes in size, and the other 10% are 10 megabytes, it would be important to know how well the system can handle either scenario.
The messaging domain to be used, and how it will be used. Does the entire application use one p2p queue? Are there many queues? Is it pub/sub with 1,000 topics? One-to-many, many-to-one, or many-to-many?
The message delivery modes to be used. Persistent? Nonpersistent? Durable subscribers? Transacted messages? A mixture? What is the mixture?
Any vendor can make any software product run faster, provided the company has the right amount of time, proper staffing, commitment, and enough hardware to analyze and test a real-world deployment environment.
The simplest scenario for a vendor to optimize is the fastest performance throughput possible with one or two clients connected. This is also the easiest scenario to test, but is not the kind of testing we recommend; for one thing, it's difficult to imagine a realistic application that only has one or two clients. More complex testing scenarios that better match your system's real-world environment are preferable.
It is important to know ahead of time if the vendor you have chosen will support the requirements of your application when it is deployed. Because JMS is a standard, you may switch JMS vendors at any time. However, you may soon find yourself building vendor-specific extensions and configurations into your application. It's always possible to change vendors, if you're willing to expend some effort. However, if you wait to find out whether or not your application scales, you may no longer be able to afford the time to switch to another vendor.
This is not intended to imply that using JMS is a risky proposition. These same issues apply to any distributed infrastructure, whether third-party or home-grown, whether it is based on a MOM or based on CORBA, COM, EJB, or RMI, and whether it is based on an established vendor or an emerging one. Everything should be sized and tested prior to deployment.
The most important thing to realize is this:
performanceWithOneClient != performanceWithManyClients;
Many issues come into play once a message server starts to scale up to a large number of clients. New bottlenecks appear under heavy load that would never have occurred otherwise. Examples include thread contention, overuse of object allocation, garbage collection, and overflow of shared internal buffers and queues.
A vendor may have chosen to optimize message throughput with hundreds or thousands of concurrent clients at the expense of optimizing for throughput with one client. Likewise, a vendor may have optimized for a small number of clients at the expense of scalability with larger client populations.
The best approach is to start with something small and build up the number of clients and the number of messages in increments. For example, run a test with 10 senders and 10 receivers, and 100,000 messages. Next try 100 senders and 100 receivers, and run a test with 1,000,000 messages. Try as many clients as you can, within the limitations of the hardware you have available, and watch for trends.
It is extremely important to measure both the send rates and the receive rates of the messages being pumped through the messaging system. If the send rate far exceeds the receive rate, what is happening to the messages? They are being buffered at the JMS provider level. That's OK, right? That is what a messaging product does—it queues things. In some cases that may be acceptable based on the predetermined throughput requirements of your application, and the predictable size and duration of the surges and spikes in the usage of the application. If these factors are not extremely predictable, it is important to measure the long-term effects of unbalanced send and receive rates.
In reality everything has a limit. If the send rate far exceeds the receive rate, the messages are filling up in-memory queues and eventually overflowing the memory limits of the system, or perhaps the in-memory queues are overflowing to disk storage, which also has a limit. The closer the system gets to its hardware limits, the more the JVM and the operating system thrash to try to compensate, further limiting the JMS provider's ability to deliver messages to its consumers.
The hardware required to perform testing varies from vendor to vendor. You should have the hardware necessary to do a full-scale test, or be prepared to purchase the hardware as soon as possible. If the JMS provider's deployment architecture uses one or more server processes (as in a hub and spoke model), then a powerful server (like a quad-processor) and lightweight clients are appropriate. If the vendor's architecture requires that the persistence and routing functionality be located on the client machine, then many workstations may be required.
If you have limited hardware for testing, do the best you can to run a multi-client load test within the limitations of your hardware. You typically won't see any reasonably indicative results until you have at least 20 JMS clients. You must therefore be able to find a machine or a set of machines that can handle at least that much.
Assuming your client population will be large, truly indicative results start showing up with over 100 JMS clients. Your goal should be to use as many clients as possible within the limits of the testing hardware and to see whether the message throughput gets better or gets worse. A good guideline is to stop adding clients when the average resource utilization on each test machine (both clients and servers) approaches 80 percent CPU or memory use. At 80 percent, you realistically measure the throughput capacity of the JMS provider for a given class of machine and eliminate the possibility of having exceeded the limits of your hardware.
If the CPU or memory utilization does not approach its maximum, and the message throughput does not continue to improve as you add clients, then the bottleneck is probably disk I/O or network throughput. Disk I/O is most likely to be the bottleneck if you are using persistent messaging.
Building a test bed suitable for simulating a proper deployment environment itself can be a moderately sized effort. Most JMS vendors provide a performance test environment freely downloadable from their web site. In most cases, they provide a test bed sufficient for testing with one client.[1]
Testing your application over a long period of time is very important. After all, it is expected to perform continuously once deployed. Verifying that the middleware behaves reliably is the first step toward ensuring long-term application reliability.
Once you have a multi-client test bed in place, try running it for an extended period of time to ensure that the performance throughput is consistent. Start out by running the test bed while you can be there to monitor the behavior. Any long-term trends are likely to be detected in the first few hours. Things to watch for are drops in performance throughput, increase in memory usage, CPU usage, and disk usage. When you feel comfortable with the results you see, you may progressively start running tests overnight, over the weekend, or over a week.
The term " memory leak" refers to a condition that can happen when new memory gets allocated and never freed over a period of time, usually through a repeated operation, such as repeatedly pumping messages through a messaging system. Eventually, the system runs out of available memory; as a result, it will perform badly and may eventually crash.
Although Java has built-in memory management through garbage collection, it is an oversimplification to think that garbage collection permanently solves the problem of memory links. Garbage collection works effectively only when the developer follows explicit rules for the scoping of Java objects. A Java object can be garbage collected only if it has gone out of scope and there are no other objects currently referencing it. Even the best code can contain memory leaks, if the developer has mistakenly overlooked an object reference in a collection class that never goes out of scope.
Therefore, you need to monitor for memory leaks during testing. Memory that leaks in small increments may be not be noticable at first, but eventually these leaks could seriously impact performance. To detect them quickly, it helps to use a memory leak detection tool like OptimizeIt! or Jprobe. Even if the JMS provider and other third-party Java products you are using contain obfuscated classes, tools like these still help you prove that your memory requirements are growing (possibly the result of a memory leak), which is a good start .