QA Tool Review: JMeter
January 02, 2010
DescriptionApache JMeter is open source software, a 100% pure Java desktop application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.
Apache JMeter may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. You can use it to make a graphical analysis of performance or to test your server/script/object behavior under heavy concurrent load.
- Free & Open Source
- Distributed testing
- A little buggy
- Reporting is lacking
- Free and Open Source
- Programmed in Java
Ease of Use
Ease of InstallationInstall Java, download the zip file and unzip it. Run the .bat file to run the program and you�re set.
Setting up a TestAdd a thread group to the test plan. Configure the thread group by determining ramp-up period and how many threads you want running at the end (see Figure 1). Set up a proxy to record your test (Right-click Workbench � Add � Non-test elements � HTTP Proxy server). After the proxy is set up you can tell your browser to route requests through the proxy (default localhost:8080). The proxy will record all requests and store them in the Thread Group as samplers. Add a listener to record your test results and you�re good to go. Overall it�s a very straightforward, well-documented process.
Figure 1 - Create thread group
Running a TestRun � Start or Ctrl+R. Bam!
Technical Knowledge NeededVery little technical knowledge is needed. It does require the ability to learn how to do something from the documentation since it is not very intuitive.
PlatformsJMeter is a Java program, and thus runs wherever Java can. It requires Java 1.4 or greater.
Recording a testJMeter provides a proxy server through which you can record tests (see Figure 2). It�s fairly simple to set up, and it does a good job recording every request sent through it. You can choose which kinds of files to include or exclude in the recording, which is great because you probably don�t need to record all .gif, .png, etc. files as separate requests. Set it up by right-clicking the Workbench and select Add � Non-test elements � HTTP Proxy Server. It can then be configured to group the samplers however you want, and add the instructions for which URL patterns to include or exclude. See Figure 2 for an example set up. Once it is set up, just click start and tell your web browser to direct traffic through the proxy. The samplers will be created and you are well on your way to having your dream test.
Because it uses a proxy, it cannot detect HTTPS traffic, so any secure requests will have to be created manually.
Figure 2 - Set up a Proxy Server
ScriptingJMeter supports BeanShell scripting. I don�t know anything about it, so I can�t really rate its ability, but supposedly it exists.
ReportingThere are several reports that can be viewed within JMeter that provide a moderate level of detail. The data listed below can be recorded in a CSV file after each test. The built-in reports you can view in the GUI during test runs are lacking. Some of them don�t work. Some are useful (see Figure 3). The list below includes all information available in the CSV file:
- Start time (ms)
- Load time (ms)
- Sampler name
- Response code
- Response message
- Thread name
- Content type (e.g. text, bin)
- Size in bytes
Figure 3 - Distribution Graph
Trend ReportingJMeter�s listeners do several things, including trend graphing (see Figure 4). In addition, the exported CSV file contains detailed information for each request, including the time that the request started and how long it took. From this, with some Excel magic, it would be possible to create a trend graph.
Figure 4 - Trend graph
IntegrationThe only integration JMeter supports is exporting test results to CSV.
Data FunctionsUsing a CSV Data Set config element, you can easily create user defined variables using the information from a CSV file (See Figure 5).
Figure 5. CSV Data Set Config
Load Generation / Distributed Load GenerationJMeter offers distributed testing. I have not tested it, but from reviews it appears that in order to use multiple load generation tools, the GUI must be used. However, it is possible to run tests on one computer without the GUI and just record the results to a file. This minimizes system resource use and allows you to generate a higher load from one machine.
DocumentationThe documentation for JMeter is extensive. The JMeter website has a ton of documentation, great tutorials, a great user manual, and clear Javadocs. There are also many other website with great tutorials and documentation for getting started with JMeter.
User InterfaceTypical Java Swing GUI. Ugly but functional.
Performance v. Stress TestingJMeter offers both constant-load performance testing and ramp-up stress testing. The tests are set up to run for a certain number of repetitions.
- Excel problem with JMeter CSV files: When saving test results to a csv, Excel likes to show big numbers (like the start time) in engineering notation (i.e., 1.24E10). If you open the file and save it, it will save it in engineering notation. When you open the file again, several significant digits will be lost, making the start time data essentially useless. This can be overcome by changing the stuff from a CSV to XLSX or by formatting the number as a regular number.
- You can�t save both the Test Plan and WorkBench at the same time. You have to save each separately. I also couldn�t get them to open together. I could open either the workbench or the test plan file, but if I tried to do both, the second one I opened would overwrite the first.
HistoryStefano Mazzocchi of the Apache Software Foundation was the original developer of JMeter. He wrote it primarily to test the performance of Apache JServ (a project that has since been replaced by the Apache Tomcat project). JMeter has been redesigned to enhance the GUI and to add functional-testing capabilities.
Frequency of ReleasesJMeter has a new release approximately 1-2 times per year. Most of these releases are minor bug-fix versions. The most recent version (2.3.4) was released in June, 2009.
Ohloh.net says that JMeter is experiencing decreasing year-over-year development activity.
I'm Ryan Schuetzler, a husband and father, professor at the University of Nebraska at Omaha, and professional nerd. You can follow me on twitter, but there's not much there.