Welcome!

IBM Cloud Authors: Elizabeth White, Don MacVittie, Pat Romanski, Liz McMillan, Yeshim Deniz

Related Topics: IBM Cloud

IBM Cloud: Article

Performance Testing Web Services

Strategies and best practices

The successful development of scalable Web services on WebSphere (or any other application server) requires thorough performance testing. Applying a well-designed, consistent performance testing methodology throughout the development life-cycle is key in satisfying a Web services application's performance requirements.

This article provides strategies for creating load test scenarios that reflect tendencies of the real world environment. To help you apply these strategies, it introduces best practices for organizing and executing load tests, and suggests how these best practices fit into a Web services application's development life-cycle. It also demonstrates how to utilize performance monitors offered by WebSphere and the operating system, as well as how to analyze load testing results. The bulk of the discussion assumes that WSDL is used to describe the Web service's interface, and that SOAP and HTTP are used for the messaging and transport layers, respectively.

Creating a Test Plan

Incorporating Performance Testing into the Web Service Development Life-Cycle
Although performance and scalability are critical parts of any non-trivial Web service, performance testing is often left for the later stages of the development process. Introducing performance testing at the early stages of development, or even prototyping, brings the following benefits:

  • Performance-related bugs (such as concurrency problems, memory leaks, and unreleased resources) are caught immediately after they are introduced, which helps isolate the culprit to the recently checked-in code or recent infrastructure changes.
  • New features' performance costs can be evaluated and quantified.
  • Project management has constant awareness of the application performance and has more confidence in meeting the release dates. This, in turn, greatly reduces the risk of missed deadlines and overspending on expensive enterprise application profiling tools and consultants.

Creating Load Test Scenarios
Performance tests should be started after the functional tests execute successfully. The best way to perform load testing is to have the load test application's virtual users run the complete functional test suite. Load tests should identify and report all transport, message, and application errors.

The usefulness of the performance test results is directly related to how closely the tests emulate request streams that the Web services application will be handling once it is deployed in the production environment. To design load test scenarios with real world value mixes, request types, sequences, and intensities, it is helpful to consider the load test scenario categories shown in Figure 1.

Type of Use (Level 3 head)
Depending on the type of deployment, your Web services can be exposed to various types of SOAP clients. These clients may produce unexpected, erroneous, and even malicious types of requests. Your load test scenarios should include profiles that simulate this type of user. The more your Web service is exposed to the outside ("uncontrolled") world, the greater the probability of a non-regular usage. The "misuse" and "malicious use" categories may include invalid SOAP requests as well as valid requests with unusual or unexpected values or request sizes.

For example, if your service uses arrays or complex types, examine your WSDL and create load test scenarios that emulate requests with expected average and maximum possible element counts, as well as element counts that exceed the allowed maximum.

 


<xsd:complexType name="IntArray">
  <xsd:sequence>
    <xsd:element name="arg" type="xsd:int" maxOccurs="100"/>
  </xsd:sequence>
</xsd:complexType>

Measure service performance with various sizes of client requests and server responses. If the expected request sizes and their probabilities are known (for example, based on log analysis), then create the request mix accordingly. If such data is unavailable, test with the best-, average-, and worst-case scenarios to cover the full performance spectrum.

Emulation Mode (Level 3 head)
A Web service may or may not support the notion of a user. More generically, it may be stateful or stateless. Your decision to use either "virtual user" or "request per second" request emulation mode should be based on this criteria. For example, the load of a stateless search engine exposed as a Web service is best expressed in terms of a number of requests per second, because the notion of a virtual user is not well-defined in this case. A counter example of a stateful Web service is one that supports customer login, such as a ticket reservation service. In this context, it makes more sense to use virtual user emulation mode.

If your Web service is stateless and you have chosen the request per second approach, make sure that you select a load test tool that supports this mode. If a load test tool can sustain only the scheduled number of users, the effective request injection rate may vary substantially based on the server response times. Such a tool will not be able to accurately emulate the desired request sequence. If the number of users is constant, the request injection rate will be inversely proportionate to the server processing time and will be likely to fluctuate, sometimes dramatically, during the test. Web services load test tools such as Parasoft SOAPtest offer both "virtual user" and "request per second" modes.

When load testing stateful Web services, such as services that support the notion of a user, make sure that you are applying appropriate intensity and concurrency loads. Load intensity can be expressed in request arrival rate; it affects system resources required to transfer and process client requests, such as CPU and network resources. Load concurrency, on the other hand, affects system resources required to keep the data associated with logged-in users or other stateful entities - session objects in memory, open connections, or used disk space. A concurrent load of appropriate intensity could expose synchronization errors in your Web services application. You can control the ratio between load intensity and concurrency by changing the virtual user think time in your load test tool.

Content Type (Level 3 head)
When load testing Web services, it is easy to overlook the fact that SOAP clients may periodically refresh the WSDL, which describes the service, to get updates on the service parameters it is about to invoke. The probability of such updates may vary depending on the circumstances. The test team can analyze access logs or make reasonable predictions based on the nature of the service. If the WSDL access factor (the probability of WSDL access per service invocation) is high and WSDL size is compatible with the combined average size of request and response, then network utilization will be noticeably higher in this scenario, as compared to the one without WSDL refresh. If your Web service's WDSLs are generated dynamically, the high WSDL access factor will affect server utilization as well. On the other hand, if your WSDLs are static, you can offload your application server by moving the WSDL files to a separate Web server that is optimized for serving static pages. Such a move creates increased capacity for processing Web service requests.

Type of Load (Level 3 head)
To ensure that your Web services application can handle the challenges it will face once it is deployed in production, you test its performance with various load intensities and durations. Performance requirement specifications should include metrics for both expected average and peak loads. After you have run average and peak load scenarios, conduct a stress test. A stress test should reveal the Web services application's behavior under extreme circumstances, which would cause your application to start running out of resources, such as database connections or disk space. Your application should not experience an unrecoverable crash under this stress.

Performance degradation - even dramatic degradation - is acceptable in this context, but the application should return to normal after the load has been reduced to the average. If the application does not crash under stress, verify that the resources utilized during the stress have been released. A comprehensive performance testing plan will also include an endurance test that verifies the application's ability to run for an extended period of time. Endurance tests are run for hours or days, and could reveal slow resource leaks that are not noticeable during regular tests. Slow memory leaks are among the most common - if they are present in a Java environment, these leaks could lead to a java.lang.OutOfMemoryError and the crash of the application server instance.

Creating a Real World Value Mix
To better verify the robustness of your Web service, you should use your load test tool to generate a wide variety of values inside SOAP requests. This mix can be achieved, for example, by using multiple value data sources (such as spreadsheets or databases), or by having the values of the desired range dynamically generated (scripted) and then passed to virtual users that simulate SOAP clients. By using this approach in load tests of sufficient duration and intensity, you can test your Web service with an extended range and mix of argument values that will augment your functional testing. Depending on the circumstances, it may be advisable to run the mixed request load test after all known concurrency issues have been resolved. If errors start occurring after the variable request mix has been introduced, inspect error details and create functional tests using the values that caused your Web service to fail during load testing. These newly created functional tests should become part of your functional test suite.

Web Services Design Considerations
The load test scenario creation practices previously mentioned help reveal performance problems that have already been introduced into a Web services application. However, it's equally important to prevent potential performance problems at the design and development stages. By focusing on prevention as well as detection, you reduce the number and severity of performance problems that you would have to identify and repair later, which in turn streamlines the development process and conserves development/testing resources. This section offers some tips on designing Web services in a way that prevents performance problems.

Minimize the Number of Service Requests and Accumulated Traffic Size
Many current Web services development software packages, including WebSphere Studio Application Developer, offer tools that greatly automate the exposure of Java code via Web services. Although these tools are effective development instruments, indiscriminate use of such tools can cause performance problems. For instance, assume that we want to use the following Java interface for customer registration:

 


public interface Customer {
	void setName(String first,String last);
	void setAddress(Address address);
	void setTelephone(String phone);
	void setAccountNo(String account);
}

If the Customer interface is exposed as a Web service, it will take four SOAP calls to complete the customer registration operation. We can reduce the number of remote calls and accumulated request processing time and traffic size by introducing a single register operation as follows:

void register
(String firstName,String lastName,Address address,String phone,String account);

An even better approach would be to introduce a CustomerRegistrationData type that will hold all of the parameters necessary for registration:

void register(CustomerRegistrationData data);

When designing a Web services interface, it is advisable to define a set of high-level actions that the Web service user would want to perform. Then, create a set of SOAP calls that perform the desired high-level actions with the minimal request quantity and traffic volume between the client and the server.

Minimize Potential Performance Problems
As previously discussed, your service can be exposed to erroneous or malicious types of requests. It's often possible to reduce the performance degradation caused by such requests. For example, assume that your Web service's WSDL was auto-generated and has complex types that allow an unlimited number of elements as a result of a default conversion of a Java array, or a vector.

 


<xsd:complexType name="stringArray">
  <xsd:sequence>
    <xsd:element name="id" type="xsd:string" maxOccurs="unbounded"/>
  </xsd:sequence>
</xsd:complexType>

Your application business logic, however, may have a well-defined limit of the number of elements in the complex type:

<xsd:element name="id" type="xsd:string" maxOccurs="20"/>

Setting a limit will cause erroneous or malicious messages to be rejected during the early validation stages of request processing, thereby minimizing the performance penalty for servicing such requests.

Other Considerations

In certain circumstances, you can gain performance benefits using another optimization technique: turning off XML schema validation and performing custom validation and parameter deserialization. If you choose such custom SOAP message processing, SAX-style parsers are a preferred choice for most situations, but especially for processing large messages. SAX parsers are faster and more memory-efficient than their DOM equivalents because they do not create a tree object model of the XML file in memory, but rather represent the XML document as a series of events.

Following a layered application design approach offers several advantages. One such advantage is that it helps you isolate the components responsible for the performance drop by turning on or off various application layers (such as security and access control) or persistence layers by stubbing out database calls.

What Are We Measuring?

Making Sense of Response Time Measurements
Response time is one of the major parameters that characterize Web service quality. Although measuring a time interval may seem straightforward, it may be trivial to decide where to place this measurement, how to interpret it, and how to estimate its accuracy. Most load testing tools measure the elapsed time between sending a request and receiving a response from the server. Although measuring response time on the client side better reflects simulated user experience, it might depend heavily on the network environment, where the particular load testing experiment is performed and can yield different results with different network configurations. For this reason, it is beneficial to monitor the server processing time as well. WebSphere Application Server offers such monitors (see Figure 2). Comparing server processing times with the virtual user recorded response times gives a good estimate of the delays caused by the network.

Eliminating Sources of Incorrect Results on the Test Client
Load testing tools that emulate hundreds to thousands or more virtual users may cause heavy resource utilization - or even resource starvation - on test client machines. This can not only cause measurement errors, but also distort the stream of requests generated by the load test tool. For example, CPU saturation, paging, and long JVM garbage collection cycles (if the load test client is a Java application) may cause bursts of requests instead of the desired uniform - Poisson - or another request distribution that the load test tool is supposed to generate. To prevent anomalies and measurement distortions caused by resource overutilization on load-generating machines, you can monitor major performance parameters, such as CPU, memory, and network utilization. If your monitoring efforts reveal that resource utilization goes higher than 75%, add more hardware to avoid those anomalies.

Overutilization of load test client resources can also result in test failures, which can be erroneously interpreted as server faults. Figure 3 presents an example of a load test run that has errors caused by socket starvation on the client, which ran on a Windows XP machine. The load test ran without errors for approximately 140 seconds, then errors started occurring in bursts of up to 7 errors per second. Error details indicate network problems:

java.net.BindException: Address already in use: connect

Adding TCP monitors on the test client revealed that the upper port on the client machine grows for the first half of the test, reaches the value of 5000, then stays at that value for the remainder of the test. The graph also shows a correlation between the upper TCP port reaching the maximum value and the beginning of errors. Further investigation revealed that, for Windows 2000 and higher, the operating system configuration restricts attempts to set up TCP connections from port 5000 and above. The upper port value is controlled by a Registry Key setting. After the Key value was increased to the maximum value of 65534, the load test ran without errors.

Conclusion

By following performance testing best practices, you can achieve more comprehensive coverage of possible performance issues that could arise during the production deployment of your Web service and increase its overall quality.

More Stories By Sergei Baranov

Sergei Baranov is a software engineer for Parasoft. He's a load test technical lead for SOAPtest - Parasoft's tool for testing Web services. His interests include distributed and concurrent computing as well as software performance issues. He holds a degree of Electrical Engineering (MSEE) from the Moscow State Institute of Radio Engineering, Electronics, and Automation.

Comments (1) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
Brian Bobak 02/08/05 06:57:22 PM EST

This was very informative.

So where exactly is the place to change the number of tcp ports that are available?

@ThingsExpo Stories
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
Large industrial manufacturing organizations are adopting the agile principles of cloud software companies. The industrial manufacturing development process has not scaled over time. Now that design CAD teams are geographically distributed, centralizing their work is key. With large multi-gigabyte projects, outdated tools have stifled industrial team agility, time-to-market milestones, and impacted P&L stakeholders.
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
Gemini is Yahoo’s native and search advertising platform. To ensure the quality of a complex distributed system that spans multiple products and components and across various desktop websites and mobile app and web experiences – both Yahoo owned and operated and third-party syndication (supply), with complex interaction with more than a billion users and numerous advertisers globally (demand) – it becomes imperative to automate a set of end-to-end tests 24x7 to detect bugs and regression. In th...
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
An increasing number of companies are creating products that combine data with analytical capabilities. Running interactive queries on Big Data requires complex architectures to store and query data effectively, typically involving data streams, an choosing efficient file format/database and multiple independent systems that are tied together through custom-engineered pipelines. In his session at @BigDataExpo at @ThingsExpo, Tomer Levi, a senior software engineer at Intel’s Advanced Analytics gr...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things’). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing? IoT is not about the devices, it’s about the data consumed and generated. The devices are tools, mechanisms, conduits. In his session at Internet of Things at Cloud Expo | DXWor...