Welcome!

IBM Cloud Authors: XebiaLabs Blog, Liz McMillan, Mehdi Daoudi, Pat Romanski, Yeshim Deniz

Related Topics: IBM Cloud

IBM Cloud: Article

Performance Testing Web Services

Strategies and best practices

The successful development of scalable Web services on WebSphere (or any other application server) requires thorough performance testing. Applying a well-designed, consistent performance testing methodology throughout the development life-cycle is key in satisfying a Web services application's performance requirements.

This article provides strategies for creating load test scenarios that reflect tendencies of the real world environment. To help you apply these strategies, it introduces best practices for organizing and executing load tests, and suggests how these best practices fit into a Web services application's development life-cycle. It also demonstrates how to utilize performance monitors offered by WebSphere and the operating system, as well as how to analyze load testing results. The bulk of the discussion assumes that WSDL is used to describe the Web service's interface, and that SOAP and HTTP are used for the messaging and transport layers, respectively.

Creating a Test Plan

Incorporating Performance Testing into the Web Service Development Life-Cycle
Although performance and scalability are critical parts of any non-trivial Web service, performance testing is often left for the later stages of the development process. Introducing performance testing at the early stages of development, or even prototyping, brings the following benefits:

  • Performance-related bugs (such as concurrency problems, memory leaks, and unreleased resources) are caught immediately after they are introduced, which helps isolate the culprit to the recently checked-in code or recent infrastructure changes.
  • New features' performance costs can be evaluated and quantified.
  • Project management has constant awareness of the application performance and has more confidence in meeting the release dates. This, in turn, greatly reduces the risk of missed deadlines and overspending on expensive enterprise application profiling tools and consultants.

Creating Load Test Scenarios
Performance tests should be started after the functional tests execute successfully. The best way to perform load testing is to have the load test application's virtual users run the complete functional test suite. Load tests should identify and report all transport, message, and application errors.

The usefulness of the performance test results is directly related to how closely the tests emulate request streams that the Web services application will be handling once it is deployed in the production environment. To design load test scenarios with real world value mixes, request types, sequences, and intensities, it is helpful to consider the load test scenario categories shown in Figure 1.

Type of Use (Level 3 head)
Depending on the type of deployment, your Web services can be exposed to various types of SOAP clients. These clients may produce unexpected, erroneous, and even malicious types of requests. Your load test scenarios should include profiles that simulate this type of user. The more your Web service is exposed to the outside ("uncontrolled") world, the greater the probability of a non-regular usage. The "misuse" and "malicious use" categories may include invalid SOAP requests as well as valid requests with unusual or unexpected values or request sizes.

For example, if your service uses arrays or complex types, examine your WSDL and create load test scenarios that emulate requests with expected average and maximum possible element counts, as well as element counts that exceed the allowed maximum.

 


<xsd:complexType name="IntArray">
  <xsd:sequence>
    <xsd:element name="arg" type="xsd:int" maxOccurs="100"/>
  </xsd:sequence>
</xsd:complexType>

Measure service performance with various sizes of client requests and server responses. If the expected request sizes and their probabilities are known (for example, based on log analysis), then create the request mix accordingly. If such data is unavailable, test with the best-, average-, and worst-case scenarios to cover the full performance spectrum.

Emulation Mode (Level 3 head)
A Web service may or may not support the notion of a user. More generically, it may be stateful or stateless. Your decision to use either "virtual user" or "request per second" request emulation mode should be based on this criteria. For example, the load of a stateless search engine exposed as a Web service is best expressed in terms of a number of requests per second, because the notion of a virtual user is not well-defined in this case. A counter example of a stateful Web service is one that supports customer login, such as a ticket reservation service. In this context, it makes more sense to use virtual user emulation mode.

If your Web service is stateless and you have chosen the request per second approach, make sure that you select a load test tool that supports this mode. If a load test tool can sustain only the scheduled number of users, the effective request injection rate may vary substantially based on the server response times. Such a tool will not be able to accurately emulate the desired request sequence. If the number of users is constant, the request injection rate will be inversely proportionate to the server processing time and will be likely to fluctuate, sometimes dramatically, during the test. Web services load test tools such as Parasoft SOAPtest offer both "virtual user" and "request per second" modes.

When load testing stateful Web services, such as services that support the notion of a user, make sure that you are applying appropriate intensity and concurrency loads. Load intensity can be expressed in request arrival rate; it affects system resources required to transfer and process client requests, such as CPU and network resources. Load concurrency, on the other hand, affects system resources required to keep the data associated with logged-in users or other stateful entities - session objects in memory, open connections, or used disk space. A concurrent load of appropriate intensity could expose synchronization errors in your Web services application. You can control the ratio between load intensity and concurrency by changing the virtual user think time in your load test tool.

Content Type (Level 3 head)
When load testing Web services, it is easy to overlook the fact that SOAP clients may periodically refresh the WSDL, which describes the service, to get updates on the service parameters it is about to invoke. The probability of such updates may vary depending on the circumstances. The test team can analyze access logs or make reasonable predictions based on the nature of the service. If the WSDL access factor (the probability of WSDL access per service invocation) is high and WSDL size is compatible with the combined average size of request and response, then network utilization will be noticeably higher in this scenario, as compared to the one without WSDL refresh. If your Web service's WDSLs are generated dynamically, the high WSDL access factor will affect server utilization as well. On the other hand, if your WSDLs are static, you can offload your application server by moving the WSDL files to a separate Web server that is optimized for serving static pages. Such a move creates increased capacity for processing Web service requests.

Type of Load (Level 3 head)
To ensure that your Web services application can handle the challenges it will face once it is deployed in production, you test its performance with various load intensities and durations. Performance requirement specifications should include metrics for both expected average and peak loads. After you have run average and peak load scenarios, conduct a stress test. A stress test should reveal the Web services application's behavior under extreme circumstances, which would cause your application to start running out of resources, such as database connections or disk space. Your application should not experience an unrecoverable crash under this stress.

Performance degradation - even dramatic degradation - is acceptable in this context, but the application should return to normal after the load has been reduced to the average. If the application does not crash under stress, verify that the resources utilized during the stress have been released. A comprehensive performance testing plan will also include an endurance test that verifies the application's ability to run for an extended period of time. Endurance tests are run for hours or days, and could reveal slow resource leaks that are not noticeable during regular tests. Slow memory leaks are among the most common - if they are present in a Java environment, these leaks could lead to a java.lang.OutOfMemoryError and the crash of the application server instance.

Creating a Real World Value Mix
To better verify the robustness of your Web service, you should use your load test tool to generate a wide variety of values inside SOAP requests. This mix can be achieved, for example, by using multiple value data sources (such as spreadsheets or databases), or by having the values of the desired range dynamically generated (scripted) and then passed to virtual users that simulate SOAP clients. By using this approach in load tests of sufficient duration and intensity, you can test your Web service with an extended range and mix of argument values that will augment your functional testing. Depending on the circumstances, it may be advisable to run the mixed request load test after all known concurrency issues have been resolved. If errors start occurring after the variable request mix has been introduced, inspect error details and create functional tests using the values that caused your Web service to fail during load testing. These newly created functional tests should become part of your functional test suite.

Web Services Design Considerations
The load test scenario creation practices previously mentioned help reveal performance problems that have already been introduced into a Web services application. However, it's equally important to prevent potential performance problems at the design and development stages. By focusing on prevention as well as detection, you reduce the number and severity of performance problems that you would have to identify and repair later, which in turn streamlines the development process and conserves development/testing resources. This section offers some tips on designing Web services in a way that prevents performance problems.

Minimize the Number of Service Requests and Accumulated Traffic Size
Many current Web services development software packages, including WebSphere Studio Application Developer, offer tools that greatly automate the exposure of Java code via Web services. Although these tools are effective development instruments, indiscriminate use of such tools can cause performance problems. For instance, assume that we want to use the following Java interface for customer registration:

 


public interface Customer {
	void setName(String first,String last);
	void setAddress(Address address);
	void setTelephone(String phone);
	void setAccountNo(String account);
}

If the Customer interface is exposed as a Web service, it will take four SOAP calls to complete the customer registration operation. We can reduce the number of remote calls and accumulated request processing time and traffic size by introducing a single register operation as follows:

void register
(String firstName,String lastName,Address address,String phone,String account);

An even better approach would be to introduce a CustomerRegistrationData type that will hold all of the parameters necessary for registration:

void register(CustomerRegistrationData data);

When designing a Web services interface, it is advisable to define a set of high-level actions that the Web service user would want to perform. Then, create a set of SOAP calls that perform the desired high-level actions with the minimal request quantity and traffic volume between the client and the server.

Minimize Potential Performance Problems
As previously discussed, your service can be exposed to erroneous or malicious types of requests. It's often possible to reduce the performance degradation caused by such requests. For example, assume that your Web service's WSDL was auto-generated and has complex types that allow an unlimited number of elements as a result of a default conversion of a Java array, or a vector.

 


<xsd:complexType name="stringArray">
  <xsd:sequence>
    <xsd:element name="id" type="xsd:string" maxOccurs="unbounded"/>
  </xsd:sequence>
</xsd:complexType>

Your application business logic, however, may have a well-defined limit of the number of elements in the complex type:

<xsd:element name="id" type="xsd:string" maxOccurs="20"/>

Setting a limit will cause erroneous or malicious messages to be rejected during the early validation stages of request processing, thereby minimizing the performance penalty for servicing such requests.

Other Considerations

In certain circumstances, you can gain performance benefits using another optimization technique: turning off XML schema validation and performing custom validation and parameter deserialization. If you choose such custom SOAP message processing, SAX-style parsers are a preferred choice for most situations, but especially for processing large messages. SAX parsers are faster and more memory-efficient than their DOM equivalents because they do not create a tree object model of the XML file in memory, but rather represent the XML document as a series of events.

Following a layered application design approach offers several advantages. One such advantage is that it helps you isolate the components responsible for the performance drop by turning on or off various application layers (such as security and access control) or persistence layers by stubbing out database calls.

What Are We Measuring?

Making Sense of Response Time Measurements
Response time is one of the major parameters that characterize Web service quality. Although measuring a time interval may seem straightforward, it may be trivial to decide where to place this measurement, how to interpret it, and how to estimate its accuracy. Most load testing tools measure the elapsed time between sending a request and receiving a response from the server. Although measuring response time on the client side better reflects simulated user experience, it might depend heavily on the network environment, where the particular load testing experiment is performed and can yield different results with different network configurations. For this reason, it is beneficial to monitor the server processing time as well. WebSphere Application Server offers such monitors (see Figure 2). Comparing server processing times with the virtual user recorded response times gives a good estimate of the delays caused by the network.

Eliminating Sources of Incorrect Results on the Test Client
Load testing tools that emulate hundreds to thousands or more virtual users may cause heavy resource utilization - or even resource starvation - on test client machines. This can not only cause measurement errors, but also distort the stream of requests generated by the load test tool. For example, CPU saturation, paging, and long JVM garbage collection cycles (if the load test client is a Java application) may cause bursts of requests instead of the desired uniform - Poisson - or another request distribution that the load test tool is supposed to generate. To prevent anomalies and measurement distortions caused by resource overutilization on load-generating machines, you can monitor major performance parameters, such as CPU, memory, and network utilization. If your monitoring efforts reveal that resource utilization goes higher than 75%, add more hardware to avoid those anomalies.

Overutilization of load test client resources can also result in test failures, which can be erroneously interpreted as server faults. Figure 3 presents an example of a load test run that has errors caused by socket starvation on the client, which ran on a Windows XP machine. The load test ran without errors for approximately 140 seconds, then errors started occurring in bursts of up to 7 errors per second. Error details indicate network problems:

java.net.BindException: Address already in use: connect

Adding TCP monitors on the test client revealed that the upper port on the client machine grows for the first half of the test, reaches the value of 5000, then stays at that value for the remainder of the test. The graph also shows a correlation between the upper TCP port reaching the maximum value and the beginning of errors. Further investigation revealed that, for Windows 2000 and higher, the operating system configuration restricts attempts to set up TCP connections from port 5000 and above. The upper port value is controlled by a Registry Key setting. After the Key value was increased to the maximum value of 65534, the load test ran without errors.

Conclusion

By following performance testing best practices, you can achieve more comprehensive coverage of possible performance issues that could arise during the production deployment of your Web service and increase its overall quality.

More Stories By Sergei Baranov

Sergei Baranov is a software engineer for Parasoft. He's a load test technical lead for SOAPtest - Parasoft's tool for testing Web services. His interests include distributed and concurrent computing as well as software performance issues. He holds a degree of Electrical Engineering (MSEE) from the Moscow State Institute of Radio Engineering, Electronics, and Automation.

Comments (1) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
Brian Bobak 02/08/05 06:57:22 PM EST

This was very informative.

So where exactly is the place to change the number of tcp ports that are available?

@ThingsExpo Stories
Five years ago development was seen as a dead-end career, now it’s anything but – with an explosion in mobile and IoT initiatives increasing the demand for skilled engineers. But apart from having a ready supply of great coders, what constitutes true ‘DevOps Royalty’? It’ll be the ability to craft resilient architectures, supportability, security everywhere across the software lifecycle. In his keynote at @DevOpsSummit at 20th Cloud Expo, Jeffrey Scheaffer, GM and SVP, Continuous Delivery Busine...
NHK, Japan Broadcasting, will feature the upcoming @ThingsExpo Silicon Valley in a special 'Internet of Things' and smart technology documentary that will be filmed on the expo floor between November 3 to 5, 2015, in Santa Clara. NHK is the sole public TV network in Japan equivalent to the BBC in the UK and the largest in Asia with many award-winning science and technology programs. Japanese TV is producing a documentary about IoT and Smart technology and will be covering @ThingsExpo Silicon Val...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
SYS-CON Events announced today that CollabNet, a global leader in enterprise software development, release automation and DevOps solutions, will be a Bronze Sponsor of SYS-CON's 20th International Cloud Expo®, taking place from June 6-8, 2017, at the Javits Center in New York City, NY. CollabNet offers a broad range of solutions with the mission of helping modern organizations deliver quality software at speed. The company’s latest innovation, the DevOps Lifecycle Manager (DLM), supports Value S...
SYS-CON Events announced today that Hitachi, the leading provider the Internet of Things and Digital Transformation, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Hitachi Data Systems, a wholly owned subsidiary of Hitachi, Ltd., offers an integrated portfolio of services and solutions that enable digital transformation through enhanced data management, governance, mobility and analytics. We help globa...
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in compute, storage and networking technologies, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology, is a premier provider of advanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/...
Amazon has gradually rolled out parts of its IoT offerings in the last year, but these are just the tip of the iceberg. In addition to optimizing their back-end AWS offerings, Amazon is laying the ground work to be a major force in IoT – especially in the connected home and office. Amazon is extending its reach by building on its dominant Cloud IoT platform, its Dash Button strategy, recently announced Replenishment Services, the Echo/Alexa voice recognition control platform, the 6-7 strategic...
@GonzalezCarmen has been ranked the Number One Influencer and @ThingsExpo has been named the Number One Brand in the “M2M 2016: Top 100 Influencers and Brands” by Analytic. Onalytica analyzed tweets over the last 6 months mentioning the keywords M2M OR “Machine to Machine.” They then identified the top 100 most influential brands and individuals leading the discussion on Twitter.
SYS-CON Events announced today that Juniper Networks (NYSE: JNPR), an industry leader in automated, scalable and secure networks, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Juniper Networks challenges the status quo with products, solutions and services that transform the economics of networking. The company co-innovates with customers and partners to deliver automated, scalable and secure network...
The age of Digital Disruption is evolving into the next era – Digital Cohesion, an age in which applications securely self-assemble and deliver predictive services that continuously adapt to user behavior. Information from devices, sensors and applications around us will drive services seamlessly across mobile and fixed devices/infrastructure. This evolution is happening now in software defined services and secure networking. Four key drivers – Performance, Economics, Interoperability and Trust ...
Data is an unusual currency; it is not restricted by the same transactional limitations as money or people. In fact, the more that you leverage your data across multiple business use cases, the more valuable it becomes to the organization. And the same can be said about the organization’s analytics. In his session at 19th Cloud Expo, Bill Schmarzo, CTO for the Big Data Practice at Dell EMC, introduced a methodology for capturing, enriching and sharing data (and analytics) across the organization...
SYS-CON Events announced today that T-Mobile will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on ...
The explosion of new web/cloud/IoT-based applications and the data they generate are transforming our world right before our eyes. In this rush to adopt these new technologies, organizations are often ignoring fundamental questions concerning who owns the data and failing to ask for permission to conduct invasive surveillance of their customers. Organizations that are not transparent about how their systems gather data telemetry without offering shared data ownership risk product rejection, regu...
Bert Loomis was a visionary. This general session will highlight how Bert Loomis and people like him inspire us to build great things with small inventions. In their general session at 19th Cloud Expo, Harold Hannon, Architect at IBM Bluemix, and Michael O'Neill, Strategic Business Development at Nvidia, discussed the accelerating pace of AI development and how IBM Cloud and NVIDIA are partnering to bring AI capabilities to "every day," on-demand. They also reviewed two "free infrastructure" pr...
With major technology companies and startups seriously embracing IoT strategies, now is the perfect time to attend @ThingsExpo 2016 in New York. Learn what is going on, contribute to the discussions, and ensure that your enterprise is as "IoT-Ready" as it can be! Internet of @ThingsExpo, taking place June 6-8, 2017, at the Javits Center in New York City, New York, is co-located with 20th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry p...
New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists will examine how DevOps helps to meet th...
With billions of sensors deployed worldwide, the amount of machine-generated data will soon exceed what our networks can handle. But consumers and businesses will expect seamless experiences and real-time responsiveness. What does this mean for IoT devices and the infrastructure that supports them? More of the data will need to be handled at - or closer to - the devices themselves.
Grape Up is a software company, specialized in cloud native application development and professional services related to Cloud Foundry PaaS. With five expert teams that operate in various sectors of the market across the USA and Europe, we work with a variety of customers from emerging startups to Fortune 1000 companies.
Financial Technology has become a topic of intense interest throughout the cloud developer and enterprise IT communities. Accordingly, attendees at the upcoming 20th Cloud Expo at the Javits Center in New York, June 6-8, 2017, will find fresh new content in a new track called FinTech.
Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more business becomes digital the more stakeholders are interested in this data including how it relates to business. Some of these people have never used a monitoring tool before. They have a question on their mind like “How is my application doing” but no id...