We explore each type of testing in the sections that follow.

Functional Testing
Functional testing is typically the first step in testing a Web service server. (If the server does not work correctly, its performance, security, interoperability, etc., are essentially irrelevant.) The goal of this testing is fairly straightforward: to ensure that the server delivers appropriate responses for the given requests. However, due to the complexity of Web services, this task is far from simple. With most Web services, it is impossible to anticipate exactly what types of requests clients will send. Enumerating all possible requests is not feasible because the space of possible inputs is either unbounded or intractably large. As a result, it is important to verify whether the server can handle a wide range of request types and parameters.

There are two main steps to each functional test:
1.   A test client sends a request to the server over an HTTP connection. This involves determining what types and ranges of requests need to be tested to determine whether the server will react appropriately to the wide variety of requests it might receive. Once you have determined what requests to send, tools such as those available in WebSphere Studio Application Developer (WSAD) can facilitate the creation and execution of test clients.
2.   The response is analyzed for correctness (either by inspection or by running the response through a tool or script that verifies conformance to a specification). This analysis can be as simple as performing a text comparison with the expected response or as complex as extracting specific information from an XML document and performing application-specific checks.The simplest possible functional test involves sending a request and checking whether the server returns a response or an error message. For example, assume we have a sample employee Web service that allows queries by last name and returns the results in the form of an XML document. The most basic functional test would involve sending a valid input parameter (a last name entered into the system) and checking whether a response or an error message was returned.

Although these types of simple tests provide an adequate way to begin testing, they cannot verify the service's more complex functionality requirements. Fully testing even this simple service's functionality requires checking for all of the following notions of failure.
1.   The attempt to open a socket to the URL of the Web service fails. This indicates a network problem or an incorrect URL or IP address.
2.   The Web service returns a fault, such as

<Fault SOAP-ENV:Server: service
'urn:EmployeeQuery' unknown>

This indicates an error caused by the server or by the client, depending on the type of fault.
3.   The Web service responds and does not return a fault, but the responding message is not readable by the client because of an interoperability issue. For example, either the server or the client (or both) might not resolve XML namespaces in accordance with the standard.
4.   A response is received, but not in the expected format. For example, the response is in an incorrect XML format or some other arbitrary text format. This type of error can be detected by validating XML with respect to an XML Schema.
5.   A response is received in the format expected, but the data contained is incorrect (for example, when we request records for Fett but receive records for Kenobi).

The symptoms of the first three problems are independent of any particular Web service because they fail at the levels of the HTTP and SOAP protocols, which are consistent across Web services. The fourth and fifth problems can also arise in any Web service, but their details, and therefore their detection, are necessarily application specific. For example, a different Web service might accept a ticker symbol and return a stock quote. In this scenario, an expected response (SOAP envelope omitted for clarity) might be

16.87

or perhaps

<Quote symbol="AMZN" value="16.87"/>

This is an example of receiving a response in an unexpected format. The details of checking for this type of error depend on the specific service because different services can have different response types.

Each of the potential failure types exhibits different symptoms when encountered in the test client. The first three problems typically result in exceptions; ideally, the client will catch these exceptions, record them as test failures, and continue testing. The format can be verified by parsing the response with a validating XML parser, or in any other way that your testing infrastructure allows. Detecting incorrect results in the correct format is the most application-specific test. It generally requires using tools that allow you to make sophisticated assertions about the service's responses, or writing code that parses the XML and tests for constraints. In the case of the incorrect last name from the employee service, the test needs to verify that the lastName attribute of each Employee element matches the last name specified for that particular query. The best way to implement this verification depends on your test client and your verification capabilities. Although WSAD does not currently provide this level of verification, its functionality can be extended with third-party tools such as Parasoft SOAPtest. Another approach is to write an XSL file that outputs an error message when applied to nonconforming outputs.

While you are performing functional testing, remember that a Web service has multiple layers and that errors can be introduced at each layer. There are transport-level errors, such as an incorrect content length specified in an HTTP header, message-level errors, such as an invalid SOAP envelope, and application-level errors, such as a getStockPrice operation returning the price for the wrong ticker symbol. Keeping the layers in mind is helpful for generating adequate test coverage, as well as for debugging failures.

Also, be aware that the WSDL can be another source of errors. If the WSDL permits a wider class of inputs than the application, it is increasingly vulnerable to erroneous input at the application level. Ideally, a service's robustness would be tested by using the type definitions from the WSDL to generate all possible inputs and send each combination to the server. In practice, this is not feasible because the input space is usually much too large. A more pragmatic goal is to cover a representative portion of the input space.

After you confirm that the server handles expected requests correctly, perform fault checking to see how it handles unexpected input. The system will inevitably be faced with unexpected requests either as a result of mistakes (such as a bad WSDL) or from attempts to breach service security. (Hackers sometimes trick applications into behaving unexpectedly by sending invalid inputs.) Performing this fault checking involves sending the service requests with illegal and/or unexpected parameters, then verifying the response with assertions, custom code, or other tool-specific verification methods. The expected service behavior in these situations can depend on the stage of development as well as whether the Web service is intended for public versus internal use. For an internal service, it might make sense for the service to display its stack trace when a runtime error occurs, because the stack trace offers very valuable information for debugging. For a publicly exposed Web service, displaying the stack trace is arguably undesirable because it provides additional information about your implementation details (details that you would prefer hackers not know).

WSAD offers considerable flexibility for producing Web service clients. A standard client (shown in Figure 1) can be generated concurrently when Web services are generated and deployed.

If needed, this standard client can be customized graphically or programmatically in the built-in JSP editor (shown in Figure 2). The combination of these client-generation and customization options provides the opportunity to perform a broad range functionality testing.

Regression Testing
After you have verified the server's functionality, rerun the functional test suite on a regular basis to ensure that modifications do not cause unexpected changes or failures. A common technique is to send various requests, manually confirm the responses, and then save them as a regression control. These regression tests can be incorporated into a regular automated build process. When regression tests are run frequently, regressions are easy to fix because they can be directly attributed to the few changes made since the last time the test was run. WSAD does not currently provide an explicit regression testing feature, but this capability can be added by extending WSAD with additional tools.

Load Testing
The next step in the server testing process is load testing. The goal of load testing is to verify the performance and functionality of the service under heavy load.

The best way to start load testing is to have multiple test clients run the complete functional test, including request submissions and response verifications. When load testing ignores the functionality verification process and focuses solely on load-rate metrics, it risks overlooking critical flaws (such as functionality problems that surface only under certain loads).

To thoroughly test the service's performance, run the functional test suite under a variety of different scenarios to check how the server handles different types of loads. For example, the test could check functionality and response time under different degrees of load increases (sudden surges versus gradual ramp-ups) or different combinations of valid and invalid requests. If the load tests reveal unacceptable performance or functionality under load, the next step is to diagnose and repair the source of the bottleneck. Sometimes, the problem is caused by a fundamental algorithmic problem in the application, and the repair could require something as painful as an application redesign and rewrite. Other times, it is caused by some part of the infrastructure (the Web server, the SOAP library, the database, and so forth). In these cases, fixing the problem might be as simple as changing a configuration or as complex as changing the architecture. Because fixing performance problems sometimes demands significant application or system changes, it is best to start load testing as soon as possible. By starting early, you can diagnose and fix any fundamental problems before it is too late to do so without a major rewriting or rebuilding nightmare.

WSAD does not currently provide load testing functionality. It allows you to generate a large load by writing custom client code that uses a loop, but this is not the preferred approach. Load testers provide more control over how the load is generated because they allow you to control parameters such as test duration and load size.

Client Testing
SOAP client developers are responsible for ensuring that the client sends requests properly. If a client sends invalid or improperly formed requests, the server usually cannot deliver the expected results. The process of testing clients is a little different from testing services because clients are the initiators of Web service interactions. This means that from a testing standpoint, there are two main things to verify: whether the client can correctly initiate an interaction by sending a request, and whether the client behaves correctly when it receives a response. Note that the second part requires inspection of the client application; it cannot generally be determined by merely observing wire traffic.

The best way to test a particular client depends on the nature of the application. If the client accesses a server that can accept "test" requests with no harmful side effects, it can directly access the live server during testing. If the server is not yet available or should not be sent test inputs, the client can access an emulated server or server stubs during testing.

No matter what type of server a client accesses, the same general principle applies: the client sends a request, the server responds, then client success or failure is determined by recording and verifying the request and/or by verifying the server response. (The same techniques and tools used to verify server functionality can be used for this purpose.) Of course, server bugs could mislead you: if the server is not operating correctly, correct client requests might result in incorrect responses, and incorrect requests might result in apparently correct responses. You can ensure that server functionality problems are not confusing your results by (1) verifying the request as well as the response, and (2) testing the simplest possible server implementations (server stubs) instead of - or in addition to - testing actual, complex servers.

After you verify that the client sends acceptable requests and can receive responses, shift to testing exceptional cases. For example, test that the client behaves properly when the server goes offline by sending the response to an invalid URL. Or use server stubs to simulate the server sending the client invalid data.

Although WSAD does not currently offer a direct client testing feature, it is possible to test a particular client by writing a test service that performs the desired analysis on the client request.

Other Testing Considerations
Functional testing and load testing are the most fundamental types of testing for Web services. Depending on the type of service being tested and its requirements, it might be necessary to address additional issues during the testing process. Some issues that might further complicate many developers' testing are interoperability, security, and UDDI registry use.

Interoperability
A driving force behind Web services is the promise of seamless interoperability for disparate programming languages, operating systems, and various runtime environments. Unfortunately, the mere adoption of technologies that promote this idea (XML, SOAP, WSDL, UDDI) does not make the promise a reality.

Ideally, interoperability would be verified by checking that a service adheres to a comprehensive, universally implemented set of standards. However, the existing W3C recommendations are still evolving. Furthermore, the technologies are flexible enough to provide implementers with a myriad of options (document versus RPC-style, SOAP encoding versus literal encoding, different array representations, different versions of HTTP, SOAP, WSDL, UDDI, etc.). Flexibility is generally beneficial, but if everyone chooses a different way to do things, it does not serve the goal of interoperability. As options proliferate, it becomes increasingly unlikely that any given vendor solution will completely conform to all aspects or options allowed by the standard.

Given the reality that not all of the standards today are fully developed or consistently implemented, one of the most pragmatic approaches to interoperability is the one taken by the Web Services Interoperability Organization (WS-I). WS-I, though not itself a standards body, intends to serve as a standards integrator by developing a core collection of profiles that are a subset of the various Web service technologies. By restricting development to technologies specified in WS-I profiles, developers can increase the odds that their systems will interoperate with other systems. Development tool companies are already working with the WS-I to develop tools that automatically check compliance with these profiles and, in the event of noncompliance, pinpoint exactly what needs to be changed to ensure compliance. Expect to see tools that check compliance with these profiles soon after the profiles are officially released.

Security
Web services security is not a single problem, but rather a host of interrelated issues. For any given application, some of the issues will be critical, while others may be of lower priority or even irrelevant. Some facets of security worth considering when deploying Web services are:

Security impacts testing requirements in two important ways. First, any security requirements for a Web service naturally translate into testing requirements. If a service requires a certain level of privacy, or if it requires that messages be authenticated in a certain way, then specific tests are needed to ensure that these security requirements are met. The second way that security impacts testing is subtler. To some extent, security schemes will complicate the process of testing and debugging the basic functionality. For example, nonintrusive monitors can often aid in functional testing as well as load testing. Encrypted traffic presents an obvious complication to this approach to testing.

UDDI
Thus far, we have not addressed how publishing and discovering services fits into testing. UDDI is not yet as mature a technology as some of the others discussed in this article, but it is evolving and gaining acceptance. Services registered in a UDDI registry that are discovered and bound dynamically have all the testing requirements that we have already discussed, plus the find and bind features require additional testing. It is helpful to consider UDDI testing in two pieces: the registry implementation and the entries within the registry. Most users will not be implementing their own UDDI registry, so the primary focus will be on testing the content of the registry. The most reliable way to test the content is to write test clients that perform inquiries on the registry, and then use the registry data to actually invoke the service. Functional testing of services can then be extended to include dynamic binding to endpoints specified in a UDDI registry. This ties together the functional testing of the registration with the service implementation. The current version of WSAD provides full UDDI support, which allows both querying and registration.

Conclusion
By integrating the discussed testing practices into the Web service development process, you can ensure that a Web service server works well with the possible types and volumes of client requests, and that a Web service client correctly accesses and retrieves whatever data a service has to offer. You can start implementing the discussed practices at any point in the development process, but if you start testing early, you will maximize your ability to prevent errors as well as detect errors. Typically, the earlier you detect a problem, the easier it is to fix it, and the less chance you and your team members have to inadvertently worsen the problem by building code or components that interact with the problematic element, or by reusing the problematic element for other servers or clients. If you start your testing as early as possible, then continue using the related tests as a regression test suite throughout development, you will not only ensure the client's or server's continued reliability, but also streamline the development process.