In short, if you don't control the network, you don't control its traffic, either. Figure 9.2 shows some of the daily network traffic fluctuation factors. Some test teams try to work around this problem by running their tests at night or early in the morning. Sometimes this works, but often they discover their company uses the network 24 hours a day. As mentioned earlier, the networks might be in use at night for large data transfers and back-ups.

If you must use a non-isolated network, try to set up a network protocol analyzer (more on how these work below) to monitor network traffic volumes. Before you do this, check with your corporate network team. Many companies restrict or forbid the use of network protocol analyzers on their networks. And, yes, they can find out if you install one anyway. The best solution remains building an isolated network.

Remember, you need repeatability to perform an effective performance and stress test. Some test teams, however, cannot get the same results twice from the same performance test, even if they don't make any adjustments to the system under test. In such an environment, you cannot effectively tune the system. If you make a change, and see better performance, you cannot know if you've found a legitimate improvement or if the environment actually factors in the solution. Testing on an open network introduces more environment variability than you can ever hope to control.

Network isolation is probably one of the most overlooked issues in performance testing, and yet it is one of the most important. If you can't know from one run to the next whether a change made improved things or not, your test dissolves into an exercise in dart-throwing as you struggle to understand your web site bottlenecks and optimal tuning.

Network Capacity
The network carries all the data for the test (and subsequently for the production environment). You need to do some network capacity planning prior to building the network, and certainly before beginning any tests. The network needs enough capacity to carry the data generated by the test. This begs the question: What data does the test generate, exactly? Here's a list of some common data packages the network handles during a performance test:

Usually the largest network impact comes from a few major sources:

However, as we've discussed before, every web site and web application differ. A good understanding of your particular web application, the kinds of pages it returns, and its interactions with other systems helps you put together a reasonable network traffic estimate.

E-Commerce Network Capacity Planning Example
e-Commerce web sites require lots of network bandwidth. As we discussed in Chapter 5, the pages returned by these sites usually contain lots of embedded pictures in the form of gifs and jpegs. Users browse the pages of the e-Commerce web site and perhaps search for groups of items ("Show me a selection of coffee pots," for example). Because these pages return items the customer probably hasn't seen before, the user's browser does not contain cached copies of the images. To build a network estimate, figure out the average and maximum page size returned to the users. Decide how frequently the maximum page size might go out to the users, and whether it merits special calculation as a worse case scenario. Again, this exercise results in estimates. Use the performance test to validate these estimates.

Let's make the following assumptions about our e-Commerce site:

• Average page size: 45KB
• Maximum page size: 70KB

45,000 bytes per page * 100 pages per second = 4.5MBps

Note that 4.5MB approaches the upper bound of sustained traffic we consider acceptable for a 100Mbps Ethernet web site. Under ideal conditions, a 100Mbps network could potentially handle 12.5MB of sustained traffic:

100Mbps / 8 bits per byte = 12.5MBps

However, Ethernet networks lose efficiency because of traffic collisions and retransmissions. Academics tell us to expect an Ethernet to support 66% of its potential capacity, or about 8.3MBps of sustained traffic. For planning purposes, we prefer to use a more conservative estimate of 5MBps of sustained traffic. This allows the network to absorb unexpected traffic spikes. Web sites operating through a switched network are the exception to this rule. A switched network acts as a point-to-point network, which makes it more efficient. When dealing with switched networks, we raise the planning estimate to 8MBps.

The 4.5MBps we estimated above tells us the peak outbound traffic on the web site. How much inbound traffic does the web site carry? Inbound traffic consists of HTTP requests, which we usually measure in terms of a few bytes. For example, if the average HTTP request requires 100 bytes, the network load generated at peak is 100 bytes/request * 100 requests/second = 10KB/second.

This is less than 1% of the outbound HTML content traffic. Keep in mind that TCP/IP generates lots of overhead packets to support "guaranteed delivery" (a hallmark of the TCP/IP protocol). So, as a rule of thumb, we estimate inbound traffic at about 20% of outbound traffic. In this case, we'll use a planning estimate of

4.5MBps * 20% = 900KBps

Now, let's consider the transfer of data between the application servers and the application database on the network. Let's assume each user request requires the transfer of 10KB of data from the database. At peak, this gives us the following traffic:

10,000 bytes per request * 100 requests per second = 1MBps

If the web site uses HTTP session persistence, we need to account for this traffic as well. If each request generates 1KB of HTTP session traffic, we estimate the following HTTP session burden:

1,000 bytes per request * 100 requests per second = 100KBps

Lots of installations grossly underestimate the size of the HTTP session data they maintain for each user. Check the HTTP session database, and check the size of the data stored for your users to properly size the average HTTP session. See Chapter 2 for more details on HTTP session management.

Other factors also influence the amount of HTTP session data on your network. HTTP session caching combined with affinity routing reduces the data read from a persistent HTTP session store. On the other hand, if your application server vendor supports a distributed HTTP session scheme, this sometimes generates more network traffic, depending on the implementation.

The network burden estimates so far add up to the following, as shown in Table 9.1.

At 6.5MBps of sustained traffic, this web site exceeds our planning limit of 5MBps for the 100Mbps Ethernet. The web site needs a more sophisticated network plan. We might consider a switched 100Mbps network rated at 8MBps, but this doesn't give the web site a lot of room for growth or for unexpected load peaks. A gigabit network might be a better fit for this web site.

Network Components
Networks consist of more than cable. Any number of switches, routers, load balancers, and other equipment make up the test environment network. The brands and types used largely depend on what's available for the test, as well as company standards. Frequently the test team reuses network equipment from other test projects or receives the equipment as a loan from a production group. Often, the equipment becomes a part of the test network without anyone really understanding how it works or how to configure it properly. The end result is a piece of equipment that may impact web site performance in ways difficult to detect without specialized monitoring equipment and skills.

Keep in mind the following questions when dealing with network components:

Network Protocol Analyzers and Network Monitoring
A network protocol analyzer monitors traffic flows across a network and allows you to find out exactly how much traffic passes over the network during a test. If permitted, we highly recommend you use a network protocol analyzer to validate the network load during testing. Don't be surprised if your test generates a very different network load than you originally estimated. Often your estimates fail to consider all the factors at play during execution. Also, you may find the page sizes you used for your estimates incorrect.

Network protocol analyzers vary greatly in sophistication and expense. The most expensive network protocol analyzers cost tens of thousands of dollars and work with high-speed networks. These high-end network protocol analyzers usually come with tools for analyzing the network traffic at various levels in the protocol stack. For test teams on a more limited budget, many free or inexpensive network protocol analyzers exist, and they usually run on a machine already connected to the network. They provide limited functionality and don't always work with high-speed networks. However, for providing a gauge of network activity on many classes of networks, these tools work just fine. Just be sure whatever tool you pick works with the network it will monitor. This requires checking the tool's tolerance for network type and network speed. Chapter 12 discusses these tools in more detail, and Appendix C contains a list of some vendors of these products.

Warning: Many companies actively monitor their networks for network protocol analyzer activity. Some companies consider a network protocol analyzer a breach in their security and do not allow them. We've visited companies where using a network protocol analyzer on a company network results in instant job termination. Please take this warning seriously; obtain permission before inserting a network protocol analyzer into a network.

The Servers
The network connects all the components of your web site. Let's next go up a level and consider the server components you need in the test environment.

Application Server Machines
In terms of configuring the web application servers, strive to make the test configuration as close to the deployment configuration as possible. If you're deploying to a farm of four-way Sun machines with 8GB of RAM, you should use the same equipment during your testing, if at all possible.

Even though the Java web application server might port to different platforms, web applications do not perform and scale the same on a one-way NT box as on a four-way Sun box. If you must choose between fewer servers for your test or smaller servers, pick fewer servers. When you use fewer servers, scale the other components of the test (databases, HTTP servers, and so on) proportionally as well.

If you deploy the web application on multiple machines in a cluster, then test on multiple machines in a cluster as well. Get a representative number: If you plan to deploy on more than four machines, test on more than two. If your target environment contains eight application server machines or more, then four application servers in your test environment should suffice, as long as you scale the rest of the test web site proportionally to match.

We often encounter test teams trying to test and tune all the software used in their web site (application servers, databases, HTTP servers, etc.) on a single machine. However, this technique doesn't work if the production web site actually uses multiple machines. For example, many production web sites separate their HTTP servers from their application servers. Often the production team places the HTTP servers in a DMZ with firewalls in front and behind, as shown in Figure 9.3. It is impossible to successfully performance test for this configuration with a single server machine. If the web site uses SSL, for example, the HTTP server needs so much CPU for encryption/decryption that it chokes the performance of the application server.

Database Servers
Most production web sites use mid-tier database servers to hold application data or data specific to the web site (such as an HTTP session database). Often the web applications access these databases on every user request. Despite their central role in the operation of the web site, the mid-tier databases sometimes receive little or no tuning before the site enters production deployment. Often the test team lacks the database administrator (DBA) skills required to tune the database properly. Thus the team ignores the database unless they stumble across a specific problem in production. The problems we see frequently with databases usually fall into one of two broad categories: Poor software configuration or poor hardware configuration.

Poor Database Software Configuration: Indexing
By far the biggest database tuning problem we encounter is poor table indexing. Usually the problem starts this way: The test team receives a database backup from the production database staff. They dutifully load the database definition and the data itself, but never bother to build indexes for the tables they've just loaded. In fact, they may not even know what an index is, much less how to build one. Chapter 6 discusses how to put together the performance test team, including DBA skills needed to tune environments using databases.

The index allows the database software to find elements in a table without scanning the table repeatedly. This saves tremendous amounts of resource, particularly CPU. Also worth noting, the DBA may build new indexes and remove old ones as usage patterns change over time. New indexes apply even when web applications use existing databases. Often these databases contain indexes tailored for existing applications, but they might require new ones to better support the web application. By monitoring database reports, the DBA determines the web application's usage patterns and makes appropriate adjustments. The DBA may also review the SQL used in the web application to find out where indexes might be most beneficial.

Poor Database Software Configuration:
Internal Resources
Web applications receive many simultaneous requests. In turn, they make a proportionally large number of simultaneous requests to the database servers supporting them. These database servers need enough resources to support large volumes of simultaneous requests. These resources include things such as buffer pools, cursors, and sockets to support high-bandwidth operation.

Regrettably, we often see high-concurrency web sites struggling to pull data from databases tuned for small, fat client applications. New applications and usage patterns require a fresh look at the tuning parameters for the database. Do not assume the database is tuned properly because the DBA made a few tweaks some years ago for the usage patterns of a fat client application.

Poor Database Software Configuration: Caching
Particularly for sites with enormous application databases, the database cache becomes very important for optimal site performance. The cache holds the results of the most common queries and makes an impact on sites with large catalogs of items but a few frequently accessed "best sellers." These items return from the cache quickly without an expensive retrieval from the hard disk.

Caching helps some web sites, but a few cannot take advantage of this feature. For example, if every query submitted by the web site is unique (if, say, every query contained the user's account number), the cache may not return a hit even though the query may return items retrieved many times before. The web site team might want to work with the DBA to build queries better able to use the caching mechanisms available.

Poor Database Hardware Configuration: I/O Management
Eventually, all databases interact with the hard disks to read or write data. Database tuning often focuses on the CPU required by the database server without focusing on how to manage the storage required by the same database. We sometimes see very large multiple-processor database machines spending most of their time waiting to access one tiny hard disk. Adding CPU does not solve disk I/O problems.

If the database server spends most of its time waiting to read or write from the disk, tune the I/O resources. If the database supports it, try adding a multiplatter disk array to the database server. By using multiple disks, the database spreads out the read and write operations for better simultaneous access. The database reads and writes spread out across multiple points rather than queuing up on a single disk. Figure 9.4 shows an example using multiple disk platters.

Likewise, the hard disks themselves often contain tunable features. Some disks allow the system administrator to specify buffering to the disk, which may also improve performance. Some databases write more efficiently to disk than even the native file system itself. For example, IBM's DB2 database product provides mechanisms for "raw" I/O management (DB2 bypasses the file system to write directly to the disk) and for multidisk data writing.

Finally, when spreading the data to multiple hard disks, don't forget about the database logs. Databases keep detailed logs for rollback and recovery purposes. Every action taken by the database must be logged, and this requires a write to the hard disk. "Striping" these logs across multiple hard disks, if supported by the database, often improves disk I/O wait times.

SIDEBAR
About The Book
Performance Analysis for Java Web Sites by Joines, Willenborg, and Hygh

ISBN 0201844540
$44.99 464 pp.
Publication date:
September 10, 2002
Publisher: Addison- Wesley Professional

Stacy Joines, Ruth Willenborg, and Ken Hygh