The rush to architect a new e-business environment required IT departments to meld existing infrastructure with new Web-based systems. To achieve speed to market, IT environments were forged by necessity and quickly retooled to enable Internet-based functionality. Often, these hastily deployed enterprise environments can be complicated, cumbersome, and expensive to maintain. Added system demands, coupled with shifting business requirements, can cause instability and performance degradation that ripples from back-end systems to front-end user interfaces - resulting in a negative impact on the bottom line.

To extend the value of their IT investments, businesses are turning to IBM's WebSphere, a universal Internet software platform, to support flexible IT initiatives and provide the performance and scalability required for e-business. No longer confined to "test-tube" pilot projects, sophisticated application server technologies are becoming a staple in production environments. To make performance benchmarks a reality, key IT stakeholders are increasingly focused on learning how to manage and optimize the performance and availability of the WebSphere Application Server (WAS).

Managers are confronted with a number of products and solutions to enhance WAS performance and availability. Before IT executives can make a best-value assessment, it is critical to consider the merits and capabilities of the underlying management methodology. The foundation of any effective systems management initiative is monitoring functionality. A successful management approach must scrutinize and search for complexities and irregularities in transactions throughout the WebSphere environment, including containers, the Java Virtual Machine (JVM), and Enterprise JavaBeans (EJBs), as well as inside applications that have "borrowed" code from other products, protocols, messages, and the machines used to drive the whole process.

There are several approaches to monitoring operations inside the WAS and across the entire WebSphere environment. The three most frequently used are application instrumentation, the "spy" JavaBean, and the JVM Profiler Interface (JVMPI). While each option has its merits, it is important to choose a solution that can be used throughout the the entire life cycle - design, testing, deployment, and management. Harnessing a single solution for application performance will enable developers and IT managers to work in tandem to align WebSphere applications with business value.

Instrumentation Tools
To optimize application performance, it is important that an instrumentation tool support the requirements of diverse user groups, including developers, testers, and IT operators. Developers and testers require the ability to drill down to the application for coding and tuning requirements. Conversely, IT administrators concerned with overall performance and availability require an enterprise application view.

Ideally, a solution should be able to provide instrumentation without requiring application code changes or compilation. The tool should also provide the ability to enable and disable instrumentation dynamically to control overhead. To achieve a complete picture of overall application performance, organizations must be able to track performance and availability metrics of all key WebSphere environment components.

Once a performance problem is raised, the instrumentation user can drill down to the individual EJB, servlet, or JavaServer Page (JSP) to identify components that have unacceptable response times and high request rates. Each component's performance can be further broken down by type of delay, e.g., EJB, servlet, Java message service, Java naming directory interface, user-defined delay, etc.

CPU usage must also be monitored to gauge true system performance. High CPU usage could be indicative of insufficient hardware to handle the server's throughput or runaway processes. When measuring CPU consumption in conjunction with other WAS performance data, it is critical to ensure that there is a direct correlation between increased workload and CPU use, without suffering unreasonable response times. To accurately determine CPU capacity, maximize the processor speed - close to 100 percent of CPU - while driving increased throughput. Measuring the point of diminished returns for performance and throughput helps IT managers determine requirements for additional processor speed, additional servers for horizontal scaling, or additional application servers for vertical scaling.

With multiple views of application performance, organizations can monitor and assess system health across the enterprise, as well as throughout the life cycle - from development to testing to deployment to management. For example, predeployment use of instrumentation and JVMPI thread-level metrics can help developers and testers find application defects and potential bottleneck components. However, this level of detail would most likely be too resource-intensive and provide too much information for IT management. After deployment, IT management requires summary-level performance and availability information that will facilitate straightforward monitoring of the overall health of the application server and application response times.

To provide a real-world example, an application was performing poorly during a test of WebSphere on OS/390. A developer using instrumentation was able to pinpoint an SQL query at the root of the poor performance. The application was issuing an SQL query that was not properly indexed, which caused each query to scan an entire 2 million-line database table, resulting in response-time delays. To eliminate this problem, the developer first identified the SQL query as the largest delay for the application, then drilled down to view the specific SQL call, analyzed the SQL statements causing the delays, and quickly resolved the issue. Quick resolution of performance slowdowns not only saves time and money but can help increase customer satisfaction.

With instrumentation tools, historical metrics can be stored and used to identify trends and to plan for new initiatives such as migrating legacy records to a new database or identifying areas to increase system productivity. For example, if a new application has an existing sister application that operates in the same environment, organizations should collect data from the existing application to gauge the expected performance of the new application. This approach allows the organization to tune and configure the application to sidestep any performance challenges encountered with the existing application.

However, instrumentation is not a cure-all. The best practice is to use a solution that offers multiple monitoring methods to allow you to look at your WAS from the inside out and outside in. Additionally, relying on only one method of data collection may not be appropriate for the different departments responsible for testing and deploying the application.

The Spy JavaBean: Undercover and Inside the Application
A "spy" JavaBean is a bridging device employed by certain application server management tools. Under this approach, these products leverage an EJB that makes architected J2EE and operating system calls to derive operation metrics for itself and for other artifacts running inside the server. This approach has significant merit in that it sidesteps the requirement to instrument the application itself, as it uses defined interfaces to provide overall metrics. The spy bean approach can also use standard APIs to collect performance metrics. WAS provides performance APIs on certain platforms, including JMX, PMI, and EPM, that can be used to monitor the health of an application server. Additionally, events written to the application server log can be automatically monitored for exception alerts.

However, there are limitations to spy bean application monitoring because the container must be healthy for the spy bean to function properly. Should the container fall into disrepair, the spy bean would lose the ability to communicate with external alert systems, and the manager would be left flying blind. Another drawback is that the spy bean may be tightly coupled to an individual application and could be expensive to maintain. Similarly, performance APIs provide metrics limited to the WAS and do not provide visibility into the end-to-end application performance. Because APIs typically change with technology enhancements, the monitoring tool will likely require updates to support new WebSphere versions.

Using JVMPI
The JVMPI allows for a third approach. Providing for a two-way function-call interface between the JVM and an in-process profiler agent, the JVMPI leverages the virtual machine to notify the profiler agent of various events corresponding to heap allocation, thread start, etc. Alternately, the profiler agent issues controls and requests information through the JVMPI. For example, the profiler agent can turn on/off an event notification based on the needs of the profiler front end. The user can select which metrics to collect from the JVMPI.

The profiler agent also provides the ability to identify lock contention in the JVM by providing detailed metrics about JVM threads. Users are able to quickly identify the threads with the highest CPU percentage used and those with the highest percentage waiting to enter monitor and waiting for an object. High values indicate that work is being delayed because of contention for Java locks. Once an individual thread is identified, further detail is available to determine whether the delay was caused by synchronization, indicating that another thread had exclusive access to the object; or by a wait for notification, indicating that a thread was waiting for another thread.

However, the profiler is not recommended for use in a production environment because its performance overhead limits application response times and scalability. While JVM profiling provides detailed accounting of thread locks, garbage collection, heap sizes, and other thread-related information, it can be resource-intensive and doesn't provide the complete answer to the question of which objects are impacted.

Using WAS to Achieve Business Objectives
WAS is not an island. By definition, the application server is an integration platform. As such, developers and managers must consider application performance across the application server and its connected platforms when creating, deploying, and managing WebSphere environments. Organizations can only gain real insight into e-efficiency, business impact, and ROI by managing end-to-end application performance.

It is important to understand where a Web transaction bogs down. Is it the front end, the back end, or the application server in the middle? To know that, you have to monitor the application server and its connected programs, applications, and databases. Therefore, selecting a stand-alone management product that cannot interface with deployed distributed and legacy applications is a shortsighted approach. While these tools may work well in limited pilots, they are incapable of scaling to support real-world applications that span diverse operating-system and platform ecosystems.

Providing choices in data collection gives an organization ultimate flexibility in adapting an application server monitor to address organizational requirements. As WebSphere initiatives progress from testing to production phases, using a single performance monitoring approach is important to providing continuity as the application evolves through the product life cycle. As all business becomes e-business, selecting the most appropriate application server monitoring and management approach promises to drive not just the success of e-business initiatives but the viability of the organization itself, by ensuring that Web transactions meet customer demand.