Welcome!

IBM Cloud Authors: Elizabeth White, Dalibor Siroky, William Schmarzo, John Worthington, Liz McMillan

Related Topics: Weblogic

Weblogic: Article

Analyzing Java Application Problems

Using Java thread dumps to assess the problem

This article discusses troubleshooting techniques for Java applications by analyzing Java thread dumps. We can use thread dumps to analyze situations like application hang, poor application response times, and application crash. Before getting into the details of analyzing the thread dumps, let's look briefly at the thread dump itself.

A Java thread dump is a snapshot of all the Java threads in a running Java application. It shows information like the current stack trace, state, and name of the threads. The list of threads includes the threads created by the JVM itself (to do housekeeping work like garbage collection, signal handling, etc.) and the threads created by the application.

You can get a thread dump by sending a SIGQUIT signal to the JVM. In case of Unix operating systems (Solaris/Linux/HP-Unix etc), this can be done by the "kill -3 <pid>" command. In the case of Windows, you can do a "ctrl-break" in the command window to get the thread dump. The thread dump is printed to the stdout or stderr of the JVM. The application will continue running normally after printing the dump. When you send the SIGQUIT signal to the JVM, the signal handler of the JVM responds to this signal by printing the thread dump. You can take thread dumps at any point while the application is running.

A Sample Thread Dump
Listing 1 is a sample thread dump from a single-threaded application that is using the Sun JVM 1.4.1. The thread named "main" is the main application thread. All other threads are created by the JVM to do housekeeping work. While analyzing application-level problems, we generally focus only on the application threads. Let's analyze the stack trace of the "main" thread from Listing 1

"main" prio=5 tid=0x002358B8 nid=0x7f8 runnable [6f000..6fc40]
at test.method1(test.java:10)
at test.main(test.java:5)

From this code snippet you can see that a thread stack trace has a name, thread priority (prio=5), state (runnable), source code line numbers, and the method calls. You can conclude from this stack trace that the thread "main" is executing some code in the method "method1" of class "test". The call to this method came from the method "main" of the same class. You can also see the exact source code line numbers in those methods.

Before moving on to analyzing thread dumps from more complex scenarios, let's discuss the different state of threads that we normally see in the thread dumps and what they mean.

  • Runnable: Either running or ready to run when it gets its CPU turn. JRockit thread dumps refer to this state as ACTIVE.
  • Waiting on monitor: Either sleeping or waiting on an object for said period of time or waiting to be notified by another thread. This will happen when any of the sleep() methods on Thread object or wait() method on an Object is called.

    For example, in BEA WebLogic Server the idle execute threads are in this condition and they wait until a socket reader thread notifies them of some new work to be done. The stack trace will look like this:

    "ExecuteThread: '2' for queue: 'weblogic.admin.RMI'"
    daemon prio=5 tid=0x1752F040 nid=0x180c in Object.wait()
    [1887f000..1887fd8c]
    at java.lang.Object.wait(Native Method)
    waiting on <04134D98> (a weblogic.kernel.ExecuteThread)
    at java.lang.Object.wait(Object.java:426)

    Some other versions of the JVM call this state CW. JRockit and refers to this state as WAITING.

  • Waiting for monitor entry: Waiting to lock an object (some other thread may be holding the lock). This happens if two or more threads try to execute some synchronized code blocks/methods of an object at the same time. Please note that the lock is always for an object and not for individual methods. That is, if a thread has to execute a synchronized method of an object, it has to first lock that object.
The following is a sample stack trace of a thread in this condition:

"ExecuteThread: '24' for queue: 'DisplayExecuteQueue'" daemon prio=5
tid=0x5541b0 nid=0x3b waiting for monitor entry [49b7f000..49b7fc24]
at
weblogic.cluster.replication.ReplicationManager.createSecondary
(ReplicationManager.java:908)
- waiting to lock <6c4b9130> (a java.lang.Object)
at
weblogic.cluster.replication.ReplicationManager.updateSecondary
(ReplicationManager.java:715)

In this snippet you can see that this thread holds the lock on an object (6c408700) and is waiting to lock another object (6c4b9130).

Some other version of a JVM may not give the object IDs lock information in the stack trace. In those cases, we can guess that the thread is waiting to lock the object from the state of the thread. The same state may also be called as "MW". JRockit refers to this state as LOCKED.

Now, let's see some thread dumps from different scenarios and analyze them to find the problem in the application.

Thread Dump from a Deadlocked Application
In Listing 2 (only part of the thread dump is shown), you can see that the JVM has found the deadlock and it has given that information along with the thread dump. It clearly says that the "ExecuteThread: '47' for queue: 'default'" is waiting to lock an object that has been locked by "ExecuteThread: '57' for queue: 'default'". (This thread dump is from a WebLogic Server. These threads are WebLogic Server's worker threads, which process the client requests.) At the same time, ExecuteThread: '47' has locked an object for which ExecuteThread: '57' is waiting. This is a deadlock. Both the threads will never move from this state, waiting for the other thread to release the lock.

In Listing 2, Execute thread: 1 is waiting to lock the "ConnectionScavanger" object, but this has been locked by thread 47, which is in a deadlock with thread 57 as mentioned before. Therefore, Execute thread: 1 also won't move further. This will eventually result in a hang because the other execute threads in the server will also try to get the lock for the same objects locked by Execute thread: 47 and 57 at some later time.

In some JVMs, the JVM will not give the deadlock information. It will just give the thread dump. In that case, we can arrive at this same conclusion by looking at the state and stack trace of the threads.

In this case the problem was due to a bug in WebLogic that was later fixed.

Other Hang Scenarios
In some scenarios, you may see that most of the threads are "runnable" but the server may appear hung and won't respond to client requests. If the threads are doing IO, they may be blocked in read() - the state will be "runnable" (database or some other network response may be poor). If this is the case, you need to check the health of the database and network.

Other reasons for a "runnable" thread to be stuck at the same place could be:

  • An infinite looping in the code. This may also lead to high CPU usage.
  • JVM garbage collection may be running for a long time, taking most of the CPU.
  • The JVM process might have run out of file descriptors.
If the application code calls wait() or sleep(), the threads won't move for the specified sleep time - the state of the threads in this case will be "Waiting on Monitor". In this scenario, check the application code.

A lot of contention for a single object may also slow down the performance of the system, which may eventually lead to a hang-like situation. In this scenario, the state of most of the threads is "Waiting for monitor entry". The application code needs to be investigated to reduce the contention on that particular object. The threads may also be waiting to get a response from another server and the other server may be hung for a different reason and it may not be able to send the response to this server.

In all of the above scenarios, it is also required that you take multiple thread dumps with a few seconds interval so that you can compare the state of a particular thread in all the thread dumps and you can decide whether the thread is moving or stuck.

JVM Crash Scenarios
A JVM crash can happen due to:

  1. Bugs in the JVM library
  2. Incorrect use of JNI APIs by the application
  3. Bugs in the native modules used by the application - native database drivers, etc.
Normally, when the JVM crashes it gives the stack trace of the Java thread that caused the crash. Some JVMs give a complete thread dump (stack trace of all the threads) before exiting. Our interest in those thread dumps is to find which thread was running when the crash happened. This thread is called as a "current thread". If the JVM gives a complete thread dump, it marks the "current thread". Therefore, generally it's easy to find the thread that caused the crash.

Note that the crash dump usually goes to stdout/stderr. The JVM may also generate a core file, which will be in binary format.

Sample Crash Dump
If you look at the crash dump in Listing 3 you can see that the thread that caused the crash has the following stack trace:

Current Java thread:
at com.aaa.bbb.qqq.Direct.setString(Native Method)
at com.aaa.bbb.qqq.PreparedStatement.setString(PreparedStatement.java:51)
- locked <0x45029c60> (a com.aaa.bbb.qqq.PreparedStatement)

The last method executed (setString) was a "native method" in the class "com.aaa.bbb.qqq.Direct". Therefore, we can guess that this crash is due to the native code in the implementation of the setString method. It could also be due to the JVM library code that has the implementation for the JNI API.

You can also see that the JVM gave the following in the dump:

Unexpected Signal : 11 occurred at PC=0x403AC9D1
Function=(null)+0x403AC9D1
Library=/opt/sunjdk/1.4.1_01-b01/jre/lib/i386/client/libjvm.so
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002E6
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi

This means that the code that caused the crash (PC=0x403AC9D1) is part of the JVM library: "/opt/sunjdk/1.4.1_01-b01/jre/lib/i386/client/libjvm.so". This doesn't mean that this is necessarily a JVM bug. Even though the instruction that caused the crash is in the JVM code, the crash could be due to a JNI call made by the application code (in this case, the implementation of the native method "setString") that could have passed some incorrect arguments, which caused the crash in the JVM module.

If you don't have the "current thread" info in the dump, then we need to analyze core dumps. This is out of the scope of this document.

More Stories By Sathish Santhanam

Sathish Santhanam is a developer relations engineer in the Weblogic/JRockit backline support team. He has more than six years of experience in software development, testing, and support; is a Sun-certified Java programmer and BEA certified developer; and is an expert in debugging server hang, performance, and JVM crash problems.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
Gemini is Yahoo’s native and search advertising platform. To ensure the quality of a complex distributed system that spans multiple products and components and across various desktop websites and mobile app and web experiences – both Yahoo owned and operated and third-party syndication (supply), with complex interaction with more than a billion users and numerous advertisers globally (demand) – it becomes imperative to automate a set of end-to-end tests 24x7 to detect bugs and regression. In th...
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
An increasing number of companies are creating products that combine data with analytical capabilities. Running interactive queries on Big Data requires complex architectures to store and query data effectively, typically involving data streams, an choosing efficient file format/database and multiple independent systems that are tied together through custom-engineered pipelines. In his session at @BigDataExpo at @ThingsExpo, Tomer Levi, a senior software engineer at Intel’s Advanced Analytics gr...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things’). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing? IoT is not about the devices, it’s about the data consumed and generated. The devices are tools, mechanisms, conduits. In his session at Internet of Things at Cloud Expo | DXWor...
Everything run by electricity will eventually be connected to the Internet. Get ahead of the Internet of Things revolution. In his session at @ThingsExpo, Akvelon expert and IoT industry leader Sergey Grebnov provided an educational dive into the world of managing your home, workplace and all the devices they contain with the power of machine-based AI and intelligent Bot services for a completely streamlined experience.
SYS-CON Events announced today that Synametrics Technologies will exhibit at SYS-CON's 22nd International Cloud Expo®, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Synametrics Technologies is a privately held company based in Plainsboro, New Jersey that has been providing solutions for the developer community since 1997. Based on the success of its initial product offerings such as WinSQL, Xeams, SynaMan and Syncrify, Synametrics continues to create and hone inn...