Thread Dump Analysis

How to conduct Thread Dump Analysis

There are several thread dump analysis tools are available to carry out the investigation on Threads and find out the root cause of the issue. Out of those, I found 2 thread dump analysers which are really good.

Thread Dump Analysis Tools:

  • fastThread: This is an online thread dump analyzer. It has a simple and easy dashboard view. fastThread helps to provide a detailed view of blocked, runnable, timed waiting, waiting and repeating stack traces. fastThread parses and analyzes thread dumps written in any language that runs on the JVM. Another good thing is that you just need to upload the application thread dump files & review the reports instantly in one click.
  • jstack.review: Another online thread dump analyzer. The additional feature in this tool is that if you face the problem in uploading the thread dump file then you can paste the thread dump file content in the given thread dump section and analyse it. It also gives the option to compare and analyse the dumps.

To understand the thread dump analysis practically, we will use ‘fastThread’ tool. We will see how to identify the deadlocks, blocked thread, waiting thread etc. with the help of fastThread tool. fastThread tool provides detailed information about each thread along with their states. It analyses the thread dump file and helps to find out the deadlock or any other problematic issue. 

How to analyse Thread Dump using fastThread?

Step 1. Browser http://fastthread.io

fastThread - Home Page
Figure 01: fastThread – Home Page

Step 2. Browse the thread dump file (.zip, .gz, .tdump format) using “Choose File” button.

Upload Thread Dump
Figure 02: Upload Thread Dump

Step 3. Click “Analyze”. You can see the file ‘Uploading’ is in progress as illustrated in Figure 03.

Figure 03: File uploading is in progress

Step 04: After clicking “Analyze”, fastThread takes a few seconds to scan and analyse the dump and prepare the dashboard. 

fastThread - Progress Bar
Figure 04: Thread Dump scanning and analysis

Step 05: On the completion of step 04, fastThread shows the result of the analysed thread dump. You can see multiple tabs in the left pane and the detailed view of each tab in the right pane of the dashboard. The result contains Thread count Summary, Identical stack trace, most used methods, CPU Threads, GC Threads etc. that provides you with the detailed information of each thread and identified issues.

Figure 05: fastThread Dashboard

Thread Dump Analysis:

There are different possible scenarios which may occur in the production as well as the test environment. In this section, we will discuss some common and important scenarios which will give you a knowledge of how to identify the issue and conclude the investigation. We will also get some tips at the end of each scenario.

For a better understanding purpose, we will divide the analysis into 3 scenarios which are:

  1. Normal Scenario
  2. Blocked Thread Scenario
  3. Deadlock Scenario

Normal Scenario

So, let’s start with a Normal scenario where all the threads are performing well without any issue. Refer to the Thread Count Summary on the fastThread dashboard:

Thread Count Summary
Figure 06: Thread Count Summary

None of the thread is blocked due to a resource crunch. 17 threads are running. 10 threads locked the resources and waiting for their chance to execute or response from other thread execution. Remaining 10 threads are either parked or sleep or made to wait using respective methods. The same information is available in percentage in the pie chart (shown in the right).

Click “View Details” to get more information about each thread.

Figure 07: Thread details

As stated above, there is no blocked thread hence transitive graph is not available

Figure 08: Transitive Graph

Deadlock Identification: Refer to the Dead Lock tab. We found that fastThread did not identify any deadlock. See below:

Figure 09: Deadlock

Conclusion: 

All the threads are working fine without having any issue or crunch of the resources. Hence no blocked thread is found, also no deadlock reported.

Tip: 

In case you do not want to use the fastThread and analyse the dump manually then follow the below steps:

Step 1. Open the thread dump file in Notepad++.

Step 2. Look at the first thread and get the resource ID locked by it. Example: Java2D Disposer thread locked the resource 0x00000000f03eb540. See below:

"Java2D Disposer" #17 daemon prio=10 os_prio=2 tid=0x0000000017816000 nid=0x30dc in Object.wait() [0x000000001946f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x00000000f03eb540> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at sun.java2d.Disposer.run(Disposer.java:148)
        at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:
- None

Step 3. Select that resource ID (0x00000000f03eb540) and then search in the thread dump file. If only 1 resource ID is present then it shows that only 1 thread locked the given resource.

Step 4. In case you found more than one occurrence in the thread dump file then check whether some other threads are waiting for the same resource. This situation causes the thread (s) blocked. We will discuss such a situation in the next scenario.

Blocked Thread Scenario

To continue the thread dump analysis, now we will move to the next part which will cover the analysis of blocked threads. The meaning of blocked thread is that a thread locks a resource for a long time and restrict other threads to use that particular resource, hence the restricted threads are moved into the Blocked state and named as Blocked Threads. Let’s try to understand with an example. Refer to the below fastThread dashboard which shows the status of 3 threads.

Blocked Thread in Thread Dump
Figure 10: Blocked Threads

Thread Count Summary shows that out of 3, two threads are in the blocked state whereas 1 thread is in the runnable state which needs an investigation.

Now, refer to the transitive graph and read the message:

com.nbp.theplatform.threaddump.ThreadBlockedState is blocking 2 threads

Figure 11: Transitive Graph

We can see pool-1-thread-1 is blocking pool-1-thread-2 and pool-1-thread-3.

When we navigate to Identical Stack Trace of Runnable thread then we can see more information about the thread and locked resources. thread-2 and thread-3 have a dependency on thread-1 locked resources.

Figure 12: Stack Traces

Deadlock Identification: Refer to the Dead Lock tab. We found that fastThread did not identify any deadlock. See below:

Deadlock Identification
Figure 13: Deadlock Identification

Conclusion: 

It has been identified that thread-2 and thread-3 are waiting for <0x0000000750a000b0> which is locked by thread-1 and hence it leads the blocking of thread-2 and thread-3. 

Tip: 

In case you do not want to use the fastThread and analyse the dump manually then follow the below steps:

Step 1. Open the thread dump file in Notepad++.

Step 2. Look at the first thread and get the resource ID locked by it. Example: Java2D Disposer thread locked resource 0x00000000f03eb540. See below:

"Java2D Disposer" #17 daemon prio=10 os_prio=2 tid=0x0000000017816000 nid=0x30dc in Object.wait() [0x000000001946f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x00000000f03eb540> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at sun.java2d.Disposer.run(Disposer.java:148)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

        - None

Step 3. Select that resource ID (0x00000000f03eb540) and then search in the thread dump file. If only 1 resource ID is present then it shows that only 1 thread locked the given resource.

Step 4. In case more that one occurrence you found in the thread dump file then check whether some other threads are waiting for the same resource and if yes, then what are their state? If those threads having the state as Blocked then you can carry out further investigation. Another situation of the Blocked thread is Deadlock. We will discuss it in the next scenario.

Deadlock Scenario

Deadlock is a situation when the locked resources are required by some other threads who locked some resources which are needed by other threads. Such a cyclic condition restrict to complete the execution of any of the threads, resulting in none of the thread completes its execution and system is halted. Refer to the below fastThread dashboard which shows the status of 3 threads.

Figure 14: fastThread Dashboard

Now, look at “Complex DeadLocks” section:

Conclusion:

A circular complex deadlock is created by all three threads. Thread-1 locked <0x00000007d58f5e48> and waiting for <0x00000007d58f5e60> which is locked by Thread-2 and it is waiting for <0x00000007d58f5e78> which is locked by Thread-3 and waiting for <0x00000007d58f5e48> which is again locked by Thread-1. Hence it creates a deadlock situation and stops the system.

Tip: 

In case you do not want to use the fastThread and analyse the dump manually then follow the below steps:

Step 1. Open the thread dump file in Notepad++.

Step 2. Look at the first thread and get the resource ID locked by the thread. 

Step 3. Select that resource ID and then search in the thread dump file. If only 1 resource ID is present in the file then it shows that only 1 thread locked the given resource.

4. In case more than one occurrence you found in the thread dump file then check whether some other threads are waiting for the same resource and are they in the Blocked state? If yes, then pick the locked resource ID of the Blocked Thread and search in the thread dump file and repeat step 3 and 4. If the resources are interdependently locked by the threads then it shows that the threads are in deadlock and needs an investigation to prevent such circumstances in the future.


You may be interested:


Leave a Reply

Your email address will not be published. Required fields are marked *