Summary
Under normal circumstances, you can get the Flow process thread dump by logging into Flow and running:
ectool getServerStatus --diagnostics 1
However, if you are trying to troubleshoot a hanging Flow process, the process will not be responsive to API commands.
Solution
You can get the thread dump directly through Linux commands.
Note: for thread activity analysis, you will need to produce multiple thread dumps spaced a short time apart (typically 30 seconds) in order to properly monitor thread activity over a time span. If support has requested thread dumps from your system, please send 5-10 dumps.
Getting the PID
First, you need to acquire the process ID of the Java service wrapper for Flow. This PID is stored in the Flow directory (/opt/electriccloud/electriccommander by default) under server/commander.pid. E.g. if the file contains the number "964," then the wrapper PID is 964. You can confirm this is the correct PID by running:
ps -ef | grep -i <PID> | grep -i commander | grep -i java
Your output will look something like this:
vagrant 982 964 15 21:51 ? 00:01:12 /opt/electriccloud/electriccommander/jre/bin/java ... (the rest is ommitted)
Note that 982 is the Flow child process PID and 964 is the parent wrapper PID we found in commander.pid. Using either the parent or the child PID will produce the same results, as the parent wrapper process simply passes the command to the child Commander process.
Getting the thread dump
Run:
kill -3 <PID>
The -3 option sends the signal that will tell the process to create a thread dump. DO NOT run kill with any other signal option, or you might terminate the Flow process.
Repeat this step 5-10 times to get a full set of thread dumps over a time span.
The thread dump is now logged to logs/commander-service.log (or logs/service.log in older versions) in the Flow root directory. It can then be cleaned and opened with tools such as Samurai.
Alternative Solution
The thread dump can also be obtained with the JDK jstack utility. This utility can be used on both Linux and Windows machines but may require additional installations. See KBEC-00052 for details on this method.
See Also
KBEC-00052 - Installing JDK and taking a jstack threads dump or stack dump
Applies to
- Product versions: all
- OS versions: Linux
Actually the above KB article just mentions how to get one thread dump logged to the logs/service.log in the Commander directory. But to analyze we need multiple thread dumps as mentioned below:
A.The ProcessID of the Commander wrapper process is stored in /opt/electriccloud/electriccommander/server/commander.pid file In this file if you see the content is "1052" that means the Commander wrapper process ID is 1052.
B.Run the command from the shell prompt:kill -3 <PID_in_/opt/electriccloud/electriccommander/server/commander.pid>For example if the PID is 1052 run:"kill -3 1052"
Similarly the agent thread dump can be got using <PID_in_/opt/electriccloud/electriccommander/logs/agent/agent.pid>
C.Wait for 30 seconds ( or longer if the issue or hang is for a longer duration . For example: like 1 or 2 minutes if the issue is related to commander not logging to commander.log )
D.Do step B
E. Do step C and D , 10 times
All the thread dumps will be logged to the /opt/electriccloud/electriccommander/logs/service.log
Please send the following to Electric Cloud Support :
1.Zip and send the service.log file ( and if it rolled over then the recent server*.log.zip file which got rolled over).
2. Commander log files.
Here are the steps for Windows platform:
To get the thread dump as explained in http://wrapper.tanukisoftware.com/doc/english/prop-thread-dump-control-code.html do steps below:
A.Run the command:
"sc control CommanderServer 255"
This will log the thread dump to C:\ProgramData\Electric Cloud\ElectricCommander\logs\service.log.
B.Wait for 30 seconds ( or longer if the issue or hang is for a longer duration . For example: like 1 or 2 minutes if the issue is related to commander not logging to commander.log ).
C.Do step A
D. Do step B and C , 10 times
All the thread dumps will be logged to the C:\ProgramData\Electric Cloud\ElectricCommander\logs\service.log.
Please send the following to Electric Cloud Support :
1.Zip and send the service.log file ( and if it rolled over then the recent server*.log.zip file which got rolled over).
2. Commander log files.