Getting Heapdumps on the Solaris

1.6.0

To produce a heapdump in the event of an OutOfMemoryError:

-XX:+HeapDumpOnOutOfMemoryError

Heapdumps can be taken on demand from the new improved jmap tool which uses the built in heapdumper code (unlike the jmap tools in 1.4.2 and 5.0). The histogram option in Java™ 6 also uses a much improved built in mechanism. To manually create a heapdump, use the following command:

jmap -dump:live,format=b,file= [JVM PID]

To print class histograms, issuing a SIGQUIT - kill -3 [JVM PID] will trigger a class histogram to be written to native_stdout.log:

-XX:+PrintClassHistogram

SDK 1.5.0_16 or later

To force a heapdump by sending the Java Virtual Machine (JVM) a SIGQUIT - kill -3 [JVM PID]:

-XX:+HeapDumpOnCtrlBreak

To produce a heapdump in the event of an OutOfMemoryError:

-XX:+HeapDumpOnOutOfMemoryError

To print class histograms without using jmap, issuing a SIGQUIT - kill -3 [JVM PID] will trigger a class histogram to be written tonative_stdout.log:

-XX:+PrintClassHistogram

SDK 1.5.0_07 to 1.5.0_15

Note: Manual generation of heapdumps is still not implemented. -XX:+HeapDumpOnCtrlBreak only works on 1.5.0_16 or later.

To produce a heapdump in the event of an OutOfMemoryError:

-XX:+HeapDumpOnOutOfMemoryError

To print class histograms without using jmap, issuing a SIGQUIT - kill -3 [JVM PID] will trigger a class histogram to be written tonative_stdout.log:

-XX:+PrintClassHistogram

SDK 1.4.2_12 or later

In 1.4.2_12, Sun implemented a built-in heapdump facility, with a new command line option.

To force a heapdump by sending the Java Virtual Machine (JVM) a SIGQUIT - kill -3 [JVM PID]:

-XX:+HeapDumpOnCtrlBreak

To produce a heapdump in the event of an OutOfMemoryError:

-XX:+HeapDumpOnOutOfMemoryError

To print class histograms without using jmap, issuing a SIGQUIT - kill -3 [JVM PID] will trigger a class histogram to be written tonative_stdout.log:

-XX:+PrintClassHistogram

SDK 1.5.0 GA to 1.5.0_06
SDK 1.4.2_09 to 1.4.2_11

Note: Due to limitations using jmap, the most reliable solution for getting a heapdump will be to upgrade the JDK. Click here for more information.

The jmap executable is contained in the same directory as the Java executable (for example: [path to JDK]/bin), and can be run in a number of ways:

The following produces a heapdump in the current working directory, namedheap.bin:

jmap -heap:format=b [JVM PID]

The following produces a heapdump in the same way as the previous command, but from a core file rather than a running process:

jmap -heap:format=b [Java executable] [core file]

The following produces a summary of the state of the heap, including details of the heap configuration (although this frequently seems to be incorrect):

jmap -heap [JVM PID] > heapsummaryfilename

The following walks the heap and produces a list of all object types, sorted by total size:

jmap -histo [JVM PID] > histofilename

The following produces classloader statistics for the permanent generation. Among other things, the results detail the size and number of classes loaded by each classloader:

jmap -permstat [JVM PID] > histofilename

MAT(Memory Analyzer) Querying Objects from a Heap Dump…

I happen to use the Memory Analyzer built-in Object Query Language (OQL) on heapdump recently, which I find it very useful.

These are the some of the queries I used while looking into jms connection leak in customer application.

SELECT toString(clientAddress.value) AS s FROM weblogic.jms.frontend.FEConnection

SELECT toString(clientAddress.value) AS s FROM weblogic.jms.frontend.FEConnection WHERE (toString(clientAddress.value)="/")

There is good blog on MAT Object Query Language (OQL) at

http://dev.eclipse.org/blogs/memoryanalyzer/2008/09/03/about-querying-objects-from-a-heap-dump/

Getting client IP in application while using loadbalancer

If you are using loadbalancer with weblogic, by default all requests it sends to WebLogic will appear to come from loadbalancer IP. Your Java applications that call HttpRequest.getRemoteAddr() will return the ip address of the hardware loadblancer or software loadbalancer like proxy plugin

If you are connecting through proxy plugin ie
client -> webserver + weblogic http plugin -> weblogic instances

You need to have the WeblogicPluginEnabled attribute turned on so that a call to getRemoteAddr will return the address of the browser client from the proprietary WL-Proxy-Client-IP header instead of the web server.You can set this flag from Admin Console, either on Advanced Option of a Server or a Cluster.

If you are using hardware loadblancer ie..

client - > Hardware LB -> weblogic instances

You need to set the WL-Proxy-Client-IP header in loadbalancer level in addition to turning on WeblogicPluginEnabled attribute at weblogic layer.

Weblogic server applying patch offline

If you don't have access to Internet from production box or if you have to apply the same patches on multiple installations, best option would be to download the patch on windows and apply the patch in target env using offline mode.

You need to make sure bsu version(BEA_HOME/utils/bsu.sh -version) matches on both env(source and target) before applying the patch. If not download the bsu client through MetaLink/My Oracle Support. Search on Patch 8248099 and run (java -jar it on target installation.)

Instructions on applying patch offline.

Copy the cache_dir from you windows env(or any other env where you can download online using bsu) to some folder in solaris (-patch_download_dir), and run the smart update as shown below

Applying patch.

$bsu.sh -prod_dir=WL_HOME -patch_download_dir=<-patch_download_dir> -patchlist= -install -verbose

Removing the Patch

$bsu.sh -prod_dir=WL_HOME -patchlist=== -remove

Checking the applied patches:

$bsu -prod_dir= -status=applied -verbose -view

Note: WL_HOME is the weblogic92(incase of weblogic 9.x) or wlserver_10.3(incase of 10.3) directory

Troubleshooting Weblogic server hung issue

Thread dumps are critical for root cause analysis and are often required to effectively resolve application issues. At times of application problems, thread dumps need to be collected at least 3~5 times with 10~15 second interval. Create a convenient script to generate thread dumps from the JVM running WLS server. The following script is a working example and can be further customized to fit specific needs. Run the script to take thread dumps before shutting down the WLS server instance at times of problems.

#! /usr/bin/ksh

# This shell scripts takes a pid and kill -3 the pid to take thread dump.

if [[ 1 != $# ]]

then

echo "Usage: $0 "

exit 1

fi

let j=1

# The loop controls the number of times thread dumps are taken

while [[ $j -le 5 ]]

do

echo kill -3 $1

kill -3 $1

# take a small pause before getting the next thread dump

sleep 15

let j=$j+1

done

Multicast issues, Server droping out of the cluster

If you are using weblogic cluster you need to make sure that you do not have any multicast issues. This is very significant If your application is using session replication or has

any distributed destination.

You can confirm that multicast traffic is working using simple multicast test using utility shipped with weblogic installation.

The MulticastTest utility helps you debug multicast problems when configuring a WebLogic Cluster. The utility sends out multicast packets and returns

information about how effectively multicast is working on your network.

Here is the syntax of multicastTest utility

java utils.MulticastTest -N name -A address

You need to run this test in each boxes with unique name identifier(-n) and a common multicast address. Make sure that multicast address used is unique in subnet and this should be

different from the one weblogic cluster is configured to use

Example.

For instance if you want to test multicast traffic between machine A and B

set the classpath using setDomainEnv.sh

$. ./setDomainEnv.sh (this is located under domain/bin)

run following command in Box A

$ java utils.MulticastTest -N serverA -A 237.155.155.1

Set up to send and receive on Multicast on Address 237.155.155.1 on port 7001

Will send a sequenced message under the name serverA every 2 seconds.

Received message 506 from serverA

Received message 533 from serverB

I (serverA) sent message num 507

Received message 507 from serverA

Received message 534 from serverB

I (serverA) sent message num 508

Received message 508 from serverA

Received message 535 from serverB

run following command in Box B once you set the classpath

$ java utils.MulticastTest -N serverB -A 237.155.155.1

If the multicast is traffic is working fine you can see the multicast messages being exchanged between these 2 nodes as shown above. You need to involve

your network system admistrator to investigate rootcause if the muticast traffic is not working.