Troubleshooting Out of Memory errors in WebSphere

Troubleshooting Out of Memory errors

Effects of running Out  of Memory

1.     The garbage collection (GC) process struggles to free memory. GC is running all the time

2.     The long and continuous GC cycles cause high CPU in the application server

3.     The Application server is not able to process request as fast as they come in. This creates queuing in the Web and Application Servers.

4.     The JVM eventually stops responding and crashes. Requests are failed over to the next server.

When an Out of Memory occurs, three primary pieces of evidence are left at the scene.

1.     Verbose garbage collection log (How it happened)

2.     Heapdump (What was in memory when it happened)

3.     Javacore (What was running when it  happened).

Tools use to analyse the three evidence

1.     Garbage collection log - The IBM Support Assistance (ISA) provides the Garbage Collection and Memory Visualizer Tool to open the verboseGC log file

2.     Java heapdump  (heapdump.phd)  - Use Memory Analyzer Tool (MAT) in IBM Support  Assistant (ISA)

3.     Javacore - IBM Thread and Monitor Dump Analyzer for Java

Categorization Out of Memory problems

1.     Java heap exhaustion - The JVM cannot allocate an object because it is out of memory and no more memory could be made available by the garbage collector.

2.     Large object allocation - The application requesting a very large object  which Java cannot accommodate in the heap.

3.     Native memory allocation failure  - The memory space for the operating system process that correspond to Java has two main areas

The Java Heap which contains the instances of Java objects and is maintained by Garbage Collection

The Native Heap which contains - Compiled JIT code, Malloc allocation by application JNI code,

Common scenarios that could  lead to Out of Memory

1. Typically, heap exhaustion is caused by
               - Large categories and lack of pagination or filtering (“show all”)
               - Improperly sized cache (in-memory cache is too large)
               - Unbounded search
               - Scheduler processing a large job
               - Processing large backend messages
               - Improperly sized Java heap (too small)
2. Typically, Out of Memory due to a large object allocation is caused by
               - A 3rd-party catalog integration returning all products at once
               - Inbound web service receiving large messages
3. Typically, a native memory error is caused by
              - Improperly sized Java heap ( too big )

Usefuls links;

1. IBM Support Assistant http://www-01.ibm.com/software/support/isa/

2. WebSphere Application Server out-of-memory Mustgather http://www-01.ibm.com/support/search.wss?rs=180&tc=SSEQTP&tc1=SSCMPCY&q=MustGatherDocument

3. IBM SDK - Diagnosis documentation http://www.ibm.com/developerworks/java/jdk/diagnosis/ 

4.  https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014821664