| Troubleshooting and Performance tuning with Tomcat and Sun |
 |
 |
 |
| Written by Administrator |
| Saturday, 02 January 2010 11:11 |
Java Spring Source Seminar, 01/04/209I recently attended a seminar run by Tomcat guru Filip Hanik that was specifically about the Spring Source version of Tomcat, tc Server. The presentation centred on Performance Tuning and Troubleshooting of Tomcat in production environments on unix/linux servers. As there are few options within Tomcat for performace tuning or troubleshooting, most tuning is done within the JVM, the presentation is relevant to all Java application servers. The tools used in the presentation for troubleshooting bottlenecks, crashes, etc were all tools that are installed on Unix/Linux systems, with reference to equivalents on Windows.
Java Heap Size vs Process Heap SizeOne of the common misconceptions about the Java Heap space is that it is the total amount of memory allocated to Java. This is in fact the process heap, which contains the Java Object Heap and everything else Java uses to run, which includes:- Permanent Space, Code Generation, Socket Buffers, Thread stacks, Direct Memory Space, JNI Code, Garbage Collection and JNI allocated memory. The JVM manages the heap space and is controlled through the -Xms and -Xmx settings. There are inherent limits to the amount of memory that can be allocated to a process heap space. On a 32-bit JVM and server this is about 2GB, a 32-bit JVM on a 64-bit it is about 3.7GB. On a 64-bit JVM/server the number is sufficiently large number that it is not a problem currently... Allocating a JVM heap size close to these limits will be detrimental to performance as it reduces the amount of memory available for the other elements in the java process. If you allocate a Heap Size too close to the maximum process size on your machine, you are likely to get an error such as: java.lang.OutOfMemoryError: unable to create new native thread which indicates that there are not enough system resources to perform the requested task withing the process heap. Possible solutions are to reduce –Xmx (max memory), -Xss (thread stack space) or resolve any thread leak issues within the application. Components of the Java HeapA Sun Java heap (and most others) is made up of several generations. The two main generations are the Young Generation and the Old Generation. The Young Generation is further broken down into the Eden Space and the Survivor Spaces. As a general rule, the Young Generation should be between 25% and 30% of the allocated heap size. This can be achieved by the following Java options: -Xmn=[fixed value] -XX:NewRatio=[ratio] - dynamic sizing -XX:MaxNewSize/-XX:NewSize - similar to -Xmx/-Xms Minor Garbage Collections occur when the Young Generation fills up, or more specifically, when a request is made by the application to load an object into memory which does not fit in the Eden space. Any objects not cleaned up are moved into one of the Survivor spaces. The main purpose of minor GCs is to alleviate fragmentation in the young generation – providing a continuous space of memory for new objects. The old Generation contains the Tenured space where objects which survive the Young Generation reside until they are ultimately garbage collected when no longer needed. A Full GC is triggered when the old generation reaches its capacity or when certain conditions are met in the young generation. Out Of Memory errors are typically more descriptive in Java 1.5+. They will specify if it is the heap space or perm gen, etc. A useful flag to use is -XX:+HeapDumpOnOutOfMemoryError. This creates a file describing the condition of the Java heap on out of memory allowing you to narrow down the cause. The Permanent Generation is where classes and other static objects are loaded. Default values for java 1.5/1.6 are 4MB initial, 64MB max. This is often too small and can be increased by using -XX:PermSize= (initial) and – -XX:MaxPermSize= (max). Increasing the Perm Gen space by 64MB increments is recommended and is extremely unlikely to exceed 256MB. Out of Memory errors in the Perm Gen can often be caused by simply insufficient space. A common cause of out of memory errors in the perm gen are memory leaks when classes are reloaded, usually through redeployment. Class loaders are removed and recreated during the reloading of the class. If for some reason a class is being referenced by another loader it may fail to be removed resulting in duplicate classes being loaded. It is also possible that some libraries may invoke the reloading of a class so it does not need to be explicitly done by the application or sysadmin. Garbage CollectionGarbage collection is an important element of all java tuning. Most garbage collection is single threaded, meaning that the application pauses while it is in progress. There are several options available to reduce the pause, but at some point there will be a pause. As the garbage collections get larger, the pauses get longer. In general Minor GCs will take tens of milliseconds and Major GCs can take much longer. It is important to decide what sort of pause is acceptable to the application users. Is a 5 second pause every couple of minutes (or longer) OK, or is the maximum pause acceptable is 1 second, but occurring much more frequently. Options in Java 1.5 and 1.6 for controlling Garbage Collections are: -XX:+UseConcMarkSweepGC This option will dedicate some threads of the application to continuously garbage collect. In effect, this smooths out the garbage collection but increases the load on the CPU. There will still be major GC’s but should occur less frequently. The Major GC’s will still be single threaded and therefore still cause a pause in the application. -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycleMin=0 -XX:CMSIncrementalDutyCycle=10 -XX:+UseParNewGC This will enable the use of parallel garbage collection in the new generation, achieving much the same result as UseConcMarkSweepGC, but only in the new generation. -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -XX:MaxGCPauseMillis=250 -XX:MaxGCMinorPauseMillis=100 These parameters set the maximum time that the application can pause while performing a major or minor garbage collection. In turn this will reduce the amount of memory freed up by the GC and thus make them more frequent. It may be that it is unacceptable for the application to pause for 5 seconds every major GC, but frequent 250 millisecond collections will not cause a noticeable pause to the end user, thus making the application appear more consistent. There are several options for troubleshooting issues with Garbage collection. The following Java flags will all provide useful information about the GC process.. -Xloggc:[log directory]/gc.log -XX:+PrintGCDetails -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC The first two options are perhaps the most useful. The first will output a log file wit details about each garbage collection, both minor and major. The logs will contain a timestamp (the number of seconds since the java process started), the details of each collection (start generation or heap size and the amount cleared) and the time it has taken to perform the garbage collection. If the heap is full and performing continuous garbage collections, it can be shown with simple mathematics by calculating the GC time and the timestamp. This will show you the percentage of time available to the application (as opposed to garbage collection) and the amount of memory that is used up within that time. It is important to remember that a Garbage Collection occurs when an object is requested that does not fit into the available space. This is different to a garbage collection being triggered when the space within the heap is full. This distinction may be useful when troubleshooting as thread dumps will show you which threads are trying to construct or load items. This will give you clues as to which processes are being blocked… Thread DumpsThread Dumps are a very useful tool to determine the state of the application. It displays the details of all current threads within the JVM. It provides information about the current thread activities, any thread locks and blocked threads. Each active thread shows a trace of all calls from where the thread started to the current point of execution. NOTE: It is very important to take more than one thread dump. What will look like a blocked or non-responsive process in a thread dump could be normal or just slow operation. Take two or three thread dumps 30 seconds to 1 minute apart. On a Unix/Linux system a thread dump can be generated by executing the following command: kill -3 On windows using Ctrl+Break will generate the same result. Note: The thread dump will be directed to standard out. If the Java process was started with a script that redirects standard out to /dev/null, you will not be able to get a thread dump. The thread dump will typically contain the following information (for example): "AWT-EventQueue-0" Thread Name prio=6 priority (1-10) tid=0x0498e800 tid (C++ pointer to OSThread object) nid=0x828 nid (Kernel Thread Identifier) in Object.wait() Thread Action [0x052cf000..0x052cfb68] Address space Of these the most interesting are the “nid” and thread action. The nid is a mapping to the thread within the kernel. Running the following command in Linux will show all threads within the JVM that are using more than 0% CPU: ps -eL -o pid,%cpu,lwp | grep -i `ps -ef |grep -v grep |grep java|awk '{print $2}'` |grep -v 0.0 Threads dumps can help you identify threads waiting for a lock, dead-locks (two or more threads waiting for each other), threads taking all CPU (and what the thread is trying to do), a snapshot of the memory usage of the JVM and the causes of unresponsive JVM. Several loader classes, such as calls, in a thread dump can indicate the JVM is overloading with new object creation and/or garbage collection.
|