Java Sanity Tune the problem, not the symptom.

1. Introduction

In order to support a production J2EE Application server, it is critical to have the some basic settings in place that allow you to diagnose problems when they occur. I call this "productionalizing" your application server.

These are the "must haves" or "Best Practice" settings that at least give you enough information to begin your analysis. You would enable these and leave on all the time because they have minimal overhead and the benefits for having them on far outweigh the risks for turning them on. In short, if you experience a problem with your J2EE Application Server, you will be blind without them.

2. Logging Parameters

I start with logging because this is a base requirement for the other items in this list. At the bare minimum, your J2EE Application Server has a "standard out" log. This is the same log that gets written to by the System.out.println(); java method. This log is created by redirecting standard out and standard error of your Java Virtual Machine to a file. If you currently redirect this to /dev/null, you will need to correct that now.

This log is where critical things like thread dumps and verbose GC output will be printed for many Java implementations.

Here is how you would do it for a Unix system when creating a custom korn or bourne shell script to start your application server:

#!/bin/ksh
[...]
java [arguments]  >/logs/appserver/appserver.out  2>&1

Because the above example uses ">" to direct output, it truncates the log each time you run your start script. For this reason, your start script should automatically back up the file prior to running the "java" command.

You will want to configure the other logs for your application server to a similar location. You can probably just leave the log levels at their default settings for now.

Warning: it is advisable to have monitoring, rotation and retention intervals for all logs to ensure your file system does not fill up. I strongly suggest having a file system designated for logging so runaway logging doesn't accidentally impair other aspects of the system.

3. Java Parameters

Because your J2EE Application Server runs inside a Java Virtual Machine, all tuning that applies to any Java program also applies to your Application Server. Java developers do not explicitly manage memory (e.g. they don't have to write code that frees memory). Java uses automatic garbage collection to free up memory. Java garbage collection is an area that pops up very frequently as a trouble spot for J2EE Applications. Garbage collection frequency and duration can provide great clues into the overall health of your application server. Before we can talk about analyzing garbage collection, you've got to ensure you capture this data. The following table provides the 'bare minimum' settings for monitoring garbage collection.

3.1 Enabling standard out logs and verbose GC in BEA WebLogic

WebLogic provides several mechanisms for starting the admin and managed servers. Examples are provided for a couple of those options.

Updating startWebLogic.sh and startManagedWebLogic.sh

Updating a nodemanager configuration for this purpose

3.2 Enabling standard out logs and verbose GC in Apache Tomcat

Tomcat provides a "startup.sh" script that you call to start tomcat. It already preserves the standard out log to the "catalina.out" file by default. All you need to do is add the java options for verbose GC and determine some way to preserve copies of catalina.out each time you restart tomcat. For Apache Tomcat, you provide the verbose GC options by setting the JAVA_OPTS environment variable. Here is an example script that accomplishes just that: