Foglight Application Performance Monitoring
This is Pedro Pacheco; I am a solutions architect with the Foglight North America team. I was recently working with a customer who wanted to monitor a Java 7 web application deployed on Tomcat 7.
Our product team is about to release a new version of our Foglight Java EE cartridge which will include, among many other improvements, official support for Java 7 and Tomcat 7. Meanwhile, using the power and flexibility of Foglight and our Java EE cartridge I was able to instrument their environment using a nifty workaround.
What I am about to share with you is something that will work, but it is not officially supported. If you have questions or problems with it, our excellent support team won’t be able to help but you might find other experts on the Foglight community that might be able to lend some assistance. If you choose to implement my suggestions, do so at your own risk. I’d also discourage you from implementing it on a production environment. The officially supported platform is coming soon!
For clarity, I will split this (minimal) effort into three separate tasks: configuration changes to support Java 7, “instrumenting” Tomcat 7 using our x-agent technology and putting it all together.
This work-around was tested on RHEL 6 64 bit, Java HotSpot 7.0 64 bit and Tomcat 7.0.22. Small variations of versions will probably work.
Java 7 was release by Sun (oops by Oracle) in July 2011 and it brought some significant improvements when compared to previous releases of the Java technology, such as:
These are exciding changes and developers are likely to want to take advantage of it right away
(Please visit this site for more information about Java 7).
If nothing else, strings in switch statements is likely simplify coding task leading to cleaner, more efficient and direct code. But I digress; this article is about Foglight monitoring Java 7, not Java 7 itself.
Very important reminder: before making the changes described on this section, please back-up your configuration files. Also, be careful when changing the regular expressions. Any mistakes will cause Foglight to misbehave or crash. Make sure you closely monitor the pre instrumentation and monitoring logs under /<your Foglight Agent Manager home>/Quest_Software/Foglight/fglam/agents/JavaEE/5.7.2-572-20110727-2127/logs for any problems.
In order to support Java 7, you will have to make configuration changes via the Foglight Management Server (FMS) UI. Let’s start by opening the left drawer and selecting the Administration/Agents/Agent Properties dashboard as display on figure 1.
Figure 1 Agent Properties
Now let’s change the agent_collectior_jvm.config file. Select JavaEE/Advance/agent_collector.config as show on figure 2. Change the regular expression /^sun-1\..*/: "collector/jvm-sun-15" to /^sun-1\..*/: "collector/jvm-sun-15". Don’t forget to save your changes.
Figure 2 – Changing agent collector configuration
Now let’s change the object tracking configuration. Select JavaEE/Advance/agent_instrumentor_object-tracking-instrumentor.config as show on figure 3. Change the regular expression /^1\.[5-6]\./: "instrumentor/object-tracking-instrumentor-15" to /^1\.[5-7]\./: "instrumentor/object-tracking-instrumentor-15". Don’t forget to save your changes.
Figure 3 - Changing object tracking configuration
Next we will change the agent type pre-instrumentation configuration. Select JavaEE/Advance/agent_type_preinstrumentor.config as show on figure 4. Change the regular expression /^preinstrumentor 1[5|6]/: "type/preinstrumentor-15" to /^preinstrumentor 1[5|6|7]/: "type/preinstrumentor-15". Once again, save your changes.
Figure 4 - Changing agent type pre-instrumentation
We are almost done. Next configuration file that needs changing is the JVM delegates configuration. Select JavaEE/Advance/ agent_collector_jvm-delegates.config as show on figure 5. This time we will have to make two changes.
First change the regular expression
/.*-1\.\.0/: "com.quest.pas.agent15.plugin.collector.jvm.JmxJvmMetricsDelegate" to
Now change the regular expression
/^sun-1\.\./: "com.quest.pas.agent.plugin.collector.jvm.SunJvmMetricsDelegate" to
Just in case you forgot, save your changes.
Figure 5 - Changing JVM delegate configuration
This next step is optional. If this is a new installation and you’d like to see details breakdowns of your request, select recording.config as shown on figure 6 and change the RequestDetailDefault to ("TierBreakdown", "ServerBreakdown", "ApplicationBreakdown","ComponentTechnologyBreakdown").
In my humble opinion, together with garbage collection metrics, heap metrics and traces, the request breakdown is a very important trouble shooting tool.
Figure 6 - Adding component breakdown information
If you don’t already have, download CartridgeForJavaEE_Install.pdf available on support.quest.com.
Follow the instructions on the section Installing the Java EE Agent, but stop short of making the configuration changes as described on the Tomcat Server section. We will use the x-agent instead. Please see next session for details.
If it all went well, once the Java Deployment Manager agent is started, you will see green signs next to agent name Administration/Agents/Agent Status. Do wait a few seconds and refresh the dashboard using the refresh button; sometimes it takes a few seconds for the agent status to refresh. If instead of a green sign you see a red sign, chances are the configuration changes are wrong. Go back and revise the configuration changes, or look under the agent logs /<your Foglight Agent Manager home>/Quest_Software/Foglight/fglam/agents/JavaEE/5.7.2-572-20110727-2127/logs for possible hints of what has gone wrong.
Figure 7 - Healthy deployment manager running Java 7
Some people will refer to the x-agent as the generic java agent. While this may be a good description, I believe it hides the true capabilities of what the x-agent can do. Instead of generic, I prefer to call it the stand-alone agent. Many of our customers have applications that are deployed directly on the OS (client or server side) without the benefit (and constraints) of an application server. The x-agent is the technology used to instrument and monitor such applications. It is an integral part of the Java EE cartridge and does not require any special licenses. Official support for the x-agent is provided via our Professional Services Organization and requires a special support agreement. Please consult your account manager is you have any questions.
When compared to the amount of changes required to support Java 7, supporting Tomcat 7 is a piece of cake. All you need to do is edit your catalina.sh/catalina.bat located on <your Tomcat 7 home>\bin directory. Before proceeding, please make a backup copy of your catalina.sh/catalina.bat.
Locate the 2nd occurrence of the following script line on your catalina.sh/catalina.bat file.
if [ "$1" = "debug" ] ; then
Immediately before this line, please include the following script segment:
export QUEST_DEPLOYMENT_DIRECTORY=<your Foglight Agent Manger> /agents/JavaEE
if [ "$1" != "stop" ]; then
if [ -f "$QUEST_DEPLOYMENT_DIRECTORY/integrate.sh" ]; then
export PERFORMASURE_OPTIONS=-Dquest.agent.appserverinfo=TomcatLocalhost:Generic:700:Tomcat_Xgroup -Dquest.debug=0
echo Java EE agent not enabled
Make sure to change <your Foglight home> to the location of your Foglight agent manager installation.
If your startup procedure does not include the default catalina.sh/catalina.bat, you will have to find a proper place to include the script segment on your startup script. As a rule-of-thumb, you will place the script segment after JAVA_HOME and CATALINA_HOME have been set, but before java is invoke. On the default catalina.sh/catalina.bat, this can identify by the following script segment:
eval exec \"$_RUNJAVA\" \"$LOGGING_CONFIG\" $JAVA_OPTS $CATALINA_OPTS \
-Djava.endorsed.dirs=\"$JAVA_ENDORSED_DIRS\" -classpath \"$CLASSPATH\" \
As a result of our pre-instrumentation (i.e. integrate.sh), the JAVA_OPTS system variable is updated. If you have a custom startup script, make sure JAVA_OPTS is passed as a command line argument to the java executable.
In the end of your editing efforts, your catalina.sh should look something like this:
Figure 8 - Catalina.sh with x-agent instrumentation
So now we are ready for Java 7 and Tomcat 7. Just start your tomcat using startup.sh or your own custom startup script. During the first run it is a good idea to tail a few log files and pay attention to the output startup.sh is echoing on the terminal.
A healthy startup should echo on terminal something like this:
Figure 9 - Healthy instrumented Tomcat 7 startup
If you don’t see something similar to figure 9, than your configuration changes may not have worked. Review your catalina.sh, make sure JAVA_OPTS is been properly populate and passed to the java executable. A properly instrumented Tomcat 7 should yield process information (ps –ef | grep java) similar to that of figure 10.
Figure 10 - Process information
Another good check is to verify if the x-agent log has been created and there are no errors reported. You can do this by checking /<your Foglight Agent Manager home>/Quest_Software/Foglight/fglam/agents/JavaEE/5.7.2-572-20110727-2127/logs for a file called <your host name>-TomcatLocalhost-Generic_<time stamp>.log.
As if we haven’t done enough checks, one last check you can perform is to verify that a new agent has been created. You can do this by using the Foglight Management Server UI and navigate to Administration/Agents/Agent Status. In there you should see something like this:
Figure 11 - Tomcat x-agent is running!
If you have questions or comments please leave me a message!