JVM Debugging without a Debugger - debugging

Question

Newish to JVM debugging.
I've worked on supporting other products based on VxWorks written in C/C++. Within that environment we were able to do symbol lookups on a live system and peek or poke memory to get an idea what the software was doing or to alter it when other "normal" configuration options weren't available.
Now I'm supporting java applications. For issues that aren't readily reproducible within our labs, we are reduced to recompiling with additional instrumentation and performing binary replacements to gather more information to understand what is happening within the sw.
These are always on applications that folks don't take kindly to restarting.
Is there any similar type of debugging that can be taken for JVM applications? These are running on customer sites where using a conventional debugger is not an option for a number of reasons.
Please no lectures on how the application is poorly designed for support-ability. That's a given, we're just a couple of guys in support that have to figure it out the best we can.
thanks,
Abe

Answer 1

I've been in similar situation, where stopping application in debugger triggered timeouts on lower layers, and adding instrumentation was annoying because of restarts.
For me the solution was to add more logging statements.
I've used slf4j API, with logback implementation, I've enabled JMX on JVM and used it to enable/disable logging as needed and/or change classes that were logged.
If you use slf4j/logback the right way, there is very little overhead for disabled log statement, so I was able to use them liberally.
This way I could turn on "debug instrumentation", without annoying users with restarts.
Now some code:
This is testbed for experiment
package pl.gov.mofnet.giif.logondemand;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LogOnDemand {
private static final Logger log = LoggerFactory.getLogger(LogOnDemand.class);
public static void main(String []args) throws InterruptedException {
for (int i = 0; i < 1000; i++) {
log.debug("i: {}", i);
Thread.sleep(1000);
}
}
}
This file needs to be put in default package (or anywhere on classpath).
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<jmxConfigurator />
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%date [%thread] %-5level %logger{25} - %msg%n</Pattern>
</layout>
</appender>
<root level="info">
<appender-ref ref="console" />
</root>
</configuration>
This code has compile time dependencies on org.slf4j:slf4j-api:1.7.7 and runtime dependency on ch.qos.logback:logback-classic:1.1.2
Run this code, in Java 7 you do not have to enable JMX explicitly if you connect to JMX from same machine. You will see that there is no output on console save for initial Logback configuration messages.
Start jconsole, connect to this process, in MBeans tab you will find ch.qos.logback.classic node, deep below it there are operations.
Edit parameters to setLoggerLevel, set p1 to package name of your class, in this case pl.gov.mofnet.giif and p2 to debug. Press setLoggerLevel button to run this operation, you should see log messages on console. To disable logging reset logger level to info or higher.