How to enable GC logging for Apache HiveServer2/MetaStore server/WebHCat server, while preventing log file overwrites and capping disk space usage - hadoop

Question

We recently decided to enable GC logging for three Apache Hive-related servers: HiveServer2, Hive MetaStore server, and WebHCat server. This is on a number of clusters (exact version varies) as a aid to looking into Hive-related memory and garbage collection problems. While doing this, we want to avoid two problems we know might happen:
overwriting of the log file when a server restarts for any reason
the logs using too much disk space, leading to disks getting filled
When Java GC logging starts for a process it seems to replace the content of any file that has the same name. This means that unless you are careful, you will lose the GC logging, perhaps when you are more likely to need it.
If you keep the cluster running long enough, log files will fill up disk unless managed. Even if GC logging is not currently voluminous we want to manage the risk of an unusual situation arising that causes the logging rate to suddenly spike up.

Answer 1

How you do this will be a little different for HiveServer2 and Hive MetaStore server vs WebHCat server. I'll answer both. In both cases you'll need to set some JVM parameters when starting the servers.
For HiveServer2 and Hive MetaStore server the Java parameters will be set in hive-env.sh. It appears these need to be set in HADOOP_OPTS, but we'll need to be careful to limit the scope of the change to the two server (and not other cases in which hive-env.sh is run).
For WebHCat server, you will also be setting HADOOP_OPTS but there doesn't seem to be a "env.sh" file in the distribution invoked when starting the server, so you'll need to set it in the environment prior to running WebHCat server. However, in Ambari, there is a webhcat-env config group.
Now lets discuss the JVM parameters to include in these cases.
To enable GC logging to a file, you will need to add -verbose:gc -Xloggc:<log-file-location>.
You need to give the log file name special consideration to prevent overwrites whenever the servers are restarted. It seems like you need to have a unique name for every invocation so appending a timestamp seems like the best option. You can include something like `date +'%Y%m%d%H%M'` to add a timestamp. In this example, it is in the form of YYYYMMDDHHMM. In some versions of Java you can put "%t" in your log file location and it will be replaced by the server start up timestamp formatted as YYYY-MM-DD_HH-MM-SS.
Now onto managing use of disk space. I'll be happy if there is a simpler way than what I have.
First, take advantage of Java's built-in GC log file rotation. -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M is an example of enabling this rotation, having up to 10 GC log files from the JVM, each of which is no more than approx 10MB in size. 10 x 10MB is 100MB max usage.
With the GC log file rotation in place with up to 10 files, '.0', '.1', ... '.9' will be added to the file name you gave in Xloggc. .0 will be first and after it reaches .9 it will replace .0 and continue on in a round robin manner. In some versions of Java '.current' will be additionally put on the end of the name of the log file currently being written to.
Due to the unique file naming we apparently have to have to avoid overwrites, you can have 100MB per server process invocation, so this is not a total solution to managing disk space used by the Hive server logs. You will end up with a set of up to 10 GC log files on each server invocation -- this can add up over time. The best solution (under *nix) to that would seem to be to use the logrotate utility (or some other utility) to periodically clean up the GC logs that have not been modified in the last N days.
Be sure to do the math and make sure you will have enough disk space. Note that on some master servers you might be running all three kinds of servers.
People frequently want more details and context in their GC logs than the default, so consider adding in -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps.
Putting this together for HiveServer2 and Hive MetaStore server, you might add something this to hive-env:
# note that $SERVICE is a string saying what it is that is being started
if [[ "$SERVICE" == "hiveserver2" || "$SERVICE" == "metastore" ]]; then
TIMESTAMP=`date +'%Y%m%d%H%M'`
# GC log location/name prior to .n addition by log rotation
HIVE_SERVERS_GC_LOG_NAME="{{hive_log_dir}}/hive-$SERVICE-gc.log-$TIMESTAMP"
HIVE_SERVERS_GC_LOG_ENABLE_OPTS="-verbose:gc -Xloggc:$HIVE_SERVERS_GC_LOG_NAME"
HIVE_SERVERS_GC_LOG_ROTATION_OPTS="-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M"
HIVE_SERVERS_GC_LOG_FORMAT_OPTS="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps"
HIVE_SERVERS_GC_LOG_OPTS="$HIVE_SERVERS_GC_LOG_ENABLE_OPTS $HIVE_SERVERS_GC_LOG_ROTATION_OPTS $HIVE_SERVERS_GC_LOG_FORMAT_OPTS"
export HADOOP_OPTS="$HADOOP_OPTS $HIVE_SERVERS_GC_LOG_OPTS"
fi
This takes advantage of $SERVICE, set by Hive, to determine what kind of process is being started and to only set the environment variables when it is HiveServer2 or MetaStore server.
In the above, you can change {{hive_log_dir}} to wherever you want the GC logs to go (you probably want it to go the the same place as server's main logs). You can change the log file naming too.
If you are managing your Hadoop cluster with Apache Ambari, then these changes would be in Hive service > Configs > Advanced > Advanced hive-env > hive-env template. With Ambari, {{hive_log_dir}} will be automatically replaced with the Hive Log Dir defined a few rows above the field.
Now for WebHCat server being manually invoked you can set the following in the environment beforehand:
TIMESTAMP=`date +'%Y%m%d%H%M'`
# GC log location/name prior to .n addition by log rotation
WEBHCAT_GC_LOG_NAME="{{templeton_log_dir}}/webhcat-server-gc.log-$TIMESTAMP"
WEBHCAT_GC_LOG_ENABLE_OPTS="-verbose:gc -Xloggc:$WEBHCAT_GC_LOG_NAME"
WEBHCAT_GC_LOG_ROTATION_OPTS="-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M"
WEBHCAT_GC_LOG_FORMAT_OPTS="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps"
WEBHCAT_GC_LOG_OPTS="$WEBHCAT_GC_LOG_ENABLE_OPTS $WEBHCAT_GC_LOG_ROTATION_OPTS $WEBHCAT_GC_LOG_FORMAT_OPTS"
HADOOP_OPTS="$HADOOP_OPTS $WEBHCAT_GC_LOG_OPTS"
If you are managing your Hadoop cluster with Apache Ambari, then you would do similar to changes in Hive service > Configs > Advanced > Advanced webhcat-env > webhcat-env template. With Ambari, {{templeton_log_dir}} will be automatically replaced with the WebHCat Log Dir defined a couple rows above the field. The one change to the above Bash lines I did was to replace -server- in the log file name with -$$- since I don't know if WebHCat server is the only thing that webhcat-env is run for (this both prevents implying that is the server for sure and prevents conflicts by adding the PID).
In all these cases GC logging will start happening upon server restart. Consider a rolling restart for HiveServer2 and Hive MetaStore.