I am just curious about the logging for Elasticsearch, (running 7.17.3) as they appear much larger then what I typically expect for application logging. I have run into issues where the logs have filled my logging partition as they are allows to grow so large.
I get that it duplicates the logs to have them in both JSON and human readable. But why retain 2GB of compressed logs (in each case) that seems a little excessive.
There are a number of other logs (deprecation, slowlog* (I understand these are effectively off until you set the thresholds)) that combined can take up to 30GB of space at capacity according to the log4j 2 config for Elasticsearch.
And then there are the gc.logs (set with the jvm.options file) that can retain another 2 GB.
It looks like default settings could potential retain 36 GB of logs. I am wondering why this is the case and if it is necessary?
Related
I am indexing a large amount of daily data ~160GB per index into elasticsearch. I am facing this case where I need to update almost all the docs in the indices with a small amount of data(~16GB) which is of the format
id1,data1
id1,data2
id2,data1
id2,data2
id2,data3
.
.
.
My update operations start happening at 16000 lines per second and in over 5 minutes it comes down to 1000 lines per second and doesnt go up after that. The update process for this 16GB of data is currently longer than the time it takes for my entire indexing of 160GB to happen
My conf file for the update operation currently looks as follows
output
{
elasticsearch {
action => "update"
doc_as_upsert => true
hosts => ["host1","host2","host3","host4"]
index => "logstash-2017-08-1"
document_id => "%{uniqueid}"
document_type => "daily"
retry_on_conflict => 2
flush_size => 1000
}
}
The optimizations I have done to speed up indexing in my cluster based on the suggestions here https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html are
Setting "indices.store.throttle.type" : "none"
Index "refresh_interval" : "-1"
I am running my cluster on 4 instances of the d2.8xlarge EC2 instances. I have allocated 30GB of heap to each nodes.
While the update is happening barely any cpu is used and the load is very less as well.
Despite everything the update is extremely slow. Is there something very obvious that I am missing that is causing this issue? While looking at the threadpool data I find that the number of threads working on bulk operations are constantly high.
Any help on this issue would be really helpful
Thanks in advance
There are a couple of rule-outs to try here.
Memory Pressure
With 244GB of RAM, this is not terribly likely, but you can still check it out. Find the jstat command in the JDK for your platform, though there are visual tools for some of them. You want to check both your Logstash JVM and the ElasticSearch JVMs.
jstat -gcutil -h7 {PID of JVM} 2s
This will give you a readout of the various memory pools, garbage collection counts, and GC timings for that JVM as it works. It will update every 2 seconds, and print headers every 7 lines. Spending excessive time in the FCT is a sign that you're underallocated for HEAP.
I/O Pressure
The d2.8xlarge is a dense-storage instance, and may not be great for a highly random, small-block workload. If you're on a Unix platform, top will tell you how much time you're spending in IOWAIT state. If it's high, your storage isn't up to the workload you're sending it.
If that's the case, you may want to consider provisioned IOP EBS instances rather than the instance-local stuff. Or, if your stuff will fit, consider an instance in the i3 family of high I/O instances instead.
Logstash version
You don't say what version of Logstash you're using. Being StackOverflow, you're likely to be using 5.2. If that's the case, this isn't a rule-out.
But, if you're using something in the 2.x series, you may want to set the -w flag to 1 at first, and work your way up. Yes, that's single-threading this. But the ElasticSearch output has some concurrency issues in the 2.x series that are largely fixed in the 5.x series.
With elasticsearch version 6.0 we had an exactly same issue of slow updates on aws and the culprit was slow I/O. Same data was upserting on a local test stack completely fine but once in cloud on ec2 stack, everything was dying after an initial burst of speedy inserts lasting only for few minutes.
Local test stack was a low-spec server in terms of memory and cpu but contained SSDs.
s3 stack was EBS volumes with default gp2 300 IOPS.
Converting the volumes to type io1 with 3000 IOPS solved the issue and everything got back on track.
I am using amazon aws elasticsearch service version 6.0 . I need heavy write/insert from serials of json file to the elasticsearch for 10 billion items . The elasticsearch-py bulk write speed is really slow most of time and occasionally high speed write . i tried all kinds of methods , such as split json file to smaller pieces , multiprocess read json files , parallel_bulk insert into elasticsearch , nothing works . Finally , after I upgraded io1 EBS volume , everything goes smoothly with 10000 write IOPS .
We recently decided to enable GC logging for three Apache Hive-related servers: HiveServer2, Hive MetaStore server, and WebHCat server. This is on a number of clusters (exact version varies) as a aid to looking into Hive-related memory and garbage collection problems. While doing this, we want to avoid two problems we know might happen:
overwriting of the log file when a server restarts for any reason
the logs using too much disk space, leading to disks getting filled
When Java GC logging starts for a process it seems to replace the content of any file that has the same name. This means that unless you are careful, you will lose the GC logging, perhaps when you are more likely to need it.
If you keep the cluster running long enough, log files will fill up disk unless managed. Even if GC logging is not currently voluminous we want to manage the risk of an unusual situation arising that causes the logging rate to suddenly spike up.
How you do this will be a little different for HiveServer2 and Hive MetaStore server vs WebHCat server. I'll answer both. In both cases you'll need to set some JVM parameters when starting the servers.
For HiveServer2 and Hive MetaStore server the Java parameters will be set in hive-env.sh. It appears these need to be set in HADOOP_OPTS, but we'll need to be careful to limit the scope of the change to the two server (and not other cases in which hive-env.sh is run).
For WebHCat server, you will also be setting HADOOP_OPTS but there doesn't seem to be a "env.sh" file in the distribution invoked when starting the server, so you'll need to set it in the environment prior to running WebHCat server. However, in Ambari, there is a webhcat-env config group.
Now lets discuss the JVM parameters to include in these cases.
To enable GC logging to a file, you will need to add -verbose:gc -Xloggc:<log-file-location>.
You need to give the log file name special consideration to prevent overwrites whenever the servers are restarted. It seems like you need to have a unique name for every invocation so appending a timestamp seems like the best option. You can include something like `date +'%Y%m%d%H%M'` to add a timestamp. In this example, it is in the form of YYYYMMDDHHMM. In some versions of Java you can put "%t" in your log file location and it will be replaced by the server start up timestamp formatted as YYYY-MM-DD_HH-MM-SS.
Now onto managing use of disk space. I'll be happy if there is a simpler way than what I have.
First, take advantage of Java's built-in GC log file rotation. -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M is an example of enabling this rotation, having up to 10 GC log files from the JVM, each of which is no more than approx 10MB in size. 10 x 10MB is 100MB max usage.
With the GC log file rotation in place with up to 10 files, '.0', '.1', ... '.9' will be added to the file name you gave in Xloggc. .0 will be first and after it reaches .9 it will replace .0 and continue on in a round robin manner. In some versions of Java '.current' will be additionally put on the end of the name of the log file currently being written to.
Due to the unique file naming we apparently have to have to avoid overwrites, you can have 100MB per server process invocation, so this is not a total solution to managing disk space used by the Hive server logs. You will end up with a set of up to 10 GC log files on each server invocation -- this can add up over time. The best solution (under *nix) to that would seem to be to use the logrotate utility (or some other utility) to periodically clean up the GC logs that have not been modified in the last N days.
Be sure to do the math and make sure you will have enough disk space. Note that on some master servers you might be running all three kinds of servers.
People frequently want more details and context in their GC logs than the default, so consider adding in -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps.
Putting this together for HiveServer2 and Hive MetaStore server, you might add something this to hive-env:
# note that $SERVICE is a string saying what it is that is being started
if [[ "$SERVICE" == "hiveserver2" || "$SERVICE" == "metastore" ]]; then
TIMESTAMP=`date +'%Y%m%d%H%M'`
# GC log location/name prior to .n addition by log rotation
HIVE_SERVERS_GC_LOG_NAME="{{hive_log_dir}}/hive-$SERVICE-gc.log-$TIMESTAMP"
HIVE_SERVERS_GC_LOG_ENABLE_OPTS="-verbose:gc -Xloggc:$HIVE_SERVERS_GC_LOG_NAME"
HIVE_SERVERS_GC_LOG_ROTATION_OPTS="-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M"
HIVE_SERVERS_GC_LOG_FORMAT_OPTS="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps"
HIVE_SERVERS_GC_LOG_OPTS="$HIVE_SERVERS_GC_LOG_ENABLE_OPTS $HIVE_SERVERS_GC_LOG_ROTATION_OPTS $HIVE_SERVERS_GC_LOG_FORMAT_OPTS"
export HADOOP_OPTS="$HADOOP_OPTS $HIVE_SERVERS_GC_LOG_OPTS"
fi
This takes advantage of $SERVICE, set by Hive, to determine what kind of process is being started and to only set the environment variables when it is HiveServer2 or MetaStore server.
In the above, you can change {{hive_log_dir}} to wherever you want the GC logs to go (you probably want it to go the the same place as server's main logs). You can change the log file naming too.
If you are managing your Hadoop cluster with Apache Ambari, then these changes would be in Hive service > Configs > Advanced > Advanced hive-env > hive-env template. With Ambari, {{hive_log_dir}} will be automatically replaced with the Hive Log Dir defined a few rows above the field.
Now for WebHCat server being manually invoked you can set the following in the environment beforehand:
TIMESTAMP=`date +'%Y%m%d%H%M'`
# GC log location/name prior to .n addition by log rotation
WEBHCAT_GC_LOG_NAME="{{templeton_log_dir}}/webhcat-server-gc.log-$TIMESTAMP"
WEBHCAT_GC_LOG_ENABLE_OPTS="-verbose:gc -Xloggc:$WEBHCAT_GC_LOG_NAME"
WEBHCAT_GC_LOG_ROTATION_OPTS="-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M"
WEBHCAT_GC_LOG_FORMAT_OPTS="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps"
WEBHCAT_GC_LOG_OPTS="$WEBHCAT_GC_LOG_ENABLE_OPTS $WEBHCAT_GC_LOG_ROTATION_OPTS $WEBHCAT_GC_LOG_FORMAT_OPTS"
HADOOP_OPTS="$HADOOP_OPTS $WEBHCAT_GC_LOG_OPTS"
If you are managing your Hadoop cluster with Apache Ambari, then you would do similar to changes in Hive service > Configs > Advanced > Advanced webhcat-env > webhcat-env template. With Ambari, {{templeton_log_dir}} will be automatically replaced with the WebHCat Log Dir defined a couple rows above the field. The one change to the above Bash lines I did was to replace -server- in the log file name with -$$- since I don't know if WebHCat server is the only thing that webhcat-env is run for (this both prevents implying that is the server for sure and prevents conflicts by adding the PID).
In all these cases GC logging will start happening upon server restart. Consider a rolling restart for HiveServer2 and Hive MetaStore.
We recently decided to enable GC logging for Hadoop MapReduce2 History Server on a number of clusters (exact version varies) as a aid to looking into history-server-related memory and garbage collection problems. While doing this, we want to avoid two problems we know might happen:
overwriting of the log file when the MR2 History server restarts for any reason
the logs using too much disk space, leading to disks getting filled
When Java GC logging starts for a process it seems to replace the content of any file that has the same name. This means that unless you are careful, you will lose the GC logging, perhaps when you are more likely to need it.
If you keep the cluster running long enough, log files will fill up disk unless managed. Even if GC logging is not currently voluminous we want to manage the risk of an unusual situation arising that causes the logging rate to suddenly spike up.
You will need to set some JVM parameters when starting the MapReduce2 History Server, meaning you need to make some changes to mapred-env.sh. You could set the parameters in HADOOP_OPTS, but that would have a broader impact than just the History server, so instead you will probably want to set them in HADOOP_JOB_HISTORYSERVER_OPTS.
Now lets discuss the JVM parameters to include in those.
To enable GC logging to a file, you will need to add -verbose:gc -Xloggc:<log-file-location>.
You need to give the log file name special consideration to prevent overwrites whenever the server is restarted. It seems like you need to have a unique name for every invocation so appending a timestamp seems like the best option. You can include something like `date +'%Y%m%d%H%M'` to add a timestamp. In this example, it is in the form of YYYYMMDDHHMM. In some versions of Java you can put "%t" in your log file location and it will be replaced by the server start up timestamp formatted as YYYY-MM-DD_HH-MM-SS.
Now onto managing use of disk space. I'll be happy if there is a simpler way than what I have.
First, take advantage of Java's built-in GC log file rotation. -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M is an example of enabling this rotation, having up to 10 GC log files from the JVM, each of which is no more than approx 10MB in size. 10 x 10MB is 100MB max usage.
With the GC log file rotation in place with up to 10 files, '.0', '.1', ... '.9' will be added to the file name you gave in Xloggc. .0 will be first and after it reaches .9 it will replace .0 and continue on in a round robin manner. In some versions of Java '.current' will be additionally put on the end of the name of the log file currently being written to.
Due to the unique file naming we apparently have to have to avoid overwrites, you can have 100MB per History server invocation, so this is not a total solution to managing disk space used by the server's GC logs. You will end up with a set of up to 10 GC log files on each server invocation -- this can add up over time. The best solution (under *nix) to that would seem to be to use the logrotate utility (or some other utility) to periodically clean up the GC logs that have not been modified in the last N days.
Be sure to do the math and make sure you will have enough disk space.
People frequently want more details and context in their GC logs than the default, so consider adding in -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps.
Putting this together, you might add something this to mapred-env:
## enable GC logging for MR2 History Server:
TIMESTAMP=`date +'%Y%m%d%H%M'`
# GC log location/name prior to .n addition by log rotation
JOB_HISTORYSERVER_GC_LOG_NAME="{{mapred_log_dir_prefix}}/$USER/mapred-jobhistory-gc.log-$TIMESTAMP"
JOB_HISTORYSERVER_GC_LOG_ENABLE_OPTS="-verbose:gc -Xloggc:$JOB_HISTORYSERVER_GC_LOG_NAME"
JOB_HISTORYSERVER_GC_LOG_ROTATION_OPTS="-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M"
JOB_HISTORYSERVER_GC_LOG_FORMAT_OPTS="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps"
JOB_HISTORYSERVER_GC_LOG_OPTS="$JOB_HISTORYSERVER_GC_LOG_ENABLE_OPTS $JOB_HISTORYSERVER_GC_LOG_ROTATION_OPTS $JOB_HISTORYSERVER_GC_LOG_FORMAT_OPTS"
export HADOOP_JOB_HISTORYSERVER_OPTS="$HADOOP_JOB_HISTORYSERVER_OPTS $JOB_HISTORYSERVER_GC_LOG_OPTS"
You may find that you already have a reference to HADOOP_JOB_HISTORYSERVER_OPTS so you should replace or add onto that.
In the above, you can change {{mapred_log_dir_prefix}}/$USER to wherever you want the GC logs to go (you probably want it to go the the same place as MapReduce history server logs). You can change the log file naming too.
If you are managing your Hadoop cluster with Apache Ambari, then these changes would be in MapReduce2 service > Configs > Advanced > Advanced mapred-env > mapred-env template. With Ambari, {{mapred_log_dir_prefix}} will be automatically replaced with the Mapreduce Log Dir Prefix defined a few rows above the field.
GC logging will start happening upon server restart the server, so you may need to have a short outage to enable this.
We recently decided to enable GC logging for Hadoop YARN ResourceManager and ApplicationTimeline servers on a number of clusters (exact version varies) as a aid to looking into YARN-related memory and garbage collection problems. While doing this, we want to avoid two problems we know might happen:
overwriting of the log file when a YARN RM or AT server restarts for any reason
the logs using too much disk space, leading to disks getting filled
When Java GC logging starts for a process it seems to replace the content of any file that has the same name. This means that unless you are careful, you will lose the GC logging, perhaps when you are more likely to need it.
If you keep the cluster running long enough, log files will fill up disk unless managed. Even if GC logging is not currently voluminous we want to manage the risk of an unusual situation arising that causes the logging rate to spike up.
You will need to set some JVM parameters when starting the YARN servers, meaning you need to make some changes to yarn-env.sh. You could set the parameters in YARN_OPTS, but that would have a broader impact than just the ResourceManager and ApplicationTimeline servers, so instead you will probably want to set then in both YARN_RESOURCEMANAGER_OPTS and YARN_TIMELINESERVER_OPTS.
Now lets discuss the JVM parameters to include in those.
To enable GC logging to a file, you will need to add -verbose:gc -Xloggc:<log-file-location>.
You need to give the log file name special consideration to prevent overwrites whenever the server is restarted. It seems like you need to have a unique name for every invocation so appending a timestamp seems like the best option. You can include something like `date +'%Y%m%d%H%M'` to add a timestamp. In this example, it is in the form of YYYYMMDDHHMM. In some versions of Java you can put "%t" in your log file location and it will be replaced by the server start up timestamp formatted as YYYY-MM-DD_HH-MM-SS.
Now onto managing use of disk space. I'll be happy if there is a simpler way than what I have.
First, take advantage of Java's built-in GC log file rotation. -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M is an example of enabling this rotation, having up to 10 GC log files from the JVM, each of which is no more than approx 10MB in size. 10 x 10MB is 100MB max usage.
With the GC log file rotation in place with up to 10 files, '.0', '.1', ... '.9' will be added to the file name you gave in Xloggc. .0 will be first and after it reaches .9 it will replace .0 and continue on in a round robin manner. In some versions of Java '.current' will be additionally put on the end of the name of the log file currently being written to.
Due to the unique file naming we apparently have to have to avoid overwrites, you can have 100MB per RM or AT server process invocation, so this is not a total solution to managing disk space used by the YARN GC logs. You will end up with a set of up to 10 GC log files on each server invocation -- this can add up over time. The best solution (under *nix) to that would seem to be to use the logrotate utility (or some other utility) to periodically clean up the GC logs that have not been modified in the last N days.
Be sure to do the math and make sure you will have enough disk space. Note that you might be running a ResourceManager and a ApplicationTimeline on the same master server.
People frequently want more details and context in their GC logs than the default, so consider adding in -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps.
Putting this together, you might add something this to yarn-env:
# this function takes the name of a YARN component as input and returns the JVM options
# to enable GC logging for the component. The option string is set in the variable named
# as the second arg
function yarn_gc_log_opts_for_component()
{
# get function args
local component_name=$1
local __resultvar=$2
# calculate GC log name
local timestamp_str=`date +'%Y%m%d%H%M'`
local gc_log_name="{{yarn_log_dir_prefix}}/$USER/yarn-${component_name}-gc.log-${timestamp_str}"
# calculate GC logging options for enablement, rotation, and format
local gc_log_enable_opts="-verbose:gc -Xloggc:$gc_log_name"
local gc_log_rotation_opts="-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M"
local gc_log_format_opts="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps"
# combing these options and return the result
local gc_log_opts="$gc_log_enable_opts $gc_log_rotation_opts $gc_log_format_opts"
eval $__resultvar="'$gc_log_opts'"
}
# get options for GC logging for YARN ResourceManager servers and put them into use
yarn_gc_log_opts_for_component "resourcemanager" YARN_RESOURCEMANAGER_GC_LOG_OPTS
export YARN_RESOURCEMANAGER_OPTS="$YARN_RESOURCEMANAGER_OPTS $YARN_RESOURCEMANAGER_GC_LOG_OPTS"
# get options for GC logging for YARN AT servers and put them into use
yarn_gc_log_opts_for_component "timelineserver" YARN_TIMELINESERVER_GC_LOG_OPTS
export YARN_RESOURCEMANAGER_OPTS="$YARN_RESOURCEMANAGER_OPTS $YARN_RESOURCEMANAGER_GC_LOG_OPTS"
In the above, you can change {{yarn_log_dir_prefix}}/$USER to wherever you want the GC logs to go (you probably want it to go the the same place as YARN RM and AT server logs). You can change the log file naming too.
If you are managing your Hadoop cluster with Apache Ambari, then these changes would be in YARN service > Configs > Advanced > Advanced yarn-env > yarn-env template. With Ambari, {{yarn_log_dir_prefix}} will be automatically replaced with the YARN Log Dir Prefix defined a few rows above the field.
GC logging will start happening upon server restart.
I have been started solr for long time approx 2 weeks then I saw that Solr ate around 22 GB from 28 GB RAM of my Server.
While checking status of Solr, using bin/solr -i it throws -bash: cannot create temp file for here-document: No space left on device
I stopped the Solr, and restarted the solr. It is working fine.
What's the problem actually. Didn't get?
And what is the solution for that?
I never want that Solr gets stop/halt while running.
First you should check the space on your file system. For example using df -h. Post the output here.
Is there any mount-point without free space?
2nd: find out the reason, why there is no space left. Your question handles two different thing: no space left on file system an a big usage of RAM.
Solr stores two different kind of data: the search index an the data himself.
Storing the data is only needed, if you like to output the documents after finding them in index. For example if you like to use highlighting. So take a look at your schema.xml an decide for every singe field, if it must be stored or if "indexing" the field is enough for your needs. Use the stored=true parameter for that.
Next: if you rebuild the index: keep in mind, that you need double space on disc during index rebuild.
You also could think about to move your index/data files to an other disk.
If you have solved you "free space" problem on disc, so you probably don't have an RAM issue any more.
If there is still a RAM problem, please post you java start parameter. There you can define, how much RAM is available for Solr. Solr needs a lot of virtual RAM, but an moderate size of physical RAM.
And: you could post the output of your logfile.