I have configured the following settings in postgreSQL 13.
logging_collector = on
log_rotation_size='100MB'
log_truncate_on_rotation = on
log_filename ='postgresql-%Y-%m-%d.log'
My issue is when the log file size reached 100MB, it will continue to append on it, I think it is because of the log_filename. Is there anyway I can rename the filename when it reached the log_rotation_size?
I need to set the log_filename with this format (without the time) so that whenever I restart the service, the log will still be in the same log file.
Do I have to run some script or services on the background so that the program is able to monitor the data/logs folder and rename the file when the log file size reaches the limit?
As the documentation says:
However, truncation will occur only when a new file is being opened due to time-based rotation, not during server startup or size-based rotation.
Truncating the log file in your case would mean to lose recent log information, so PostgreSQL won't do it.
I can think of no better way than a cron job that removes the log file when it approaches the limit. Then size based log rotation will create the file again.
Related
I tried a crazy thing about setting attributes to session for a client request.
for(int i=0; i<i+1; i++) {
session.setAttribute("dumpData" + i , new Object().toString());
System.out.println("$$$$$$$$--- Count = " + i);
}
I just want to know how much the session can take. But while looking into log file for the count of attributes, I found a strange thing log file rotation..., this was also in the log file content. First I searched in the entire project for any file(s) which is logging log file rotation..., but there were no files. So I was wondering whether the GlassFish is automatically rotating the log files. If yes can any body explain or show any useful links with explanation.
Yes, Glassfish is automatically rotating the log files.
From Oracle GlassFish Server Administration Guide
Oracle GlassFish Server by default rotates log files when they reach 2
MB in size. However, you can change the default rotation settings. For
example, you can change the file size at which the server rotates the
log file or you can configure a server to rotate log files based on a
time interval. In addition to changing when rotation occurs, you can
also:
Specify the maximum number of rotated files that can accumulate.
By default, Oracle GlassFish Server does not limit the number of
rotated log files that are retained. However, you can set a limit.
After the number of log files reaches this limit, subsequent file
rotations delete the oldest rotated log file.
Rotate the log file manually.
A manual rotation forces the immediate rotation of the target log
file.
Have a look at /GLASSFISH_FOLDER/glassfish/domains/domain1/config/logging.properties. It should be easy to understand. For detailed explanations have a look at the Glassfish Server Administration Guide.
See also:
Turn off Glassfish Log Rotation
We recently decided to enable GC logging for Hadoop MapReduce2 History Server on a number of clusters (exact version varies) as a aid to looking into history-server-related memory and garbage collection problems. While doing this, we want to avoid two problems we know might happen:
overwriting of the log file when the MR2 History server restarts for any reason
the logs using too much disk space, leading to disks getting filled
When Java GC logging starts for a process it seems to replace the content of any file that has the same name. This means that unless you are careful, you will lose the GC logging, perhaps when you are more likely to need it.
If you keep the cluster running long enough, log files will fill up disk unless managed. Even if GC logging is not currently voluminous we want to manage the risk of an unusual situation arising that causes the logging rate to suddenly spike up.
You will need to set some JVM parameters when starting the MapReduce2 History Server, meaning you need to make some changes to mapred-env.sh. You could set the parameters in HADOOP_OPTS, but that would have a broader impact than just the History server, so instead you will probably want to set them in HADOOP_JOB_HISTORYSERVER_OPTS.
Now lets discuss the JVM parameters to include in those.
To enable GC logging to a file, you will need to add -verbose:gc -Xloggc:<log-file-location>.
You need to give the log file name special consideration to prevent overwrites whenever the server is restarted. It seems like you need to have a unique name for every invocation so appending a timestamp seems like the best option. You can include something like `date +'%Y%m%d%H%M'` to add a timestamp. In this example, it is in the form of YYYYMMDDHHMM. In some versions of Java you can put "%t" in your log file location and it will be replaced by the server start up timestamp formatted as YYYY-MM-DD_HH-MM-SS.
Now onto managing use of disk space. I'll be happy if there is a simpler way than what I have.
First, take advantage of Java's built-in GC log file rotation. -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M is an example of enabling this rotation, having up to 10 GC log files from the JVM, each of which is no more than approx 10MB in size. 10 x 10MB is 100MB max usage.
With the GC log file rotation in place with up to 10 files, '.0', '.1', ... '.9' will be added to the file name you gave in Xloggc. .0 will be first and after it reaches .9 it will replace .0 and continue on in a round robin manner. In some versions of Java '.current' will be additionally put on the end of the name of the log file currently being written to.
Due to the unique file naming we apparently have to have to avoid overwrites, you can have 100MB per History server invocation, so this is not a total solution to managing disk space used by the server's GC logs. You will end up with a set of up to 10 GC log files on each server invocation -- this can add up over time. The best solution (under *nix) to that would seem to be to use the logrotate utility (or some other utility) to periodically clean up the GC logs that have not been modified in the last N days.
Be sure to do the math and make sure you will have enough disk space.
People frequently want more details and context in their GC logs than the default, so consider adding in -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps.
Putting this together, you might add something this to mapred-env:
## enable GC logging for MR2 History Server:
TIMESTAMP=`date +'%Y%m%d%H%M'`
# GC log location/name prior to .n addition by log rotation
JOB_HISTORYSERVER_GC_LOG_NAME="{{mapred_log_dir_prefix}}/$USER/mapred-jobhistory-gc.log-$TIMESTAMP"
JOB_HISTORYSERVER_GC_LOG_ENABLE_OPTS="-verbose:gc -Xloggc:$JOB_HISTORYSERVER_GC_LOG_NAME"
JOB_HISTORYSERVER_GC_LOG_ROTATION_OPTS="-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M"
JOB_HISTORYSERVER_GC_LOG_FORMAT_OPTS="-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps"
JOB_HISTORYSERVER_GC_LOG_OPTS="$JOB_HISTORYSERVER_GC_LOG_ENABLE_OPTS $JOB_HISTORYSERVER_GC_LOG_ROTATION_OPTS $JOB_HISTORYSERVER_GC_LOG_FORMAT_OPTS"
export HADOOP_JOB_HISTORYSERVER_OPTS="$HADOOP_JOB_HISTORYSERVER_OPTS $JOB_HISTORYSERVER_GC_LOG_OPTS"
You may find that you already have a reference to HADOOP_JOB_HISTORYSERVER_OPTS so you should replace or add onto that.
In the above, you can change {{mapred_log_dir_prefix}}/$USER to wherever you want the GC logs to go (you probably want it to go the the same place as MapReduce history server logs). You can change the log file naming too.
If you are managing your Hadoop cluster with Apache Ambari, then these changes would be in MapReduce2 service > Configs > Advanced > Advanced mapred-env > mapred-env template. With Ambari, {{mapred_log_dir_prefix}} will be automatically replaced with the Mapreduce Log Dir Prefix defined a few rows above the field.
GC logging will start happening upon server restart the server, so you may need to have a short outage to enable this.
I'm currently running a ruby script which logs its HTTP traffic to stdout. Since I wanted the logs to be persistent, I redirected the output to a log file with ruby ruby_script.rb >> /var/log/ruby_script.log. However, the logs are now getting very large so I wanted to implement logrotate using the following:
"/var/log/ruby_script.log" {
missingok
daily
rotate 10
dateext
}
However, after running logrotate --force -v ruby_script where "ruby_script" is the name of the logrotate.d configuration file, no new file is created for the script to write to, and it writes to the rotated file instead. I'm guessing this behavior happens because the file descriptor that is passed by >> sticks to the file regardless of moving it, and is unrelated to the filename after the first call. Thus, my question is, what is the correct way to achieve the functionality I'm looking for?
Take a look at option copytruncate.
From man logrotate:
copytruncate: Truncate the original log file to zero size in place after creating a copy, instead of moving the old log file and
optionally creating a new one. It can be used when some program cannot be told to close its logfile and thus might
continue writing (appending) to the previous log file forever. Note that there is a very small time slice between
copying the file and truncating it, so some logging data might be lost. When this option is used, the create option
will have no effect, as the old log file stays in place.
Users on our network copy files on the server in a directory called "DropBox" with AFP connection, simply dragging them with the Finder.
A script running on the server checks periodically for the presence of new files inside "DropBox" and then moves them with mv into other directories.
How can the script check if a file is being copied (and wait for the process to complete before moving it away)?
I've tried with fuser filename with no success. If the file copy is issued by a remote machine fuser reports that no process is using the file.
There are two approaches to this problem:
1. flag the filename to tell the file is partial:
When a file is written (moved/copied) into the DropBox .part extension is added to the name. When the writing is over the .part extension is removed.
Other processes operate on files only if the .part extension is not present.
2. check the file size; if it's constant then the write operation is ended (maybe)
This is less accurate and more complicated that (1) but offer a good guess if option 1 is not viable for whatever reason.
For each file in the dropbox the time the size isn't changed is recorded.
When a file increases in size the time value is reset to zero.
When a file is not being written the time values remains constant.
If a file isn't grown for at least (30 seconds, 1 minute, the more time you can afford the better it is of course) then write operation are over and the file can be managed.
This approach fails when a transfer in interrupted.
Is there a way to determine change frequency of a file?
The situation is i have a log file which will be rolling all the time, in that way i can say my application is running .
if it's not writing any then i can say there's some problem.
So instead of using tail and see manually if the logs are rolling , how can i check if the log is rolling programmatically like analysing it for 2 mins and checking if logs are being written?
Is there a way to track the change interval by using stat in some program kinda ???
i mean i can take 2 mins as parameter,at first storing mtime and after 2mins checking with new time and confirming it's changed, but i need to know the frequency kinda like x modifications/time or number.of.lines written/sec kinda
A better idea would be to have inotify, gamin, or FAM notify you when the file has been modified.
On a Unix system, the stat() family of functions will obtain a file's metadata. The st_mtime member of the struct stat structure will give you the time of last modification.
Also on a Unix system, sending a signal 0 to a process will tell you if the process is still alive without affecting the process.