I have a script which runs every 15 minutes (checks if a new file is present and emails it) [00, 15, 30, 45]
Sometimes there are files being transferred to this location at the very last second (HH:MM:59seconds)
E.g.
Let's say file modified timestamp is (16:14:59)
Script next run is (16:15:00)
The script is not being able to pick that file so is never sent via email
Part of the code:
check=`find . -mmin -15`
If you schedule your script to start at 16:15:00.0000000, it will not start and finish in that exact nanosecond. Likewise, disk writes are not atomic, and starting to write a file at 16:14:59.9999999 does not mean it'll finish and have a timestamp before 16:15:00.
It will take hundreds or maybe thousands of milliseconds before all the OS processes are executed, initialized and scheduled, and all the disk reads finish. The exact time it takes to finish this is unpredictable.
That means that one of your jobs may run at 16:15:00.13 while the next runs at 16:30:01.22, leaving a second long gap where you can lose files.
Instead of checking "modified in the last 15 minutes" you should check for "modified since the last run of the script" (keeping track of the last processed filename or modification date), or at the very least, "modified on or after 16:15:00 but strictly before 16:30:00".
Related
I'm using Tailfile processor to fetch logs from a cluster(3 nodes) scheduled to run every minute. The log file name changes for every hour
I was confused on which Tailing mode should I use . If I use Single File it is not fetching the new file generated after 1 hour. If I use the multifile, It is fetching the file after 3rd minute of file name change which is increasing the size of the file. what should be the rolling filename for my file and which mode should I use.
Could you please let me know. Thank you
Myfilename:
retrieve-11.log (generated at 11:00)- this is removed but single file mode still checks for this file
after 1 hour retrieve-12.log (generated at 12:00)
My Processor Confuguration:
Tailing mode: Multiple Files
File(s) to Tail: retrieve-${now():format("HH")}.log
Rolling Filename Pattern: ${filename}.*.log
Base Directory: /ext/logs
Initial Start Position: Beginning of File
State Location: Local
Recursive lookup: false
Lookup Frequency: 10 minutes
Maximum age: 24 hours
Sounds like you aren't really doing normal log file rolling. That would be, for example, where you write to logfile.log then after 1 day, you move logfile.log to be logfile.log.1 and then write new logs to a new, empty logfile.log.
Instead, it sounds like you are just writing logs to a different file based on the hour. I assume this means you overwrite each file every 24h?
So something like this might work?
EDIT:
So given that you are doing the following:
At 10:00, `retrieve-10.log` is created. Logs are written here.
At 11:00, `retrieve-11.log` is created. Logs are now written here.
At 11:10, `retrieve-10.log` is moved.
TailFile is only run every 10 minutes.
Then targeting a file based on the hour won't work. At 10:00, your tailFile only reads retrieve-10.log. At 11:00 your tailFile only reads retrieve-11.log. So worst case, you miss 10 minuts of logs between 10:50 and 11:00.
Given that another process is cleaning up the old files, there isn't going to be a back log of old files to worry about. So it sounds like there's no need to set the hour specifically.
tailing mode: multiple files
files to tail: /path/retrieve-*.log
With this, at 10:00, tailFile tails retrieve-9.log and retrieve-10.log. At 10:10, retrieve-9.log is removed and it tails retrieve-10.log. At 11:00 it tails retrieve-10.log and retrieve-11.log. At 11:10, retrieve-10.log is removed and it tails retrieve-11.log. Etc.
Hello datastage savvy people here.
Two days in a row, the same single datastage job failed (not stopping at all).
The job tries to create a hashfile using the command /logiciel/iis9.1/Server/DSEngine/bin/mkdbfile /[path to hashfile]/[name of hashfile] 30 1 4 20 50 80 1628
(last trace in the log)
Something to consider (or maybe not ? ) :
The [name of hashfile] directory exists, and was last modified at the time of execution) but the file D_[name of hashfile] does not.
I am trying to understand what happened to prevent the same incident to happen next run (tonight).
Previous to this day, this job is in production since ever, and we had no issue with it.
Using Datastage 9.1.0.1
Did you check the job log to see if captured an error? When a DataStage job executes any system command via command execution stage or similar methods, the stdout of the called job is captured and then added to a message in the job log. Thus if the mkdbfile command gives any output (success messages, errors, etc) it should be captured and logged. The event may not be flagged as an error in the job log depending on the return code, but the output should be there.
If there is no logged message revealing cause of directory non-create, a couple of things to check are:
-- Was target directory on a disk possibly out of space at that time?
-- Do you have any Antivirus software that scans directories on your system at certain times? If so, it can interfere with I/o. If it does a scan at same time you had the problem, you may wish to update the AV software settings to exclude the directory you were writing dbfile to.
I have a shell script that collects some data and send it to destination. Part of the data should be copied every 5 minutes and other every 20 minutes. How can this be achieved in a single script? As of now i'm collecting the data every 5 minutes by scheduling with cron.
Best practice would be to use to separate files with two different cron entries. If you need to reutilize part of your code consider using functions.
If you still want to do it in only one file, you should run it every 5 minutes and on each run check whether or not you should execute the other part (every 20 min) or not.
modulo=$((`date +%_M)` % 20))
//do whatever has to be done every 5min
[...]
//check for modulo of current minute / 20
if [ $modulo -eq 0 ]; then
echo Current minute is `date +%_M)`, must execute part 2
//whatever has to be done every 20min
else
//do nothing
fi;
The reason why the variable modulo is defined in the first line is because what has to be done every 5min can potentially take longer than 1min to execute so by the time it is done minute is no longer 20 but 21. Defining the variable before is a safeguard to overcome this.
I have created a dialplan which takes the call and save the start and end time of the call. I used ${CDR(start)} to get the start time of the call but when I used ${STRFTIME(${EPOCH},,%Y-%m-%d %H:%M:%S)} instead it gave me a 10 or more seconds difference as compared to ${CDR(start)}.Part of my code:
same => n,NoOp(------${CDR(start)}----${STRFTIME(${EPOCH},,%Y-%m-%d %H:%M:%S)}--)
the above line is second line in my dialplan.From the docs ${CDR(start)} give start time of call and ${STRFTIME(${EPOCH},,%Y-%m-%d %H:%M:%S)} gives current time .But I place the line as second line of my dialplan so its nearly start of call so both ${CDR(start)} and ${STRFTIME(${EPOCH},,%Y-%m-%d %H:%M:%S)} should be equal approximately but the output was like
NoOp(------2015-10-25 12:30:10----2015-10-25 12:30:21--)
Why there is so much diffrence in both? and also I dont know why the value of ${CDR(end)} is empty.
${CDR(start)} - is a time of call start.
${STRFTIME(${EPOCH},,%Y-%m-%d %H:%M:%S)} - current time, when this command was executed.
If you use EPOCH variable after hangup, it may points to the end of call.
${CDR(end)} can be empty if call is active now and will be filled after hangup.
About CDR variables: https://wiki.asterisk.org/wiki/display/AST/CDR+Variables
${CDR(start)} is not time of current call start, but time of current cdr start. It can be changed after transfer from queue, ResetCDR, ForkCDR command
EPOCH always give current linux time. After hangup it can show end of call, but also can show different time if processing was long enoght.
I need to be able to determine if a file is older than one hour and if so, email, text, or page to notify that our process has started. I need any file that ends with .eob
You can use DateTime date = File.GetCreationTime(str_Path);
str_path is the path of the .eob file.
Then compare date variable with DateTime.Now()
and if its greater than 1 hour send your email or whatever you want to do next
EDIT
To accomplish this you will have to make a small c#/vb .Net application that will, when compiled, make a .exe. Then you can schedukle a job in windows to execute your small app.