I am trying to write a shell script , which will write the output of another script in a file and it will keep writing to that upto a certain point and then it will overwrite the file so that file size will remain within a well bounded range.
while true
do
./runscript.sh > test.txt
sleep 1
done
I have tried to use infinite loop and sleep so that it will keep overwrite that file.
But, it shows a different behaviour. Till the point command is running , the filesize keeps on increasing. But, when i stop the command, the file size get reduce.
How can i keep overwriting the same file and maintain the file size along with it.
use truncate -s <size> <file> to shrink the file when its size is out of your boundary
I will do with below script
#!/bin/sh
Logfile=test.txt
minimumsize=100000 # define the size you want
actualsize=$(wc -c <"$Logfile")
if [[ $actualsize -ge $minimumsize ]]; then
rm -rf "$Logfile"
sh ./runscript.sh >> test.txt
else
#current_date_time="`date +%Y%m%d%H%M%S`"; #add this to runscript.sh to track when it was written
#echo "********Added at :$current_date_time ********" #add this to runscript.sh to track when it was written
sh ./runscript.sh >> test.txt
fi
I can try with the option for generating the new file once the old one
is full. … How can make the
script to generate the new file and write to it.
The following script, let's call it chop.sh, does that; you use it by feeding the output to it, specifying the desired file size and name as arguments, e. g. ./runscript.sh|chop.sh 999999 test.txt.
File=${2?usage: $0 Size File}
Size=$1
while
set -- `ls -l "$File" 2>/dev/null` # 5th column is file size
[ "$5" -lt "$Size" ] || mv "$File" "$File"-old
read -r && echo "$REPLY" >>"$File"
do :
done
The old (full) file would then be named test.txt-old.
Related
Below is a bash script to move files around and rename them. The problem is it doesn't work when there is more than one file in the directory. I'm assuming because the last parameter in the mv command is a file. Any suggestions?
'#!/bin/bash'
'INPUTDIR="/home/southern-uniontn/S001007420"'
'OUTPUTDIR="/mnt/edi-06/southern-uniontn/flats-in"'
'BACKUPDIR="/backup/southern-uniontn/S001007420"'
YEAR=`date +%Y`
MONTH=`date +%m`
DAY=`date +%d`
HOUR=`date +%H`
MINUTE=`date +%M`
######## Do some error checking #########
# Does backup dir exist?
if [ ! -d $BACKUPDIR/$YEAR ]
then
mkdir $BACKUPDIR/$YEAR
fi
if [ ! -d $BACKUPDIR/$YEAR/$MONTH ]
then
mkdir $BACKUPDIR/$YEAR/$MONTH
fi
if [ ! -d $BACKUPDIR/$YEAR/$MONTH/$DAY ]
then
mkdir $BACKUPDIR/$YEAR/$MONTH/$DAY
fi
if [[ $(find $INPUTDIR -type f | wc -l) -gt 0 ]];
then
###### Rename the file, move it to Backup, then copy to the Output Directory #####
for f in $INPUTDIR/*
do
echo "`date` - Move recurring txt flat file to BackupDir for Union TN from Southern"
mv $INPUTDIR/* $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt
sleep 2
echo "`date` - Copy backup file to the Union TN Output Directory"
cp $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt $OUTPUTDIR/
done;
fi
Some notes:
Get out of the habit of using ALLCAPS variable names, leave those as reserved
by the shell. One day you'll write PATH=something and then wonder
why your script is
broken.
mkdir -p can create parent directories, and will not error if the dir already exists
store the filenames in an array. Then the shell does not have to duplicate
the work, and you don't need to count how many there are: if there are no
files, the loop has zero iterations
if you want to keep the same directory hierarchy in the outputdir,
you need to do that by hand.
use read to get the date parts
with bash v4.2+, printf can be used instead of calling out to date
use magic value "-1" to mean "now".
printf '%(%Y-%m-%d)T\n' -1 prints "2021-10-25" (as of the day I write this)
This is, I think, what you want:
#!/bin/bash
inputdir='/home/southern-uniontn/S001007420'
outputdir='/mnt/edi-06/southern-uniontn/flats-in'
backupdir='/backup/southern-uniontn/S001007420'
read year month day hour minute < <(printf '%(%Y %m %d %H %M)T\n' -1)
# create backup dirs if not exists
date_dir="$year/$month/$day"
mkdir -p "$backupdir/$date_dir"
mkdir -p "$outputdir/$date_dir"
mapfile -t files < <(find $inputdir -type f)
for f in "${files[#]}"
do
###### Rename the file, move it to Backup, then copy to the Output Directory #####
backup_file="UnionTN-S001007420-$year$month$day-$hour$minute.txt"
printf '%(%c)T - Move recurring txt flat file to backupdir for Union TN from Southern\n' -1
mv "$f" "$backupdir/$date_dir/$backup_file"
printf '%(%c)T - Copy backup file to the Union TN Output Directory\n' -1
cp "$backupdir/$date_dir/$backup_file" "$outputdir/$date_dir/$backup_file"
done
When using a glob with mv, the target must be an existing directory, and all matching files will be moved inside that directory.
In your case,
mv $INPUTDIR/* $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt
tells mv to move all file inside the $INPUTDIR/* directory to a directory named $BACKUPDIR/$YEAR/$MONTH/$DAY/UnionTN-S001007420-$YEAR$MONTH$DAY-$HOUR$MINUTE.txt.
I'm not sure what you're trying to do, but I hope this help.
Some more advice you could use:
Don't put the shebang (the first line beginning with "#") and the first three variable declarations inside single-quotes.
Some argue it is more portable and better to write /usr/bin/env bash instead of /bin/bash in the shebang
if [ CONDITION ] /then ACTION /fi statements can be simplified by writing [ CONDITION ] && ACTION
You reduce your likely hood of encountering unexpected behaviour when double-quoting your strings and variable (i.e. write "${year}/${month}/" instead of $year/$month.
No need to call mkdir a, followed by mkidr a/b, then mkdir a/b/c and so on, you can just call mkdir -p a/b/c. The p flag tells mkdir to create parent directories if they don't already exist.
It is unnecessary to validate the existence of a directory before calling mkdir since mkdir already validates that for you.
As pointed out by commenters, all-caps variables are conventions for special POSIX related variables. You should use another type of casing.
You could use date to do the formatting for you: date +%Y/%m/%d will print 2021/10/25
Strings without interpolation can have single-quotes.
(Optional, prevent undesired behaviors) Put set -e at the beginning of your scripts, after the shebang, to tell bash to halt if an error is encountered
And finally, use man <command_name> for built-in documentation!
My requirement is to replace file only when it is not being accessed. I have following snippet:
if [ -f file ]
then
while true
do
if [ -n "$(fuser "file")" ]
then
echo "file is in use..."
else
echo "file is free..."
break
fi
done
fi
{
flock -x 3
mv newfile file
} 3>file
But I have a doubt that I am not handling concurrency properly. Please give some insights and possible way to achieve this.
Thanks.
My requirement is to replace file only when it is not being accessed.
Getting requirements right can be hard. In case your actual requirement is the following, you can boil down the whole script to just one command.
My guess on the actual requirement (not as strict as the original):
Replace file without disturbing any programs reading/writing file.
If this is the case, you can use a very neat behavior: In Unix-like systems file descriptors always point to the file (not path) for which they where opened. You can move or even delete the corresponding path. See also How do the UNIX commands mv and rm work with open files?.
Example:
Open a terminal and enter
i=1; while true; do echo $((i++)); sleep 1; done > file &
tail -f file
The first command writes output to file and runs in the background. The second command reads the file and continues to print its changing content.
Open another terminal and move or delete file, for instance with
mv file file2
echo overwritten > otherFile
mv otherFile file2
rm file2
echo overwritten > file
echo overwritten > file2
While executing these commands have a look at the output of tail -f in the first terminal – it won't be affected by any of these commands. You will never see overwritten.
Solution For New Requirement:
Because of this behavior you can replace the whole script with just one mv command:
mv newfile file
Consider lsof.
mvWhenClear() {
while [[ -f "$1" ]] && lsof "$1"
do sleep $delay
done
mv "$1" "$2" # still allows race condition
}
I am running quantum chemical calculations by providing the command molcas -f file.input. I now have need for putting the molcas -f into a script that also tails the last 100 lines of the generated file.log, for me to quickly confirm that everything finished the way it's supposed to. So I want to run the script run.sh:
#!/bin/bash
molcas -f [here read the file.input from command line]
tail -100 [here read the file.log]
The question is how I can make the script read the argument I give, and then find on its own the output file (which has the same filename, but with a different extension).
Follow-up
Say I have a bunch of numbered files file-1, file-2, ..., file-n. I would save time if I instead of running
./run.sh file-1.input file-1.log
I run
./run.sh n n
or
./run.sh n.input n.log
assuming that the actual filename and placement of the number n is given in the script. Can that be done?
With this code:
#!/bin/bash
molcas -f "$1"
tail -100 "$2"
You will need to execute the script run.sh as follows:
./run.sh file.input file.log
to be hornest I have/had no clue over molcas, so I jumed to this side to get basic understandings.
The syntax shoould look like this ...
#!/bin/bash
# waiting for input
read -p "Enter a filename (sample.txt): " FILE
# checking for existing file
if [ -e "$FILE" ]; then
read -p "Enter a command for moculas: " CMD
else
echo "Sorry, \"${FILE}\" was not found. Exit prgramm."
exit 1
fi
# I am not sure how this command works.
# maybe you have to edit this line by your self.
molcas $FILE -f "$CMD"
# checking for programm-errors
ERRNO=$?
if [ "$ERRNO" != "" ] && [ "$ERRNO" -gt 0 ]; then
echo "Molcas returned an error. Number: ${ERRNO}"
exit 1
fi
# cuts off the fileending (For example: sample.txt gets sample)
FILENAME="${FILE%.*}"
# checking other files
ERRFILE="${FILENAME}.err"
tail -n 100 $ERRFILE
LOGFILE="${FILENAME}.log"
tail -n 100 $LOGFILE
exit 0
I would have posted more, but its not clear what to do with this data.
Hope this helps a bit.
I have the following bash script, that I launch using the terminal.
dataset_dir='/home/super/datasets/Carpets_identification/data'
dest_dir='/home/super/datasets/Carpets_identification/augmented-data'
# if dest_dir does not exist -> create it
if [ ! -d ${dest_dir} ]; then
mkdir ${dest_dir}
fi
# for all folder of the dataset
for folder in ${dataset_dir}/*; do
curr_folder="${folder##*/}"
echo "Processing $curr_folder category"
# get all files
for item in ${folder}/*; do
# if the class dir in dest_dir does not exist -> create it
if [ ! -d ${dest_dir}/${curr_folder} ]; then
mkdir ${dest_dir}/${curr_folder}
fi
# for each file
if [ -f ${item} ]; then
# echo ${item}
filename=$(basename "$item")
extension="${filename##*.}"
filename=`readlink -e ${item}`
# get a certain number of patches
for i in {1..100}
do
python cropper.py ${filename} ${i} ${dest_dir}
done
fi
done
done
Given that it needs at least an hour to process all the files.
What happens if I change the '100' with '1000' in the last for loop and launch another instance of the same script?
Will the first process count to 1000 or will continue to count to 100?
I think the file will be readonly when a bash process executes it. But you can force the change. The already running process will count to its original value, 100.
You have to take care about the results. You are writing in the same output directory and have to expect side effects.
"When you make changes to your script, you make the changes on the disk(hard disk- the permanent storage); when you execute the script, the script is loaded to your memory(RAM).
(see https://askubuntu.com/questions/484111/can-i-modify-a-bash-script-sh-file-while-it-is-running )
BUT "You'll notice that the file is being read in at 8KB increments, so Bash and other shells will likely not load a file in its entirety, rather they read them in in blocks."
(see https://unix.stackexchange.com/questions/121013/how-does-linux-deal-with-shell-scripts )
So, in your case, all your script is loaded in the RAM memory by the script interpretor, and then executed. Meaning that if you change the value, then execute it again, the first instance will still have the "old" value.
I needed to convert several pnm image files to jpeg with pnmtojpeg. So I used this script, which I named 'pnm2jpg':
for f in *.pnm;
do pnmtojpeg -quality=85 "$f" > "${f%.pnm}.jpg";
done
This works very nicely. However, I would like to adapt it further so that it can be used for a single file as well.
In other words, if no files are specified in the command line, then process all the files.
$ pnm2jpg thisfile.pnm # Process only this file.
$ pnm2jpg # Process all pnm files in the current directory.
Your insight is greatly appreciated- Thank you.
Something like:
#!/bin/bash
if [[ -z "$1" ]]; then
for f in *.pnm; do
pnmtojpeg -quality=85 "$f" > "${f%.pnm}.jpg"
done
else
pnmtojpeg -quality=85 "$1" > "${1%.pnm}.jpg"
fi
If you execute pnm2jpg without an argument the if block is processed.
if you execute pnm2jpg thisfile.pnm the else block is processed.