I need to control files in a folder... The script has to wait the file until it exists...
These files have the name... The format is file_d01_YYYY-MM-DD_HH:00:00. For example:
file_d01_2018-11-12_00:00:00
file_d01_2018-11-12_01:00:00
And so on, for 7 days ahead.
!/bin/bash
ZZ=`date +%k`
date=$(date +%Y%m%d)
if [[ $ZZ -ge 2 ]] && [[ $ZZ -lt 14 ]] ; then #03:45 UTC
ZZ=00
PARAM=`date +%Y%m%d`$ZZ
PARAM2=`date +%Y-%m-%d`
elif [[ $ZZ -ge 14 ]] && [[ $ZZ -lt 23 ]] ; then #15:45 UTC
ZZ=12
PARAM=`date +%Y%m%d`$ZZ
PARAM2=`date +%Y-%m-%d`
fi
rundir=/home/$PARAM/wrfv3
dir=/home/$PARAM
data=$(date +%Y-%m-%d)
data1=$(date -d "1 day" +%Y-%m-%d)
data2=$(date -d "2 day" +%Y-%m-%d)
data3=$(date -d "3 day" +%Y-%m-%d)
data4=$(date -d "4 day" +%Y-%m-%d)
data5=$(date -d "5 day" +%Y-%m-%d)
data6=$(date -d "6 day" +%Y-%m-%d)
days=( "$data" "$data1" "$data2" "$data3" "$data4" "$data5" "$data6" ) #array of days
values=( {00..23} ) #array of 24 hours
echo ${#values[#]}
# Here, using to loops, I check if files exist...for every day and hour
for day in "${days[#]}"; do
for value in "${values[#]}"; do
echo file_d01_${day}_${value}:00:00
while [ ! -f $rundir/file_d01_2018-11-15_20:00:00 ] # while file doesn't exist...wait...and repeat checking till it exists...
do
echo "waiting for the file"
sleep 10
done
echo "file exists"
sleep 5
done
done
I receive always "waiting for the file"...and they exist... where is the problema in the code?
You should add the double quotes "" to protect the path. It's a good practice. Also bash expansion escapes the : character, so maybe it is an issue in your context (not in the one i did the test).
while [ ! -e "$rundir/file_d01_2018-11-15_20:00:00" ]
I would suggest to follow those steps:
Protect the path with double quotes "" (not simple ones, otherwise $rundir won't be expanded)
Write echo "waiting for the file $rundir/file_d01_2018-11-15_20:00:00" to see what path you're testing
Additionally, use -e to see any changes (-e checks for a path existence, not only a regular file one)
Note: the brackets [ ] invokes in fact test. So, man test will give you the operators you can use and their meanings. Also nowadays bash has double brackets [[ ]] as built-in operators, more powerful, which can be used instead.
The code in the question contains:
echo file_d01_${day}_${value}:00:00
while [ ! -f $rundir/file_d01_2018-11-15_20:00:00 ]
It's calculating a file name, echoing it, and then checking for a (probably different) fixed, unchanging, file name. It should check for the calculated file name. For example:
file=file_d01_${day}_${value}:00:00
echo "$file"
while [ ! -f "$rundir/$file" ]
To make debugging easier, it would be better to have:
filepath=$rundir/file_d01_${day}_${value}:00:00
echo "$filepath"
while [ ! -f "$filepath" ]
The full code in the question has many issues (starting with a broken shebang line). It's a good idea to make Bash code Shellcheck-clean.
Related
my bash code basically just needs to generate paths with dates as folder names. But gets stuck at 19820101 for some reason. Really cant quite figure out what's the special case with 1982? Any idea why this is happening?
DTZ=19540101
while [[ $DTZ -le 19850101 ]]
do
echo username#servername:/path/filename_${DTZ}TO0300Z >> afile
DTZ=$(date +%Y%m%d -d "$DTZ+1 months")
done
I can't repro, but try using a more explicit date format.
DTZ=19540101
while [[ $DTZ -le 19850101 ]]
do
echo username#servername:/path/filename_${DTZ}TO0300Z >> afile
DTZ=$(date +%Y%m%d -d "${DTZ:0:4}-${DTZ:4:2}-${DTZ:6:s} + 1 month")
done
The ${string:offset:length} parameter expansion is a Bash-only feature, but so is the [[ conditional you were already using.
I need to write a while loop to check for a file existence.
My requirement is: check for the file only for 5 minutes. If file come in that path within 5 minutes exit the loop and continue rest of the script otherwise exit from the script after 5 minutes with an error 'file not found'. I wrote the code like this :
SOURCEFILE=/path/*file.csv
StartTime=$(date +'%s')
TimeSpan=300
EndTime=$((StartTime + TimeSpan))
while [[ ! -f ${SOURCEFILE} && $(date +'%s') < ${EndTime} ]]
do
echo "inside loop"
sleep 25
done
echo "outside loop"
But with this while loop, even if the file is present in the mentioned path, it is going inside loop and will exit only after 300 seconds. I am beginner in shell scripting and I am not able to understand the issue. I am using ksh.
I could also tell you that it works find with while [ ! -f {SOURCEFILE} ] only. But whenever I add any && condition to while loop , then the -f is not working properly.
The SOURCEFILE=/path/*file.csv is wrong in your case. It can't be evaluated right with the -f flag.
An easy solution would be to use find or ls and count the result:
find /path/ -name "*file.csv" -type f
# then count the result...
Now I think there is a logic issue with the operators precedence. To force evaluation of the ! for the -f only, use parenthesis. Here is what works for me, and you must adapt it a little to match the * before file.csv:
while [[ ( ! -f file.csv ) && $(date +'%s') < ${EndTime} ]]
do
echo "inside loop"
sleep 25
...
There are some more explanation on this answer. The "and" operator precedes the "not", that's why you had the issue.
Your primary issue is getting the asterisk (*) to expand at the 'right time'.
It doesn't help that the [ ] and [[ ]] constructs behave differently, especially when it comes to if/when to expand that asterisk. [You can peruse the google search for 'ksh single bracket vs double bracket' for more details.]
Try running the following to see the differences between single/double brackets and unquoted/single-quoted/double-quoted variable:
SOURCEFILE=/path/*file.csv
set -x
[ ! -f ${SOURCEFILE} ] && echo 'missing'
[ ! -f '${SOURCEFILE}' ] && echo 'missing'
[ ! -f "${SOURCEFILE}" ] && echo 'missing'
[[ ! -f ${SOURCEFILE} ]] && echo 'missing'
[[ ! -f '${SOURCEFILE}' ]] && echo 'missing'
[[ ! -f "${SOURCEFILE}" ]] && echo 'missing'
NOTE: Notice which tests expand the asterisk and which are looking for a (literal) asterisk in the name.
NOTE: Try adding a space to your file name (eg, *file XX.csv) and run the above tests ... tricky, tricky, tricky ...
For this particular case ... asterisk/wildcard in file name, no spaces, ksh ... you'll likely be ok with something like:
while [[ ! -f ${SOURCEFILE} ]] && [[ $(date +'%s') < ${EndTime} ]]
I have a list of 400 folders with the following pattern:
backup_01_01_2013/
backup_01_02_2013/
backup_01_03_2013/
...
backup_08_25_2014/
A new folder is created every day via a cron job.
I want to delete all folders EXCEPT:
Keep the 30 most recent folders
Keep the first of every month
How can I delete all the unnecessary folders using Bash in Linux?
So, assuming they're all in the same directory.
I'll point out I tested portions of this script, but not all of it, so you'll probably want to do some modification/testing before you actually run it - lest you end up with all your directories removed.
# Find the date 30 days ago.
recent_dirs=`date -d "-30 days" +%d-%m-%Y`
rec_month=`echo "${recent_dirs}" | cut -d '-' -f2`
rec_day=`echo "${recent_dirs}" | cut -d '-' -f1`
rec_month=`echo "${recent_dirs}" | cut -d '-' -f3`
# Go to your "home" directory for the directories.
cd /path/to/home/directory
for i in *; do
# Check to see if the element is a directory.
if [ -d "${i}" ]; then
echo "Processing directory - ${i} ... "
# Determine the date information for the directory.
cur_month=`cat "${i}" | cut -d '_' -f1`
cur_day=`cat "${i}" | cut -d '_' -f2`
# Keep all directories from the first of the month.
if [ "${first_day}" -eq "01" ]; then continue; fi
# Keep all directories from the current month and current year.
if [ "${cur_month}" -eq "${rec_month}" ]; then
if [ "${cur_year}" -eq "${rec_year}" ]; then continue; fi
fi
# Keep all directories from last month, but newer than today's DAY. I'm not
# a pro at bash arithmetic, so you might have to goof with the subshell
# math to get it to work quite right.
if [ "${cur_month} -eq $(( expr $rec_month - 1 )) ]; then
if [ "${cur_day}" -gt "${rec_day}" ]; then continue; fi
fi
# So at this point, I think we've dealt with everything, except the case where
# the current month is January, and so you're trying to save December's
# directories from last year. I think this handles it.
if [ "${rec_month}" -eq "01" ]; then
if [ "${cur_month} -eq "12" ]; then
if [ "${cur_year}" -eq $(( expr ${rec_year} - 1 )) ]; then continue; fi
fi
fi
# If we haven't stopped processing the directory by now, it's time
# remove our directory.
rm -fr "${i}"
else
echo "Skipping non-directory "${i}"...
do
exit 1
Something this won't do is deal with months with 31 days in it, so you may end up in many cases having saved 31 directories, instead of 30. I get the impression that you're trying to do a cleanup, and not a strict compliance routine, though...
Quick & dirty - try 'ls -ltr' combined with tail -30 to get all but 30. run a for loop and rm -rf
I have some operations to do on files last modified on a specific date. I would like to get the date, stock it in a string, then split it to test if the day corresponds to what I want.
So far, I've been trying things like that:
#!/bin/bash
for i in {45..236}; do
nom=M$i
chem=/Users/nfs/helene/soft/metAMOS-1.5rc3/$nom.fastq/Assemble/out
if [ -e $chem ]; then
IN= $(date -r $chem)
arr=(${IN//\ / })
if [[ ${arr[1]} == 'juin' && ${arr[2]} == '10' ]]; then
echo $nom
#cp $chem/proba.faa /Users/nfs/helene/metagenomes/DB/$nom.faa
fi
fi
done
exit 0
But it seems like the date isn't well stocked in $IN, and I'm not sure about the space-spliting either..
Perhaps the simple mistake is that you didn't place your assignment adjacent to =. It must be:
IN=$(date -r $chem)
And here's a simplified suggestion:
#!/bin/bash
for i in {45..236}; do
nom="M${i}"
chem="/Users/nfs/helene/soft/metAMOS-1.5rc3/${nom}.fastq/Assemble/out"
if [[ -e $chem ]]; then
read month day < <(exec date -r "$chem" '+%b %d')
if [[ $month == 'Jun' && $day == 10 ]]; then
echo "$nom"
# cp "$chem/proba.faa" "/Users/nfs/helene/metagenomes/DB/$nom.faa"
fi
fi
done
exit 0
* See date --help for a list of formats.
* <() is a form of Process Substitution. Check Bash's manual for it.
* Always place your arguments around double quotes when they have variables to avoid word splitting.
Say I have a filename ABC.20131212.XX.xml where XX is HH (hour). I need to get the value of XX and compare it to the current hour of the system time. If it's equal, then, i'll rename that file. So if it's 12pm (12:00), I should get the file with ABC.20131212.12.xml and rename it. How can I achieve the comparison in shell script?
Here's how to get started in pure bash without starting external commands:
a=ABC.20131212.XX.xml
# Strip off .xml
b=${a%.xml}
# Extract last 2 characters
xx=${b: -2}
and the hour can be got with
time=`date +'%H'`
Comparison and rename can be done like this
if [[ $time -eq $xx ]]
then
mv something somewhere
fi
Here's how you can do it using Bash built-in regex matching:
file=foobar.11.xml
if [[ $file =~ .*([0-9][0-9])\.xml ]]; then
if [[ "${BASH_REMATCH[1]}" -eq $(date "+%I") ]]; then
# $file has current hour in its name
fi
fi
Use the +%H option with date for comparing to a 24-hour time system.
Here is the Script you can use(Change path accordingly):
`
!/bin/bash
Path=/Your/Path/Here/*
for l in $Path
do
a=$l
b=${a%.xml}
xx=${b: -2}
time=date +"%I"
if [[ $time -eq $xx ]]
then
mv $l /New/file/name/path/newfile.txt
fi
done
exit 0`