bashscript for changing wput uploaddirectory on every 5000 passage - bash

how i can change the uploaddirectory of wput on every 5000 passage?
whats wrong in my code?
only for the record; i need infnite loop i want run this script 24h 365 days!
every 5000 passage one number higher in directory like incoming1-infinite
best regards
haug
#!/bin/bash
for (( ; ; ))
do
no=1
while [ $no -le 5000 ]
do
perl job.pl false false 1
cd comp
for fname in *.jpg;
do
mv -i "$fname" ${RANDOM}${RANDOM}.jpg
done
for fname in *.jpg;
do
mv "$fname" $(echo "$fname" | sha1sum | cut -f1 -d' ').jpg
done
if [ $no = '5000' ]
then
echo $no
echo $n
n=$(( n+1 )) # <-- this one not do what i need :-(
else
wput *.jpg ftp://user:pass#ip:port/incoming$n/ # <-- here is the important part 2, i need it to change one directory higher every 5000 passage for example; incoming1 until infinite
rm -rf *.jpg
cd ..
fi
no=`expr $no + 1`
done
done

cd comp
for ((no=1; ; no++))
do
n=`expr $no / 5000`
perl job.pl false false 1
for fname in *.jpg;
# ... rest of the script
done

Just calculate the remainder of division by 5000 and if it is equal 0 (or 1 or anything < 5000) do what you need.
if [ $((no%5000)) = 0 ]
then
...
fi

Just put a for-loop that executes 5,000 times inside your infinite loop (which can keep track of the directory sequence number for you):
#!/bin/bash
for (( n=0; ; n++ )); do
for (( no=1; no < 5000; no++ )); do
perl job.pl false false 1
( # Subshell so we don't have to worry about returning
# to the previous directory.
cd comp
# No need to rename each file twice.
for fname in *.jpg; do
mv "$fname" $(echo "${RANDOM}${RANDOM}.jpg" | sha1sum | cut -f1 -d' ').jpg
done
wput *.jpg ftp://user:pass#ip:port/incoming$n
)
done
done

Related

bash: sed processing of the files including leading-zeros

I am using a bash script to loop over the files pre-defined in several groups of the array within the directory in order to edit the file in case if its exist (for example in the 1st group there are 100 files arranged from 0001 to 0100, in the second group - 50 files arranged from 0001 to 0050 etc).
#an array for the groups
systems=(one two three four)
# loop over the groups
for file in "${systems[#]}"; do
i="1"
# introduce K var because the files are numbered as 0001 ... 0100
k=$(printf '%03d' $i)
while [ $i -le 100 ]; do
if [ ! -f "${output}/${file}_${k}.pdb" ]; then
echo 'File '${output}/${file}_${k}.pdb' does not exits!'
break
else
## edit file via SED
# to add i-th number on the first string of the file and substitute smth on the last string;
sed -i -e '1 i\MODEL '$i'' -e 's/TER/ENDMDL/g' ${output}/${file}_${k}.pdb
((i++))
fi
done
done
This script doesn't not work on the stage of SED editing, but if I omit usage of the leading zeros in the names of files and use just i-th index within the script, everything works fine:
# loop over the groups
for file in "${systems[#]}"; do
i="1"
# put k into comment since filles arranged from 1 to 100 without leading zeros;
#k=$(printf '%03d' $i)
while [ $i -le 100 ]; do
# the filles arranged from 1 to 100
if [ ! -f "${output}/${file}_${i}.pdb" ]; then
echo 'File '${output}/${file}_${i}.pdb' does not exits!'
break
else
## edit file via SED
# to add i-th number on the first string of the file
sed -i -e '1 i\MODEL '$i'' -e 's/TER/ENDMDL/g' ${output}/${file}_${i}.pdb
((i++))
fi
done
done
k is assigned before the loop with i
i="1"
# introduce K var because the files are numbered as 0001 ... 0100
k=$(printf '%03d' $i)
while [ $i -le 100 ]; do
...
((i++))
...
done
Move the assignment to k inside the loop.
Alternative:
for ((i=1;i<100;i++)); do
k=$(printf '%03d' ${i})
...
NOTE: have made several edits.
No answers of my own here - just compiling into a single block of code, incorporating the the answers of jas (at his request) and Walter A who likely hit the real problem -
for file in "${systems[#]}"
do for ((i=1;i<100;i++))
do printf -v enumerated "${output}/${file}_%04d.pdb" $i
if [[ -f "$enumerated" ]
then sed -i -e "1 i\\MODEL $i" -e 's/TER/ENDMDL/g' $enumerated
else echo "file not found: '$enumerated''
fi
done
done
Depending on what else is in your directory structure, you might also try this:
for stub in "${systems[#]}"
do for file in "$output/${stub}"_[0-9][0-9][0-9][0-9].pdb
do sed -i -e "1 i\\MODEL ${file//[^0-9]/}" -e 's/TER/ENDMDL/g' "$file"
done
done

How to list files with words exceeding n characters in all subdirectories

I have to write a shell script that creates a file containing the name of each text files from a folder (given as parameter) and it's subfolders that contain words longer than n characters (read n from keyboard).
I wrote the following code so far :
#!/bin/bash
Verifies if the first given parameter is a folder:
if [ ! -d $1 ]
then echo $1 is not a directory\!
exit 1
fi
Reading n
echo -n "Give the number n: "
read n
echo "You entered: $n"
Destination where to write the name of the files:
destinatie="destinatie"
the actual part that i think it makes me problems:
nr=0;
#while read line;
#do
for fisier in `find $1 -type f`
do
counter=0
for word in $(<$fisier);
do
file=`basename "$fisier"`
length=`expr length $word`
echo "$length"
if [ $length -gt $n ];
then counter=$(($counter+1))
fi
done
if [ $counter -gt $nr ];
then echo "$file" >> $destinatie
fi
done
break
done
exit
The script works but it does a few more steps that i don't need.It seems like it reads some files more than 1 time. If anyone can help me please?
Does this help?
egrep -lr "\w{$n,}" $1/* >$destinatie
Some explanation:
\w means: a character that words consist of
{$n,} means: number of consecutive characters is at least $n
Option -l lists files and does not print the grepped text and -r performs a recursive scan on your directory in $1
Edit:
a bit more complete version around the egrep command:
#!/bin/bash
die() { echo "$#" 1>&2 ; exit 1; }
[ -z "$1" ] && die "which directory to scan?"
dir="$1"
[ -d "$dir" ] || die "$dir isn't a directory"
echo -n "Give the number n: "
read n
echo "You entered: $n"
[ $n -le 0 ] && die "the number should be > 0"
destinatie="destinatie"
egrep -lr "\w{$n,}" "$dir"/* | while read f; do basename "$f"; done >$destinatie
This code has syntax errors, probably leftovers from your commented-out while loop: It would be best to remove the last 3 lines: done causes the error, break and exit are unnecessary as there is nothing to break out from and the program always terminates at its end.
The program appears to output files multiple times because you just append to $destinatie. You could simply delete that file when you start:
rm "$destinatie"
You echo the numbers to stdout (echo "$length") and the file names to $destinatie (echo "$file" >> $destinatie). I do not know if that is intentional.
I found the problem.The problem was the directory in which i was searching.Because i worked on the files from the direcotry and modified them , it seems that there remained some files which were not displayed in file explorer but the script would find them.i created another directory and i gived it as parameter and it works. Thank you for your answers
.

Infinite while-loop in BASH script

I'm really struggling to see why this while-loop never ends, when the loop starts, my variable LOC is set to Testing/, which is a directory I created to test this program, it has the following layout:
I want the loop to end once all Directories have had the "count" function applied to them.
Here are the things I have tried;
I've checked my count function, and it doesn't produce an infinite loop
I've tried running through the algorithm by hand
PARSE=1
LOC=$LOC/
count
AVAILABLEDIR=$(ls $LOC -AFl | sed "1 d" | grep "/$" | awk '{ print $9 }')
while [ $PARSE = "1" ]
do
if [[ ${AVAILABLEDIR[#]} == '' ]]; then
PARSE=0
fi
DIRBASE=$LOC
for a in ${AVAILABLEDIR[#]}; do
LOC="${DIRBASE}${a}"
LOCLIST="$LOCLIST $LOC"
count
done
for a in ${LOCLIST[#]}; do
TMPAVAILABLEDIR=$(ls $a -AFl | sed "1 d" | grep "/$" | awk '{ print $9 }')
PREPEND=$a
if [[ ${TMPAVAILABLEDIR[#]} == '' ]]; then
continue
fi
for a in ${TMPAVAILABLEDIR[#]}; do
TMPAVAILABLEDIR2="$TMPAVAILABLEDIR2 ${PREPEND[#]}${a}"
done
NEWAVAILABLEDIR="$NEWAVAILABLEDIR $TMPAVAILABLEDIR2"
done
AVAILABLEDIR=$NEWAVAILABLEDIR
NEWAVAILABLEDIR=''
LOC=''
done
I am really struggling, and any input would be greatly appreciated, I've been trying to figure this out for the last couple of hours.
You should try to run the script with argument -x, or write it into the first line:
#!/bin/bash -x
Then it tells you everything it does.
In that case, you might notice two errors:
You never reset TMPAVAILABLEDIR2
You do ls on regular files as well.
If you really must avoid recursion, try this. It completely recursion-free:
#!/bin/bash
count() {
echo counting "$1"
}
todo=(Testing)
while test ${#todo[#]} != 0
do
doit=("${todo[#]}")
todo=()
for dir in "${doit[#]}"
do
for entry in "$dir"/* # If the directory is empty, this shows an entry named "*"
do
test -e "$entry" || continue # Skip the entry "*" of an empty directory
count "$entry"
test -d "$entry" || continue
todo+=("$entry")
done
done
done
You wrote you want to perform "count" on all directories.
Look at the options of find:
find $LOC -type d | while read dir; do
cd $LOC
cd ${dir}
count
done
Or shorter (when your function count accepts a directory as parameter 1):
find $LOC -type d | xargs count
I now see you do not want to use find or ls -R (recursive function). Then you should make your own recursive function, something like
function parseDir {
ls -d */ $1 | while read dir; do
count
parseDir $1/$dir
done
}
I have no idea if this will work, but it’s an interesting question I couldn't stop thinking about.
while true ; do
for word in "$(echo *)" ; do
if [[ -d "$word" ]] ; then
d[$((i++))]="$PWD"/"$word"
elif [[ -f "$word" ]] ;then
f[$((j++))]="$PWD"/"$word"
fi
done
[[ $k -gt $i ]] && cd ..
cd "$d[$((k++))]" || break
done

problem with while loop

here is the script I wrote but it seems it has problem with while ! while suppose to compare the content of K with SDATE and while they are not equal go to the loop !
for d in \
$(sed -nre 's/.*\[(..)\/(...)\/(....):(..:..:..) .*/\1 \2 \3 \4/p' thttpd.log | date +%s -f-);
do echo $d >s1; done
time=$(expr 60 \* 60 \* 24 \* 5)
EDATE=`tail -1 s1`
SDATE=$[$EDATE - $time]
time=$(expr 60 \* 60 \* 24 \* 5)
EDATE=`tail -1 s1`
SDATE=$[$EDATE - $time]
k=`tail -1 s1`
echo $k
echo $SDATE
while [$k -ne $SDATE](k and SDATE contain numbers)
do
k=`tail -1 s1`
sed '1d' < s1 > tempfile
mv s1 s1.old
mv tempfile s1
echo $K| awk '{print strftime("%d/%m/%Y:%T",$1)}'|tee -a ass
done
The problem is that you do not have spaces around [ or ]. Which causes BASH to parse the line incorrectly.
With the following line, BASH will attempt to run the program [$K, probably not what you are intending.
while [$k -ne $SDATE]
What you need to have is the following:
while [ $k -ne $SDATE ]
Try this:
while [[ $k != $SDATE ]]
Ah, shell programming is so touchy...
k=0
while [ $k != 4 ]; do echo $k; k=`expr $k + 1`; done
Works and prints 0, 1, 2, 3 on separate lines as expected and this essentially looks like what you have except for spaces, newlines, and sources of values. Maybe it is a string versus value problem but I did not think shell variables had types.
I would try stripping what you have back until it works then add back what you want it to do.

bash loop between two given dates

I'm trying to create a script that will loop through files that have their filenames written in the following format: yyyymmdd.hh.filename.
The script is called with:
./loopscript.sh 20091026.00 23
./loopscript.sh 20091026.11 15
./loopscript.sh 20091026.09 20091027.17
The need is for the script to check each hour between those two given dates/hours.
e.g.
cat 20091026.00.filename |more
cat 20091026.01.filename |more
...
cat 20091026.23.filename |more
cat 20091027.01.filename |more
cat 20091027.02.filename |more
...
and so on.
any idea how to go about this? I don't have any difficulty with standard 0 - x loops. or simple for loops. Just not sure how to go about the above.
How about this:
#!/bin/bash
date1=$1
date2=$2
#verify dates
if ! date -d "$date1" 2>&1 > /dev/null ;
then echo "first date is invalid" ; exit 1
fi
if ! date -d "$date2" 2>&1 > /dev/null ;
then echo "second date is invalid" ; exit 1
fi
#set current and end date
current=$(date -d "$date1")
end=$(date -d "$date2 +1 hours")
#loop over all dates
while [ "$end" != "$current" ]
do
file=$(date -d "$current" +%Y%m%d.%H)
cat $file."filename" | more
current=$(date -d "$current +1 hours")
done
To process each file between two given date/hours, you can use the following:
#!/usr/bin/bash
#set -x
usage() {
echo 'Usage: loopscript.sh <from> <to>'
echo ' <from> MUST be yyyymmdd.hh or empty, meaning 00000000.00'
echo ' <to> can be shorter and is affected by <from>'
echo ' e.g., 20091026.00 27.01 becomes'
echo ' 20091026.00 20091027.01'
echo ' If empty, it is set to 99999999.99'
echo 'Arguments were:'
echo " '${from}'"
echo " '${to}'"
}
# Check parameters.
from="00000000.00"
to="99999999.99"
if [[ ! -z "$1" ]] ; then
from=$1
fi
if [[ ! -z "$2" ]] ; then
to=$2
fi
## Insert this to default to rest-of-day when first argument
## but no second argument. Basically just sets second
## argument to 23 so it will be transformed to end-of-day.
#if [[ ! -z "$1"]] ; then
# if [[ -z "$2"]] ; then
# to=23
# fi
#fi
if [[ ${#from} -ne 11 || ${#to} -gt 11 ]] ; then
usage
exit 1
fi
# Sneaky code to modify a short "to" based on the start of "from".
# ${#from} is the length of ${from}.
# $((${#from}-${#to})) is the length difference between ${from} and ${to}
# ${from:0:$((${#from}-${#to}))} is the start of ${from} long enough
# to make ${to} the same length.
# ${from:0:$((${#from}-${#to}))}${to} is that with ${to} appended.
# Voila! Easy, no?
if [[ ${#to} -lt ${#from} ]] ; then
to=${from:0:$((${#from}-${#to}))}${to}
fi
# Process all files, checking that they're inside the range.
echo "From ${from} to ${to}"
for file in [0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].[0-9][0-9].* ; do
if [[ ! ( ${file:0:11} < ${from} || ${file:0:11} > ${to} ) ]] ; then
echo " ${file}"
fi
done
When you create the files 20091026.00.${RANDOM} through 20091028.23.${RANDOM} inclusive, this is a couple of sample runs:
pax> ./loopscript.sh 20091026.07 9
From 20091026.07 to 20091026.09
20091026.07.21772
20091026.08.31390
20091026.09.9214
pax> ./loopscript.sh 20091027.21 28.02
From 20091027.21 to 20091028.02
20091027.21.22582
20091027.22.30063
20091027.23.29437
20091028.00.14744
20091028.01.6827
20091028.02.10366
pax> ./loopscript.sh 00000000.00 99999999.99 # or just leave off the parameters.
20091026.00.25772
20091026.01.25964
20091026.02.21132
20091026.03.3116
20091026.04.6271
20091026.05.14870
20091026.06.28826
: : :
20091028.17.20089
20091028.18.13816
20091028.19.7650
20091028.20.20927
20091028.21.13248
20091028.22.9125
20091028.23.7870
As you can see, the first argument must be of the correct format yyyymmdd.hh. The second argument can be shorter since it inherits the start of the first argument to make it the correct length.
This only attempts to process files that exist (from ls) and of the correct format, not every date/hour within the range. This will be more efficient if you have sparse files (including at the start and the end of the range) since it doesn't need to check that the files exist.
By the way, this is the command that created the test files, if you're interested:
pax> for dt in 20091026 20091027 20091028 ; do
for tm in 00 01 02 ... you get the idea ... 21 22 23 ; do
touch $dt.$tm.$RANDOM
done
done
Please don't type that in verbatim and then complain that it created files like:
20091026.you.12345
20091028.idea.77
I only trimmed down the line so it fits in the code width. :-)
One possible solution: convert dates into standard Unix representation of "Seconds passed since the epoch" and loop, increasing this number by 3600 (number of seconds in an hour) each iteration. Example:
#!/bin/bash
# Parse your input to date and hour first, so you get:
date_from=20090911
hour_from=10
date_to=20091026
hour_to=01
i=`date --date="$date_from $hour_from:00:00" +%s`
j=`date --date="$date_to $hour_to:00:00" +%s`
while [[ $i < $j ]]; do
date -d "1970-01-01 $i sec" "+%Y%m%d.%H"
i=$[ $i + 3600 ]
done

Resources