I'm trying to create a shell script that shall convert certain Files according to some rules. The files are organized like this:
file_0000.adat
file_0001.adat
file_0002.adat
file_0003.adat
...
file_0010.adat
file_0011.adat
...
file_9999.adat
My current script looks like this:
#!/bin/sh
lauf=$(expr 0)
for filename in ../distData/file_*.adat
do
...
lauf=$(expr $lauf + 1)
done;
But now I only want the files with numbers from 0000 to 1500. For the conversion I need the number of the file (done by the variable lauf in the script). Can't figure out how to do this right now.
Thanks for replys.
Try something like:
for i in $(seq -w 1500) ; do
if [ -f ../distData/file_${i}.adat ] ; then
# do whatever, the file number is ${i}
fi
done
I tried something like this to generate the number pattern of the file.
You can put the other part of the code to concatenate with file name and search for the file name
START=0
MAX=1500
while [ $START -le $MAX ]
do
PAT=$(printf "%04d" $START)
echo $START $PAT
START=`expr $START + 1`
done
FILES=$(ls ../distData/file_{0000..1500}.adat 2>/dev/null)
for FILE in ${FILES[#]}; do
#$FILE exists, do what you want
done
the redirect of stderr to /dev/null is to suppress output messages for files in the range that do not exist
Related
I need split big *.csv file for several smaller. Currently there is 661497 rows, I need each file with max. 40000. I've tried solution that I found on Github but with no success:
FILENAME=/home/cnf/domains/cnf.com.pl/public_html/sklep/dropshipping-pliki/products-files/my_file.csv
HDR=$(head -1 ${FILENAME})
split -l 40000 ${FILENAME} xyz
n=1
for f in xyz*
do
if [[ ${n} -ne 1 ]]; then
echo ${HDR} > part-${n}-${FILENAME}.csv
fi
cat ${f} >> part-${n}-${FILENAME}.csv
rm ${f}
((n++))
done
The error I get:
/home/cnf/domains/cnf.com.pl/public_html/sklep/dropshipping-pliki/download.sh: line 23: part-1-/home/cnf/domains/cnf.com.pl/public_html/sklep/dropshipping-pliki/products-files/my_file.csv.csv: No such file or directory
thanks for help!
Keep in mind FILENAME contains both a directory and a file so later in the script when you build the new filename you get something like:
part-1-/home/cnf/domains/cnf.com.pl/public_html/sklep/dropshipping-pliki/products-files/tyre_8.csv.csv
One quick-n-easy fix would be split the directory and filename into 2 separate variables, eg:
srcdir='/home/cnf/domains/cnf.com.pl/public_html/sklep/dropshipping-pliki/products-files'
filename='tyre_8.csv'
hdr=$(head -1 ${srcdir}/${filename})
split -l 40000 "${srcdir}/${filename}" xyz
n=1
for f in xyz*
do
if [[ ${n} -ne 1 ]]; then
echo ${hdr} > "${srcdir}/part-${n}-${filename}"
fi
cat "${f}" >> "${srcdir}/part-${n}-${filename}"
rm "${f}"
((n++))
done
NOTES:
consider using lowercase variables (using uppercase variables raises the possibility of problems if there's an OS variable of the same name)
wrap variable references in double quotes in case string contains spaces
don't need to add a .csv extension on the new filename since it's already part of $filename
I have to write a shell script that creates a file containing the name of each text files from a folder (given as parameter) and it's subfolders that contain words longer than n characters (read n from keyboard).
I wrote the following code so far :
#!/bin/bash
Verifies if the first given parameter is a folder:
if [ ! -d $1 ]
then echo $1 is not a directory\!
exit 1
fi
Reading n
echo -n "Give the number n: "
read n
echo "You entered: $n"
Destination where to write the name of the files:
destinatie="destinatie"
the actual part that i think it makes me problems:
nr=0;
#while read line;
#do
for fisier in `find $1 -type f`
do
counter=0
for word in $(<$fisier);
do
file=`basename "$fisier"`
length=`expr length $word`
echo "$length"
if [ $length -gt $n ];
then counter=$(($counter+1))
fi
done
if [ $counter -gt $nr ];
then echo "$file" >> $destinatie
fi
done
break
done
exit
The script works but it does a few more steps that i don't need.It seems like it reads some files more than 1 time. If anyone can help me please?
Does this help?
egrep -lr "\w{$n,}" $1/* >$destinatie
Some explanation:
\w means: a character that words consist of
{$n,} means: number of consecutive characters is at least $n
Option -l lists files and does not print the grepped text and -r performs a recursive scan on your directory in $1
Edit:
a bit more complete version around the egrep command:
#!/bin/bash
die() { echo "$#" 1>&2 ; exit 1; }
[ -z "$1" ] && die "which directory to scan?"
dir="$1"
[ -d "$dir" ] || die "$dir isn't a directory"
echo -n "Give the number n: "
read n
echo "You entered: $n"
[ $n -le 0 ] && die "the number should be > 0"
destinatie="destinatie"
egrep -lr "\w{$n,}" "$dir"/* | while read f; do basename "$f"; done >$destinatie
This code has syntax errors, probably leftovers from your commented-out while loop: It would be best to remove the last 3 lines: done causes the error, break and exit are unnecessary as there is nothing to break out from and the program always terminates at its end.
The program appears to output files multiple times because you just append to $destinatie. You could simply delete that file when you start:
rm "$destinatie"
You echo the numbers to stdout (echo "$length") and the file names to $destinatie (echo "$file" >> $destinatie). I do not know if that is intentional.
I found the problem.The problem was the directory in which i was searching.Because i worked on the files from the direcotry and modified them , it seems that there remained some files which were not displayed in file explorer but the script would find them.i created another directory and i gived it as parameter and it works. Thank you for your answers
.
so here is my dilemma. I have a command in the form:
grdpaste infile.grd infile.grd -Goutfile.grd
I have a series of folders in the same directory that each contain a file named infile.grd. I want to iterate through all the folder so that the first run combines infile.grd from the first and second folder, and then the second combines outfile.grd from the first run and infile.grd from the third folder, and so on. I do not know how many folders exist, and the final product should contain the combination of all the infiles.
I think I can use a counter to control the combination parts (I did it earlier in my script), but I do not know how to make a for loop that takes one file from one folder and the other file from the next folder, without knowing the names of the folders. I hope this makes sense, thanks much.
AM
If grdpaste will accept an empty input file in a sane way then the following should work:
lastfile=dummy.grd
touch "$lastfile"
for infile in */infile.grd; do
_outfile=outfile$((i++)).grd
grdpaste "$lastfile" "$infile" -G"$_outfile"
lastfile=$_outfile
done
If it can't then the above loop needs to be modified to store the first name it sees in $lastfile and do nothing else that first loop through... something like this:
lastfile=
for infile in */infile.grd; do
[ -z "$lastfile" ] && { lastfile=$infile; continue; }
_outfile=outfile$((i++)).grd
grdpaste "$lastfile" "$infile" -G"$_outfile"
lastfile=$_outfile
done
solution posted below. For complete code, see moravi project here.
for folder in */
do
ls "$folder" | sed 's/e/e/' >"${folder%/}.tmp"
done
for file in *.tmp
do
lat=$(echo $file | awk -F "." '{print $1}')
count=0
while read line
do
count=$(( $count + 1 ))
if [ "$count" = "1" ]
then
declare "tmp_${count}=$line"
elif [ "$count" = "2" ]
then
declare "tmp_${count}=$line"
prod="P"$(( ${count} - 1 ))".grd"
grdpaste ./${lat}/${tmp_1} ./${lat}/${tmp_2} -G./${lat}/${prod} -V
elif [ "$count" > "2" ]
then
r="tmp_"${count}
declare "r=$line"
pprod="P"$(( ${count} - 2 ))".grd"
prod="P"$(( ${count} - 1 ))".grd"
grdpaste ./${lat}/${r} ./${lat}/${pprod} -G./${lat}/${prod} -V
to_paste=${prod}
fi
done <$file
done
rm *.tmp
I've a .txt file which contains
abc.com
google.com
....
....
yahoo.com
And I'm interested in loading it to a bash script as a list (i.e. Domain_List=( "abc.com" "google.com" .... "yahoo.com") ). Is it possible to do?
Additional information, once the list is obtained it is used in a for loop and if statements.
for i in "${Domain_list[#]}
do
if grep -q "${Domain_list[counter]}" domains.log
....
....
fi
....
let counter=counter+1
done
Thank you,
Update:
I've changed the format to Domain_list=( "google.com .... "yahoo.com" ), and using source Doamin.txt allows me to use Domain_list as a list in the bash script.
#!/bin/bash
counter=0
source domain.txt
for i in "${domain_list[#]}"
do
echo "${domain_list[counter]}"
let counter=counter+1
done
echo "$counter"
Suppose, your datafile name is web.txt. Using command substitution (backtics) and cat, the array can be built. Pl. see the following code,
myarray=(`cat web.txt`)
noofelements=${#myarray[*]}
#now traverse the array
counter=0
while [ $counter -lt $noofelements ]
do
echo " Element $counter is ${myarray[$counter]}"
counter=$(( $counter + 1 ))
done
Domain_list=()
while read addr
do
Domain_list+=($addr)
done < addresses.txt
That should store each line of the text file into the array.
I used the source command, and it works fine.
#!/bin/bash
counter=0
source domain.txt
for i in "${domain_list[#]}"
do
echo "${domain_list[counter]}"
let counter=counter+1
done
echo "$counter"
There's no need for a counter if we're sourcing the list from a file. You can simply iterate through the list and echo the value.
#!/bin/bash
source domain.txt
for i in ${domain_list[#]}
do
echo $i
done
I've found most of the questions of this kind where the change in name has been same for the entire set of files in that directory.
But i'm here presented with a situation to give a different name to every file in that directory or just add a different prefix.
For Example, I have about 200 files in a directory, all of them with numbers in their filename. what i want to do is add a prefix of 1 to 200 for every file. Like 1_xxxxxxxx.png,2_xxxxxxxx.png...........200_xxxxxxxx.png
I'm trying this, but it doesnt increment my $i everytime, rather it gives a prefix of 1_ to every file.
echo "renaming files"
i=1 #initializing
j=ls -1 | wc -l #Count number of files in that dir
while [ "$i" -lt "$j" ] #looping
do
for FILE in * ; do NEWFILE=`echo $i_$FILE`; #swapping the file with variable $i
mv $FILE $NEWFILE #doing the actual rename
i=`expr $i+1` #increment $i
done
Thanks for any suggestion/help.
To increment with expr, you definitely need spaces( expr $i + 1 ), but you would probably be better off just doing:
echo "renaming files"
i=1
for FILE in * ; do
mv $FILE $((i++))_$FILE
done
i=1
for f in *; do
mv -- "$f" "${i}_$f"
i=$(($i + 1))
done