I need to move over 100,000 img's from 1 server to another via a mount point, i have a .csv with them listed and im looking to script it
the csv looks like this
"images1\002_0001\thumb",53717902.jpg,/www/images/002_0001/thumb/
"images1\002_0001\thumb",53717901.jpg,/www/images/002_0001/thumb/
"images1\002_0001\thumb",53717900.jpg,/www/images/002_0001/thumb/
comma separated we have source name and destination
I was thinking of using awk to create each as a variable
SOURCE=`awk -F ',' '{ print $1 }' test.csv`
IMGNAME=`awk -F ',' '{ print $2 }' test.csv`
DEST=`awk -F ',' '{ print $3 }' test.csv`
this is where im getting stuck, my loop
while read line
do
cp $SOURCE${IMGNAME} $DEST
done <test.csv
this has copied the first name it finds into all the directories
You could use what you have and move the variable declaration into the loop referencing $line, or you could use IFS, as suggested below.
while IFS=, read -r src filename dest
do
cp $src${filename} $dest
done <test.csv
There are many way to do it, some example
If you have no spaces in the directories string: you can do even from shell
sed -E 's/"/cp /; s/",/\// ; s/,/ /;s/\\/\//g' test.csv | /bin/bash
It better if check it before you try. You speak about a lot of files...
sed -E 's/"/cp /; s/",/\// ; s/,/ /;s/\\/\//g' test.csv | less
It can happen that you have spaces in the string of the directory name like My Windows Like Dir Name. In this case you need double quotes (there are the double quote even for this reason maybe...)
You can do it using only awk(always from the shell)
awk -F',' '{gsub(/"/, "", $1); gsub(/\\/, "/", $1); print "cp \""$1"/" $2"\" \"" $3"\""}' test.csv | /bin/bash
or that is equivalent
awk -F',' '{gsub(/"/, "", $1); gsub(/\\/, "/", $1); printf ("cp \"%s/%s\" \"%s\"\n",$1,$2,$3)}' test.csv | /bin/bash
Check it always in advance, avoiding the last pipe |/bin/bash, putting maybe | head -n 10 to have only the first 10 lines.
The script can be written:
while IFS=, read -r SOURCE IMGNAME DEST
do
SOURCE=( ${SOURCE//\\/\/} ) # Here you need to change "\" in "/"
SOURCE=( ${SOURCE//\"/} ) # Here I like to kill ""
cp "${SOURCE}/${IMGNAME}" "$DEST" # Here I put again ""
done <test.csv
Note: I think you need to change "\" windows style in "/" unix style. So I required to the substitution rules.
Related
The problem is the following I have a file that each line has this form:
id|lastName|firstName|gender|birthday|joinDate|IP|browser
i want to sort alphabetically all the firstnames in that file and print them one on each line but each name only once
i have created the following program but for some reason it creates an infinite loop:
array1=()
while read LINE
do
if [ ${LINE:0:1} != '#' ]
then
IFS="|"
array=($LINE)
if [[ "${array1[#]}" != "${array[2]}" ]]
then
array1+=("${array[2]}")
fi
fi
done < $3
echo ${array1[#]} | awk 'BEGIN{RS=" ";} {print $1}' | sort
NOTES
if [ ${LINE:0:1} != '#' ] : this command is used because there are comments in the file that i dont want to print
$3 : filename
array1 : is used for all the seperate names
Wow, there's a MUCH simpler and cleaner way to achieve this, without having to mess with the IFS variable or using arrays. You can use "for" to do this:
First I created a file with the same structure as yours:
$ cat file
id|lastName|Douglas|gender|birthday|joinDate|IP|browser
id|lastName|Tim|gender|birthday|joinDate|IP|browser
id|lastName|Andrew|gender|birthday|joinDate|IP|browser
id|lastName|Sasha|gender|birthday|joinDate|IP|browser
#id|lastName|Carly|gender|birthday|joinDate|IP|browser
id|lastName|Madson|gender|birthday|joinDate|IP|browser
Here's the script I wrote using "for":
#!/bin/bash
for LINE in `cat file | grep -v "^#" | awk -F'|' '{print$3}' | sort -u`
do
echo $LINE
done
And here's the output of this script:
$ ./script.sh
Andrew
Douglas
Madson
Sasha
Tim
Explanation:
for LINE in `cat file`
Creates a loop that reads each line of "file". The commands between ` are run by linux, for example, if you wanted to store the date inside of a variable you could use "VARDATE=`date`".
grep -v "^#"
The option -v is used to exclude results matching the pattern, in this case the pattern is "^#". The "^" character means "line begins with". So grep -v "^#" means "exclude lines beginning with #".
awk -F'|' '{print$3}'
The -F option switches the column delimiter from the default (the default is a space) to whatever you put between ' after it, in this case the "|" character.
The '{print$3}' prints the 3rd column.
sort -u
And the "sort -u" command to sort the names alphabetically.
I have a file test.txt like below spaces in between each record
service[1.1],parttion, service[1.2],parttion, service[1.3],parttion, service[2.1],parttion, service2[2.2],parttion,
Now I want to rearrange it as below into a output.txt
COMPOSITES=parttion/service/1.1,parttion/service/1.2,parttion/service/1.3,parttion/service/2.1,parttion/service/2.2
I've tried:
final_str=''
COMPOSITES=''
# Re-arranging the composites and preparing the composite property file
while read line; do
partition_val="$(echo $line | cut -d ',' -f 2)"
composite_temp1_val="$(echo $line | cut -d ',' -f 1)"
composite_val="$(echo $composite_temp1_val | cut -d '[' -f 1)"
version_temp1_val="$(echo $composite_temp1_val | cut -d '[' -f 2)"
version_val="$(echo $version_temp1_val | cut -d ']' -f 1)"
final_str="$partition_val/$composite_val/$version_val,"
COMPOSITES=$COMPOSITES$final_str
done <./temp/test.txt
We start with the file:
$ cat test.txt
service[1.1],parttion, service[1.2],parttion, service[1.3],parttion, service[2.1],parttion, service2[2.2],parttion,
We can rearrange that file as follows:
$ awk -F, -v RS=" " 'BEGIN{printf "COMPOSITES=";} {gsub(/[[]/, "/"); gsub(/[]]/, ""); if (NF>1) printf "%s%s/%s",NR==1?"":",",$2,$1;}' test.txt
COMPOSITES=parttion/service/1.1,parttion/service/1.2,parttion/service/1.3,parttion/service/2.1,parttion/service2/2.2
The same command split over multiple lines is:
awk -F, -v RS=" " '
BEGIN{
printf "COMPOSITES=";
}
{
gsub(/[[]/, "/")
gsub(/[]]/, "")
if (NF>1) printf "%s%s/%s",NR==1?"":",",$2,$1
}
' test.txt
Here's what I came up with.
awk -F '[],[]' -v RS=" " 'BEGIN{printf("COMPOSITES=")}/../{printf("%s/%s/%s,",$4,$1,$2);}' test.txt
Broken out for easier reading:
awk -F '[],[]' -v RS=" " '
BEGIN {
printf("COMPOSITES=");
}
/../ {
printf("%s/%s/%s,",$4,$1,$2);
}' test.txt
More detailed explanation of the script:
-F '[],[]' - use commas or square brackets as field separators
-v RS=" " - use just the space as a record separator
'BEGIN{printf("COMPOSITES=")} - starts your line
/../ - run the following code on any line that has at least two characters. This avoids the empty field at the end of a line terminating with a space.
printf("%s/%s/%s,",$4,$1,$2); - print the elements using a printf() format string that matches the output you specified.
As concise as this is, the format string does leave a trailing comma at the end of the line. If this is a problem, it can be avoided with a bit of extra code.
You could also do this in sed, if you like writing code in line noise.
sed -e 's:\([^[]*\).\([^]]*\).,\([^,]*\), :\3/\1/\2,:g;s/^/COMPOSITES=/;s/,$//' test.txt
Finally, if you want to avoid external tools like sed and awk, you can do this in bash alone:
a=($(<test.txt))
echo -n "COMPOSITES="
for i in "${a[#]}"; do
i="${i%,}"
t="${i%]*}"
printf "%s/%s/%s," "${i#*,}" "${i%[*}" "${t#*[}"
done
echo ""
This slurps the contents of test.txt into an array, which means your input data must be separated by whitespace, per your example. It then adds the prefix, then steps through the array, using Parameter Expansion to massage the data into the fields you need. The last line (echo "") is helpful for testing; you may want to eliminate it in practice.
I have a problem creating a script that reads specific value from all the files of an entire folder
I have a number of email files in a directory and I need to extract from each file, 2 specific values.
After that I have to put them into a new file that looks like that:
--------------
To: value1
value2
--------------
This is what I want to do, but I don't know how to create the script:
# I am putting the name of the files into a temp file
`ls -l | awk '{print $9 }' >tmpfile`
# use for the name of a file
`date=`date +"%T"
# The first specific value from file (phone number)
var1=`cat tmpfile | grep "To: 0" | awk '{print $2 }' | cut -b -10 `
# The second specific value from file(subject)
var2=cat file | grep Subject | awk '{print $2$3$4$5$6$7$8$9$10 }'
# Put the first value in a new file on the first row
echo "To: 4"$var1"" > sms-$date
# Put the second value in the same file on the second row
echo ""$var2"" >>sms-$date
.......
and do the same for every file in the directory
I tried using while and for functions but I couldn't finalize the script
Thank You
I've made a few changes to your script, hopefully they will be useful to you:
#!/bin/bash
for file in *; do
var1=$(awk '/To: 0/ {print substr($2,0,10)}' "$file")
var2=$(awk '/Subject/ {for (i=2; i<=10; ++i) s=s$i; print s}' "$file")
outfile="sms-"$(date +"%T")
i=0
while [ -f "$outfile" ]; do outfile="sms-$date-"$((i++)); done
echo "To: 4$var1" > "$outfile"
echo "$var2" >> "$outfile"
done
The for loop just goes through every file in the folder that you run the script from.
I have added added an additional suffix $i to the end of the file name. If no file with the same date already exists, then the file will be created without the suffix. Otherwise the value of $i will keep increasing until there is no file with the same name.
I'm using $( ) rather than backticks, this is just a personal preference but it can be clearer in my opinion, especially when there are other quotes about.
There's not usually any need to pipe the output of grep to awk. You can do the search in awk using the / / syntax.
I have removed the cut -b -10 and replaced it with substr($2, 0, 10), which prints the first 10 characters from column 2.
It's not much shorter but I used a loop rather than the $2$3..., I think it looks a bit neater.
There's no need for all the extra " in the two output lines.
I sugest to try the following:
#!/bin/sh
RESULT_FILE=sms-`date +"%T"`
DIR=.
fgrep -l 'To: 0' "$DIR" | while read FILE; do
var1=`fgrep 'To: 0' "$FILE" | awk '{print $2 }' | cut -b -10`
var2=`fgrep 'Subject' "$FILE" | awk '{print $2$3$4$5$6$7$8$9$10 }'`
echo "To: 4$var1" >>"$RESULT_FIL"
echo "$var2" >>"$RESULT_FIL"
done
I'm having some trouble getting this code to work, and I have no idea why it's not, Maybe one of you gurus can lend me a hand.
To begin with I have two CSV files structured as such:
Book1.csv:
Desc,asset,asset name,something,waiver,waiver name,init date,wrong date,blah,blah,target
akdhfa,2014,adskf,kadsfjh,123-4567,none,none,none,none,none,BOOP
Book2.csv
Desc,asset,asset name,something,waiver,waiver name,init date,wrong date,blah,blah,target
akdhfa,2014,adskf,kadsfjh,123-4567,none,none,none,none,none
(Lack of "BOOP" on the second book)
What I want is to scan Book1.csv for column 11. If it's there, find the matching row in Book2.csv based on asset and waiver. Then simply append the target to that line.
Here's what I've tried so far:
#!/bin/bash
oldIFS=IFS
IFS=$'\n'
HOME=($(cat Book1.csv))
for i in "${HOME[#]}"
do
target=`echo $i | cut -d "," -f 11`
asset=`echo $i | cut -d "," -f 2`
waiv=`echo $i | cut -d "," -f 5`
if [ "$target" != "target" ]
then
sed -i '/*${asset}*${waiv}*/s/$/,${target}/' Book2.csv
fi
done
IFS=oldIFS
Everything seems to be working except for the sed command. Any suggestions?
You are using
sed -i '/*${asset}*${waiv}*/s/$/,${target}/' Book2.csv
which means that the variables are not expanded (the ' quotes "hide" them).
Also the * needs something "in front of it" - probably you meant to use .* (otherwise you are looking for "any number of repeats of the last character in asset, etc.).
Just change it to
sed -i "/.*${asset}.*${waiv}.*/s/$/,${target}/" Book2.csv
Now the variables will be replaced with their value before the sed command runs, and the quantifier (*) should work properly, as it has something to quantify (.)...
You're using single quotes, which inhibit variable expansion. Change to double quotes.
This awk might be tidier:
awk -F, -v OFS=, '
NR == FNR {boop[$2,$5] = $11; next}
NF != 11 {$11 = boop[$2,$5]}
{print}
' Book1.csv Book2.csv > tmpfile && mv tmpfile Book2.csv
awk does not have a -i option.
I am trying to right a bash script to do multiple checks and searches for a CMS my company uses. I trying to implement a function for a user to be able to search for a certain macro call and the function return all the files that contain the call, the line the macro is called on, and the actual code in the macro call. What I have seems to be getting screwed up by the fact I am using a for loop to format the output. Here's the snippet of the script I am working on:
elif [ "$choice" = "2" ]
then
echo -e "\n What macro call are we looking for $name?"
read macrocall
for i in $(grep -inR "$macrocall" $sitepath/templates/macros/); do
file=$(echo $i | cut -d\: -f1 | awk -F\/ '{ print $NF }')
line=$(echo $i | cut -d\: -f2)
calltext=$(echo $i | cut -d\: -f3-)
echo -e "\nFile: $file"
echo -e "\nLine: $line"
echo -e "\nMacro Call from file: $calltext"
done
fi
the current script runs the first few fields until it gets a a space and then everything gets all screwy. Anybody have any idea how I can have the for loops deliminator to be each result of the grep? any suggestions would be helpful. Let me know if any of you need more info. Thanks!
The right way to do this would be more like:
printf "\n What macro call are we looking for %s?" "$name"
read macrocall
# ensure globbing is off and set IFS to a newline after saving original values
oSET="$-"; set -f; oIFS="$IFS"; IFS=$'\n'
awk -v macrocall="$macrocall" '
BEGIN { lc_macrocall = "\\<" tolower(macrocall) "\\>" }
tolower($0) ~ lc_macrocall {
file=FILENAME
sub(/.*\//,"",file)
printf "\n%s\n", file
printf "\n%d\n", FNR
printf "\nMacro Call from file: %s\n", $0
}
' $(find "$sitepath/templates/macros" -type f -print)
# restore original IFS and globbing values
IFS="$oIFS"; set +f -"$oSET"
This solves the problem of having spaces in your file names as originally requested, but also handles globbing characters in your file names, and the various typical echo issues.
You can set the internal field separator $IFS (which is normally set to space, tab and newline) to just newline to get around this problem:
IFS="\n"