I have a folder with the following files:
DA-001-car.jpg
DA-001-dog.jpg
DA-001-coffee.jpg
DA-002-house.jpg
DA-003-coffee.jpg
DA-003-cat.jpg
...
I want to generate this (CSV) output:
SKU, IMAGE
DA-001, "DA-001-car.jpg, DA-001-dog.jpg, DA-001-coffee.jpg"
DA-002, "DA-001-house.jpg"
DA-003, "DA-001-coffee.jpg, DA-001-cat.jpg"
I tried to program this in Bash:
#!/bin/bash
echo "SKU, FILE" >> tmp.csv
for file in /home/calvin/test/*.jpg
do
SKU_NAME="${file##*/}"
echo ${SKU_NAME:0:6}, \"inner for-loop?, ?, ?\" >> tmp.csv
done
uniq tmp.csv output.csv
As you can see I'm a noob as for programming :)
Please help me out, thanks in advance!
This will do the trick. This requires GNU awk to output in ascending order. If you don't care about the order, you can use any old awk and remove the PROCINFO line
#!/bin/bash
awk -F- '
BEGIN{
print "SKU, IMAGE"
}
{
sep=!a[$2]?"":", "
a[$2]=a[$2] sep $0
}
END{
PROCINFO["sorted_in"] = "#ind_str_asc" # GNU only feature
for(i in a){print "DA-" i ", " "\"" a[i] "\""}
}' <(find /home/calvin/test -type f -name "*.jpg" -printf "%f\n") > ./tmp.csv
Example Output
$ cat ./tmp.csv
SKU, IMAGE
DA-001, "DA-001-coffee.jpg, DA-001-car.jpg, DA-001-dog.jpg"
DA-002, "DA-002-house.jpg"
DA-003, "DA-003-coffee.jpg, DA-003-cat.jpg"
If the filenames don't contain spaces, you can use sed instead of an inner loop:
printf '%s\n' *.jpg \
| cut -f1,2 -d- \
| sort -u \
| while IFS= read -r sku ; do
echo "$sku",\"$(echo "$sku"* | sed 's/ /, /')\"
done
With the inner loop, you can switch to printf from echo. Sed is used to remove the trailing comma.
printf '%s\n' *.jpg \
| cut -f1,2 -d- \
| sort -u \
| while IFS= read -r sku ; do
printf %s "$sku, \""
for f in "$sku"* ; do
printf '%s, ' "$f"
done | sed 's/, $//'
printf '"\n'
done
If you don't want to parse the output of ls and run sort, you can store the prefixes in an associative array:
#!/bin/bash
declare -A prefix
for jpg in *.jpg ; do
p1=${jpg%%-*}
jpg=${jpg#*-}
p2=${jpg%%-*}
prefix[$p1-$p2]=1
done
for sku in "${!prefix[#]}" ; do
printf '%s, "' "$sku"
for f in "$sku"* ; do
printf '%s, ' "$f"
done | sed 's/, $//'
printf '"\n'
done
awk '
BEGIN {
OFS = ", "
print "SKU", "IMAGE"
for (i=1; i<ARGC; i++) {
curr = fname = ARGV[i]
sub(/-[^-]+$/,"",curr)
if ( curr != prev ) {
if ( i > 1 ) {
print prev, "\"" fnames "\""
}
prev = curr
fnames = ""
}
fnames = (fnames == "" ? "" : fnames OFS) fname
}
print prev, "\"" fnames "\""
exit
}
' /home/calvin/test/*.jpg
SKU, IMAGE
DA-001, "DA-001-car.jpg, DA-001-coffee.jpg, DA-001-dog.jpg"
DA-002, "DA-002-house.jpg"
DA-003, "DA-003-cat.jpg, DA-003-coffee.jpg"
As result of all the replies and advises I'm using this code to achieve the desired output:
#!/bin/bash
echo "SKU, IMAGES" >> output.csv
ls *.jpg | cut -f1,2 -d- | sort -u | while read SKU
do
echo $SKU, \"$(echo "$SKU"* | sed 's/ /, /g')\" >> output.csv
done
Thanks all!
I have this script, and i have multiple results.
How can the multiple results be output in one variable?
Example: results from first loop is English, second loop is Italian.
I need final Results in a Variable : English Italian
for i in $(ls -l $1/$2/);do
if [[ $i =~ .*\.idx$ ]];then
tr -d '\r' < $1/$2/$i > $1/$2/newfile
rm -f $1/$2/$i
mv $1/$2/newfile $1/$2/$i
results=$(cat $1/$2/$i |awk '/^# alt:/ { a[$3] } END { for (l in a) { printf("%s%s", c, l); c = " " } printf("\n") }')
echo "Results for $i : $results"
fi
done
Alternatively, just accumulate your results in another variable:
RESULTS=""
for i in $(ls -l $1/$2/);do
if [[ $i =~ .*\.idx$ ]];then
tr -d '\r' < $1/$2/$i > $1/$2/newfile
rm -f $1/$2/$i
mv $1/$2/newfile $1/$2/$i
results=$(cat $1/$2/$i |awk '/^# alt:/ { a[$3] } END { for (l in a) { printf("%s%s", c, l); c = " " } printf("\n") }')
RESULTS="$RESULTS$results "
fi
done
echo ${RESULTS%" "} #Get rid of the trailing space
You can get all results in a single variable if you use the following syntax:
variable=$(command)
You might find it useful to put your code in a function and then call it like this:
function command() {
# your code from above
}
results=$( command "$#" )
I am trying to flip the contents of any sentence vertically. So each chars of any string will get printed vertically in same line. For Example:
Sample Text: This is an Example
Output expected: T i a E
h s n x
i a
s m
p
l
e
In following direction I am trying to achieve this but not able to yet.
echo "Input provided by user is $#"
for i in $(seq 1 $#); do
echo ${!i} | sed 's/./ &/g' | xargs |tr ' ' '\n'
done
Current output:
T
h
i
s
i
s
a
n
E
x
a
m
p
l
e
Also, This is also not helping
echo Print text vertically | fold -c -w1
T
h
i
s
i
s
a
n
E
x
a
m
p
l
e
More alternatives which did not worked :
#echo "Input provided by user is $#"
for i in $(seq 1 $#); do
content[i]=$(echo ${!i}|fold -c -w1)
#echo ${content[i]}
done
echo ${content[#]}
max variable holds the max length among all words. For your text it would be: length('Example') which is 7 (maximum out of lengths of all words)
Using an awk script file:
$ awk -f script.awk <<< "This is an Example"
TiaE
hsnx
i a
s m
p
l
e
And here is the script:
{
max=0
for(i=1;i<=NF;i++)
max=length($i)>max?length($i):max;
for(j=1;j<=max;j++)
{
for(i=1;i<=NF;i++)
{
temp=substr($i, j, 1);
printf temp==""?" ":temp
}
printf "\n"
}
}
#!/bin/bash
function abc(){
maxIteration=0;
for i in $(seq 1 $#); do
j=$(echo ${!i})
if [ $maxIteration -lt ${#j} ]
then
maxIteration=${#j};
fi
done
COUNTER=0;
while [ $COUNTER -lt $maxIteration ]; do
for i in $(seq 1 $#); do
j=$(echo ${!i})
if [ ${#j} -gt $COUNTER ]
then
echo ${j:$COUNTER:1} | tr '\n' ' ';
else
echo " " | tr '\n' ' ';
fi
done
echo -e "\n"
let COUNTER=COUNTER+1
done
}
abc $#| grep .
I had created some similar script before. A short but complete POC:
#!/bin/bash
count=0
max=0
#first determine the longest string so we can later pad shorter strings with spaces
for i in $(echo "$1" | xargs -d: -i echo {})
do
size=$(echo $i | wc -c)
if [[ $size > $max ]]
then
max=$size
fi
done
files=""
#then echo the strings vertically inside the tmp files
for i in $(echo "$1" | xargs -d: -i echo {})
do
res=$(echo $i | sed 's/./ &/g' | xargs |tr ' ' '\n' > /tmp/$count.out)
#and add spaces at the end
add_space=$((max-$(echo $i | wc -c)))
for space in $(seq 0 $add_space)
do
echo " " >> /tmp/$count.out
done
files=$files" $count.out"
count=$((count+1))
done
#and finally print them side by side
pr -t -J -m -w 70 -S" " $files
I create tmp files under /tmp, echo the string vertical and later use pr to print it out.
% ./s.sh "This is an Example"
T i a E
h s n x
i a
s m
p
l
e
Because Perl is fun:
perl -a script.pl <<< 'This is an Example'
T i a E
h s n x
i a
s m
p
l
e
And the script:
#F = map { [/./g] } #F;
while (grep #{$_}, #F) {
printf "%s ", shift #{$_} || ' ' for #F;
print "\n"
}
Alternative script:
perl -pe '$r.=$/while/\S/&&s/(\S)(\S*)|\B/$r.=($1||$").$";$2/ge}{$_=$r' \
<<< 'This is an Example'
T i a E
h s n x
i a
s m
p
l
e
Could someone help me figure why is the variable "$valorbase" in my script being incremented. It should continue the process for the next line when "$valorbase=8", though it happens only for the first line. Then it starts incrementing the numbers and dont stop when value 8 is reached.
Thanks!
#!/bin/bash
while read -r line <&3
do
valorbase=8
valor=0
echo "$line"
echo "Valor:"
read -r valor
echo "$valor" >&5
echo "||||||""$valor""|" >&6
valordasoma=$(awk -F"|" '{ sum += $1 } END { print sum }' < soma.txt)
var=$(awk -v o1=$valordasoma -v t1=$valorbase 'BEGIN { print (o1 >= t1)?"1":"0" }' < soma.txt)
if [[ $var -ge 1 ]]
then
echo "Valor da soma > que, ou = ao valor base"
echo "Repete comando para a linha"
else [[ $var -lt 1 ]]
until [[ $var -eq 1 ]]
do
echo "Valor:"
read -r valor
echo "$valor" >&5
echo "||||||""$valor""|" >&6
valordasoma=$(awk -F"|" '{ sum += $1 } END { print sum }' < soma.txt)
var=$(awk -v o1=$valordasoma -v t1=$valorbase 'BEGIN { print (o1 >= t1)?"1":"0" }' < soma.txt)
if [[ $var -eq 1 ]]
then
> soma.txt
else
:
fi
done
fi
done 3<resultado.txt 5>soma.txt 6>partidasdobradas.txt
resultado.txt
TEST| 31|02/05/2015|6.1.1.01.001|2.1.1.01.005||8.100|DIVERSOS|||N||| ||S|CB|||05|||||||31.000|N|N|||0.000|
TEST| 34|03/05/2015|6.1.1.01.002|2.1.1.01.005||6.900|DIVERSOS|||N||| ||S|CB|||05|||||||34.000|N|N|||0.000|
TEST| 36|03/05/2015|6.1.1.01.002|2.1.1.01.005||8.900|DIVERSOS|||N||| ||S|CB|||05|||||||36.000|N|N|||0.000|
TEST| 38|04/05/2015|6.1.1.01.001|2.1.1.01.005||13.490|DIVERSOS|||N||| ||S|CB|||05|||||||38.000|N|N|||0.000|
TEST| 64|12/05/2015|6.1.1.01.001|2.1.1.01.005||6.780|DIVERSOS|||N||| ||S|CB|||05|||||||64.000|N|N|||0.000|
When you write to soma.txt using >&5 inside the loop, it writes to the file at the current position in the file. When you truncate the file with > soma.txt, the current position isn't reset, so future writes go into the middle of the file, with a bunch of nulls at the beginning.
You should truncate the file before the loop, then use 5>>soma.txt as the redirection of the loop, so it will write in append mode. This will make it automatically seek to the current end of the file.
#!/bin/bash
>soma.txt
while read -r line <&3
do
valorbase=8
valor=0
echo "$line"
echo "Valor:"
read -r valor
echo "$valor" >&5
echo "||||||""$valor""|" >&6
valordasoma=$(awk -F"|" '{ sum += $1 } END { print sum }' < soma.txt)
var=$(awk -v o1=$valordasoma -v t1=$valorbase 'BEGIN { print (o1 >= t1)?"1":"0" }' < soma.txt)
if [[ $var -ge 1 ]]
then
echo "Valor da soma > que, ou = ao valor base"
echo "Repete comando para a linha"
else [[ $var -lt 1 ]]
until [[ $var -eq 1 ]]
do
echo "Valor:"
read -r valor
echo "$valor" >&5
echo "||||||""$valor""|" >&6
valordasoma=$(awk -F"|" '{ sum += $1 } END { print sum }' < soma.txt)
var=$(awk -v o1=$valordasoma -v t1=$valorbase 'BEGIN { print (o1 >= t1)?"1":"0" }' < soma.txt)
if [[ $var -eq 1 ]]
then
> soma.txt
else
:
fi
done
fi
done 3<resultado.txt 5>>soma.txt 6>partidasdobradas.txt
The first part of my script works on a debian wheezy box:
OUTPUT_DIR=/share/es-ops/Build_Farm_Reports/WorkSpace_Reports
BASE=/export/ws
TODAY=`date +"%m-%d-%y"`
HOSTNAME=`hostname`
WORKSPACES=( "bob_avail" "bob_used" "mel_avail" "mel_used" "sideshow-ws2_avail" "sideshow-ws2_used" )
if ! [ -f $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == "sideshow" ]; then
echo "$HOSTNAME" > $OUTPUT_DIR/$HOSTNAME.csv # with a linebreak
separator="," # defined empty for the first value
for v in "${WORKSPACES[#]}"
do
echo -n "$separator$v" >> $OUTPUT_DIR/$HOSTNAME.csv # append, concatenated, the separator and the value to the file
#separator="," # comma for the next values
done
echo >> $OUTPUT_DIR/$HOSTNAME.csv # add a linebreak (if you want it)
WORKSPACES2=( "bob" "mel" "sideshow-ws2" )
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
elif [ $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == "sideshow" ]; then
WORKSPACES2=( "bob" "mel" "sideshow-ws2" )
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
else
:
fi
and produces daily output like this each time the cron goes off at 3:00AM -8GMT:
sideshow
,bob_avail,bob_used,mel_avail,mel_used,sideshow-ws2_avail,sideshow-ws2_used
09-20-14,470400,1032124,661826,1032124,43443,1032108
09-20-15,470400,1032124,661826,1032124,43443,1032108
09-20-16,470400,1032124,661826,1032124,43443,1032108
But for some reason when I try to run it on these other 3 debian squeeze boxes I get triple commas between values:
case "$HOSTNAME" in
simpsons) WORKSPACES=(bart_avail bart_used homer_avail home_used lisa_avail lisa_used \
marge_avail marge_used releases_avail releases_used rt-private_avail rt-private_used \
simpsons-ws0_avail simpsons-ws0_used simpsons-ws1_avail simpsons-ws1_used simpsons-ws2_avail \
simpsons-ws2_used vsimpsons-ws_avail vsimpsons-ws_used) ;;
moes) WORKSPACES=(barney_avail barney_used carl_avail carl_used lenny_avail lenny_used moes-ws2_avail moes-ws2_used) ;;
flanders) WORKSPACES=(flanders-ws0_avail flanders-ws0_used flanders-ws1_avail flanders-ws1_used flanders-ws2_avail \
flanders-ws2_used maude_avail maude_used ned_avail ned_used rod_avail rod_used todd_avail \
todd_used to-delete_avail to-delete_used) ;;
esac
if ! [ -f $OUTPUT_DIR/$HOSTNAME.csv ]; then
echo "$HOSTNAME" > $OUTPUT_DIR/$HOSTNAME.csv # with a linebreak
separator="," # defined empty for the first value
for v in "${WORKSPACES[#]}"
do
echo -n "$separator$v" >> $OUTPUT_DIR/$HOSTNAME.csv # append, concatenated, the separator and the value to the file
#separator="," # comma for the next values
done
echo >> $OUTPUT_DIR/$HOSTNAME.csv # add a linebreak (if you want it)
case "$HOSTNAME" in
simpsons) WORKSPACES2=(bart homer lisa marge releases rt-private simpsons-ws0 simpsons-ws1 simpsons-ws2 vsimpsons-ws) ;;
moes) WORKSPACES2=(barney carl lenny moes-ws2) ;;
flanders) WORKSPACES2=(flanders-ws0 flanders-ws1 flanders-ws2 maude ned rod todd to-delete) ;;
esac
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
elif [ $OUTPUT_DIR/$HOSTNAME.csv ]; then
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
else
:
fi
which looks like this:
simpsons
,bart_avail,bart_used,homer_avail,home_used,lisa_avail,lisa_used,marge_avail,marge_used,releases_avail,releases_used,rt-private_avail,rt-private_used,simpsons-ws0_avail,simpsons-ws0_used,simpsons-ws1_avail,simpsons-ws1_used,simpsons-ws2_avail,simpsons-ws2_used,vsimpsons-ws_avail,vsimpsons-ws_used
09-21-14,,,43417,1154259,,,2669,1195007,,,3427,1194249,,,2948,162602,,,128174,281377,,,967520,991870,,,85,168836,,,11995,1011937,,,780184,199511,,,14251,22408
Can you guys help me reduce the 3 commas to just 1 between values?
On these 3 boxes (simpsons, moes, and flanders), the only way to get the right avail and used values is to run awk like this:
df -m /export/ws/maude | awk '{if (NR!=1) {print $3, $2}}'
which looks like this:
492 163306
Otherwise if you run it like this:
df -m /export/ws/maude | awk '{print $3, $2}'
you get this:
Used 1M-blocks
492 163306
I fixed the triple comma issue with a work around:
OUTPUT_DIR=/share/es-ops/Build_Farm_Reports/WorkSpace_Reports
BASE=/export/ws
TODAY=`date +"%m-%d-%y"`
HOSTNAME=`hostname`
WORKSPACES=( "bob_avail" "bob_used" "mel_avail" "mel_used" "sideshow-ws2_avail" "sideshow-ws2_used" )
if ! [ -f $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == "sideshow" ]; then
echo "$HOSTNAME" > $OUTPUT_DIR/$HOSTNAME.csv # with a linebreak
separator="," # defined empty for the first value
for v in "${WORKSPACES[#]}"
do
echo -n "$separator$v" >> $OUTPUT_DIR/$HOSTNAME.csv # append, concatenated, the separator and the value to the file
#separator="," # comma for the next values
done
echo >> $OUTPUT_DIR/$HOSTNAME.csv # add a linebreak (if you want it)
WORKSPACES2=( "bob" "mel" "sideshow-ws2" )
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
elif [ $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == "sideshow" ]; then
WORKSPACES2=( "bob" "mel" "sideshow-ws2" )
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
else
:
fi
case "$HOSTNAME" in
simpsons) WORKSPACES=(bart_avail bart_used homer_avail home_used lisa_avail lisa_used marge_avail marge_used releases_avail releases_used rt-private_avail rt-private_used simpsons-ws0_ava
il simpsons-ws0_used simpsons-ws1_avail simpsons-ws1_used simpsons-ws2_avail simpsons-ws2_used vsimpsons-ws_avail vsimpsons-ws_used) ;;
moes) WORKSPACES=(barney_avail barney_used carl_avail carl_used lenny_avail lenny_used moes-ws2_avail moes-ws2_used) ;;
flanders) WORKSPACES=(flanders-ws0_avail flanders-ws0_used flanders-ws1_avail flanders-ws1_used flanders-ws2_avail flanders-ws2_used maude_avail maude_used ned_avail ned_used rod_avail ro
d_used todd_avail todd_used to-delete_avail to-delete_used) ;;
esac
if ! [ -f $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == `hostname` ]; then
echo "$HOSTNAME" > $OUTPUT_DIR/$HOSTNAME.csv # with a linebreak
separator="," # defined empty for the first value
for v in "${WORKSPACES[#]}"
do
echo -n "$separator$v" >> $OUTPUT_DIR/$HOSTNAME.csv # append, concatenated, the separator and the value to the file
#separator="," # comma for the next values
done
echo >> $OUTPUT_DIR/$HOSTNAME.csv # add a linebreak (if you want it)
case "$HOSTNAME" in
simpsons) WORKSPACES2=(bart homer lisa marge releases rt-private simpsons-ws0 simpsons-ws1 simpsons-ws2 vsimpsons-ws) ;;
moes) WORKSPACES2=(barney carl lenny moes-ws2) ;;
flanders) WORKSPACES2=(flanders-ws0 flanders-ws1 flanders-ws2 maude ned rod todd to-delete) ;;
esac
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
sed -i s/,,,/,/g "$OUTPUT_DIR/$HOSTNAME.csv"
elif [ $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == `hostname` ]; then
case "$HOSTNAME" in
simpsons) WORKSPACES2=(bart homer lisa marge releases rt-private simpsons-ws0 simpsons-ws1 simpsons-ws2 vsimpsons-ws) ;;
moes) WORKSPACES2=(barney carl lenny moes-ws2) ;;
flanders) WORKSPACES2=(flanders-ws0 flanders-ws1 flanders-ws2 maude ned rod todd to-delete) ;;
esac
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
sed -i s/,,,/,/g "$OUTPUT_DIR/$HOSTNAME.csv"
else
:
fi
I just put:
sed -i s/,,,/,/g "$OUTPUT_DIR/$HOSTNAME.csv"
I really wish I would have thought of how to do the awk part that eliminated the triple commas in the first place but at least my script works now.