I have been attempting to extract a CSV file full of URL's of images (about 1000).
Each row is a specific product with the first cell labelled "id".
I have taken the ID of each line in excel and created directories for them using a loop with mkdir.
My issue now is that I can't seem to figure out how to download the image, and then immediately store it into these folder's.
What I am attempting here is to use wget by concatenating "fold_name" and "EXT" to get it like a directory "/name_of_folder", and then getting the links to the images (in cell 5,6,7 and 8) and then using wget from these cells, into the directory.
Can anyone assist me with this?
I think this should be straight forward enough.
Thank you!
#!/usr/bin/bash
EXT='/'
while read line
do
fold_name= cut -d$',' -f1
concat= "%EXT" + "%fold_name"
img1= cut -d$',' -f5
img2= cut -d$',' -f6
img3= cut -d$',' -f7
img4= cut -d$',' -f8
wget -O "%img1" "%concat"
wget -O "%img2" "%concat"
wget -O "%img1" "%concat"
wget -O "%img2" "%concat"
done < file.csv
You might use -P switch to designate target directory, consider following simple example using some files from test-images/png repository
mkdir -p black
mkdir -p gray
mkdir -p white
wget -P black https://raw.githubusercontent.com/test-images/png/main/202105/cs-black-000.png
wget -P gray https://raw.githubusercontent.com/test-images/png/main/202105/cs-gray-7f7f7f.png
wget -P white https://raw.githubusercontent.com/test-images/png/main/202105/cs-white-fff.png
will lead to following structure
black
cs-black-000.png
gray
cs-gray-7f7f7f.png
white
cs-white-fff.png
You should use variables names that are less ambiguous.
You need to provide the directory as part of the output filename.
"%" is not a bash variable designator. That is a formatting directive (for bash, awk, C, etc.).
The following will provide what you want.
#!/usr/bin/bash
DBG=1
INPUT="${1}"
INPUT="file.csv"
cat >"${INPUT}" <<"EnDoFiNpUt"
#topic_1,junk01,junk02,junk03,img_101.png,img_102.png,img_103.png,img_104.png
#topic_2,junk04,junk05,junk06,img_201.png,img_202.png,img_203.png,img_204.png
#
topic_1,junk01,junk02,junk03,https://raw.githubusercontent.com/test-images/png/main/202105/cs-black-000.png,https://raw.githubusercontent.com/test-images/png/main/202105/cs-gray-7f7f7f.png,https://raw.githubusercontent.com/test-images/png/main/202105/cs-white-fff.png
EnDoFiNpUt
if [ ${DBG} -eq 1 ]
then
echo -e "\n Input file:"
cat "${INPUT}" | awk '{ printf("\t %s\n", $0 ) ; }'
echo -e "\n Hit return to continue ..." ; read k
fi
REPO_ROOT='/tmp'
grep -v '^#' "${INPUT}" |
while read line
do
topic_name=$(echo "${line}" | cut -f1 -d\, )
test ${DBG} -eq 1 && echo -e "\t topic_name= ${topic_name} ..."
folder="${REPO_ROOT}/${topic_name}"
test ${DBG} -eq 1 && echo -e "\t folder= ${folder} ..."
if [ ! -d "${folder}" ]
then
mkdir "${folder}"
else
rm -f "${folder}/"*
fi
if [ ! -d "${folder}" ]
then
echo -e "\n Unable to create directory '${folder}' for saving downloads.\n Bypassing 'wget' actions ..." >&2
else
test ${DBG} -eq 1 && ls -ld "${folder}" | awk '{ printf("\n\t %s\n", $0 ) ; }'
url1=$(echo "${line}" | cut -d\, -f5 )
url2=$(echo "${line}" | cut -d\, -f6 )
url3=$(echo "${line}" | cut -d\, -f7 )
url4=$(echo "${line}" | cut -d\, -f8 )
test ${DBG} -eq 1 && {
echo -e "\n URLs extracted:"
echo -e "\n\t ${url1}\n\t ${url2}\n\t ${url3}\n\t ${url4}"
}
#imageFile1=$( basename "${url1}" | sed 's+^img_+yourImagePrefix_+' )
#imageFile2=$( basename "${url2}" | sed 's+^img_+yourImagePrefix_+' )
#imageFile3=$( basename "${url3}" | sed 's+^img_+yourImagePrefix_+' )
#imageFile4=$( basename "${url4}" | sed 's+^img_+yourImagePrefix_+' )
imageFile1=$( basename "${url1}" | sed 's+^cs-+yourImagePrefix_+' )
imageFile2=$( basename "${url2}" | sed 's+^cs-+yourImagePrefix_+' )
imageFile3=$( basename "${url3}" | sed 's+^cs-+yourImagePrefix_+' )
test ${DBG} -eq 1 && {
echo -e "\n Image filenames assigned:"
#echo -e "\n\t ${imageFile1}\n\t ${imageFile2}\n\t ${imageFile3}\n\t ${imageFile4}"
echo -e "\n\t ${imageFile1}\n\t ${imageFile2}\n\t ${imageFile3}"
}
test ${DBG} -eq 1 && {
echo -e "\n WGET process log:"
}
### This form of wget does NOT work for me, although man page says it should.
#wget -P "${folder}" -O "${imageFile1}" "${url1}"
### This form of wget DOES work for me
wget -O "${folder}/${imageFile1}" "${url1}"
wget -O "${folder}/${imageFile2}" "${url2}"
wget -O "${folder}/${imageFile3}" "${url3}"
#wget -O "${folder}/${imageFile3}" "${url3}"
test ${DBG} -eq 1 && {
echo -e "\n Listing of downloaded files:"
ls -l /tmp/topic* 2>>/dev/null | awk '{ printf("\t %s\n", $0 ) ; }'
}
fi
done
The script is adapted for what I had to work with. :-)
i have a code:
L12(){
echo -e "/tftpboot/log/archive/L12/*/*$sn*L12*.log /tftpboot/log/diag/*$sn*L12*.log"
command="| grep -v hdd"
}
getlog(){
echo $(ls -ltr $(${1}) 2>/dev/null `${command}` | tail -1)
}
however $command does not seem to be inserting | grep -v hdd correctly
i need $command to be either empty or | grep
is there a simple solution to my issue or should i go for different approach
edit:
there may be another problem in there
i am loading a few "modules"
EVAL.sh
ev(){
case "${1}" in
*FAIL*) paint $red "FAIL";;
*PASS*) paint $green "PASS";;
*)echo;;
esac
result=${1}
}
rackinfo.sh (the "main script")
#! /bin/bash
#set -x
n=0
for src in $(ls modules/)
do
source modules/$src && ((n++))
## debugging
# source src/$src || ((n++)) || echo "there may be an issue in $src"
done
## debugging
# x=($n - $(ls | grep src | wc -l))
# echo -e "$x plugin(s) failed to laod correctly"
# echo -e "loaded $n modules"
########################################################################
command=cat
tests=("L12" "AL" "BI" "L12-3")
while read sn
do
paint $blue "$sn\t"
for test in ${tests[#]}
do
log="$(ev "$(getlog ${test})")"
if [[ -z ${log} ]]
then
paint $cyan "${test} "; paint $red "!LOG "
else
paint $cyan "${test} ";echo -ne "$log "
fi
done
echo
done <$1
the results i get are still containing "hdd" for L12()
Set command to cat as a default.
Also, it's best to use an array for commands with arguments, in case any of the arguments is multiple words.
There's rarely a reason to write echo $(command). That's essentially the same as just writing command.
#default command does nothing
command=(cat)
L12(){
echo -e "/tftpboot/log/archive/L12/*/*$sn*L12*.log /tftpboot/log/diag/*$sn*L12*.log"
command=(grep -v hdd)
}
getlog(){
ls -ltr $(${1}) 2>/dev/null | "${command[#]}" | tail -1)
}
I run the below script in a reference env and get the correct output in the output file, but when I run the same in production the output file is empty.
I tried debugging the using set -x, and understand that the for loop is not getting executed in the prod env.
please suggest what might be the issue.
#!/bin/ksh
DIR=/some/log/dir/error
DATE=$1
OUTPUT=/some/logs/dir/scripts/output/output.csv
. /calling/env/setupscript.ksh
for file in $(find $DIR 2>/dev/null| grep $ASOF | grep -i something1 | grep -vi someotherthing2 | grep -iv someotherthing3 | grep -vi someotherthing4 | grep -vi someotherthing4 )
do
echo "Checking file $file..."
if [[ -z "$DONE" ]] then
head -1 $file | read line
echo "File name;Filter name;Type;Scenario;$line" > $OUTPUT.tmp
DONE=1
fi
fullfile=$file
file=$(basename $file)
file=${file#SPR_RPT_}
file=${file#SPR_BY__}
echo $file | awk -F_ '{ print $(NF-2)" "$(NF-1) }' | read type scen
filter=${file%%_$type\_*}
grep ERROR $fullfile | while read line
do
echo "$(basename $fullfile);$filter;$type;$scen;$line" >> ${OUTPUT}.tmp
done
done
I have a issue with my Script, i am just trying to fingure out if my screen session is running or not (line 19).
The rest of the script is working.
#!/bin/bash
echo $PATH // /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
echo "0"
content=$(wget http://interwebs.com/index.php?page=count -q -O -)
z=$(($content / 5))
z=$(($z + 1))
echo $z // 4
lockfile=/var/tmp/mylock
if ( set -o noclobber; echo "$$" > "$lockfile") 2> /dev/null; then
trap 'rm -f "$lockfile"; exit $?' INT TERM EXIT
# do stuff here
x=1
count=0
while [ $x -le $z ]
do
$req ="$(ps -ef | grep -i mystatus$count | grep -v grep)"
if [ "$req" = "" ]; then
# run bash script
screen -amds mystatus$count /usr/bin/wget --spider interwebs.com/index.php?page=cronwhatsoever$(( $count +1))-$(( $count +5))
else
echo "Cron running"
fi
x=$(( $x + 1 ))
count=$(( $count +5))
done
# clean up after yourself, and release your trap
rm -f "$lockfile"
trap - INT TERM EXIT
else
echo "Lock Exists: $lockfile owned by $(cat $lockfile)"
fi
sleep 15
It returns line 19: =: command not found. Actually running:
ps -ef | grep -i bukkit | grep -v grep
Works without issues if i run it directly in my Terminal, so any idea how to solve this issue?
I guess it something PATH related but grep is located in /bin/grep.
$req ="$(ps -ef | grep -i mystatus$count | grep -v grep)"
should be
req="$(ps -ef | grep -i mystatus$count | grep -v grep)"
Don't use $ on the left-hand side of an assignment, and you must not have spaces around the =
filein="users.csv"
IFS=$'\n'
if [ ! -f "$filein" ]
then
echo "Cannot find file $filein"
else
#...
groups=(`cut -d: -f 6 "$filein" | sed 's/ //'`)
fullnames=(`cut -d: -f 1 "$filein"`)
userid=(`cut -d: -f 2 "$filein"`)
usernames=(`cut -d: -f 1 "$filein" | tr [A-Z] [a-z] | awk '{print substr($1,1,1) $2}'`)
#...
for group in ${groups[*]}
do
grep -q "^$group" /etc/group ; let x=$?
if [ $x -eq 1 ]
then
groupadd "$group"
fi
done
#...
x=0
created=0
for user in ${usernames[*]}
do
useradd -n -c ${fullnames[$x]} -g "${groups[$x]}" $user 2> /dev/null
if [ $? -eq 0 ]
then
let created=$created+1
fi
#...
echo "${userid[$x]}" | passwd --stdin "$user" > /dev/null
#...
echo "Welcome! Your account has been created. Your username is $user and temporary
password is \"$password\" without the quotes." | mail -s "New Account for $user" -b root $user
x=$x+1
echo -n "..."
sleep .25
done
sleep .25
echo " "
echo "Complete. $created accounts have been created."
fi
I'm guessing the problem is that you're trying to capture command output in arrays without actually using command substitution. You want something like this:
groups=( $( cut... ) )
Note the extra set of parentheses with $ in front of the inner set.