I am getting a ShellCheck warning [SC2045] for the second line in the code below. Is it fine to ignore it as I am making sure the directory is not empty before I try the last ls?
if [ "$(ls -A "$retryDir")" ] ; then
for thisRetryFile in $(ls "$retryDir"/*.tar.gz) ; do
scp -o ConnectTimeout=30 "$thisRetryFile" \
"$remoteUser#$remoteHost:$remotePath" >> "$BACKUPLOG"
done
fi
UPDATE:
After reading the post comments. I have changed the line to:
for thisRetryFile in "$retryDir"/*.tar.gz ; do
This has removed the warnings.
Use the loop with a glob, and also set nullglob to avoid executing scp in case the pattern doesn't match anything.
And you don't need the outer if condition either,
since the for with nullglob effectively takes care of that:
shopt -s nullglob
for thisRetryFile in "$retryDir"/*.tar.gz; do
scp -o ConnectTimeout=30 "$thisRetryFile" \
"$remoteUser#$remoteHost:$remotePath" >> "$BACKUPLOG"
done
If you want to catch the case when no file matches the pattern,
you can write like this, without using shopt -s nullglob:
for thisRetryFile in "$retryDir"/*.tar.gz; do
if [ -f "$thisRetryFile" ]; then
scp -o ConnectTimeout=30 "$thisRetryFile" \
"$remoteUser#$remoteHost:$remotePath" >> "$BACKUPLOG"
break
else
echo "warn: no tar.gz file in dir: $retryDir"
fi
done
This is safer. Try it.
if [ "$(ls -A "$retryDir")" ] ; then
for thisRetryFile in ${retryDir}'/*.tar.gz' ; do
scp -o ConnectTimeout=30 "$thisRetryFile" "$remoteUser#$remoteHost:$remotePath" >> "$BACKUPLOG"
done
fi
Regards!
Related
I got some code from here that works pretty well until I get "Argument list too long"
I am NOT a developer and pretty old too :) so if it is not much to ask please explain.
Is there a way the expand DIRCMD like eval does and pass each of the commands one at the time so eval does not break?
for (( ifl=0;ifl<$((NUMFIRSTLEVELDIRS));ifl++ )) { FLDIR="$(get_rand_dirname)"
FLCHILDREN="";
for (( ird=0;ird<$((DIRDEPTH-1));ird++ )) {
DIRCHILDREN=""; MOREDC=0;
for ((idc=0; idc<$((MINDIRCHILDREN+RANDOM%MAXDIRCHILDREN)); idc++)) {
CDIR="$(get_rand_dirname)" ;
# make sure comma is last, so brace expansion works even for 1 element? that can mess with expansion math, though
if [ "$DIRCHILDREN" == "" ]; then
DIRCHILDREN="\"$CDIR\"" ;
else
DIRCHILDREN="$DIRCHILDREN,\"$CDIR\"" ;
MOREDC=1 ;
fi
}
if [ "$MOREDC" == "1" ] ; then
if [ "$FLCHILDREN" == "" ]; then
FLCHILDREN="{$DIRCHILDREN}" ;
else
FLCHILDREN="$FLCHILDREN/{$DIRCHILDREN}" ;
fi
else
if [ "$FLCHILDREN" == "" ]; then
FLCHILDREN="$DIRCHILDREN" ;
else
FLCHILDREN="$FLCHILDREN/$DIRCHILDREN" ;
fi
fi
}
cd $OUTDIR
DIRCMD="mkdir -p $OUTDIR/\"$FLDIR\"/$FLCHILDREN"
eval "$DIRCMD"
echo "$DIRCMD"
}
I tried echo $DIRCMD but do not get the expanded list of commands
'echo mkdir -p /mnt/nvme-test/rndpath/"r8oF"/{"rc","XXKR","p0H"}/{"5Dw0K","oT","rV","coU","uo"}/{"3m5m","uEdA","w4SJp","49"}'
I had trouble following the code, but if I understood it correctly, you dynamically generate a mkdir -p command with a brace expansion:
'mkdir -p /mnt/nvme-test/rndpath/"r8oF"/{"rc","XXKR","p0H"}/{"5Dw0K","oT","rV","coU","uo"}/{"3m5m","uEdA","w4SJp","49"}'
Which then fails when you eval it due to your OS' maximum argument limit.
To get around that, you can instead generate a printf .. command since this is a Bash builtin and not subject to the argument limit, and feed its output to xargs:
dircmd='printf "%s\0" /mnt/nvme-test/rndpath/"r8oF"/{"rc","XXKR","p0H"}/{"5Dw0K","oT","rV","coU","uo"}/{"3m5m","uEdA","w4SJp","49"}'
eval "$dircmd" | xargs -0 mkdir -p
If your xargs doesn't support -0, you can instead use printf "%s\n" and xargs mkdir -p, though it won't behave as well if your generated names contain spaces and such.
If this is for benchmarking, you may additionally be interested to know that you can now use xargs -0 -n 1000 -P 8 mkdir -p to run 8 mkdirs in parallel, each creating 1000 dirs at a time.
Some commands have many -x (x can be any English letter) options and it's some times difficult to remember all of their meanings. I can use bash's compgen -W '-a -b -c' to show possible options and I'm wondering if it's possible to also show some help message. Like this:
bash# foo -<TAB><TAB>
-a: This is option a -b: This is option b
-C: This is option c
bash#
I ever did something similar to map some of curl's single char options (like -x) to GNU style --long-options.
This is how it works:
[STEP 101] # cat curl
function _compgen_curl()
{
local cmd=$1 cur=$2 pre=$3
local -a options=( \
'' --connect-timeout \
-k --insecure \
-m --max-time \
-o --output \
-O --remote-name \
-u --user \
-U --proxy-user
-x --proxy \
-y --speed-time \
-Y --speed-limit \
)
local -a options2=()
local i short long
for ((i = 0; i < ${#options[#]}; i += 2)); do
short=${options[i]}
long=${options[i+1]}
if [[ -z $short || -z $long ]]; then
options2+=( $short$long )
else
options2+=( $short,$long )
fi
done
if [[ $cur == - ]]; then
COMPREPLY=( $( compgen -W "${options2[*]}" -- "$cur" ) )
elif [[ $cur == --* ]]; then
COMPREPLY=( $( compgen -W "${options[*]}" -- "$cur" ) )
fi
}
complete -F _compgen_curl -o bashdefault -o default curl
[STEP 102] # . ./curl
[STEP 103] # curl -<TAB><TAB>
--connect-timeout -o,--output -u,--user -y,--speed-time
-k,--insecure -O,--remote-name -x,--proxy
-m,--max-time -U,--proxy-user -Y,--speed-limit
[STEP 103] # curl -
Not exactly what you asked but you can update it for your own purpose.
(I'm not sure if bash can handle whitespaces in the completion result but at least you can use _ or -. :-)
IMO that would be a bad idea.
But if you really want it, you can do it something like this:
Note that this code is a bit crude and unless I know your use case, I cannot provide exact answer. However, it should be sufficient to give you a general idea.
Original:
complete -o nospace -F _default_completion my_command
New:
_custom_completion(){
local cur;
_get_comp_words_by_ref cur;
_default_completion
if [ ${#COMPREPLY[#]} == 1 ]; then return; fi
local _compreply=()
local reply_entry
local description
for reply_entry in ${COMPREPLY[#]}; do
description=$(generate_description_from_option "$reply_entry")
description=$(printf "%${COLUMNS}s" "$reply_entry : $description" )
_compreply+=$description
done
COMPREPLY=(${_compreply[#]})
} && complete -o nospace -F _custom_completion my_command
With this, bash should show one option per line with description in front of it. Of course, you will need to write generate_description_from_option yourself.
ble.sh provides that, besides many other features.
I'm running it under MacOS El Capitan 10.10.6
among all commands to get my current dir (path I'm running my script from) only this works for me:
FILES="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
But it's not going to work if the folder has whitespace in it's name (aka: "folder name")
How to fix this?
Thank you! )
Update: added a script:
#!/bin/bash
function check ()
{
oldsize=`wc -c <"$1"`
sleep 1
newsize=`wc -c <"$1"`
while [ "$oldsize" -lt "$newsize" ]
do
echo "Not yet..."
oldsize=`wc -c <"$1"`
sleep 1
newsize=`wc -c <"$1"`
done
if [ "$oldsize" -eq "$newsize" ]
then
echo "The file has been copied completely."
fi
}
FILES="$(dirname "${BASH_SOURCE[0]}")/*"
function main
{
for f in $FILES
do
if [[ "$f" =~ \.mkv$ ]];
then
#/////////////////////////////////////////////////////////////////////
check "$f"
(( count = count + 1 ))
g="${f/mkv/avi}"
#LOG_FILE="${g/avi/log}"
#exec > >(tee -a "${LOG_FILE}" )
#exec 2> >(tee -a "${LOG_FILE}" >&2)
now="$(date)"
printf "Current date and time %s\n" "$now"
echo "Processing $f file..."
#avconv -i "${f}" -map 0:0 -map 0:1 -codec copy -sn "${g}"
avconv -i "$f" -map 0 -codec copy "$g"
if [ $? -eq 0 ]; then
echo OK
rm "$f"
else
echo FAIL
rm "$g"
#rm "$LOG_FILE"
return
fi
fi
#/////////////////////////////////////////////////////////////////////
done
}
############
count=0
############
main
if (($count > 0)); then
open "$(dirname "${BASH_SOURCE[0]}")"
fi
exit
I am using Mac OS X 10.11.6, and I have a directory $HOME/tmp. From there, I executed:
$ cd $HOME/tmp
$ pwd
/Users/jleffler/tmp
$ mkdir -p "Spaced Out Directory "/bin
$ export PATH="$PATH:$PWD/$_"
$ cat <<'EOF' > Spaced\ Out\ \ Directory\ \ \ /bin/gorblinsky.sh
> #!/bin/bash
>
> echo "PWD=$PWD"
> DIR="$(dirname "${BASH_SOURCE[0]}")"
> echo "DIR=$DIR"
> cd "$DIR"
> pwd
> echo "PWD=$PWD"
> EOF
$ chmod +x Spaced\ Out\ \ Directory\ \ \ /bin/gorblinsky.sh
$ gorblinsky.sh
PWD=/Users/jleffler/tmp
DIR=/Users/jleffler/tmp/Spaced Out Directory /bin
/Users/jleffler/tmp/Spaced Out Directory /bin
PWD=/Users/jleffler/tmp/Spaced Out Directory /bin
$
This shows that the command $(dirname "${BASH_SOURCE[0]}") can determine the name of the directory where the source for the command is stored.
If the script was going to use the variable $DIR to specify file names, you'd need to be careful (very careful) to ensure it is always properly quoted.
For example:
cp "$DIR/gorblinksky.h" "$HOME/tmp/cobbled together name"
Modern style is to always (double) quote all variable references, even when there's nothing in them that needs protecting (see shellcheck.net for example — and Google Shell Style Guide). I'm old-school enough not to put quotes around names that can't contain spaces or metacharacters, but I guess that is just old-fashioned. For example, I shell-checked a script for playing with RCS version numbers, and it doesn't quote variables containing dotted strings of digits (9.19.2.24 — could be an IBM IPv4 address too) and I was told off for not quoting them, though the file names were already protected with quotes.
Writing a script to optimize my images for the web. Having issues with filenames and directories with spaces in the names.
Heres what I have:
read -p "Enter full path from root (/) to your site... example /var/www/public_html: " path1
echo ""
#read -p "Enter in ImageMagick quality (default is 80) if unsure enter 80: " optjpg
#echo ""
#id="$(id -u optiimage)"
cmd="id -u optiimage"
eval $cmd
id=$(eval $cmd)
tmp1="${path1}/shell/optiimage/imagemagick"
tmp2="${path1}/shell/optiimage/imagemagick/jpg"
restore1="${path1}/shell/optiimage/restore"
restore2="${path1}/shell/optiimage/restore/imagemagick/jpg"
backup1="${path1}/shell/optiimage/backup"
backup2="${path1}/shell/optiimage/backup/imagemagick/jpg"
log1="${path1}/shell/optiimage/log/imagemagick/"
DATE="$(date +%a-%b-%y-%T)"
# Need user input for www path from root
##
## Make directories
##
############################################################################################################
mkdir -p ${tmp1}
mkdir -p ${tmp2}
mkdir -p ${restore1}
mkdir -p ${restore2}
mkdir -p ${backup1}
mkdir -p ${backup2}
mkdir -p ${log1}
mkdir -p ${path1}/build
echo "Processing JPG Files"
find $path1 -iname "*jpg" | \
#write out script to put on cron for image optimization
while read file;
do
# If not equal to optimage uid
# to check username id -u optimage
if [ -u "${id}" ]; then
filebase=`basename "$file" .jpg`
dirbase=`dirname "$file"`
echo "${dirbase}/${filebase}.jpg already optimized" >> ${log1}_optimized_$DATE.log
else
#simple log for size of image before optimization
ls -s $file >> ${log1}_before_$DATE.log
#Do the following if *.jpg found
filebase=`basename $file .jpg`
dirbase=`dirname $file`
echo "cp -p ${dirbase}/${filebase}.jpg ${tmp2}" >> ${path1}/build/backup_jpg.txt
echo "chown optiimage:www-data ${filebase}.jpg" >> ${path1}/build/restore_jpg.txt #${restore1}/imagemagick.sh
echo "cp -p ${filebase}.jpg ${dirbase}/${filebase}.jpg" >> ${path1}/build/restore_jpg.txt #${restore1}/imagemagick.sh
##
## ImageMagick
## Original Command:
## convert $file -quality 80 ${filebase}.new.jpg
##########################
echo "convert ${dirbase}/${filebase}.jpg -quality 80 ${tmp2}/${filebase}.jpg" >> ${path1}/build/imagemagick.txt
echo "mogrify -strip ${tmp2}/${filebase}.jpg" >> ${path1}/build/imagemagick.txt
echo "chown optiimage:www-data ${tmp2}/${filebase}.jpg" >> ${path1}/build/owner_jpg.txt
echo "rm ${dirbase}/${filebase}.jpg" >> ${path1}/build/remove_jpg.txt
echo "cp -p ${tmp2}/${filebase}.jpg ${dirbase}/" >> ${path1}/build/migrate_jpg.txt
simple log for size of image after optimization
ls -s $file >> ${log1}_after_$DATE.log
fi
done
I have edited this with suggestions some have given me. It didn't seem to work.
This works fine if I remove directories with spaces in the names otherwise it ends the name at the space and get errors directory doesn't exist.
You need to double-quote variable substitutions. This applies inside command substitutions as well as in the top-level lexical context. The only exception to this is assignment of a string variable from another string variable, e.g. str2=$str1;, although other types of variable assignments generally need quoting, such as assigning a string variable from an array slice, even if it only slices one element, e.g. str="${#:1:1}";.
Although unlikely to be a problem here, the read builtin strips leading and trailing whitespace if you provide one or more NAMEs; you can solve that by not providing any NAMEs at all, and just letting it store the whole line in the $REPLY variable by default.
You should always use the -r option of the read builtin, as that prevents its ill-advised default behavior of doing backslash interpolation/removal on the input data.
If you don't need any kind of interpolation in a string literal, prefer the '...' syntax to "...", as the former does not do any interpolation.
Prefer the [[ ... ]] expression evaluation form to the old-style [ ... ] form, as the former syntax is slightly more powerful.
Prefer the $(...) command substitution form to the old-style `...` form, as the former syntax has more favorable nesting properties (namely, no need to escape the nested command substitution delimiters).
find "$path1" -iname '*jpeg'| \
# write out script to put on cron for image optimization
while read -r; do
file=$REPLY;
# If not equal to optimage uid
# to check username id -u optimage
if [[ -u "$id" ]]; then
filebase=$(basename "$file" .jpeg);
dirbase=$(dirname "$file");
#MYBASENAME=$(basename "$1")
echo "${dirbase}/${filebase}.jpeg already optimized" >>"${log1}_optimized_$DATE.log";
fi;
done;
;
Quote your $file variable in every place where is used:
find $path1 -iname "*jpeg" | \
while read file;
do
if [ -u "${id}" ]; then
filebase=`basename "$file" .jpeg`
dirbase=`dirname "$file"`
fi
done
So I have been struggling with this task for eternity and still don't get what went wrong. This program doesn't seem to download ANY pdfs. At the same time I checked the file that stores final links - everything stored correctly. The $PDFURL also checked, stores correct values. Any bash fans ready to help?
#!/bin/sh
#create a temporary directory where all the work will be conducted
TMPDIR=`mktemp -d /tmp/chiheisen.XXXXXXXXXX`
echo $TMPDIR
#no arguments given - error
if [ "$#" == "0" ]; then
exit 1
fi
# argument given, but wrong format
URL="$1"
#URL regex
URL_REG='(https?|ftp|file)://[-A-Za-z0-9\+&##/%?=~_|!:,.;]*[-A-Za-z0-9\+&##/%=~_|]'
if [[ ! $URL =~ $URL_REG ]]; then
exit 1
fi
# go to directory created
cd $TMPDIR
#download the html page
curl -s "$1" > htmlfile.html
#grep only links into temp.txt
cat htmlfile.html | grep -o -E 'href="([^"#]+)\.pdf"' | cut -d'"' -f2 > temp.txt
# iterate through lines in the file and try to download
# the pdf files that are there
cat temp.txt | while read PDFURL; do
#if this is an absolute URL, download the file directly
if [[ $PDFURL == *http* ]]
then
curl -s -f -O $PDFURL
err="$?"
if [ "$err" -ne 0 ]
then
echo ERROR "$(basename $PDFURL)">&2
else
echo "$(basename $PDFURL)"
fi
else
#update url - it is always relative to the first parameter in script
PDFURLU="$1""/""$(basename $PDFURL)"
curl -s -f -O $PDFURLU
err="$?"
if [ "$err" -ne 0 ]
then
echo ERROR "$(basename $PDFURLU)">&2
else
echo "$(basename $PDFURLU)"
fi
fi
done
#delete the files
rm htmlfile.html
rm temp.txt
P.S. Another minor problem I have just spotted. Maybe the problem is with the if in regex? I pretty much would like to see something like that there:
if [[ $PDFURL =~ (https?|ftp|file):// ]]
but this doesn't work. I don't have unwanted parentheses there, so why?
P.P.S. I also ran this script on URLs beginning with http, and the program gave the desired output. However, it still doesn't pass the test.