Writing a script to optimize my images for the web. Having issues with filenames and directories with spaces in the names.
Heres what I have:
read -p "Enter full path from root (/) to your site... example /var/www/public_html: " path1
echo ""
#read -p "Enter in ImageMagick quality (default is 80) if unsure enter 80: " optjpg
#echo ""
#id="$(id -u optiimage)"
cmd="id -u optiimage"
eval $cmd
id=$(eval $cmd)
tmp1="${path1}/shell/optiimage/imagemagick"
tmp2="${path1}/shell/optiimage/imagemagick/jpg"
restore1="${path1}/shell/optiimage/restore"
restore2="${path1}/shell/optiimage/restore/imagemagick/jpg"
backup1="${path1}/shell/optiimage/backup"
backup2="${path1}/shell/optiimage/backup/imagemagick/jpg"
log1="${path1}/shell/optiimage/log/imagemagick/"
DATE="$(date +%a-%b-%y-%T)"
# Need user input for www path from root
##
## Make directories
##
############################################################################################################
mkdir -p ${tmp1}
mkdir -p ${tmp2}
mkdir -p ${restore1}
mkdir -p ${restore2}
mkdir -p ${backup1}
mkdir -p ${backup2}
mkdir -p ${log1}
mkdir -p ${path1}/build
echo "Processing JPG Files"
find $path1 -iname "*jpg" | \
#write out script to put on cron for image optimization
while read file;
do
# If not equal to optimage uid
# to check username id -u optimage
if [ -u "${id}" ]; then
filebase=`basename "$file" .jpg`
dirbase=`dirname "$file"`
echo "${dirbase}/${filebase}.jpg already optimized" >> ${log1}_optimized_$DATE.log
else
#simple log for size of image before optimization
ls -s $file >> ${log1}_before_$DATE.log
#Do the following if *.jpg found
filebase=`basename $file .jpg`
dirbase=`dirname $file`
echo "cp -p ${dirbase}/${filebase}.jpg ${tmp2}" >> ${path1}/build/backup_jpg.txt
echo "chown optiimage:www-data ${filebase}.jpg" >> ${path1}/build/restore_jpg.txt #${restore1}/imagemagick.sh
echo "cp -p ${filebase}.jpg ${dirbase}/${filebase}.jpg" >> ${path1}/build/restore_jpg.txt #${restore1}/imagemagick.sh
##
## ImageMagick
## Original Command:
## convert $file -quality 80 ${filebase}.new.jpg
##########################
echo "convert ${dirbase}/${filebase}.jpg -quality 80 ${tmp2}/${filebase}.jpg" >> ${path1}/build/imagemagick.txt
echo "mogrify -strip ${tmp2}/${filebase}.jpg" >> ${path1}/build/imagemagick.txt
echo "chown optiimage:www-data ${tmp2}/${filebase}.jpg" >> ${path1}/build/owner_jpg.txt
echo "rm ${dirbase}/${filebase}.jpg" >> ${path1}/build/remove_jpg.txt
echo "cp -p ${tmp2}/${filebase}.jpg ${dirbase}/" >> ${path1}/build/migrate_jpg.txt
simple log for size of image after optimization
ls -s $file >> ${log1}_after_$DATE.log
fi
done
I have edited this with suggestions some have given me. It didn't seem to work.
This works fine if I remove directories with spaces in the names otherwise it ends the name at the space and get errors directory doesn't exist.
You need to double-quote variable substitutions. This applies inside command substitutions as well as in the top-level lexical context. The only exception to this is assignment of a string variable from another string variable, e.g. str2=$str1;, although other types of variable assignments generally need quoting, such as assigning a string variable from an array slice, even if it only slices one element, e.g. str="${#:1:1}";.
Although unlikely to be a problem here, the read builtin strips leading and trailing whitespace if you provide one or more NAMEs; you can solve that by not providing any NAMEs at all, and just letting it store the whole line in the $REPLY variable by default.
You should always use the -r option of the read builtin, as that prevents its ill-advised default behavior of doing backslash interpolation/removal on the input data.
If you don't need any kind of interpolation in a string literal, prefer the '...' syntax to "...", as the former does not do any interpolation.
Prefer the [[ ... ]] expression evaluation form to the old-style [ ... ] form, as the former syntax is slightly more powerful.
Prefer the $(...) command substitution form to the old-style `...` form, as the former syntax has more favorable nesting properties (namely, no need to escape the nested command substitution delimiters).
find "$path1" -iname '*jpeg'| \
# write out script to put on cron for image optimization
while read -r; do
file=$REPLY;
# If not equal to optimage uid
# to check username id -u optimage
if [[ -u "$id" ]]; then
filebase=$(basename "$file" .jpeg);
dirbase=$(dirname "$file");
#MYBASENAME=$(basename "$1")
echo "${dirbase}/${filebase}.jpeg already optimized" >>"${log1}_optimized_$DATE.log";
fi;
done;
;
Quote your $file variable in every place where is used:
find $path1 -iname "*jpeg" | \
while read file;
do
if [ -u "${id}" ]; then
filebase=`basename "$file" .jpeg`
dirbase=`dirname "$file"`
fi
done
Related
I have written a shell script (bash) which runs some commands. It has an option to not to run the commands but to echo them to the screen. By default, the output of these commands is redirected to /dev/null but there is another option to show the output on the screen.
I use a function to check for the value of these variables and run the commands or simulate them:
runmaybe() {
if [[ true = $dry_run ]]; then
echo "Simulating '$#'"
else
if [[ true = $verbose ]]; then
$#
else
$# > /dev/null
fi
fi
}
The function is working but I had some issues with complex commands such as:
runmaybe eval svn cp $url $root/tags/$ntag -m \"Tagging revision with $ntag\"
I had to add the eval to prevent wordsplitting so svn gets the right value for the -m option.
I have some other complex commands in that script such as:
runmaybe vzctl exec 1 "( cd /var/wwww/vhosts/myhost ; php cron.php )"
runmaybe ssh -t user#$host "vzctl exec $vmid \"( /usr/local/bin/myscript )\"" 2>/dev/null
runmaybe rsync --delete --exclude=\"**/.svn/\" --exclude=\"**/.git/\" --include=*.exe --numeric-ids -a $vOpt -H $LOCAL_VM$dir $host:$REMOTE_VM$dir
Although the script is working right now, I wonder if there is a better way of doing this task.
The problem is in unquoted expansion of $#. As a rule of a thumb, if you see $, you should put it inside ". Unquoted expansions undergo word splitting and filename expansions - to prevent them, quote the expansion.
I had to add the eval
eval is evil. Do not use it.
runmaybe() {
if [[ true = $dry_run ]]; then
echo "Simulating '$*'"
# or better quote with printf in some corner cases
printf "Simulating:"
printf " %q" "$#"
printf "\n"
elif [[ true = $verbose ]]; then
"$#"
else
"$#" > /dev/null
fi
}
runmaybe svn cp "$url" "$root/tags/$ntag" -m "Tagging revision with $ntag"
runmaybe rsync --delete --exclude="**/.svn/" --exclude="**/.git/" --include="*.exe" --numeric-ids -a "$vOpt" -H "$LOCAL_VM$dir" "$host:$REMOTE_VM$dir"
I have a small script for Mac where I'm adding printers. It works fine but I think I could make it simpler or at least it would be interesting to know a different solution.
while IFS= read -r line;
do
if [[ $line == *"Printer_E1"* ]]
then
if [[ "$FIND_PRINTERS" =~ "$PRINTER_E1_IP" ]];
then
echo "found printer e1"
else
echo "adding printer e1"
"$LPADMIN" -p "$PRINTER_E1_IP" -v "lpd://$PRINTER_E1_IP" -L "$PRINTER_E1_LOCATION" -P "$PRINTER_E1_PPD" -E -o printer-is-shared=false -D "$PRINTER_E1_NAME"
echo "adding printer e1 done"
fi
fi
done <<< "$AD_GROUPS"
The content of $AD_GROUPS is:
Printer_E0
Printer_E1
Printer_E2
Printer_E3
Printer_E4
Printer_Strasse
Printer_Wien
I have such a loop for 5 printers, so 5 times that just with different variables.
How could I do that with one loop? (or how can I make that different or simpler)?
Something like this:
while IFS= read -r printer; do
[[ "$FIND_PRINTERS" =~ "${printer}_IP" ]] && \
echo "Found ${printer}" && continue
echo "Adding ${printer}..."
"$LPADMIN" -p "${printer}_IP" \
-v "lpd://${printer}_IP" \
-L "${printer}_LOCATION" \
-P "${printer}_PPD" -E -o printer-is-shared=false \
-D "${printer}_NAME" \
&& echo "Done"
done <<< "$AD_GROUPS"
I assume your variable FIND_PRINTERS has some printers you want to skip, that you have already set the parameters (IP, LOCATION etc) related to each printer.
We use the variable inside double quotes into there, so it expands to what you want for the various commands. Also I have simplified the if condition then command to condition && command and also continue moves to next iteration.
I got some code from here that works pretty well until I get "Argument list too long"
I am NOT a developer and pretty old too :) so if it is not much to ask please explain.
Is there a way the expand DIRCMD like eval does and pass each of the commands one at the time so eval does not break?
for (( ifl=0;ifl<$((NUMFIRSTLEVELDIRS));ifl++ )) { FLDIR="$(get_rand_dirname)"
FLCHILDREN="";
for (( ird=0;ird<$((DIRDEPTH-1));ird++ )) {
DIRCHILDREN=""; MOREDC=0;
for ((idc=0; idc<$((MINDIRCHILDREN+RANDOM%MAXDIRCHILDREN)); idc++)) {
CDIR="$(get_rand_dirname)" ;
# make sure comma is last, so brace expansion works even for 1 element? that can mess with expansion math, though
if [ "$DIRCHILDREN" == "" ]; then
DIRCHILDREN="\"$CDIR\"" ;
else
DIRCHILDREN="$DIRCHILDREN,\"$CDIR\"" ;
MOREDC=1 ;
fi
}
if [ "$MOREDC" == "1" ] ; then
if [ "$FLCHILDREN" == "" ]; then
FLCHILDREN="{$DIRCHILDREN}" ;
else
FLCHILDREN="$FLCHILDREN/{$DIRCHILDREN}" ;
fi
else
if [ "$FLCHILDREN" == "" ]; then
FLCHILDREN="$DIRCHILDREN" ;
else
FLCHILDREN="$FLCHILDREN/$DIRCHILDREN" ;
fi
fi
}
cd $OUTDIR
DIRCMD="mkdir -p $OUTDIR/\"$FLDIR\"/$FLCHILDREN"
eval "$DIRCMD"
echo "$DIRCMD"
}
I tried echo $DIRCMD but do not get the expanded list of commands
'echo mkdir -p /mnt/nvme-test/rndpath/"r8oF"/{"rc","XXKR","p0H"}/{"5Dw0K","oT","rV","coU","uo"}/{"3m5m","uEdA","w4SJp","49"}'
I had trouble following the code, but if I understood it correctly, you dynamically generate a mkdir -p command with a brace expansion:
'mkdir -p /mnt/nvme-test/rndpath/"r8oF"/{"rc","XXKR","p0H"}/{"5Dw0K","oT","rV","coU","uo"}/{"3m5m","uEdA","w4SJp","49"}'
Which then fails when you eval it due to your OS' maximum argument limit.
To get around that, you can instead generate a printf .. command since this is a Bash builtin and not subject to the argument limit, and feed its output to xargs:
dircmd='printf "%s\0" /mnt/nvme-test/rndpath/"r8oF"/{"rc","XXKR","p0H"}/{"5Dw0K","oT","rV","coU","uo"}/{"3m5m","uEdA","w4SJp","49"}'
eval "$dircmd" | xargs -0 mkdir -p
If your xargs doesn't support -0, you can instead use printf "%s\n" and xargs mkdir -p, though it won't behave as well if your generated names contain spaces and such.
If this is for benchmarking, you may additionally be interested to know that you can now use xargs -0 -n 1000 -P 8 mkdir -p to run 8 mkdirs in parallel, each creating 1000 dirs at a time.
I am a beginner and trying to write a script that takes a config file (example below) and sets the rights for the users, if that user or group doesn´t exist, they get added.
For every line in the file, I am cutting out the user or the group and check if they exist.
Right now I only check for users.
#!/bin/bash
function SetRights()
{
if [[ $# -eq 1 && -f $1 ]]
then
for line in $1
do
var1=$(cut -d: -f2 $line)
var2=$(cat /etc/passwd | grep $var1 | wc -l)
if [[ $var2 -eq 0 ]]
then
sudo useradd $var1
else
setfacl -m $line
fi
done
else
echo Enter the correct path of the configuration file.
fi
}
SetRights $1
The config file looks like this:
u:TestUser:- /home/temp
g:TestGroup:rw /home/temp/testFolder
u:TestUser2:r /home/temp/1234.txt
The output:
grep: TestGroup: No such file or directory
grep: TestUser: No such file or directory
"The useradd help menu"
If you could give me a hint what I should look for in my research, I would be very grateful.
Is it possible to reset var1 and var2? Using unset didn´t work for me and I couldn´t find variables could only be set once.
It's not clear how you are looping over the contents of the file -- if $1 contains the file name, you should not be seeing the errors you report.
But anyway, here is a refactored version which hopefully avoids your problems.
# Avoid Bash-only syntax for function definition
SetRights() {
# Indent function body
# Properly quote "$1"
if [[ $# -eq 1 && -f "$1" ]]
then
# Read lines in file
while read -r acl file
do
# Parse out user
user=${acl#*:}
user=${user%:*}
# Avoid useless use of cat
# Anchor regex correctly
if ! grep -q "^$user:" /etc/passwd
then
# Quote user
sudo useradd "$user"
else
setfacl -m "$acl" "$file"
fi
done <"$1"
else
# Error message to stderr
echo Enter the correct path of the configuration file. >&2
# Signal failure to the caller
return 1
fi
}
# Properly quote argument
SetRights "$1"
I am missing my bash aliases in fish, and don't want to manually convert all of them to fish functions.
How to get them all from bash to fish?
Bonus points if:
the solution supports an iterative process, as in: i can easily change the aliases in bash, and re-convert/re-import them into fish
the solution also imports bash functions
Convert bash aliases to bash scripts
I decided to do this instead of approach below, and putting the scripts into ~/bin/, which is in my PATH. This allows me to use them from whatever shell i am currently executing, and it prevents potential problems with quoting.
use like so:
# converts all bash aliases to script files
convert_bash_aliases_to_scripts
# removes all scripts previously converted by this script
convert_bash_aliases_to_scripts clean
bash conversion script:
#!/bin/bash
# Convert bash aliases to bash scripts.
#
# Copyright 2018 <hoijui.quaero#gmail.com>, licensed under the GPL-3.0+
#
# Usage:
# convert_bash_aliases_to_scripts # converts all bash aliases to script files
# convert_bash_aliases_to_scripts clean # removes all scripts previously converted by this script
COLOR_RED=$'\e[0;31m'
COLOR_ORANGE=$'\e[0;33m'
COLOR_BLUE=$'\e[0;34m'
COLOR_BLUE_LIGHT=$'\e[1;34m'
COLOR_GREEN=$'\e[0;32m'
COLOR_BROWN=$'\e[0;33m'
COLOR_YELLOW=$'\e[1;33m'
COLOR_WHITE=$'\e[1;37m'
COLOR_CYAN=$'\e[0;36m'
COLOR_PURPLE=$'\e[0;35m'
COLOR_GRAY=$'\e[1;30m'
COLOR_GRAY_LIGHT=$'\e[0;37m'
COLOR_NONE=$'\e[m' # No Color
OUTPUT_DIR=~/bin/converted/aliases
LINKS_DIR=~/bin
README_FILE_NAME="README.md"
README_FILE="$OUTPUT_DIR/$README_FILE_NAME"
if [ "$1" = "clean" ]
then
for script_file in $(find "$LINKS_DIR" -maxdepth 1 -type l)
do
conv_script_file="$OUTPUT_DIR/$(basename $script_file)"
if [ -e $conv_script_file ] && [ "$(readlink --canonicalize $script_file)" = "$(realpath $conv_script_file)" ]
then
script_name=$(basename $script_file)
echo "removing converted bash alias-script: $script_name"
rm $conv_script_file \
&& rm $script_file
fi
done
rm $README_FILE 2> /dev/null
rmdir $OUTPUT_DIR 2> /dev/null
exit 0
fi
SOURCE_FILES="${HOME}/.bashrc ${HOME}/.bash_aliases"
mkdir -p $OUTPUT_DIR
echo -e "# Bash alias conversion scripts\n\nsee $0\n\nWARNING: Do NOT manually edit files in this directory. instead, copy them to $LINKS_DIR (replacing the symbolic link that already exists there), and edit that new file.\nIf you edit the files in this dir, it will be replaced on the next (re)conversion from aliases." \
> $README_FILE
AUTO_IMPORT_WARNING="# WARNING Do NOT edit this file by hand, as it was auto-generated from a bash alias, and may be overwritten in the future. please read ${README_FILE}"
function _is_link_to {
local file_link=$1
local file_target=$2
test -e $file_target \
&& test "$(readlink --canonicalize $file_link)" = "$(realpath $file_target)"
return $?
}
for source_file in $SOURCE_FILES
do
IFS=$'\n'
for a in $(cat $source_file | grep "^alias")
do
a_name="$(echo "$a" | sed -e 's/alias \([^=]*\)=.*/\1/')"
a_command="$(echo "$a" | sed -e 's/alias \([^=]*\)=//' -e 's/[ \t]*#.*$//')"
if echo "${a_command:0:1}" | grep -q -e "[\'\"]"
then
# unquote
a_command_start=1
let a_command_end="${#a_command} - 2"
else
# leave as is
a_command_start=0
let a_command_end="${#a_command}"
fi
script_file="$LINKS_DIR/$a_name"
conv_script_file="$OUTPUT_DIR/$a_name"
# Check whether the script already exists.
# If so, we skip importing it, unless it is just a link to a previously imported script.
log_action="none"
log_action_color="${COLOR_NONE}"
log_content=""
if [ -e $script_file ] && ! $(_is_link_to $script_file $conv_script_file)
then
log_action="skipped (exists)"
log_action_color="${COLOR_ORANGE}"
log_content=""
else
if [ -e $script_file ]
then
log_action="reimporting"
log_action_color="${COLOR_BLUE}"
else
log_action="importing"
log_action_color="${COLOR_GREN}"
fi
# write the script file to a temporary location
conv_script_file_tmp="${conv_script_file}_BAK"
echo "#!/bin/bash" > $conv_script_file_tmp
echo -e "$AUTO_IMPORT_WARNING" >> $conv_script_file_tmp
echo -e "#\n# Imported bash alias '$a_name' from file '$source_file'" >> $conv_script_file_tmp
cat >> "${conv_script_file_tmp}" <<EOF
${a_command:${a_command_start}:${a_command_end}} \${#}
EOF
if diff -N ${conv_script_file_tmp} ${conv_script_file} > /dev/null
then
log_content="no change"
log_content_color="${COLOR_NONE}"
else
log_content="changed"
log_content_color="${COLOR_GREEN}"
fi
log_content=$(printf "%s %10s -> %s${COLOR_NONE}" "${log_content_color}" "${log_content}" "$a_command")
mv "${conv_script_file_tmp}" "${conv_script_file}"
# make the script executable
chmod +x $conv_script_file
# remove the link if it already exists (in case of reimport)
rm $script_file 2> /dev/null
# .. and re-create it as local symbolic link
# to the function in the imports dir
ln --symbolic --relative $conv_script_file $script_file
fi
printf "%s%20s: %-25s${COLOR_NONE}%s\n" "${log_action_color}" "${log_action}" "$a_name" "${log_content}"
done
done
Deprecated: Creating fish wrappers that execute bash code
Below is a script that creates fish script wrappers for the local bash aliases: For each bash alias, it takes the contents, and creates a fish alias/script that executes the code in bash sub-shell.
It is not optimal, but is sufficient for most of my aliases.
WARNING It might happen that the imported function acts differently then in bash. You may loose data or accidentally DDOS your coworkers when using them.
use like so:
# imports (or reimports) all bash aliases into fish functions, permanently
import_bash_aliases
# removes all fish functions previously imported by this script
import_bash_aliases clean
save this in ~/.config/fish/functions/import_bash_aliases.fish:
#!/usr/bin/fish
# Fish function to import bash aliases
#
# Copyright 2018 <hoijui.quaero#gmail.com>, licensed under the GPL-3.0+
#
# This script is based on a script from Malte Biermann,
# see: https://glot.io/snippets/efh1c4aec0
#
# WARNING: There is no guarantee that the imported aliases work the same way
# as they do in bash, so be cautious!
#
# Usage:
# import_bash_aliases # imports (or reimports) all bash aliases into fish functions, permanently
# import_bash_aliases clean # removes all fish functions previously imported by this script from bash aliases
function import_bash_aliases --description 'Converts bash aliases to .fish functions.\nThis might be called repeatedly, and will not override functions that are already defined in fish, except they are merely an older import from this script.'
set -l FISH_FUNCTIONS_DIR ~/.config/fish/functions
set -l BASH_IMPORTS_DIR_NAME bash-imports
set -l BASH_IMPORTS_DIR $FISH_FUNCTIONS_DIR/$BASH_IMPORTS_DIR_NAME
set -l README_FILE $BASH_IMPORTS_DIR/README.md
if test "$argv[1]" = "clean"
for fun_file in (find $FISH_FUNCTIONS_DIR -maxdepth 1 -name '*.fish')
set -l imp_fun_file $BASH_IMPORTS_DIR/(basename $fun_file)
if test -e $imp_fun_file ; and test (readlink --canonicalize $fun_file) = (realpath $imp_fun_file)
set -l fun_name (basename $fun_file '.fish')
echo "removing imported bash alias/function $fun_name"
rm $imp_fun_file
and rm $fun_file
and functions --erase $fun_name
end
end
rm $README_FILE ^ /dev/null
rmdir $BASH_IMPORTS_DIR ^ /dev/null
return 0
end
set -l SOURCE_FILES ~/.bashrc ~/.bash_aliases
mkdir -p $BASH_IMPORTS_DIR
echo -e "# Bash alias imports\n\nsee `$argv[0]`\n\nWARNING: Do NOT manually edit files in this directory. instead, copy them to $FISH_FUNCTIONS_DIR (replacing the symbolic link that already exists there), and edit that new file.\nIf you edit the files in this dir, it will be replaced on the next (re)import from bash aliases." \
> $README_FILE
set -l UNUSED_STUB_MSG "The bash alias corresponding to this function was NOT imported, because a corresponding function already exists at %s\n"
set -l AUTO_IMPORT_WARNING "# WARNING Do NOT edit this file by hand, as it was auto-generated from a bash alias, and may be overwritten in the future. please read {$README_FILE}"
function _fish_func_exists
set -l fun_name $argv[1]
# This also detects in-memory functions
functions --query $fun_name
# This also detects script files in the functions dir
# that do not contain a function wiht the same name
or test -e "$FISH_FUNCTIONS_DIR/$fun_name.fish"
return $status
end
function _is_link_to
set -l file_link $argv[1]
set -l file_target $argv[2]
test -e $file_target
and test (readlink --canonicalize $file_link) = (realpath $file_target)
return $status
end
for source_file in $SOURCE_FILES
for a in (cat $source_file | grep "^alias")
set -l a_name (echo $a | sed -e 's/alias \([^=]*\)=.*/\1/')
set -l a_command (echo $a | sed -e 's/alias \([^=]*\)=//' -e 's/[ \t]*#[^\'\"]\+$//')
set -l fun_file "$FISH_FUNCTIONS_DIR/$a_name.fish"
set -l imp_fun_file "$BASH_IMPORTS_DIR/$a_name.fish"
# Check whether the function already exists.
# If so, we skip importing it, unless it is just a link to a previously imported function.
if _fish_func_exists $a_name; and not _is_link_to $fun_file $imp_fun_file
set_color red
printf "%20s: %-25s\n" "skipping (exists)" $a_name
set_color normal
#printf $UNUSED_STUB_MSG $fun_file > $imp_fun_file
else
set_color green
printf "%20s: %-25s -> %s\n" "(re-)importing" $a_name $a_command
set_color normal
# remove the link, in case of re-importing
rm $fun_file ^ /dev/null
# write the function file
echo "#!/usr/bin/fish" > $imp_fun_file
echo "\
$AUTO_IMPORT_WARNING
function $a_name -d 'bash alias "$a_name" import'
bash -c $a_command' '\$argv''
end
" \
>> $imp_fun_file
# make the script executable
chmod +x $imp_fun_file
# .. and re-create it as local symbolic link
# to the function in the imports dir
ln --symbolic --relative $imp_fun_file $fun_file
end
end
end
# (re-)load all the functions we just defined
exec fish
end