I am looking for a way to declare an array that will work in bash and zsh.
I know I can do simply this in bash:
file_list=$(example_command)
And in zsh I can make it work like this:
file_list=($(example_command))
I know I can just do it with an if statement if I want to, but hope to do it without:
if [ `basename $SHELL`=bash ]; then
file_list=$(example_command)
elif [ `basename $SHELL`=zsh ]; then
file_list=($(example_command))
else
echo "ERROR: UNKNOWN SHELL"
fi
I do not really care about other shells (eg sh). Just looking for something working in zsh and bash which is not too exotic looking.
Your assumptions about bash are wrong. file_list=$(example_command) does not create an array there. file_list=( $(example_command) ) does create an array in both shells, though it's not good practice to do so (can't deal with files with spaces, files whose names could be expanded as a glob, etc).
The following is a good-practice approach (insofar as you can call handling filenames in a newline-delimited stream good-practice when newlines can be literals in filenames) that works in both shells:
file_list=( )
while IFS= read -r line; do file_list+=( "$line" ); done < <(example_command)
Importantly, to use an array in a way that works in both shells, you need to expand it as "${file_list[#]}", not $file_list.
Related
I need to verify that all images mentioned in a csv are present inside a folder. I wrote a small shell script for that
#!/bin/zsh
red='\033[0;31m'
color_Off='\033[0m'
csvfile=$1
imgpath=$2
cat $csvfile | while IFS=, read -r filename rurl
do
if [ -f "${imgpath}/${filename}" ]
then
echo -n
else
echo -e "$filename ${red}MISSING${color_Off}"
fi
done
My CSV looks something like
Image1.jpg,detail-1
Image2.jpg,detail-1
Image3.jpg,detail-1
The csv was created by excel.
Now all 3 images are present in imgpath but for some reason my output says
Image1.jpg MISSING
Upon using zsh -x to run the script i found that my CSV file has a BOM at the very beginning making the image name as \ufeffImage1.jpg which is causing the whole issue.
How can I ignore a BOM(byte-order marker) in a while read operation?
zsh provides a parameter expansion (also available in POSIX shells) to remove a prefix: ${var#prefix} will expand to $var with prefix removed from the front of the string.
zsh also, like ksh93 and bash, supports ANSI C-like string syntax: $'\ufeff' refers to the Unicode sequence for a BOM.
Combining these, one can refer to ${filename#$'\ufeff'} to refer to the content of $filename but with the Unicode sequence for a BOM removed if it's present at the front.
The below also makes some changes for better performance, more reliable behavior with odd filenames, and compatibility with non-zsh shells.
#!/bin/zsh
red='\033[0;31m'
color_Off='\033[0m'
csvfile=$1
imgpath=$2
while IFS=, read -r filename rurl; do
filename=${filename#$'\ufeff'}
if ! [ -f "${imgpath}/${filename}" ]; then
printf '%s %bMISSING%b\n' "$filename" "$red" "$color_Off"
fi
done <"$csvfile"
Notes on changes unrelated to the specific fix:
Replacing echo -e with printf lets us pick which specific variables get escape sequences expanded: %s for filenames means backslashes and other escapes in them are unmodified, whereas %b for $red and $color_Off ensures that we do process highlighting for them.
Replacing cat $csvfile | with < "$csvfile" avoids the overhead of starting up a separate cat process, and ensures that your while read loop is run in the same shell as the rest of your script rather than a subshell (which may or may not be an issue for zsh, but is a problem with bash when run without the non-default lastpipe flag).
echo -n isn't reliable as a noop: some shells print -n as output, and the POSIX echo standard, by marking behavior when -n is present as undefined, permits this. If you need a noop, : or true is a better choice; but in this case we can just invert the test and move the else path into the truth path.
I need to determine which command a shell alias resolves to in bash, programmatically; i.e., I need to write a bash function that will take a name potentially referring to an alias and return the "real" command it ultimately refers to, recursing through chains of aliases where applicable.
For example, given the following aliases:
alias dir='list -l'
alias list='ls'
where my function is dereference_alias,
dereference_alias list # returns "ls"
dereference_alias dir # also returns "ls"
Is there some builtin I don't know about that does this neatly, or shall I resign myself to scraping the output of alias?
Here's a version I wrote that does not rely on any external commands and also handles recursive aliases without creating an infinite loop:
# Arguments:
#
# $1 Command to compact using aliases
#
function command-to-alias()
{
local alias_key
local expansion
local guess
local command="$1"
local search_again="x"
local shortest_guess="$command"
while [[ "${search_again:-}" ]]; do
unset search_again
for alias_key in "${!BASH_ALIASES[#]}"; do
expansion="${BASH_ALIASES[$alias_key]}"
guess="${command/#"$expansion"/$alias_key}"
test "${#guess}" -lt "${#shortest_guess}" || continue
shortest_guess="$guess"
search_again="x"
done
command="$shortest_guess"
done
echo "$command"
}
Here's how I'm doing it, though I'm not sure it's the best way:
dereference_alias () {
# recursively expand alias, dropping arguments
# output == input if no alias matches
local p
local a="$1"
if [[ "alias" -eq $(type -t $a) ]] && p=$(alias "$a" 2>&-); then
dereference_alias $(sed -re "s/alias "$a"='(\S+).*'$/\1/" <<< "$p")
else
echo $a
fi
}
The major downsides here are that I rely on sed, and my means of dropping any arguments in the alias stops at the first space, expecting that no alias shall ever point to a program which, for some reason, has spaces in its name (i.e. alias badprogram='A\ Very\ Bad\ Program --some-argument'), which is a reasonable enough assumption, but still. I think that at least the whole sed part could be replaced by maybe something leveraging bash's own parsing/splitting/tokenization of command lines, but I wouldn't know where to begin.
I am creating a bash shell script that will rename a file extension without having to specify the old file extension name. If I enter "change foo *" to the Terminal in Linux, it will change all file extension to foo.
So lets say I've got four files: "file1.txt", "file2.txt.txt", "file3.txt.txt.txt" and "file4."
When I run the command, the files should look like this: "file1.foo", "file2.txt.foo", "file3.txt.txt.foo" and "file4.foo"
Can someone look at my code and correct it. I would also appreciate it if someone can implement this for me.
#!/bin/bash
shift
ext=$1
for file in "$#"
do
cut=`echo $FILE |sed -n '/^[a-Z0-9]*\./p'`
if test "${cut}X" == 'X'; then
new="$file.$ext"
else
new=`echo $file | sed "s/\(.*\)\..*/\1.$ext/"`
fi
mv $file $new
done
exit
Always use double quotes around variable substitutions, e.g. echo "$FILE" and not echo $FILE. Without double quotes, the shell expands whitespace and glob characters (\[*?) in the value of the variable. (There are cases where you don't need the quotes, and sometimes you do want word splitting, but that's for a future lesson.)
I'm not sure what you're trying to do with sed, but whatever it is, I'm sure it's doable in the shell.
To check if $FILE contains a dot: case "$FILE" in *.*) echo yes;; *) echo no;; esac
To strip the last extension from $FILE: ${FILE%.*}. For example, if $FILE is file1.txt.foo, this produces file1.txt. More generally, ${FILE%SOME_PATTERN} expands to $FILE with a the shortest suffix matching SOME_PATTERN stripped off. If there is no matching suffix, it expands to $FILE unchanged. The variant ${FILE%%SOME_PATTERN} strips the longest suffix. Similarly, ${FILE#SOME_PATTERN} and ${FILE##SOME_PATTERN} strip a suffix.
test "${TEMP}X" == 'X' is weird. This looks like a misremembered trick from the old days. The normal way of writing this is [ "$TEMP" = "" ] or [ -z "$TEMP" ]. Using == instead of = is a bash extension. There used to be buggy shells that might parse the command incorrectly if $TEMP looked like an operator, but these have gone the way of the dinosaur, and even then, the X needs to be at the beginning, because the problematic operators begin with a -: [ "X$TEMP" == "X" ].
If a file name begins with a -, mv will think it's an option. Use -- to say “that's it, no more options, whatever follows is an operand”: mv -- "$FILE" "$NEW_FILE".
This is very minor, but a common (not universal) convention is to use capital letters for environment variables and lowercase letters for internal script variables.
Since you're using only standard shell features, you can start the script with #!/bin/sh (but #!/bin/bash works too, of course).
exit at the end of the script is useless.
Applying all of these, here's the resulting script.
#!/bin/sh
ext="$1"; shift
for file in "$#"; do
base="${file%.*}"
mv -- "$file" "$base.$ext"
done
Not exactly what you are asking about, but have a look at the perl rename utility. Very powerful! man rename is a good start.
Use: for file in *.gif; do mv $file ${file%.gif}.jpg; done
Or see How to rename multiple files
For me this worked
for FILE in `ls`
do
NEW_FILE=${FILE%.*}
NEW_FILE=${NEW_FILE}${EXT}
done
I just want to tell about NEW_FILE=${FILE%.*}.
Here NEW_FILE gets the file name as output. You can use it as you want.
I tested in bash with uname -a = "Linux 2.4.20-8smp #1 SMP Thu Mar 13 17:45:54 EST 2003 i686 i686 i386 GNU/Linux"
I have written a small bash script called "isinFile.sh" for checking if the first term given to the script can be found in the file "file.txt":
#!/bin/bash
FILE="file.txt"
if [ `grep -w "$1" $FILE` ]; then
echo "true"
else
echo "false"
fi
However, running the script like
> ./isinFile.sh -x
breaks the script, since -x is interpreted by grep as an option.
So I improved my script
#!/bin/bash
FILE="file.txt"
if [ `grep -w -- "$1" $FILE` ]; then
echo "true"
else
echo "false"
fi
using -- as an argument to grep. Now running
> ./isinFile.sh -x
false
works. But is using -- the correct and only way to prevent code/option injection in bash scripts? I have not seen it in the wild, only found it mentioned in ABASH: Finding Bugs in Bash Scripts.
grep -w -- ...
prevents that interpretation in what follows --
EDIT
(I did not read the last part sorry). Yes, it is the only way. The other way is to avoid it as first part of the search; e.g. ".{0}-x" works too but it is odd., so e.g.
grep -w ".{0}$1" ...
should work too.
There's actually another code injection (or whatever you want to call it) bug in this script: it simply hands the output of grep to the [ (aka test) command, and assumes that'll return true if it's not empty. But if the output is more than one "word" long, [ will treat it as an expression and try to evaluate it. For example, suppose the file contains the line 0 -eq 2 and you search for "0" -- [ will decide that 0 is not equal to 2, and the script will print false despite the fact that it found a match.
The best way to fix this is to use Ignacio Vazquez-Abrams' suggestion (as clarified by Dennis Williamson) -- this completely avoids the parsing problem, and is also faster (since -q makes grep stop searching at the first match). If that option weren't available, another method would be to protect the output with double-quotes: if [ "$(grep -w -- "$1" "$FILE")" ]; then (note that I also used $() instead of backquotes 'cause I find them much easier to read, and quotes around $FILE just in case it contains anything funny, like whitespace).
Though not applicable in this particular case, another technique can be used to prevent filenames that start with hyphens from being interpreted as options:
rm ./-x
or
rm /path/to/-x
While writing a bash script to help creating polaroid thumbnail using Imagick's convert commmand. I encounter a problem. Although, I manage to work around with this (actually, because convert is flexible enough), I still want to know how to solve this without such specific workaround.
So basically, the bash script will get a caption value which may contain space. I want to use that caption as parameter of convert. If the caption is empty (''), I will not use the option '-caption' for convert command. Like this:
CAPTION="Is this Cute?" # The actual value will be tacked from the parameter of this bash.
IN_FILE="resources/puppy.png"
OUTFILE="resources/puppy_polaroid.png"
# If CAPTION is not empty, reformat CAPTION
if [ "$CAPTION" != "" ]; then CAPTION="-caption \"$CAPTION\""; fi
# otherwise, do not use '-caption' add all
COMMAND="convert $CAPTION \"$IN_FILE\" \"$OUTFILE\""
echo "Command: $COMMAND" #This echo a value command
`$COMMAND`
The echo echoes the value command that can be copied can pasted in a terminal and run. BUT the bash does not run. How I can do this?
NOTE: In case of convert, -caption "" do the job. I know this and currently use this as work around.
Thanks in advance for helps.
EDIT: From the answer, here is the code that work for me now.
... # Get CAPTION and GRAVITY from parameters
if [ "$CAPTION" != "" ]; then ARGS_CAPTION=(-caption "$CAPTION"); fi
if [ "$GRAVITY" != "" ]; then ARGS_GRAVITY=(-gravity "$GRAVITY"); fi
if [ ! -f "$IN_FILE" ]; then echo "The input file does not exist: '$IN_FILE'"; exit; fi
if [ "$OUTFILE" == "" ]; then OUTFILE=${IN_FILE%.*}-${IN_FILE#*.}-polaroid.png; fi
ARGS=("${ARGS_CAPTION[#]}" -thumbnail 480x480 -border 5x5 -pointsize 60 "${ARGS_GRAVITY[#]}" +polaroid -thumbnail 120x120)
echo convert "${ARGS[#]}" "$IN_FILE" "$OUTFILE";
convert "${ARGS[#]}" "$IN_FILE" "$OUTFILE"
I hope that this will be useful for those seeking similar solution.
You'll want to read entry 050 in the BASH FAQ:
I'm trying to put a command in a variable, but the complex cases always fail!
Variables hold data. Functions hold code. Don't put code inside variables! There are many situations in which people try to shove commands, or command arguments, into variables and then run them. Each case needs to be handled separately.
...
I'm constructing a command based on information that is only known at run time
The root of the issue described above is that you need a way to maintain each argument as a separate word, even if that argument contains spaces. Quotes won't do it, but an array will. (We saw a bit of this in the previous section, where we constructed the addrs array on the fly.)
If you need to create a command dynamically, put each argument in a separate element of an array. A shell with arrays (like Bash) makes this much easier. POSIX sh has no arrays, so the closest you can come is to build up a list of elements in the positional parameters. Here's a POSIX sh version of the sendto function from the previous section:
Use an array, as so:
#!/bin/bash
# ^^^ - note the shebang line explicitly using bash, not /bin/sh
CAPTION="Is this Cute?" # The actual value will be tacked from the parameter of this bash.
IN_FILE="resources/puppy.png"
OUTFILE="resources/puppy_polaroid.png"
extra_args=( )
if [[ $CAPTION ]] ; then
extra_args+=( -caption "$1" )
fi
convert "${extra_args[#]}" "$INFILE" "$OUTFILE"
This construct presumes that you're potentially going to be appending numerous extra arguments. Note that += is unsupported in some older versions of bash which are still present on systems deployed in the field (most notably RHEL4). For such older releases it can be necessary to write extra_args=( "${extra_args[#]}" -caption "$1" ) to append to an array.
Putting backticks around $COMMAND on the last line causes the script to try to execute the output of the command rather than the command itself.
$ c='echo hi'
$ `$c`
hi: command not found
This will work:
if [[ "$CAPTION" != "" ]]
then
convert -caption "$CAPTION" "$IN_FILE" "$OUTFILE"
else
convert "$IN_FILE" "$OUTFILE"
fi
CAPTION="$1"
IN_FILE="resources/puppy.png"
OUTFILE="resources/puppy_polaroid.png"
case "$CAPTION" in
"" ) CAPTION="-caption ''";;
* ) CAPTION='-caption "$CAPTION"';;
esac
convert $CAPTION "$IN_FILE" "$OUTFILE"