Sometimes I need to quote an entire command line for future evaluation. Usually I do that with:
printf "%q " "$#"
That's short and sweet but the output look awful. Most of the time this doesn't matter but in occasions I want to show it to the user. For example, in a history of executed commands menu that allows for re-execution of entries. That being the case, I would like to quote in a more readable form (closer to what the user itself would have done if he were in charge of quoting). So this:
search 'Wordreference (eng->spa)' utter
would be preferable to this:
search Wordreference\ \(eng-\>spa\) utter
In order to get the first quoted form I could iterate "$#" and do something like what follows for each argument:
[[ $arg == *\ * ]] && arg="'"${arg//\'/\'\\\'\'}"'"
This is not difficult at all but it involves a loop, a conditional string transformation and concatenation of the result of each iteration.
I wonder if there is a more "batteries included" command to do this kind of transformation out of the box.
In the same way you use eval to later execute the string, you can use eval to print it:
eval "echo $yourstring"
This will remove the shell escapes but keep your variable intact.
Related
I try to use automator for renaming multiple files
I got this far:
the first variable must be inside the Exiftool command line
in this case I selected 2 files, but that could be 1 or 100 files
how do I make this happen? is it possible to start from array key 1 instead of array key 0 for the filenames?
The standard way to do this is to store $1 in a variable, then use shift to remove it from the argument list, and then use "$#" to get all of the remaining arguments (i.e. the original "$2" "$3" "$4" ...) Something like this:
RenameTo="$1"
shift
echo "New name: $RenameTo"
echo "files:" "$#"
I'm not sure exactly what you're trying to to with exiftool, so I won't try to give that full command.
Note that the double-quotes aren't required in zsh, but they make this portable to POSIX-compliant shells. Also, echo isn't a very good way to see what a command would do, because it looses the distinction between spaces within an argument (e.g. spaces in the new name, or within a filename) and spaces between arguments (e.g. between the filenamess in a list of them).
I have a situation where a Bash script runs and parses a user-supplied JSON file using jq. Since it's supplied by the user, it's possible for them to include values in the JSON to perform an injection attack.
I'd like to know if there's a way to overcome this. Please note, the setup of: 'my script parsing a user-supplied JSON file' cannot be changed, as it's out of my control. Only thing I can control is the Bash script.
I've tried using jq with and without the -r flag, but in each case, I was successfully able to inject.
Here's what the Bash script looks like at the moment:
#!/bin/bash
set -e
eval "INCLUDES=($(cat user-supplied.json | jq '.Include[]'))"
CMD="echo Includes are: "
for e in "${INCLUDES[#]}"; do
CMD="$CMD\\\"$e\\\" "
done
eval "$CMD"
And here is an example of a sample user-supplied.json file that demonstrates an injection attack:
{
"Include": [
"\\\";ls -al;echo\\\""
]
}
The above JSON file results in the output:
Includes are: ""
, followed by a directory listing (an actual attack would probably be something far more malicious).
What I'd like instead is something like the following to be outputted:
Includes are: "\\\";ls -al;echo\\\""
Edit 1
I used echo as an example command in the script, which probably wasn’t the best example, as then the solution is simply not using eval.
However the actual command that will be needed is dotnet test, and each array item from Includes needs to be passed as an option using /p:<Includes item>. What I was hoping for was a way to globally neutralise injection regardless of the command, but perhaps that’s not possible, ie, the technique you go for relies heavily on the actual command.
You don't need to use eval for dotnet test either. Many bash extensions not present in POSIX sh exist specifically to make eval usage unnecessary; if you think you need eval for something, you should provide enough details to let us explain why it isn't actually required. :)
#!/usr/bin/env bash
# ^^^^- Syntax below is bash-only; the shell *must* be bash, not /bin/sh
include_args=( )
IFS=$'\n' read -r -d '' -a includes < <(jq -r '.Include[]' user-supplied.json && printf '\0')
for include in "${includes[#]}"; do
include_args+=( "/p:$include" )
done
dotnet test "${include_args[#]}"
To speak a bit to what's going on:
IFS=$'\n' read -r -d '' -a arrayname reads up to the next NUL character in stdin (-d specifies a single character to stop at; since C strings are NUL-terminated, the first character in an empty string is a NUL byte), splits on newlines, and puts the result into arrayname.
The shorter way to write this in bash 4.0 or later is readarray -t arrayname, but that doesn't have the advantage of letting you detect whether the program generating the input failed: Because we have the && printf '\0' attached to the jq code, the NUL terminator this read expects is only present if jq succeeds, thus causing the read's exit status to reflect success only if jq reported success as well.
< <(...) is redirecting stdin from a process substitution, which is replaced with a filename which, when read from, returns the output of running the code ....
The reason we can set include_args+=( "/p:$include" ) and have it be exactly the same as include_args+=( /p:"$include" ) is that the quotes are read by the shell itself and used to determine where to perform string-splitting and globbing; they're not persisted in the generated content (and thus later passed to dotnet test).
Some other useful references:
BashFAQ #50: I'm trying to put a command in a variable, but the complex cases always fail! -- explains in depth why you can't store commands in strings without using eval, and describes better practices to use instead (storing commands in functions; storing commands in arrays; etc).
BashFAQ #48: Eval command and security issues -- Goes into more detail on why eval is widely frowned on.
You don't need eval at all.
INCLUDES=( $(jq '.Include[]' user-supplied.json) )
echo "Includes are: "
for e in "${INCLUDES[#]}"; do
echo "$e"
done
The worst that can happen is that the unquoted command substitution may perform word-splitting or pathname expansion where you don't want it (which is a problem in your original as well), but there's no possibility for arbitrary command execution.
I have 2 bash scripts. One is calling another.
Caller.sh
arg1="+hcpu_extra=111 bbb"
str="-y +hcpu_extra=111 bbb"
local cmd_re="(-y)(.*)"
if [[ $str =~ $cmd_re ]]
then
opt=${BASH_REMATCH[1]}
arg=${BASH_REMATCH[2]}
echo "matched $opt"
echo "matched $arg"
fi
./callee.sh -y $arg
## ./callee.sh -y $arg1
I found if I print $arg1 and $arg, they show the same value "+hcpu_extra=111 bbb" on the screen. But when I pass them respectively to the callee.sh as the argument. I got different results . So my question is , what is the difference between $arg and $arg1 from bash interpreter's point of view? .
First, the code won't work right as posted, because local can't be used except in a function.
If you remove the local or put this in a function, the only difference between arg and arg1 is that arg starts with a space (the one that was between "-y" and "+hcpu". but since you're expanding those variables without double-quotes around them, that'll be removed... unless you changed IFS to something that doesn't contain a space.
(BTW, variable references without double-quotes and changes to IFS are both things that can have weird effects, and are best avoided when possible.)
Anyway, my summary is: the posted code doesn't show the effect you've described; you appear to have left out something important. See How to create a Minimal, Complete, and Verifiable example.
In the below code, I would like to have the output of pyfg() passed exactly as echoed (i.e. with the space between -htns and crl being interpreted literally, not as whitespace, by aoeu()) to aoeu(). Of course, the problem is that in aoeu(), $1 is -htns, $2 is crl, and $3, which I don't want at all in this case, is qjkx. I know this example is thoroughly useless, but the real application to which I'm trying to apply this calls an external program in place of the below aoeu(), so I do need something like what's below.
#!/bin/bash
# pass_space_function.sh
aoeu() {
echo "$1" "$2"
}
pyfg() {
echo "-htns crl" "qjkx"
}
aoeu $(pyfg)
My running the above outputs:
$ ./pass_space_function.sh
-htns crl
My desired output is:
$ ./pass_space_function.sh
-htns crl qjkx
To be clear, I do understand exactly why my code isn't working, but that about which I'm not so sure is how to make it do what I want it to do.
EDIT:
#!/bin/bash
aoeu() {
echo 1:"$1" 2:"$2" 3:"$3"
}
pyfg() {
# These variables might be user-provided.
wvz="/usr/lib/scarychacacters_\"##$:%^&:*(){}[]; a o ;u ;::e i y f.so.4"
bm="/space space space"
snt="/var/cache/normalpath"
printf "%q %q %q" "$wvz" "$bm" "$snt"
}
aoeu $(pyfg)
That code returns, for me, 1:/usr/lib/scarychacacters_\"##\$:%\^\&:\*\(\)\{\}\[\]\;\ 2:a\ 3:o\. It's obviously splitting at the whitespace in $wvz.
The key to correct quoting lies in the understanding what happens.
That echo "-htns crl" "qjkx" for example will print just a byte stream to its stdout, so it will be just -htns crl qjkx in the end. The information that -htns crl were grouped more closely than qjkx is lost.
To avoid this loss you can use printf "%q":
pyfg() {
printf "%q %q" "-htns crl" "qjkx"
}
This will generate quoted output: -htns\ crl qjkx which means to the shell the same as "-htns crl" "qjkx" (whether the space is escaped with a backslash or quoted with double quotes does not make a difference).
The next aspect is the use of $() to pass the output of one program to the next.
The typical way is to put that in double quotes:
aoeu "$(pyfg)"
This way everything is passed without interpretation which is desirable in most cases.
In your case, however, you might want to make the output of pyfg quoted instead of quote the output of pyfg; notice the important difference: The first means that pyfg produces quoted output (as shown above), the second means that pyfg produces output which gets quoted later. The second does not help if the output of pyfg already lost the information which parts belong together.
If you now just leave away the double quotes, the output unfortunately just gets split at the spaces (i. e. first character of $IFS) even if this space is escaped with a backslash. So, instead, you need to use eval in this case to force the shell to interpret the value of $(pyfg) with the normal shell evaluation mechanism:
eval aoeu "$(pyfg)"
EDIT: This works
#!/bin/bash
# pass_space_function.sh
aoeu() {
echo $1 x $2
}
pyfg() {
echo "'-htns crl' 'qjkx'"
}
eval aoeu $(pyfg)
I have a script which uses the following logic:
if [ ! -z "$1" ]; then # if any parameter is supplied
ACTION= # clear $ACTION
else
ACTION=echo # otherwise, set it to 'echo'
fi
This works fine, as-is. However, in reading the Shell Parameter Expansion section of the bash manual, it seems this should be able to be done in a single step. However, I can't quite wrap my head around how to do it.
I've tried:
ACTION=${1:-echo} # ends up with $1 in $ACTION
ACTION=${1:+}
ACTION=${ACTION:-echo} # ends up always 'echo'
and a few ways of nesting them, but nesting seems to be disallowed as far as I can tell.
I realize I've already got a working solution, but now I'm genuinely curious if this is possible. It's something that would be straightforward with a ternary operator, but I don't think bash has one.
If this is possible, I'd like to see the logic to do this seeming two-step process, with no if/else constructs, but using only any combination of the Shell Parameter Expansion features.
Thank you.
EDIT for elderarthis:
The remainder of the script is just:
find . -name "*\?[NMSD]=[AD]" -exec ${ACTION} rm -f "{}" +
I just want ACTION=echo as a sanity check against myself, hence, passing any argument will actually do the deletion (by nullifying ${ACTION}, whereas passing no args leaves echo in there.
And I know TIMTOWTDI; I'm looking to see if it can be done with just the stuff in the Shell Parameter Expansion section :-)
EDIT for Mikel:
$ cat honk.sh
#!/bin/bash
ACTION=${1-echo}
echo $ACTION
$ ./honk.sh
echo
$ ./honk.sh foo
foo
The last needs to have ACTION='', and thus return a blank line/null value.
If I insisted on doing it in fewer than 4 lines and no sub-shell, then I think I'd use:
ACTION=${1:+' '}
: ${ACTION:=echo}
This cheats slightly - it creates a blank action rather than an empty action if there is an argument to the script. If there is no argument, then ACTION is empty before the second line. On the second line, if action is empty, set it to 'echo'. In the expansion, since you (correctly) do not quote $ACTION, no argument will be passed for the blank.
Tester (xx.sh):
ACTION=${1:+' '}
: ${ACTION:=echo}
echo $ACTION rm -f a b c
Tests:
$ sh xx.sh 1
rm -f a b c
$ sh xx.sh
echo rm -f a b c
$ sh xx.sh ''
echo rm -f a b c
$
If the last line is incorrect, then remove the colon from before the plus.
If a sub-shell is acceptable, then one of these two single lines works:
ACTION=$([ -z "$1" ] && echo echo)
ACTION=$([ -z "${1+X}" ] && echo echo)
The first corresponds to the first version shown above (empty first arguments are treated as absent); the second deals with empty arguments as present. You could write:
ACTION=$([ -z "${1:+X}" ] && echo echo)
to make the relation with the second clearer - except you're only going to use one or the other, not both.
Since the markdown notation in my comment confused the system (or I got it wrong but didn't get to fix it quickly enough), my last comment (slightly amended) should read:
The notation ${var:+' '} means 'if $var is set and is not empty, then use what follows the +' (which, in this case, is a single blank). The notation ${var+' '} means 'if $var is set - regardless of whether it is empty or not - then use what follows the +'. These other expansions are similar:
${var:=X} - set $var to X unless it already has a non-empty value.
${var:-X} - expands to $var if it has a non-empty value and expands to X if $var is unset or is empty
Dropping the colon removes the 'empty' part of the test.
ACTION=${1:-echo}
is correct.
Make sure it's near the top of your script before anything modifies $1 (e.g. before any set command). Also, it wouldn't work inside a function, because $1 would be the first parameter to the function.
Also check if $1 is set but null, in which case fix how you're calling it, or use ACTION=${1-echo} (note there is no :).
Update
Ah, I assumed you must have meant the opposite, because it didn't really make sense otherwise.
It still seems odd, but I guess as a mental exercise, maybe you want something like this:
#!/bin/bash
shopt -s extglob
ACTION=$1
ACTION=${ACTION:-echo}
ACTION=${ACTION/!(echo)/} # or maybe ACTION=${ACTION#!(echo)}
echo ACTION=$ACTION
It's not quite right: it gives ACTION=o, but I think something along those lines should work.
Further, if you pass echo as $1, it will stay as echo, but I don't think that's a bad thing.
It's also terribly ugly, but you knew that when asking the question. :-)