This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Closed 3 years ago.
I wanted to know how vulnerable is bash by code injection. So I wrote a script as simple as this:
#!/bin/bash
grep $1 $2
and saved it as greptest.sh. The quotes around the variables were dropped intentionally for vulnerability test, so grep "$1" "$2" is the preferred way.
Then I created test.txt:
sadhuer
sadjfh Hello
cusad
Hello
fgdfg
First was to show its proper use.
$ ./greptest.sh 'Hello' 'test.txt'
Output as expected:
sadjfh Hello
Hello
Then the first attack:
$ ./greptest.sh 'Hello test.txt'
Outputs the same as above. So, obviously, it does something due to missing quotes within the script - altough $2 is empty! Next try with $2 not empty for proving my assumption that $2 will be interpreted as a further input file:
$ ./greptest.sh 'Hello test.txt' 'nonexistingfile.txt'
outputs:
test.txt:sadjfh Hello
test.txt:Hello
grep: nonexistingfile.txt: No such file or directory
Then the harder attack: Trying to execute an arbitrary command:
$ ./greptest.sh 'Hello test.txt' '; ls'
outputs:
test.txt:sadjfh Hello
test.txt:Hello
grep: ;: No such file or directory
grep: ls: No such file or directory
I did not expect this. I thought the variables were subsitituted to yield
grep Hello test.txt ; ls
which should result in listing the current directory. So, is missing these quotes just ugly and error prone or a serious security concern I should care about (given the values of these parameters come from an untrusted source)?
You don't need to worry about the parameters passed to your greptest.sh bash script, as you are passing the parameters directly as parameters to the called program (grep in this case).
You need to worry about whatever calls greptest.sh injecting parameters onto that command line.
See this post for information.
One security issue of unquoted variables (though not code injection) is glob expansion DoS – a variable without quotes not only undergoes word splitting, but also globbing!
If the variable contains a pattern like */*/../../ repeated several times, like */*/../../*/*/../../, */*/../../*/*/../../*/*/../../ and so on, then the number of files that expands to increases exponentially with the length of the pattern.
Not related to quoting, but if the attacker can influence $PATH, $LD_PRELOAD, etc, then he can make grep in your script mean anything.
Related
I am trying to write a bash function that takes command as the first variable and output file as the second variable:
my_func() {
$1 ${SOME_OTHER_PARAMS} |
tee $2
}
when I run my_func "python print_hello_to_a_file ~/out.txt" "second_file.txt", it seems to output "hello" to "~/out.txt", not "out.txt" in my home directory, meaning that "~" was not expanded correctly.
I am wondering if it is possible to correctly expand "~" inside this function?
Possible? Yes, but probably not a good idea.
The basic problem: When parsing the command line, Tilde Expansion happens before Parameter Expansion. This means you can't put a tilde inside a variable and have it be replaced by a path to your home directory in the simplest case.
Minimal demo:
[user#host]$ myvar="~"
[user#host]$ echo $myvar
~
[user#host]$ echo ~
/home/user
One possible solution is to use eval to force a second round of parsing before executing the command.
[user#host]$ eval echo $myvar
/home/user
But eval is VERY DANGEROUS and you should not use it without exhausting all other possibilities. Forcing a second parsing of the command line can result in unexpected, confusing, and potentially even unsafe results if you are not extremely familiar with the parsing rules and take sufficient steps to sanitize your inputs before running them through eval.
The more standard solution is to build up your command inside a bash array.
my_func() {
tee_output="${1}"
shift
# expand the inputs all together
# SOME_OTHER_PARAMS should be an array as well
"${#}" "${SOME_OTHER_PARAMS[#]}" | tee "${tee_output}"
}
# build the command up as an array with each word as its own element
# Tilde expansion will occur here and the result will be stored for later
my_command=( "python" "print_hello_to_a_file" ~/"out.txt" )
# expand the array and pass to my_func
# Important that the tee_location goes first here, as we
# shift it off to capture the remaining arguments as a complete command
my_func "${tee_loc}" "${my_command[#]}"
But my_func still only supports simple commands with this approach - no loops or if/case statements, no file redirections, etc. This might be okay if your goal is just to decorate a variety of commands with extra parameters and tee their output somewhere.
This question already has answers here:
Why does shell ignore quoting characters in arguments passed to it through variables? [duplicate]
(3 answers)
Closed 6 years ago.
If i use this command in pipeline, it's working very well;
pipeline ... | grep -P '^[^\s]*\s3\s'
But if I want to set grep into variable like:
var="grep -P '^[^\s]*\s3\s'"
And if I put variable in pipeline;
pipeline ... | $var
nothing happens, like there isn't any matches.
Any help what am I doing wrong?
The robust way to store a simple command in a variable in Bash is to use an array:
# Store the command names and arguments individually
# in the elements of an *array*.
cmd=( grep -P '^[^\s]*\s3\s' )
# Use the entire array as the command to execute - be sure to
# double-quote ${cmd[#]}.
echo 'before 3 after' | "${cmd[#]}"
If, by contrast, your command is more than a simple command and, for instance, involves pipes, multiple commands, loops, ..., defining a function is the right approach:
# Define a function encapsulating the command...
myGrep() { grep -P '^[^\s]*\s3\s'; }
# ... and use it:
echo 'before 3 after' | myGrep
Why what you tried didn't work:
var="grep -P '^[^\s]*\s3\s'"
causes the single quotes around the regex to become a literal, embedded part of $var's value.
When you then use $var - unquoted - as a command, the following happens:
Bash performs word-splitting, which means that it breaks the value of $var into words (separate tokens) by whitespace (the chars. defined in special variable $IFS, which contains a space, a tab, and a newline character by default).
Bash also performs globbing (pathname expansion) on the resulting works, which is not a problem here, but can have unintended consequences in general.
Also, if any of your original arguments had embedded whitespace, word splitting would split them into multiple words, and your original argument partitioning is lost.
(As an aside: "$var" - i.e., double-quoting the variable reference - is not a solution, because then the entire string is treated as the command name.)
Specifically, the resulting words are:
grep
-P
'^[^\s]*\s3\s' - including the surrounding single quotes
The words are then interpreted as the name of the command and its arguments, and invoked as such.
Given that the pattern argument passed to grep starts with a literal single quote, matching won't work as intended.
Short of using eval "$var" - which is NOT recommended for security reasons - you cannot persuade Bash to see the embedded single quotes as syntactical elements that should be removed (a process appropriate called quote removal).
Using an array bypasses all these problems by storing arguments in individual elements and letting Bash robustly assemble them into a command with "${cmd[#]}".
What you are doing wrong is trying to store a command in a variable. For simplicity, robustness, etc. commands are stored in aliases (if no arguments) or functions (if arguments), not variables. In this case:
$ alias foo='grep X'
$ echo "aXb" | foo
aXb
I recommend you read the book Shell Scripting Recipes by Chris Johnson ASAP to get the basics of shell programming and then Effective Awk Programming, 4th Edition, by Arnold Robbins when you're ready to start writing scripts to manipulate text.
This question already has answers here:
How to pass the value of a variable to the standard input of a command?
(9 answers)
Closed 1 year ago.
I have a shell script:
TOPDIR=`pwd`
FOLDER=$($TOPDIR | sed 's/\//\_/g')
if [[ condition ]];then
source ~/[$FOLDER]-build/build-env.sh
fi
the TOPDIR here is /home/uname/project, so the variable FOLDER is supposed to be _home_uname_project because sed is called to replace / with _.
But it goes wrong when executing, terminal tells that /home/uname/[]-build/build-env.sh: No such file or directory which, I guess, means that FOLDER is unexpected empty in the if-then statement. Can anybody help me with figuring this out?
If you look at the output of just
$TOPDIR | sed 's/\//\_/g'
you'll realize that it's empty; it's trying to execute a command equal to the contents of $TOPDIR and pipe the output of that into sed, but there is no output in the first place.
You could do
pwd | sed 's\//_/g'
instead (no need to escape _), which would work.
Or, instead of using an external tool, you could use parameter expansion
topdir="$(pwd)"
topdir="${topdir//\//_}"
with the same result.
Notice that uppercase variable names are discouraged, as they're more likely to clash with existing, reserved names.
For the command ls, the option -1 is supposed to do run ls yielding an output with only one column (yielding one file per line). But if I put it in a script, it just shows every file jammed on one line, separated with spaces.
Script:
#!/bin/ksh
text=`ls -1`
echo $text
Folder contents:
test
|--bob
|--coolDir
|--file
|--notThisDirectoryAgain
|--script.sh
|--spaces are cool
|--thatFile
Script Output:
bob coolDir file notThisDirectoryAgain script.sh spaces are cool thatFile
Output if I run ls -1 in the terminal (not in a script)
bob
coolDir
file
notThisDirectoryAgain
script.sh
spaces are cool
thatFile
it just shows every file jammed on one line, separated with spaces.
You have to consider what it is.
When you do
text=`ls -1`
that runs the program ls and presents the output as if you typed it in. So the shell gets presented with:
ls=file1
file2
file3
etc.
The shell splits command-line tokens on whitespace, which by default includes a space, a tab, and a newline. So each filename is seen as a separate token by the shell.
These tokens are then passed into echo as separate parameters. The echo command is unaware that the newlines were ever there.
As I'm sure you know, all echo does is to write each parameter to stdout, with a space between each one.
This is why the suggestion given by #user5228826 works. Change IFS if you don't want a newline to separate tokens.
However, all you really had to do is to enclose the variable in quotes, so that it didn't get split:
echo "$text"
By the way, using `backticks` is deprecated and poor practice because it can be difficult to read, particularly when nested. If you run ksh -n on your script it will report this to you (assuming you are not using an ancient version). Use:
text=$(ls -1)
Having said all that, this is a terrible way to get a list of files. UNIX shells do globbing, this is an unnecessary use of the ls external program. Try:
text=(*) # Get all the files in current directory into an array
oldIFS="$IFS" # Save the current value of IFS
IFS=$'\n' # Set IFS to a newline
echo "${text[*]}" # Join the elements of the array by newlines, and display
IFS="$oldIFS" # Reset IFS to previous value
That's because you're capturing ls output into a variable. Bash does the same.
Everybody says eval is evil, and you should use $() as a replacement. But I've run into a situation where the unquoting isn't handled the same inside $().
Background is that I've been burned too often by file paths with spaces in them, and so like to quote all such paths. More paranoia about wanting to know where all my executables are coming from. Even more paranoid, not trusting myself, and so like being able to display the created commands I'm about to run.
Below I try variations on using eval vs. $(), and whether the command name is quoted (cuz it could contain spaces)
BIN_LS="/bin/ls"
thefile="arf"
thecmd="\"${BIN_LS}\" -ld -- \"${thefile}\""
echo -e "\n Running command '${thecmd}'"
$($thecmd)
Running command '"/bin/ls" -ld -- "arf"'
./foo.sh: line 8: "/bin/ls": No such file or directory
echo -e "\n Eval'ing command '${thecmd}'"
eval $thecmd
Eval'ing command '"/bin/ls" -ld -- "arf"'
/bin/ls: cannot access arf: No such file or directory
thecmd="${BIN_LS} -ld -- \"${thefile}\""
echo -e "\n Running command '${thecmd}'"
$($thecmd)
Running command '/bin/ls -ld -- "arf"'
/bin/ls: cannot access "arf": No such file or directory
echo -e "\n Eval'ing command '${thecmd}'"
eval $thecmd
Eval'ing command '/bin/ls -ld -- "arf"'
/bin/ls: cannot access arf: No such file or directory
$("/bin/ls" -ld -- "${thefile}")
/bin/ls: cannot access arf: No such file or directory
So... this is confusing. A quoted command path is valid everywhere except inside a $() construct? A shorter, more direct example:
$ c="\"/bin/ls\" arf"
$ $($c)
-bash: "/bin/ls": No such file or directory
$ eval $c
/bin/ls: cannot access arf: No such file or directory
$ $("/bin/ls" arf)
/bin/ls: cannot access arf: No such file or directory
$ "/bin/ls" arf
/bin/ls: cannot access arf: No such file or directory
How does one explain the simple $($c) case?
The use of " to quote words is part of your interaction with Bash. When you type
$ "/bin/ls" arf
at the prompt, or in a script, you're telling Bash that the command consists of the words /bin/ls and arf, and the double-quotes are really emphasizing that /bin/ls is a single word.
When you type
$ eval '"/bin/ls" arf'
you're telling Bash that the command consists of the words eval and "/bin/ls" arf. Since the purpose of eval is to pretend that its argument is an actual human-input command, this is equivalent to running
$ "/bin/ls" arf
and the " gets processed just like at the prompt.
Note that this pretense is specific to eval; Bash doesn't usually go out of its way to pretend that something was an actual human-typed command.
When you type
$ c='"/bin/ls" arf'
$ $c
the $c gets substituted, and then undergoes word splitting (see §3.5.7 "Word Splitting" in the Bash Reference Manual), so the words of the command are "/bin/ls" (note the double-quotes!) and arf. Needless to say, this doesn't work. (It's also not very safe, since in addition to word-splitting, $c also undergoes filename-expansion and whatnot. Generally your parameter-expansions should always be in double-quotes, and if they can't be, then you should rewrite your code so they can be. Unquoted parameter-expansions are asking for trouble.)
When you type
$ c='"/bin/ls" arf'
$ $($c)
this is the same as before, except that now you're also trying to use the output of the nonworking command as a new command. Needless to say, that doesn't cause the nonworking command to suddenly work.
As Ignacio Vazquez-Abrams says in his answer, the right solution is to use an array, and handle the quoting properly:
$ c=("/bin/ls" arf)
$ "${c[#]}"
which sets c to an array with two elements, /bin/ls and arf, and uses those two elements as the word of a command.
With the fact that it doesn't make sense in the first place. Use an array instead.
$ c=("/bin/ls" arf)
$ "${c[#]}"
/bin/ls: cannot access arf: No such file or directory
From the man page for bash, regarding eval:
eval [arg ...]:
The args are read and concatenated together into a single command.
This command is then read and executed by the shell, and its exit
status is returned as the value of eval.
When c is defined as "\"/bin/ls\" arf", the outer quotes will cause the entire thing to be processed as the first argument to eval, which is expected to be a command or program. You need to pass your eval arguments in such a way that the target command and its arguments are listed separately.
The $(...) construct behaves differently than eval because it is not a command that takes arguments. It can process the entire command at once instead of processing arguments one at a time.
A note on your original premise: The main reason that people say that eval is evil was because it is commonly used by scripts to execute a user-provided string as a shell command. While handy at times, this is a major security problem (there's typically no practical way to safety-check the string before executing it). The security problem doesn't apply if you are using eval on hard-coded strings inside your script, as you are doing. However, it's typically easier and cleaner to use $(...) or `...` inside of scripts for command substitution, leaving no real use case left for eval.
Using set -vx helps us understand how bash process the command string.
As seen in the picture, "command" works cause quotes will be stripped when processing. However, when $c(quoted twice) is used, only the outside single quotes are removed. eval can process the string as the argument and outside quotes are removed step by step.
It is probably just related to how bash semanticallly process the string and quotes.
Bash does have many weird behaviours about quotes processing:
Bash inserting quotes into string before execution
How do you stop bash from stripping quotes when running a variable as a command?
Bash stripping quotes - how to preserve quotes