Learning shell script after years of Windows scripting. I have qBittorrent-nox running in a TrueNAS 12.2 jail. qBittorrent provides a feature to run a command after a torrent download completes. I want to call a shell script to run two chown commands on the folder that is passed as a parameter. The folder will have spaces and may have ampersands, and the shell script fails as a result. The command passed according to qBittorrent's log file is this:
bash /mnt/torrents/Live/fixperms.sh "Name of the folder & description of contents"
This is the code I expected to work:
#!/usr/local/bin/bash
chown -R 1001:1006 $(printf "%q" "$1")
The command is correctly formed, but the script fails as it splits the string. The command it creates works if I echo it and execute it manually. I cannot find a way of making any shell (csh, sh, zsh and bash) NOT split the string at the spaces. I've tried it with single quotes, double quotes and backticks.
I have spent a number of hours on this and I have made no progress. I have tried all four shells, zsh splits the double-quoted string when the documentation says it shouldn't, parameter expansion works in bash but still splits the string. I have no python or perl, and no wish to install them if I can avoid it.
What am I missing?
Bash is 5.0.18, zsh is v5.8, tcsh is 6.2.00
You could do the following
VAR="Name of the folder & description of contents"; bash /mnt/torrents/Live/fixperms.sh "$VAR"
Then the string is handled with the blank spaces in it. If you for example to this in a bash, it works:
user#server:~$ VAR="test is test"
user#server:~$ ./test2.sh "$VAR"
test is test
test2.sh contains just an echo $1 so you are able to work with the entire string afterwards like
#!/usr/local/bin/bash
chown -R 1001:1006 $(printf "%q" "$1")
Related
I have this bash script that starts with
for d in /data/mydata/*; do
echo $d
filepath=$(echo $d | tr "/" "\n")
pathArr=($filepath) # fails here
echo ${pathArr[-1]}
It runs fine when I just call in on command line
./run_preprocess.sh
but when I run it using screen
screen -dmSL run_preproc ./run_preprocess.sh
it fails on that pathArr line
./run_preproc.sh: 7: ./run_preproc.sh: Syntax error: "(" unexpected (expecting "done")
is there something I need to do to protect the script code?
Based on the error, looks like you're running your script with POSIX sh, not bash. Arrays are undefined in POSIX sh.
To fix this, add a proper hashbang to your script (e.g. /usr/bin/env bash, or run the script directly with Bash interpreter (e.g. /bin/bash script.sh).
In addition (unrelated to the problem at hand), your script (or the snippet posted) has several potential issues:
variables should be quoted to prevent globbing and word splitting (e.g. consider d - one of your files - containing * -- echo $d will include a list of all files, since * will be expanded)
splitting into array with ($var) is done on any IFS character, not just newlines. IFS includes a space, tab and newline by default. Use of read -a or mapfile is recommended over ($var).
Finally, if all you're trying is get the last component in path (filename), you should consider using basename(1):
$ basename /path/to/file
file
or substring removal syntax of Bash parameter expansion:
$ path=/path/to/file
$ echo "${path##*/}"
file
I get the set of strings as input in terminal. I need to replace the ".awk" substring to ".sh" in each string using shell and then output modified string.
I wrote such script for doing this:
#!/bin/bash
while read line
do
result=${line/.awk/.sh}
echo $result
done
But it gives me an error: script-ch.sh: 6: script-ch.sh: Bad substitution.
How should I change this simple script to fix error?
UPD: ".awk" may be inside the string. For example: "file.awk.old".
If you are using Bash, then there is nothing wrong with your substitution. There is no reason to spawn an additional subshell and use a separate utility when bash substring replacement was tailor made to do that job:
$ fn="myfile.awk.old"; echo "$fn --> ${fn/.awk/.sh}"
myfile.awk.old --> myfile.sh.old
Note: if you are substituting .sh for .awk, then the . is unnecessary. A simple ${fn/awk/sh} will suffice.
I suspect you have some stray DOS character in your original script.
Not sure why it works for you and not for me.. might be the input you're giving it. It could have a space in it.
This should work:
#!/bin/bash
while read line
do
result=$(echo $line | sed 's/\.awk/\.sh/')
echo $result
done
If you run chmod +x script.sh and then run it with ./script.sh, or if you run it with bash script.sh, it should work fine.
Running it with sh script.sh will not work because the hashbang line will be ignored and the script will be interpreted by dash, which does not support that string substitution syntax.
I cannot believe I've spent 1.5 hours on something as trivial as this. I'm writing a very simple shell script which greps a file, stores the output in a variable, and echos the variable to STDOUT.
I have checked the grep command with the regex on the command line, and it works fine. But for some reason, the grep command doesn't work inside the shell script.
Here is the shell script I wrote up:
#!/bin/bash
tt=grep 'test' $1
echo $tt
I ran this with the following command: ./myScript.sh testingFile. It just prints an empty line.
I have already used chmod and made the script executable.
I have checked that the PATH variable has /bin in it.
Verified that echo $SHELL gives /bin/bash
In my desperation, I have tried all combinations of:
tt=grep 'test' "$1"
echo ${tt}
Not using the command line argument at all, and hardcoding the name of the file tt=grep 'test' testingFile
I found this: grep fails inside bash script but works on command line, and even used dos2unix to remove any possible carriage returns.
Also, when I try to use any of the grep options, like: tt=grep -oE 'test' testingFile, I get an error saying: ./out.sh: line 3: -oE: command not found.
This is crazy.
You need to use command substitution:
#!/usr/bin/env bash
test=$(grep 'foo' "$1")
echo "$test"
Command substitution allows the output of a command to replace the command itself. Command substitution occurs when a command is enclosed like this:
$(command)
or like this using backticks:
`command`
Bash performs the expansion by executing COMMAND and replacing the command substitution with the standard output of the command, with any trailing newlines deleted. Embedded newlines are not deleted, but they may be removed during word splitting.
The $() version is usually preferred because it allows nesting:
$(command $(command))
For more information read the command substitution section in man bash.
Everybody says eval is evil, and you should use $() as a replacement. But I've run into a situation where the unquoting isn't handled the same inside $().
Background is that I've been burned too often by file paths with spaces in them, and so like to quote all such paths. More paranoia about wanting to know where all my executables are coming from. Even more paranoid, not trusting myself, and so like being able to display the created commands I'm about to run.
Below I try variations on using eval vs. $(), and whether the command name is quoted (cuz it could contain spaces)
BIN_LS="/bin/ls"
thefile="arf"
thecmd="\"${BIN_LS}\" -ld -- \"${thefile}\""
echo -e "\n Running command '${thecmd}'"
$($thecmd)
Running command '"/bin/ls" -ld -- "arf"'
./foo.sh: line 8: "/bin/ls": No such file or directory
echo -e "\n Eval'ing command '${thecmd}'"
eval $thecmd
Eval'ing command '"/bin/ls" -ld -- "arf"'
/bin/ls: cannot access arf: No such file or directory
thecmd="${BIN_LS} -ld -- \"${thefile}\""
echo -e "\n Running command '${thecmd}'"
$($thecmd)
Running command '/bin/ls -ld -- "arf"'
/bin/ls: cannot access "arf": No such file or directory
echo -e "\n Eval'ing command '${thecmd}'"
eval $thecmd
Eval'ing command '/bin/ls -ld -- "arf"'
/bin/ls: cannot access arf: No such file or directory
$("/bin/ls" -ld -- "${thefile}")
/bin/ls: cannot access arf: No such file or directory
So... this is confusing. A quoted command path is valid everywhere except inside a $() construct? A shorter, more direct example:
$ c="\"/bin/ls\" arf"
$ $($c)
-bash: "/bin/ls": No such file or directory
$ eval $c
/bin/ls: cannot access arf: No such file or directory
$ $("/bin/ls" arf)
/bin/ls: cannot access arf: No such file or directory
$ "/bin/ls" arf
/bin/ls: cannot access arf: No such file or directory
How does one explain the simple $($c) case?
The use of " to quote words is part of your interaction with Bash. When you type
$ "/bin/ls" arf
at the prompt, or in a script, you're telling Bash that the command consists of the words /bin/ls and arf, and the double-quotes are really emphasizing that /bin/ls is a single word.
When you type
$ eval '"/bin/ls" arf'
you're telling Bash that the command consists of the words eval and "/bin/ls" arf. Since the purpose of eval is to pretend that its argument is an actual human-input command, this is equivalent to running
$ "/bin/ls" arf
and the " gets processed just like at the prompt.
Note that this pretense is specific to eval; Bash doesn't usually go out of its way to pretend that something was an actual human-typed command.
When you type
$ c='"/bin/ls" arf'
$ $c
the $c gets substituted, and then undergoes word splitting (see ยง3.5.7 "Word Splitting" in the Bash Reference Manual), so the words of the command are "/bin/ls" (note the double-quotes!) and arf. Needless to say, this doesn't work. (It's also not very safe, since in addition to word-splitting, $c also undergoes filename-expansion and whatnot. Generally your parameter-expansions should always be in double-quotes, and if they can't be, then you should rewrite your code so they can be. Unquoted parameter-expansions are asking for trouble.)
When you type
$ c='"/bin/ls" arf'
$ $($c)
this is the same as before, except that now you're also trying to use the output of the nonworking command as a new command. Needless to say, that doesn't cause the nonworking command to suddenly work.
As Ignacio Vazquez-Abrams says in his answer, the right solution is to use an array, and handle the quoting properly:
$ c=("/bin/ls" arf)
$ "${c[#]}"
which sets c to an array with two elements, /bin/ls and arf, and uses those two elements as the word of a command.
With the fact that it doesn't make sense in the first place. Use an array instead.
$ c=("/bin/ls" arf)
$ "${c[#]}"
/bin/ls: cannot access arf: No such file or directory
From the man page for bash, regarding eval:
eval [arg ...]:
The args are read and concatenated together into a single command.
This command is then read and executed by the shell, and its exit
status is returned as the value of eval.
When c is defined as "\"/bin/ls\" arf", the outer quotes will cause the entire thing to be processed as the first argument to eval, which is expected to be a command or program. You need to pass your eval arguments in such a way that the target command and its arguments are listed separately.
The $(...) construct behaves differently than eval because it is not a command that takes arguments. It can process the entire command at once instead of processing arguments one at a time.
A note on your original premise: The main reason that people say that eval is evil was because it is commonly used by scripts to execute a user-provided string as a shell command. While handy at times, this is a major security problem (there's typically no practical way to safety-check the string before executing it). The security problem doesn't apply if you are using eval on hard-coded strings inside your script, as you are doing. However, it's typically easier and cleaner to use $(...) or `...` inside of scripts for command substitution, leaving no real use case left for eval.
Using set -vx helps us understand how bash process the command string.
As seen in the picture, "command" works cause quotes will be stripped when processing. However, when $c(quoted twice) is used, only the outside single quotes are removed. eval can process the string as the argument and outside quotes are removed step by step.
It is probably just related to how bash semanticallly process the string and quotes.
Bash does have many weird behaviours about quotes processing:
Bash inserting quotes into string before execution
How do you stop bash from stripping quotes when running a variable as a command?
Bash stripping quotes - how to preserve quotes
I am writing a bash script which amongst many other things uses expect to automatically run a binary and install it by answering installer prompts.
I was able to get my expect script to work fine when the expect script is called in my bash script with the command "expect $expectscriptname $Parameter". However, I want to embed the expect script into the shell script instead of having to maintain two separate script files for the job. I searched around and found that the procedure to embed expect into bash script was to declare a variable like VAR below and then echo it.:
VAR=$(expect -c "
#content of expect script here
")
echo "$VAR"
1) I don't understand how echoing $VAR actually runs the expect script. Could anyone explain?
2) I am not sure how to pass $Parameter into VAR or to the echo statement. This is my main concern.
Any ideas? Thanks.
Try something like:
#!/bin/sh
tclsh <<EOF
puts $1
EOF
I don't have the expect command installed today, so I used tclsh instead.
In bash, the construct $(cmd) runs the specified command and captures its output. It's similar to the backtick notation, though there are some slight differences. Thus, the assignment to VAR is what runs the expect command:
# expect is run here
VAR=$(expect -c "
# ...
")
# This echoes the output of the expect command.
echo "$VAR"
From the bash manual page:
When the old-style backquote form
of substitution is used, backslash
retains its literal meaning except
when followed by $, , or \. The
first backquote not preceded by a
backslash terminates the command
substitution. When using the
$(command)` form, all characters
between the parentheses make up the
command; none are treated specially.
That's why it works: The bash comment character (#) isn't treated as a comment character within the $( ... ).
EDIT
Passing parameters: Just put 'em in there. For instance, consider this script:
foo="Hello, there"
bar=$(echo "
# $foo
")
echo $bar
If you run that script, it prints:
# Hello, there
Thus, the value of $foo was substituted inside the quotes. The same should work for the expect script.
Instead of a bash script and an expect script, have you considered writing just a single expect script?
Expect is a superset of Tcl, which means it is a fully functioning programming language, and you can do anything with it that you can do with bash (and for the things that you can't, you can always exec bash commands. You don't have to use expect just to "expect" things