I cannot believe I've spent 1.5 hours on something as trivial as this. I'm writing a very simple shell script which greps a file, stores the output in a variable, and echos the variable to STDOUT.
I have checked the grep command with the regex on the command line, and it works fine. But for some reason, the grep command doesn't work inside the shell script.
Here is the shell script I wrote up:
#!/bin/bash
tt=grep 'test' $1
echo $tt
I ran this with the following command: ./myScript.sh testingFile. It just prints an empty line.
I have already used chmod and made the script executable.
I have checked that the PATH variable has /bin in it.
Verified that echo $SHELL gives /bin/bash
In my desperation, I have tried all combinations of:
tt=grep 'test' "$1"
echo ${tt}
Not using the command line argument at all, and hardcoding the name of the file tt=grep 'test' testingFile
I found this: grep fails inside bash script but works on command line, and even used dos2unix to remove any possible carriage returns.
Also, when I try to use any of the grep options, like: tt=grep -oE 'test' testingFile, I get an error saying: ./out.sh: line 3: -oE: command not found.
This is crazy.
You need to use command substitution:
#!/usr/bin/env bash
test=$(grep 'foo' "$1")
echo "$test"
Command substitution allows the output of a command to replace the command itself. Command substitution occurs when a command is enclosed like this:
$(command)
or like this using backticks:
`command`
Bash performs the expansion by executing COMMAND and replacing the command substitution with the standard output of the command, with any trailing newlines deleted. Embedded newlines are not deleted, but they may be removed during word splitting.
The $() version is usually preferred because it allows nesting:
$(command $(command))
For more information read the command substitution section in man bash.
Related
Learning shell script after years of Windows scripting. I have qBittorrent-nox running in a TrueNAS 12.2 jail. qBittorrent provides a feature to run a command after a torrent download completes. I want to call a shell script to run two chown commands on the folder that is passed as a parameter. The folder will have spaces and may have ampersands, and the shell script fails as a result. The command passed according to qBittorrent's log file is this:
bash /mnt/torrents/Live/fixperms.sh "Name of the folder & description of contents"
This is the code I expected to work:
#!/usr/local/bin/bash
chown -R 1001:1006 $(printf "%q" "$1")
The command is correctly formed, but the script fails as it splits the string. The command it creates works if I echo it and execute it manually. I cannot find a way of making any shell (csh, sh, zsh and bash) NOT split the string at the spaces. I've tried it with single quotes, double quotes and backticks.
I have spent a number of hours on this and I have made no progress. I have tried all four shells, zsh splits the double-quoted string when the documentation says it shouldn't, parameter expansion works in bash but still splits the string. I have no python or perl, and no wish to install them if I can avoid it.
What am I missing?
Bash is 5.0.18, zsh is v5.8, tcsh is 6.2.00
You could do the following
VAR="Name of the folder & description of contents"; bash /mnt/torrents/Live/fixperms.sh "$VAR"
Then the string is handled with the blank spaces in it. If you for example to this in a bash, it works:
user#server:~$ VAR="test is test"
user#server:~$ ./test2.sh "$VAR"
test is test
test2.sh contains just an echo $1 so you are able to work with the entire string afterwards like
#!/usr/local/bin/bash
chown -R 1001:1006 $(printf "%q" "$1")
I have this bash script that starts with
for d in /data/mydata/*; do
echo $d
filepath=$(echo $d | tr "/" "\n")
pathArr=($filepath) # fails here
echo ${pathArr[-1]}
It runs fine when I just call in on command line
./run_preprocess.sh
but when I run it using screen
screen -dmSL run_preproc ./run_preprocess.sh
it fails on that pathArr line
./run_preproc.sh: 7: ./run_preproc.sh: Syntax error: "(" unexpected (expecting "done")
is there something I need to do to protect the script code?
Based on the error, looks like you're running your script with POSIX sh, not bash. Arrays are undefined in POSIX sh.
To fix this, add a proper hashbang to your script (e.g. /usr/bin/env bash, or run the script directly with Bash interpreter (e.g. /bin/bash script.sh).
In addition (unrelated to the problem at hand), your script (or the snippet posted) has several potential issues:
variables should be quoted to prevent globbing and word splitting (e.g. consider d - one of your files - containing * -- echo $d will include a list of all files, since * will be expanded)
splitting into array with ($var) is done on any IFS character, not just newlines. IFS includes a space, tab and newline by default. Use of read -a or mapfile is recommended over ($var).
Finally, if all you're trying is get the last component in path (filename), you should consider using basename(1):
$ basename /path/to/file
file
or substring removal syntax of Bash parameter expansion:
$ path=/path/to/file
$ echo "${path##*/}"
file
The POSIX shell standard at
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04
says in section 2.6.3:
Command substitution allows the output of a command to be substituted in place of the command name itself
This would seem to imply that it is only guaranteed to work, if you substitute for the whole command name; if you substitute for a part of it, or something else, then it may or may not work.
Experimenting:
$ echo ;
$ $(echo echo) ;
$
So far so good...
$ e$(echo cho) ;
$ echo $(echo ';')
;
$ echo $(echo foobar)
foobar
The first and third example above seem to "work" but the second "does not work". Is this all simply undocumented and random behavior, as the standard seems to imply, and in reality for some other POSIX shell, none of these three are guaranteed to "work"?
(By "work", I mean "produce the same result as if the results of the substitution were typed in the command itself on the terminal")
The reason why this:
$ echo $(echo ';')
Does not output the same as this:
$ echo ;
Is the same reason why this:
$ ;
bash: syntax error near unexpected token `;'
Does not output the same as this:
$ `echo ';'`
;: command not found
The latest case, the output from command substitution (the (?)expected-to-be(?) token ;) is interpreted as a command, because it is passed [and interpreted] as a command in a subshell, and not as a command interpreter built-in token.
As my interpretation, this isn't against the standard.
EDIT: Answering the question
You stated:
This would seem to imply that it is only guaranteed to work, if you
substitute for the whole command name; if you substitute for a part of
it, or something else, then it may or may not work.
And the POSIX standard states:
The shell shall expand the command substitution by executing command
in a subshell environment (see Shell Execution Environment) and
replacing the command substitution (the text of command plus the
enclosing "$()" or backquotes) with the standard output of the
command, removing sequences of one or more <newline> characters at the
end of the substitution.
The standard seems clear regarding to what happens. This is not a question of what's present in a command containing a command substitution, only the command that's inside the enclosing $() or or ``.
I get the set of strings as input in terminal. I need to replace the ".awk" substring to ".sh" in each string using shell and then output modified string.
I wrote such script for doing this:
#!/bin/bash
while read line
do
result=${line/.awk/.sh}
echo $result
done
But it gives me an error: script-ch.sh: 6: script-ch.sh: Bad substitution.
How should I change this simple script to fix error?
UPD: ".awk" may be inside the string. For example: "file.awk.old".
If you are using Bash, then there is nothing wrong with your substitution. There is no reason to spawn an additional subshell and use a separate utility when bash substring replacement was tailor made to do that job:
$ fn="myfile.awk.old"; echo "$fn --> ${fn/.awk/.sh}"
myfile.awk.old --> myfile.sh.old
Note: if you are substituting .sh for .awk, then the . is unnecessary. A simple ${fn/awk/sh} will suffice.
I suspect you have some stray DOS character in your original script.
Not sure why it works for you and not for me.. might be the input you're giving it. It could have a space in it.
This should work:
#!/bin/bash
while read line
do
result=$(echo $line | sed 's/\.awk/\.sh/')
echo $result
done
If you run chmod +x script.sh and then run it with ./script.sh, or if you run it with bash script.sh, it should work fine.
Running it with sh script.sh will not work because the hashbang line will be ignored and the script will be interpreted by dash, which does not support that string substitution syntax.
I'm having a little issue getting quite a simple bash script running.
The part that's working:
qstat -f $queuenum | grep -Po '(?<=PBS_O_WORKDIR=).*(?=,)'
Outputs a directory to the screen (for example):
/short/h72/SiO2/defected/2-999/3-forces/FORCES_364
All I want to do is change directory to this folder. Appending a "| cd" to the end of the above command doesn't work, and I can't quite figure out how to use the $(()) tags either.
Any help would be appreciated.
cd `qstat -f $queuenum | grep -Po '(?<=PBS_O_WORKDIR=).*(?=,)' `
Invoking your script creates a new bash shell
This shell is destroyed when your script ends.
If you use exec <scriptname> to run your script, the new bash shell is substituted for the current one. So if you add the command bash at the end of your script, you will get what you desire, but not the same bash shell.
You should use:
cd "$(qstat -f $queuenum | grep -Po '(?<=PBS_O_WORKDIR=).*(?=,)' )"
you are trying to achieve command substitution which is achieved by using either of these two syntax:
$(command)
or like this using backticks:
`command`
The first one is the preferred way as it allows nesting of command substitutions something like:
foo=$(command1 | command2 $(command3))
also you should enclose the entire command substitution in double quotes to protect you if the result of the command substitution is a string with spaces.