I have been using a shell script which used to do something like this:
output=$(ls -R /logs/{abc,cde}/"$((${date}/100))"/*/out*${date}* | grep -v "Random" )
echo $output
On running this command, I used to get the files with either abc or cde at relevant location. You can ignore other variables like date etc
However, when I modified the script to take abc,cde as command line parameter instead of hardcoding in the script, it stopped working.
I changed the command to:
output=$(ls -R /logs/{${list}}/"$((${date}/100))"/*/out*${date}* | grep -v "Random" )
where list is abc,cde
I tried a lot of different combinations of quotes etc but it does not seem to be working. Can someone help please with the correct syntax that it works properly
It doesn't work, because the expansion order prevents it.
The order of expansions is: brace expansion; tilde expansion,
parameter and variable expansion
You could solve it with using eval, but eval should be avoided at all.
Probably it's better to use find here.
find /logs -name 'abc/...' -o -name 'cde/...'
This works even with variables
Or use the regex logic.
find /logs/ -regex '/logs/\(abc\|cde\).*'
Try with this
Assign abc and cde to an array
ex -
declare -a ARR=(abc cde)
Assign it to variable like below.
output=ls -R /logs/${ARR[#]/"$((${date}/100))"/*/out*${date}* | grep -v "Random"
Related
This is a simple question but i am unable to find it in tutorials. Could anybody please explain what this statement does when executed in a bash shell within a folder containing .sh scripts. I know -i does in place editing, i understand that it will run sed on all scripts within the current directory. And i know that it does some sort of substitution. But what does this \(.*\) mean?
sed -i 's/MY_BASE_DIR=\(.*\)/MY_BASE_DIR=${MY_BASE_DIR-\1}/' *.sh
Thanks in advance.
You have an expression like:
sed -i 's/XXX=\(YYY\)/XXX=ZZZ/' file
This looks for a string XXX= in a file and captures what goes after. Then, it replaces this captured content with ZZZ. Since there is a captured group, it is accessed with \1. Finally, using the -i flag in sed makes the edition to be in-place.
For the replacement, it uses the following syntax described in Shell parameter expansion:
${parameter:-word}
If parameter is unset or null, the expansion of word is substituted.
Otherwise, the value of parameter is substituted.
Example:
$ d=5
$ echo ${d-3}
5
$ echo ${a-3}
3
So with ${MY_BASE_DIR-SOMETHING-\1} you are saying: print $MY_BAS_DIR. And if this variable is unset or null, print what is stored in \1.
All together, this is resetting MY_BASE_DIR to the value in the variable $MY_BASE_DIR unless this is not set; in such case, the value remains the same.
Note though that the variable won't be expanded unless you use double quotes.
Test:
$ d=5
$ cat a
d=23
blabla
$ sed "s/d=\(.*\)/d=${d-\1}/" a # double quotes -> value is replaced
d=5
blabla
$ sed 's/d=\(.*\)/d=${d-\1}/' a # single quotes -> variable is not expanded
d=${d-23}
blabla
Andd see how the value remains the same if $d is not set:
$ unset d
$ sed "s/d=\(.*\)/d=${d-\1}/" a
d=23
The scripts contain lines like this:
MY_BASE_DIR=/usr/local
The sed expression changes them to:
MY_BASE_DIR=${MY_BASE_DIR-/usr/local}
The effect is that /usr/local is not used as a fixed value, but only as the default value. You can override it by setting the environment variable MY_BASE_DIR.
For future reference, I would take a look at the ExplainShell website:
http://explainshell
that will give you a breakdown of the command structure etc. In this instance, let step through the details...Let's start with a simple example, let's assume that we were going to make the simple change - commenting out all lines by adding a "#" before each line. We can do this for all *.sh files in a directory with the ".sh" extension in the current directory:
sed 's/^/\#/' *.sh
i.e. Substitute beginning of line ^, with a # ...
Caveat: You did not specify the OS you are using. You may get different results with different versions of sed and OS...
ok, now we can drill into the substitution in the script. An example is probably easier to explain:
File: t.sh
MY_BASE_DIR="/important data/data/bin"
the command 's/MY_BASE_DIR=\(.*\)/MY_BASE_DIR=${MY_BASE_DIR-\1}/' *.sh
will search for "MY_BASE_DIR" in each .sh file in the directory.
When it encounters the string "MY_BASE_DIR=.*", in the file, it expands it to be MY_BASE_DIR="/important data/data/bin", this is now replaced on the right side of the expression /MY_BASE_DIR=${MY_BASE_DIR-\1}/ which becomes
MY_BASE_DIR=${MY_BASE_DIR-"/important data/data/bin"}
essentially what happens is that the substitute operation takes
MY_BASE_DIR="/important data/data/bin"
and inserts
MY_BASE_DIR=${MY_BASE_DIR-"/important data/data/bin"}
now if we run the script with the variable MY_BASE_DIR set
export MY_BASE_DIR="/new/import/dir"
the scripts modified by the sed script referenced will now substitute /important data/data/bin with /new/import/dir...
We started using shellcheck to check our scripts for errors/warnings,
Now common warning what we see in all our scripts is unquoted variables.
is there any script to correct those simple warnings/errors ?
I have below command which I use to change $VAR to ${VAR}
sed -i -r 's:\$([_a-zA-Z?][_a-zA-Z0-9]*):${\1}:g' <scriptname>
I modified it as follows,
sed -i -r 's:\$([_a-zA-Z?][_a-zA-Z0-9]*):"${\1}":g' <scriptname>
above command works fine when variables are unquoted but when they are quoted e.g. "$VAR" it changes to ""${VAR}""
any suggestion to whether continue doing it with sed or better write script to do it ?
any particular suggestions?
Edit carefully.
When you write echo "This is example ${var} in the middle of the line" you do not want to put quotes around ${var}.
You should put all variables (except PATH, PWD and some other system vars) in lowercase.
You might want to add some mappings in .vimrc, that will execute your sed first or second commandline using F4 of F5 (something like . ! ~/bin/make_my_var) making the editing easier. In make_my_var you can add logic for lowercasing the vars when they are not one of a list of exceptions.
And (edited):
You might want some more standards, perhaps use a styleguide.
I would like my bash script to check the name of the directory where it is run. Something like:
#!/bin/bash
path=eval 'pwd'
dirname=eval 'basename $path'
But it doesn't work: I get
./foo.sh: line 5: basename $path: command not found
How can I fix it? Also, once I get dirname to contain the correct dirname, I'd like to convert it to lowercase, to test it. I'm able to do this on the command line with awk:
echo $dirname | awk '{print tolower($0)}'
but how do I capture the return value into a variable?
Why not use:
#!/bin/bash
path=`pwd`
dirname=`basename $path | awk '{print tolower($0)}'`
Or if you want to do it as a one liner:
dirname=`pwd | xargs basename | awk '{print tolower($0)}'`
You can rewrite it to
dirname=eval "basename $path"
With single-quotes, you don't get shell expansion, but you want $path getting expanded.
BTW: I'd suggesst using
path=$(basename $path)
It's way more generic and better readable if you do something like
path=$(basename $(pwd))
or to get the lowercase result
path=$(basename $(pwd) | awk '{print tolower($0)}')
or
path=$(basename $(pwd) | tr 'A-Z' 'a-z' )
The form
x=y cmd
means to temporarily set environment variable x to value y and then run cmd, which is how these lines are interpreted:
path=eval 'pwd'
dirname=eval 'basename $path'
That is, they aren't doing what you seem to expect at all, instead setting an environment variable to the literal value eval and then running (or failing to find) a command. As others have said, the way to interpolate the results of a command into a string is to put it inside $(...) (preferred) or `...` (legacy). And, as a general rule, it's safer to wrap those in double quotes (as it is safer to wrap any interpolated reference in quotes).
path="$(pwd)"
dirname="$(basename "$path")"
(Technically, in this case the outer quotes aren't strictly necessary. However, I'd say it's still a good habit to have.)
B=$(echo "Some text that has CAPITAL letters " | awk '{print tolower($0)}')
eval executes command passed to it, but it returns only command exit status code, so you cannot really use it in set operator. The way to go to embed command into set operator either to use right single quotes or $()
So the script will look like this:
#!/bin/bash
curr_path=$(pwd)
echo $curr_path
curr_dir=$(basename $curr_path)
echo $curr_dir
echo $curr_dir | awk '{print tolower($0)}'
Your code doesn't work because you use single quotes rather than double quotes. Single quotes prevent variable expansion, thus $path is not expanded into the path you want to use and is taken as it is, as it if were a string.
Your awk invocation would not work for the same reason as well.
Although you could solve the problem replacing single quotes with double quotes, like this:
#!/bin/bash
path=eval "pwd"
dirname=eval "basename $path"
I would suggest using grave accents instead (). There's no reason to useeval` in this case. Plus, you can also use it to collect the return value you are interested in:
#!/bin/bash
path=`pwd`
dirname=`basename $path`
variable=`echo $dirname | awk "{print tolower($0)}"`
Here's an excerpt from my answer to What platform independent way to find directory of shell executable in shell script? which, in itself, fully answers your question aside from the lowercase part, which, in my opinion, has been duly addressed many times in other answers here.
What's unique about my answer is that when I was attempting to write it for the other question I encountered your exact problem - how do I store the function's results in a variable? Well, as you can see, with some help, I hit upon a pretty simple and very powerful solution:
I can pass the function a sort of messenger variable and dereference any explicit use of the resulting function's argument's $1 name with eval as necessary, and, upon the function routine's completion, I use eval and a backslashed quoting trick to assign my messenger variable the value I desire without ever having to know its name.
In full disclosure, though this was the solution to my problem, it was not by any means my solution. I've had several occasions to visit there before, but some of his descriptions, though probably brilliant, are a little out of my league, and so I thought others might benefit if include my own version of how this works in the previous paragraph. Though of course it was very simple to understand once I did, for this one especially, I had to think long and hard to figure out how it might work. Anyway, you can find that and more at Rich's sh tricks and I have also excerpted the relevant portion of his page below my own answer's excerpt.
...
EXCERPT:
...
Though not strictly POSIX yet, realpath is a GNU core app since 2012. Full disclosure: never heard of it before I noticed it in the info coreutils TOC and immediately thought of [the linked] question, but using the following function as demonstrated should reliably, (soon POSIXLY?), and, I hope, efficiently
provide its caller with an absolutely sourced $0:
% _abs_0() {
> o1="${1%%/*}"; ${o1:="${1}"}; ${o1:=`realpath "${1}"`}; eval "$1=\${o1}";
> }
% _abs_0 ${abs0:="${0}"} ; printf %s\\n "${abs0}"
/no/more/dots/in/your/path2.sh
EDIT: It may be worth highlighting that this solution uses POSIX parameter expansion to first check if the path actually needs expanding and resolving at all before attempting to do so. This should return an absolutely sourced $0via a messenger variable (with the notable exception that it will preserve symlinks) as efficiently as I could imagine it could be done whether or not the path is already absolute.
...
(minor edit: before finding realpath in the docs, I had at least pared down my version of [the version below] not to depend on the time field [as it does in the first ps command], but, fair warning, after testing some I'm less convinced ps is fully reliable in its command path expansion capacity)
On the other hand, you could do this:
ps ww -fp $$ | grep -Eo '/[^:]*'"${0#*/}"
eval "abs0=${`ps ww -fp $$ | grep -Eo ' /'`#?}"
...
And from Rich's sh tricks:
...
Returning strings from a shell function
As can be seen from the above pitfall of command substitution, stdout is not a good avenue for shell functions to return strings to their caller, unless the output is in a format where trailing newlines are insignificant. Certainly such practice is not acceptable for functions meant to deal with arbitrary strings. So, what can be done?
Try this:
func () {
body here
eval "$1=\${foo}"
}
Of course ${foo} could be replaced by any sort of substitution. The key trick here is the eval line and the use of escaping. The “$1” is expanded when the argument to eval is constructed by the main command parser. But the “${foo}” is not expanded at this stage, because the “$” has been quoted. Instead, it’s expanded when eval evaluates its argument. If it’s not clear why this is important, consider how the following would be bad:
foo='hello ; rm -rf /'
dest=bar
eval "$dest=$foo"
But of course the following version is perfectly safe:
foo='hello ; rm -rf /'
dest=bar
eval "$dest=\$foo"
Note that in the original example, “$1” was used to allow the caller to pass the destination variable name as an argument the function. If your function needs to use the shift command, for instance to handle the remaining arguments as “$#”, then it may be useful to save the value of “$1” in a temporary variable at the beginning of the function.
I’m trying to build a command string based to pass in a “-e” flag and another variable into a another base script being call as a subroutine and have run into a strange problem; I’m losing the “-e” portion of the string when I pass it into the subroutine. I create a couple example which illustrate the issue, any help?
This works as you would expect:
$echo "-e $HOSTNAME"
-e ops-wfm
This does NOT; we lose the “-e” because it is interpreted as a special qualifier.
$myFlag="-e $HOSTNAME"; echo $myFlag
ops-wfm
Adding the “\” escape charactor doesn’t work either, I get the correct string with the "\" in front:
$myFlag="\-e $HOSTNAME"; echo $myFlag
\-e ops-wfm
How can I prevent -e being swallowed?
Use double-quotes:
$ myFlag="-e $HOSTNAME"; echo "${myFlag}"
-e myhost.local
I use ${var} rather than $var out of habit as it means that I can add characters after the variable without the shell interpreting them as part of the variable name.
echo may not be the best example here. Most Unix commands will accept -- to mark no more switches.
$ var='-e .bashrc' ; ls -l -- "${var}"
ls: -e .bashrc: No such file or directory
Well, you could put your variable in quotes:
echo "$myFlag"
...making it equivalent to your first example, which, as you say, works just fine.
I have a question regarding the usage of functions in a command in bash. getRegex is my function, it is defined at the end of the the file. The command that I want to use is the following:
COUNT=`grep -rnE 'getRegex' $HOME/new`
Now I tried a lot of different variants but I cannot make it work, even if I split it in 2. The method works correctly if I call it the following way: getRegex. Any idea what I am missing? TIA
The key words to answer are "bash command substitution", which you could find in man bash or google.
By the way, double quotes are really important here.
#!/bin/bash
function my_func () {
echo "no"
}
string="no you don't
no you don't
no you don't
no you don't
no you don't"
COUNT="$( echo "${string}" | grep "$( my_func )" -c )"
echo "${COUNT}"
And
$> ./ok.sh
5
If you're trying to call a bash command within another bash command, the inner command (here getRegex) needs to be enclosed in backticks `` or else it will be interpreted as text. Since you here would have backticks inside backticks, you'll need to escape the inner ones. Try this:
COUNT=`grep -rnE '\`getRegex\`' $HOME/new`
But, through the wonders of POSIX, we can use a different syntax. Anywhere you use backticks, you can also use $(). So to avoid backslash emesis, you could write:
COUNT=$(grep -rnE '$(getRegex)' $HOME/new)