I want to store the contents of a file in a bash shell variable. This works fine:
$ cat hello
Hello, world!
$ F=hello
$ Q=$(cat $F)
$ echo $Q
Hello, world!
However, if the file contains an asterisk, the asterisk is replaced by a list of all files in the current directory.
How can I quote the filename to protect the asterisk? Or otherwise load the file into the shell variable?
I am aware of this question, but it doesn't work for files that contain an asterisk.
Q contains the asterisk. It is the unquoted expansion of $Q that replaces the * with a list of files.
$ Q="*"
$ echo $Q
<list of files>
$ echo "$Q"
*
The right-hand side of an assignment is not subject to path name expansion, so Q=* would work as well, and the command substitution used to read from the file is also not affected. Q=$(cat hello) works fine: you just need to quote the expansion of Q.
Related
This is a simple question but i am unable to find it in tutorials. Could anybody please explain what this statement does when executed in a bash shell within a folder containing .sh scripts. I know -i does in place editing, i understand that it will run sed on all scripts within the current directory. And i know that it does some sort of substitution. But what does this \(.*\) mean?
sed -i 's/MY_BASE_DIR=\(.*\)/MY_BASE_DIR=${MY_BASE_DIR-\1}/' *.sh
Thanks in advance.
You have an expression like:
sed -i 's/XXX=\(YYY\)/XXX=ZZZ/' file
This looks for a string XXX= in a file and captures what goes after. Then, it replaces this captured content with ZZZ. Since there is a captured group, it is accessed with \1. Finally, using the -i flag in sed makes the edition to be in-place.
For the replacement, it uses the following syntax described in Shell parameter expansion:
${parameter:-word}
If parameter is unset or null, the expansion of word is substituted.
Otherwise, the value of parameter is substituted.
Example:
$ d=5
$ echo ${d-3}
5
$ echo ${a-3}
3
So with ${MY_BASE_DIR-SOMETHING-\1} you are saying: print $MY_BAS_DIR. And if this variable is unset or null, print what is stored in \1.
All together, this is resetting MY_BASE_DIR to the value in the variable $MY_BASE_DIR unless this is not set; in such case, the value remains the same.
Note though that the variable won't be expanded unless you use double quotes.
Test:
$ d=5
$ cat a
d=23
blabla
$ sed "s/d=\(.*\)/d=${d-\1}/" a # double quotes -> value is replaced
d=5
blabla
$ sed 's/d=\(.*\)/d=${d-\1}/' a # single quotes -> variable is not expanded
d=${d-23}
blabla
Andd see how the value remains the same if $d is not set:
$ unset d
$ sed "s/d=\(.*\)/d=${d-\1}/" a
d=23
The scripts contain lines like this:
MY_BASE_DIR=/usr/local
The sed expression changes them to:
MY_BASE_DIR=${MY_BASE_DIR-/usr/local}
The effect is that /usr/local is not used as a fixed value, but only as the default value. You can override it by setting the environment variable MY_BASE_DIR.
For future reference, I would take a look at the ExplainShell website:
http://explainshell
that will give you a breakdown of the command structure etc. In this instance, let step through the details...Let's start with a simple example, let's assume that we were going to make the simple change - commenting out all lines by adding a "#" before each line. We can do this for all *.sh files in a directory with the ".sh" extension in the current directory:
sed 's/^/\#/' *.sh
i.e. Substitute beginning of line ^, with a # ...
Caveat: You did not specify the OS you are using. You may get different results with different versions of sed and OS...
ok, now we can drill into the substitution in the script. An example is probably easier to explain:
File: t.sh
MY_BASE_DIR="/important data/data/bin"
the command 's/MY_BASE_DIR=\(.*\)/MY_BASE_DIR=${MY_BASE_DIR-\1}/' *.sh
will search for "MY_BASE_DIR" in each .sh file in the directory.
When it encounters the string "MY_BASE_DIR=.*", in the file, it expands it to be MY_BASE_DIR="/important data/data/bin", this is now replaced on the right side of the expression /MY_BASE_DIR=${MY_BASE_DIR-\1}/ which becomes
MY_BASE_DIR=${MY_BASE_DIR-"/important data/data/bin"}
essentially what happens is that the substitute operation takes
MY_BASE_DIR="/important data/data/bin"
and inserts
MY_BASE_DIR=${MY_BASE_DIR-"/important data/data/bin"}
now if we run the script with the variable MY_BASE_DIR set
export MY_BASE_DIR="/new/import/dir"
the scripts modified by the sed script referenced will now substitute /important data/data/bin with /new/import/dir...
My problem boils down to this:
echo $(echo '*')
That outputs the names of all the files in the current directory.
I do not want that. I want a literal asterisk (*).
How do I do this in a generic way?
My above example is simplified. The asterisk is not literally written in my bash script - it comes from the result of another command.
So this is perhaps closer to my real situation:
echo $(my-special-command)
I just want to get the literal output of my-special-command; I do not want any embedded asterisks (or other special characters) to be expanded.
How do I do this in a general-purpose way?
I suppose I could do set -f before running the command, but how can I be sure that covers everything? That turns off pathname expansion, but what about other kinds? I have zero control over what output might be produced by my-special-command, so must be able to handle everything properly.
Just enclose the Command substitution with double quotes:
echo "$(my-special-command)"
Its called globbing, you have multiply ways to prevent it:
echo * # will expand to files / directories
echo "*" # Will print *
echo '*' # Will also print *
In your example you can simple write:
echo "$(echo '*')"
You can also turn off globbing in your script by calling it with bash -f script.sh or inside your code:
#!/usr/bin/env bash
set -f
echo *
From the "Command Substitution" section of the man page:
If the [command] substitution appears within double quotes, word splitting and
pathname expansion are not performed on the results.
By quoting the command expansion, you prevent its result, *, from undergoing pathname expansion.
$ echo "$(echo "*")"
Running this statement in OS X Terminal
for i in `ls -v *.mkv`; do echo $i; done
will successfully print out all the file names in the directory in name order with each file name on its own line.
Source: This StackOverFlow answer
However, if I run this statement in OS X Terminal
for i in 'ls -v *.mkv'; do echo $i; done
the output is "ls -v fileName1.mkv fileName2.mkv", etc. with all the file names concatenated into one long line (as opposed to each being printed on its own line).
My questions are:
What's the difference between ` and ' in bash?
Why is that difference responsible for the completely different output?
What keyboard combination produces `? (Keyboard combination)
1) Text between backticks is executed and replaced by the output of the enclosed command, so:
echo `echo 42`
Will expand to:
echo 42
This is called Command Substitution and can also be achieved using the syntax $(command). In your case, the following:
for i in `ls -v *.mkv`; do ...
Is replaced by something like (if your directory contains 3 files named a.mkv, b.mkv and c.mkv):
for i in a.mkv b.mkv c.mkv; do ...
Text between quotes or double quotes are just plain Bash strings with characters like space scaped inside them (there are other ways to quote strings in Bash and are described here):
echo "This is just a plain and simple String"
echo 'and this is another string'
A difference between using ' and " is that strings enclosed between " can interpolate variables, for example:
variable=42
echo "Your value is $variable"
Or:
variable=42
echo "Your value is ${variable}"
Prints:
Your value is 42
2) Wildcard expressions like *.mkv are replaced by the expanded filenames in a process known as Globbing. Globbing is activated using wildcards in most of the commands without enclosing the expression inside a string:
echo *.mkv
Will print:
a.mkv b.mkv c.mkv
Meanwhile:
echo "*.mkv"
prints:
*.mkv
The i variable in your for loop takes the value "ls -v *.mkv" but the echo command inside the loop body takes $i without quotes, so Bash applied globbing there, you end up with the following:
for i in 'ls -v *.mkv'; do
# echo $i
#
# which is expanded to:
# echo ls -v *.mkv (no quotes)
#
# and the globbing process transform the above into:
echo ls -v a.mkv b.mkv c.mkv
Which is just a one-line string with the file names after the globbing is applied.
3) It depends on your keyboard layout.
One trick to keep the character around is to use the program ascii, search for the character 96 (Hex 60), copy it and keep it on your clipboard (you can use parcellite or any other clipboard manager that suits your needs).
Update: As suggested by #triplee, you should check useless use of ls as this is considered a bash pitfall and there are better ways to achieve what you're trying to do.
'expression', will output the exact string in expression.
`expression`, will execute the content of the expression and echo outputs it.
For example:
x="ls"
echo "$x" --> $x
echo `$x` --> file1 file2 ... (the content of your current dir)
Backticks mean "run the thing between the backticks as a command, and then act as if I had typed the output of that command here instead". The single quotes mean, as others have said, just a literal string. So in the first case, what happens is this:
bash runs ls -v *.mkv as a command, which outputs something like:
fileName1.mkv
fileName2.mkv
bash then substitutes this back into where the backtick-surrounded command was, i.e. it effectively makes your for statement into this:
for i in fileName1.mkv fileName2.mkv; do echo $i; done
That has two "tokens": "fileName1.mkv" and "fileName2.mkv", so the loop runs its body (echo $i) twice, once for each:
echo fileName1.mkv
echo fileName2.mkv
By default, the echo command will output a newline after it finishes echoing what you told it to echo, so you'll get the output you expect, of each filename on its own line.
When you use single quotes instead of backticks, however, the stuff in between the single quotes doesn't get evaluated; i.e. bash doesn't see it as a command (or as anything special at all; the single quotes are telling bash, "this text is not special; do not try to evaluate it or do anything to it"). So that means what you're running is this:
for i in 'ls -v *.mkv'; do echo $i; done
Which has only one token, the literal string "ls -v *.mkv", so the loop body runs only once:
echo ls -v *.mkv
...but just before bash runs that echo, it expands the "*.mkv".
I glossed over this above, but when you do something like ls *.mkv, it's not actually ls doing the conversion of *.mkv into a list of all the .mkv filenames; it's bash that does that. ls never sees the *.mkv; by the time ls runs, bash has replaced it with "fileName1.mkv fileName2.mkv ...".
Similarly for echo: before running this line, bash expands the *.mkv, so what actually runs is:
echo ls -v fileName1.mkv fileName2.mkv
which outputs this text:
ls -v fileName1.mkv fileName2.mkv
(* Footnote: there's another thing I've glossed over, and that's spaces in filenames. The output of the ls between the backticks is a list of filenames, one per line. The trouble is, bash sees any whitespace -- both spaces and newlines -- as separators, so if your filenames are:
file 1.mkv
file 2.mkv
your loop will run four times ("file", "1.mkv", "file", "2.mkv"). The other form of the loop that someone mentioned, for i in *.mkv; do ... doesn't have this problem. Why? Because when bash is expanding the "*.mkv", it does a clever thing behind the scenes and treats each filename as a unit, as if you'd said "file 1.mkv" "file 2.mkv" in quotes. It can't do that in the case where you use ls because after it passes the expanded list of filenames to ls, bash has no way of knowing that what came back was a list of those same filenames. ls could have been any command.)
At the command line, this cat works as expected:
cat /home/me/path\ with\ spaces/to/file
But if I put it in a script:
#!/bin/bash
FILE="$1"
cat $FILE # cat "$FILE" gives same result
and call the script with ./script.sh "/home/me/path\ with\ spaces/to/file", I get:
cat: /home/me/path\ with\ spaces/to/file: No such file or directory
Note the escape quotes, which should be in the right places.
What gives?
Simply use double quotes to prevent word splitting:
cat "$FILE"
As an aside, upper case variable names should be reserved for shell internal variables, so you should change FILE to file.
If you are quoting the argument to your script (which is a good idea), then the file name doesn't need backslashes:
./script.sh "/home/me/path with spaces/to/file"
I am performing a grep on a file which is resulting in a single line output. This output has * as data in it. In the shell script I am trying to assign the value to a variable but * is being replaced with the file list in the current folder.
Eg:
My script name is script1.sh and I have another file script2.sh in the same directory.
The content of the script is
VAR1=`grep pattern search_file`
echo $VAR1
The intended output would be
The pattern is *
But the output I am getting is
The pattern is script1.sh script2.sh
Kindly let me know what is that I am doing wrong.
You simply need to quote the variable: echo "$VAR1"
If you look at the sequence of bash shell expansions, you'll notice that filename expansion occurs after parameter expansion. Unquoted variables will be subsequently subjected to word splitting and filename expansion.
use set -f shell option will disable globbing in sub-shells and interactive session.
use set +f to enable globbing again.
You need to escape the * with \, otherwise it treats it as a wild-card that matches filenames in the current directory.
\*