I am new in linux and AWK as well. I have a file in my home folder called testing.txt and i am trying to read the file using this awk command:
**arjun#arjun-Aspire-4741:~$ awk ´{print $1}´ testing.txt¨**
And I am getting this as output
**¨awk: ´{print
awk: ^ invalid char '�' in expression
arjun#arjun-Aspire-4741:~$ ¨**
The problem is that you've used forward ticks instead of quotes (in this case only single quotes are appropriate):
awk '{print $1}' testing.txt
instead of
awk ´{print $1}´ testing.txt
In shell, strings in double quotes " can contain expressions with special meaning (such as backticks, variables) which will be expanded before the string is processed as part of the full shell command. Strings in single quotes ' are fully escaped; to put it another way, the string is passed literally without any interpretation. That's why you should use single quotes when writing awk scripts, because the awk variable dereference operator $ is the same as in shell. There are no other valid string-delimiting characters*.
I initially thought you'd used backticks (thanks to Andras Deak for spotting my error).
Backticks have a special meaning in shell (equivalent to wrapping something in $(...)): execute this string as a command, and evaluate to its output (stdout). This is done before your main command is executed.
So, if I do
cat `echo myfile`
this turns into
cat myfile
which then executes.
You can read more about shell behaviour in a few places:
http://www.grymoire.com/Unix/Sh.html
http://www.tldp.org/LDP/Bash-Beginners-Guide/html/index.html
* ignoring that spaces are also technically string-delimiters
Related
this is perhaps one of the most discussed topics here. I tried almost all the commands and other tweaks found here, but something doesn't seems to be doing well.
i would want to replace all the double quotes in my file with whitespace/blank
I'm seeing the below error when i tried to execute this command.
sed "s/"/ \''/g' x_orbit.txt > new.tx
sed: -e expression #1, char 3: unterminated `s' command
You're close. Just use single quotes, so the shell doesn't try to expand the metacharacters in your sed command:
sed 's/"/ /g' x_orbit.txt > new.txt
You could try tr for example:
tr '"' ' ' < x_orbit.txt > new.txt
The script you provided:
sed "s/"/ \''/g' x_orbit.txt > new.tx
means:
sed # invoke sed to execute the following script:
" # enclose the script in double quotes rather than single so the shell can
# interpret it (e.g. to expand variables like $HOME) before sed gets to
# interpret the result of that expansion
s/ # replace what follows until the next /
" # exit the double quotes so the shell can now not only expand variables
# but can now do globbing and file name expansion on wildcards like foo*
/ # end the definition of the regexp you want to replace so it is null since
# after the shell expansion there was no text for sed to read between
# this / and the previous one (the 2 regexp delimiters)
\' # provide a blank then an escaped single quote for the shell to interpret for some reason
'/g' # enclose the /g in single quotes as all scripts should be quoted by default.
That is so far off the correct syntax it's kinda shocking which is why I dissected it above to try to help you understand what you wrote so you'll see why it doesn't work. Where did you get the idea to write it that way (or to put it another way - what did you think each character in that script meant? I'm asking as it indicates a fundamental misunderstanding of how quoting and escaping works in shell so it'd be good if we could help correct that misunderstanding rather than just correct that script.
When you use any script or string in shell, simply always enclose it in single quotes:
sed 'script' file
var='string'
unless you NEED to use double quotes to let a variable expand and then use double quotes unless you NEED to use no quotes to let globbing and file name expansion happen.
An awk version:
awk '{gsub(/"/," ")}1' file
gsub is used for the replace
1 is always true, so line is printed
I'm trying to extract a string from a group of strings stored in a variable, say foo, in bash:
foo="I foobar you"
using awk, but I got different results when I wrapped awk with single and double quotes:
$ echo $foo | awk '{print $2}'
$ foobar
$ echo $foo | awk "{print $2}"
$ I foobar you
Could anyone tell me why ' and " cause different results when wrapping awk?
It is because when using double-quotes the the shell tries to expand it as a positional parameter or a variable before passing it to awk. In your case it tries to expand $1 but finds no value for it, and so the expansion results to nothing and that's why you get the default print action in awk to print the whole line.
The reason why we single quote the awk commands is that we want a plain string and not modified/tampered by the shell by any means before processed by the Awk command itself which understands $1 as the first field (with default space delimiter)
According to shell quoting rules:
Single quotes (‘ ‘) protect everything between the opening and closing quotes. The shell does no interpretation of the quoted text, passing it on verbatim to the command.
As in your code {print $2} passed on as it is to awk as an action:
$ echo $foo | awk '{print $2}'
$ foobar
Double quotes (“ “) protect most things between the opening and closing quotes. The shell does at least variable and command substitution on the quoted text. Different shells may do additional kinds of processing on double-quoted text. Because certain characters within double-quoted text are processed by the shell, they must be escaped within the text. Of note are the characters ‘$’, ‘‘’, ‘\’, and ‘"’, all of which must be preceded by a backslash within double-quoted text if they are to be passed on literally to the program.
So you have to escape $ to get the value of second col as:
$ echo $foo | awk "{print \$2}"
$ foobar
otherwise awk is doing its default action of printing the whole line as in your case.
Fully agreed with #Ed Mortan regarding the best practice of quoting the shell variables. Although the double quotes are default, but if accidentally ….
$ echo "$foo" with double quotes
$ I foobar you
$ echo ‘$foo’ with single quotes
$ $foo
Remember, quoting rules are specific to shell. The above rules only apply to POSIX-complaint, Bourne-style shells (such as Bash, the GNU Bourne-Again Shell). It has nothing to do with awk, sed, grep, etc…
I'm trying to remove everything after a specific_string in a path string in Bash. I've tried using sed to no avail so far.
variable="specific_string"
input_string="/path/to/some/specific_string/specific_string.something/specific_string.something-else"
output=$(sed 's/$variable//' $input_string)
Output should be "/path/to/some/specific_string/"
Would be better if I didn't have to use commands such as sed!
The Problems
There are many problems
Variables are not evaluated inside single quotes. 's/$variable//' will be treated as a literal string, which does not contain specific_string
sed can modify text from files or STDIN, but not text given via parameters. With sed 's/...//' $input_string the /path/to/some/specific_string/.../file is opened and its content is read, instead of the path itself.
s/string// deletes only string, not the words afterwards.
Also remember to double quote your variables. cmd $variable is dangerous if the variable contains spaces. cmd "$variable" is safe.
Sed Solution
output="$(sed "s/$variable.*/$variable/" <<< "$input_string")"
GNU Grep Solution
output="$(grep -Po "^.*?$variable" <<< "$input_string")"
Pure Bash Solution
output="${input_string%%$variable*}$variable"
If you want to remove everything after "specific_string" it will remove the "/" also as it does with the following example:
output=$(echo $input_string|sed "s/${variable}.*$/${variable}/")
try with simple sed:
variable="specific_string"
input_string="/path/to/some/specific_string/specific_string.something/specific_string.something-else"
output=$(echo "$input_string" | sed "s/\(.*$variable\/\).*/\1/")
Output of variable output will be as follows.
echo $output
/path/to/some/specific_string/
I want to issue this command from the bash script
sed -e $beginning,$s/pattern/$variable/ file
but any possible combination of quotes gives me an error, only one that works:
sed -e "$beginning,$"'s/pattern/$variable/' file
also not good, because it do not dereferences the variable.
Does my approach can be implemented with sed?
Feel free to switch the quotes up. The shell can keep things straight.
sed -e "$beginning"',$s/pattern/'"$variable"'/' file
You can try this:
$ sed -e "$beginning,$ s/pattern/$variable/" file
Example
file.txt:
one
two
three
Try:
$ beginning=1
$ variable=ONE
$ sed -e "$beginning,$ s/one/$variable/" file.txt
Output:
ONE
two
three
There are two types of quotes:
Single quotes preserve their contents (> is the prompt):
> var=blah
> echo '$var'
$var
Double quotes allow for parameter expansion:
> var=blah
> echo "$var"
blah
And two types of $ sign:
One to tell the shell that what follows is the name of a parameter to be expanded
One that stands for "last line" in sed.
You have to combine these so
The shell doesn't think sed's $ has anything to do with a parameter
The shell parameters still get expanded (can't be within single quotes)
The whole sed command is quoted.
One possibility would be
sed "$beginning,\$s/pattern/$variable/" file
The whole command is in double quotes, i.e., parameters get expanded ($beginning and $variable). To make sure the shell doesn't try to expand $s, which doesn't exist, the "end of line" $ is escaped so the shell doesn't try anything funny.
Other options are
Double quoting everything but adding a space between $ and s (see Ren's answer)
Mixing quoting types as needed (see Ignacio's answer)
Methods that don't work
sed '$beginning,$s/pattern/$variable/' file
Everything in single quotes: the shell parameters are not expanded (doesn't follow rule 2 above). $beginning is not a valid address, and pattern would be literally replaced by $variable.
sed "$beginning,$s/pattern/$variable/" file
Everything in double qoutes: the parameters are expanded, including $s, which isn't supposed to (doesn't follow rule 1 above).
the following form worked for me from within script
sed $beg,$ -e s/pattern/$variable/ file
the same form will also work if executed from the shell
I am writing a shell program to output another shell program to be evalled later. Is there some common shell program to print shell escaped for a string?
I'm not sure I understand you question. But the %q option of printf might be what you are looking for.
%q Output the corresponding argument in a format that can be reused as shell input
printf %q 'C:\ProgramFiles is a Windows path;'
outputs C:\\ProgramFiles\ is\ a\ Windows\ path\;
(In this example, simple quotes are needed – comment of Gordon Davisson – but this doesn't matter if you print from a variable or the output of a command.)
You could use single quoted string as this is evaluated without any substitution.
For example the following commands are equivalent
cat abc\ hi.txt
cat 'abc hi.txt'