The following command prints the absolute path of a particular C++ header, according to where g++ believes it to be.
echo \#include\<ham/hamsterdb.h\> | g++ -M -x c++-header - | grep hamsterdb.hpp | sed -e 's/-: //' -e 's/ \\//'
On my system, this outputs: /usr/local/include/ham/hamsterdb.hpp
I run into a problem when trying to run this inside a Makefile in order to set a variable:
FILE=$(shell echo \#include\<ham/hamsterdb.h\> | g++ -M -x c++-header - | grep hamsterdb.hpp | sed -e 's/-: //' -e 's/ \\//')
.PHONY spec
spec:
#echo $(FILE)
This outputs a new line. I think it's the hash ('#') character which is messing with make; if I rewrite the FILE=... line like this:
FILE=$(shell echo \#include\<ham/hamsterdb.h\>)
the output is still nothing.
You have to escape the hash twice in order to use it inside functions: once for Make and once again for the shell.
That is,
FILE = $(shell echo \\\#include\ \<ham/hamsterdb.h\>)
Notice three backslashes instead of two as one could expect. The third backslash is needed because otherwise the first one would escape the second and the hash would be still unescaped.
UPD.
Another possible solution is to only escape the hash for Make and use Bash single quotes to prevent interpreting the hash as a shell comment. This also eliminates the need of escaping spaces, < and >:
FILE = $(shell echo '\#include <ham/hamsterdb.h>')
You need just a bit more quoting:
FILE=$(shell echo \\\#include\<ham/hamsterdb.h\> ...
^^^
Quote once for make itself, a second time for the shell (the shell needs to 'see' \#).
Related
I have being trying to write a bash script that can search recursively in a directory and replace multiple strings e.g. #{DEMO_STRING_1} etc with an environment variable e.g. $sample1.
Full script:
#!/bin/sh
find /my/path/here -type f -name '*.js' -exec sed -i \
-e 's/#{DEMO_STRING_1}/'"$sample1"'/g' \
-e 's/#{DEMO_STRING_2}/'"$sample2"'/g' \
-e 's/#{DEMO_STRING_3}/'"$sample3"'/g' \
-e 's/#{DEMO_STRING_4}/'"$sample4"'/g' \
-e 's/#{DEMO_STRING_5}/'"$sample5"'/g' \
-e 's/#{DEMO_STRING_6}/'"$sample6"'/g' \
-e 's/#{DEMO_STRING_7}/'"$sample7"'/g' \
-e 's/#{DEMO_STRING_8}/'"$sample8"'/g' \
{} +
I can not figure out how to replace strings with hashtag with curly brackets.
I tried this example: sed find and replace with curly braces or Environment variable substitution in sed but I can not figure out how to combine them.
What I am missing? I searched also for characters that need to be escaped e.g. What characters do I need to escape when using sed in a sh script? but again not the characters that I need.
The specific format is throwing the following error:
sed: bad option in substitution expression
Where am I going so wrong?
Update: Sample of environment variables:
https://www.example.com
/sample string/
12345-abcd-54321-efgh
base64 string
All the cases above are environment variables that I would like to replace. All environment variables are within double quotes.
It is important to understand that the environment variable references are expanded by the shell, as it prepares to execute the command, not by the command itself (sed in this case). The command sees only the results of the expansions.
In your case, that means that if any of the environment variables' values contain characters that are meaningful to sed in context, such as unescaped (to sed) slashes (/), then sed will attribute special significance to them instead of interpreting them as ordinary characters. For example, given a sed command such as
sed -e "s/X/${var}/" <<EOF
Replacement: X
EOF
, if the value of $var is Y then the output will be
Replacement: Y
, but if the value of $var is /path/to/Y then sed will fail with the same error you report. This happens because the sed command actually run is the same as if you had typed
sed -e s/X//path/to/Y
, which contains an invalid s instruction. Probably the best alternative would be to escape the replacement-string characters that otherwise would be significant to sed. You can do that by interposing a shell function:
escape_replacement() {
# replace all \ characters in the first argument with double backslashes.
# Note that in order to do that here, we need to escape them from the shell
local temp=${1//\\/\\\\}
# Replace all & characters with \&
temp=${temp//&/\\&}
# Replace all / characters with \/, and write the result to standard out.
# Use printf instead of echo to avoid edge cases in which the value to print
# is interpreted to be or start with an option.
printf -- "%s" "${temp//\//\\/}"
}
Then the script would use it like this:
find /my/path/here -type f -name '*.js' -exec sed -i \
-e 's/#{DEMO_STRING_1}/'"$(escape_replacement "$sample1")"'/g' \
...
Note that you probably also want to use a shebang line that explicitly specifies a shell that supports substitution references (${parameter/pattern/replacement}), because these are not required by POSIX, and you might run into a system where /bin/sh is a shell that does not support them. If you're willing to rely on Bash then that should be reflected in your shebang line. Alternatively, you could prepare a version of the escape_replacement function that does not rely on substitution references.
If you use perl - you don't need to escape anything.
With your shell variable exported you can access it via $ENV{name} inside perl.
examples:
samples=(
https://www.example.com
'/sample string/'
12345-abcd-54321-efgh
'base64 string'
$'multi\nline'
)
for sample in "${samples[#]}"
do
echo '---'
export sample
echo 'A B #{DEMO_STRING_1} C' |
perl -pe 's/#{DEMO_STRING_1}/$ENV{sample}/g'
done
echo '---'
Output:
---
A B https://www.example.com C
---
A B /sample string/ C
---
A B 12345-abcd-54321-efgh C
---
A B base64 string C
---
A B multi
line C
---
To add the -i option you can: perl -pi -e 's///'
Makefile:
.PHONY: all
SHELL:=/usr/bin/env bash
all:
$(eval x=$(shell cat file))
#echo "$x"
File:
foo
bar
Output:
foo bar
How do I get the contents of the file into the make variable without losing the newlines?
You can't do this with shell, as described in its documentation.
If you have a sufficiently new version of GNU make, you can use the file function however.
Make converts newlines from shell outputs to spaces (see here):
The shell function performs the same function that backquotes (‘`’)
perform in most shells: it does command expansion. This means that it
takes as an argument a shell command and evaluates to the output of
the command. The only processing make does on the result is to convert
each newline (or carriage-return / newline pair) to a single space. If
there is a trailing (carriage-return and) newline it will simply be
removed.
So, you cannot preserve spaces from the $(shell) command directly. That being said, make does allow multiline variables using define -- but beware, attempting to use such variables is problematic. Consider:
define x
foo
bar
endef
all:
#echo "$x"
Make expands the $x in place, and you end up with:
all:
#echo " foo
bar"
(where the newline is considered the end of the recipe line..).
Depending on what you want this for, you may be able to get around this is using a bash variable:
all:
#x=$$(cat file); \
echo $$x
Or potentially storing your output in a file, and referencing that when necessary.
all:
eval (cat file >> output.txt)
cat output.txt
(and yes, the last one is convoluted as written, but I'm not sure what you're trying to do, and this allows the output of your command to be persistent across recipe lines).
If the file contents are ensured not to contain any binary data, and if you're willing to to extra processing each time you access the variable, then you could:
foo:=$(shell cat file | tr '\n' '\1')
all:
#echo "$(shell echo "$(foo)" | tr '\1' '\n')"
Note that you cannot use nulls \0, and I suspect that probably means there's a buffer overflow bug in my copy of Make.
I am trying to find and replace a specific text content using the sed command and to run it via a shell script.
Below is the sample script that I am using:
fp=/asd/filename.txt
fd="sed -i -E 's ($2).* $2:$3 g' ${fp}"
eval $fd
and executing the same by passing the arguments:
./test.sh update asd asdfgh
But if the argument string contains $ , it breaks the commands and it is replacing with wrong values, like
./test.sh update asd $apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1.
How can I make sure that the values inside the variables are not expanded because of the $?
Updated
sh file test.sh
set -xv
fp="/asd/filename.txt"
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
text file filename.txt
hello:world
Outputs
1)
./test.sh update hello WORLD
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
++ sed -iE 's/(hello).*/hello:WORLD/g' /asd/filename.txt
2)
./test.sh update hello '$apr1$hosgaxyv$D0KXp5dCyZ2BUYCS9BmHu1'
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
++ sed -iE 's/(hello).*/hello:'\''$'\''apr1'\''$'\''hosgaxyv'\''$'\''D0KXp5dCyZ2BUYCS9BmHu1/g' /asd/filename.txt
In both the case , its not replacing the content
You don't need eval here at all:
fp=/asd/filename.txt
sed -i -E "s/(${2//'$'/'\$'}).*/\1:${3//'$'/'\$'}/g" "$fp"
The whole sed command is in double quotes so variables can expand.
I've replaced the blank as the s separator with / (doesn't really matter in the example).
I've used \1 to reference the first capture group instead of repeating the variable in the substitution.
Most importantly, I've used ${2//'$'/'\$'} instead of $2 (and similar for $3). This escapes every $ sign as \$; this is required because of the double quoting, or the $ get eaten by the shell before sed gets to see them.
When you call your script, you must escape any $ in the input, or the shell tries to expand them as variable names:
./test.sh update asd '$apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1.'
Put the command-line arguments that are filenames in single quotes:
./test.sh update 'asd' '$apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1'
must protect all the script arguments with quotes if having space and special shell char, and escape it if it's a dollar $, and -Ei instead of -iE even better drop it first for test, may add it later if being really sure
I admit i won't understant your regex so let's just get in the gist of solution, no need eval;
fp=/asd/filename.txt
sed -Ei "s/($2).*/$2:$3/g" $fp
./test.sh update asd '\$apr1\$HnIF6bOt\$9m3NzAwr.aG1Yp.t.bpIS1.'
I'm trying to escape ('\') a semicolon (';') in a string on unix shell (bash) with sed. It works when I do it directly without assigning the value to a variable. That is,
$ echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'
hello\;
$
However, it doesn't appear to work when the above command is assigned to a variable:
$ result=`echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'`
$
$ echo $result
hello;
$
Any idea why?
I tried by using the value enclosed with and without quotes but that didn't help. Any clue greatly appreciated.
btw, I first thought the semicolon at the end of the string was somehow acting as a terminator and hence the shell didn't continue executing the sed (if that made any sense). However, that doesn't appear to be an issue. I tried by using the semicolon not at the end of the string (somewhere in between). I still see the same result as before. That is,
$ echo "hel;lo" | sed 's/\([^\\]\);/\1\\;/g'
hel\;lo
$
$ result=`echo "hel;lo" | sed 's/\([^\\]\);/\1\\;/g'`
$
$ echo $result
hel;lo
$
You don't need sed (or any other regex engine) for this at all:
s='hello;'
echo "${s//;/\;}"
This is a parameter expansion which replaces ; with \;.
That said -- why are you trying to do this? In most cases, you don't want escape characters (which are syntax) to be inside of scalar variables (which are data); they only matter if you're parsing your data as syntax (such as using eval), which is a bad idea for other reasons, and best avoided (or done programatically, as via printf %q).
I find it interesting that the use of back-ticks gives one result (your result) and the use of $(...) gives another result (the wanted result):
$ echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'
hello\;
$ z1=$(echo "hello;" | sed 's/\([^\\]\);/\1\\;/g')
$ z2=`echo "hello;" | sed 's/\([^\\]\);/\1\\;/g'`
$ printf "%s\n" "$z1" "$z2"
hello\;
hello;
$
If ever you needed an argument for using the modern x=$(...) notation in preference to the older x=`...` notation, this is probably it. The shell does an extra round of backslash interpretation with the back-ticks. I can demonstrate this with a little program I use when debugging shell scripts called al (for 'argument list'); you can simulate it with printf "%s\n":
$ z2=`echo "hello;" | al sed 's/\([^\\]\);/\1\\;/g'`
$ echo "$z2"
sed
s/\([^\]\);/\1\;/g
$ z1=$(echo "hello;" | al sed 's/\([^\\]\);/\1\\;/g')
$ echo "$z1"
sed
s/\([^\\]\);/\1\\;/g
$ z1=$(echo "hello;" | printf "%s\n" sed 's/\([^\\]\);/\1\\;/g')
$ echo "$z1"
sed
s/\([^\\]\);/\1\\;/g
$
As you can see, the script executed by sed differs depending on whether you use x=$(...) notation or x=`...` notation.
s/\([^\]\);/\1\;/g # ``
s/\([^\\]\);/\1\\;/g # $()
Summary
Use $(...); it is easier to understand.
You need to use four (three also work). I guess its because it's interpreted twice, first one by the sed command and the second one by the shell when reading the content of the variable:
result=`echo "hello;" | sed 's/\([^\\]\);/\1\\\\;/g'`
And
echo "$result"
yields:
hello\;
Here is my code. I simply want to copy some files and replace a line in my Makefile. The parameter $1 is just the name of my new .tex file.
#!/bin/bash
pwd="./"
tex=".tex"
pwd=$1$tex
cp ~/TeX/matt.sty .
cp ~/TeX/mon.tex $pwd
cp ~/TeX/Makefile .
sed="sed 's/mon.tex/"$1$tex"/g' Makefile > Makefile"
$sed
I've the following error : sed: 1: "'s/mon.tex/salut.tex/g'": invalid command code '
ps: i'm using sed on Mac OS X. (so it's bsd sed)
The first argument to sed is literally 's/mon.tex/"$1$tex"/g' (with single quotes). obviously sed cannot parse that as a command.
Removing the single quotes would solve that problem but redirection (>) still won't work.
Just run the sed command directly (what's the point of the $sed variable? i don't get it)
Note: to modify a file with sed, use sed -i. Redirecting to the same file you are processing won't work.
You've doubling up your sed command line:
sed="sed 's/mon.tex/"$1$tex"/g'"
creates a sed variable. You then use this variable as a command:
$sed s/mon.tex$1$tex/g Makefile > Makefile
^^^^---here
which effectively makes your command line:
sed 's/mon.tex/"$1$tex"/g' s/mon.tex$1$tex/g Makefile > Makefile
^^^^^^^^^^^^^^^^^^^^^^^^^^--var
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^--- excess args
and now I see your question's been editted to remove that... so ignore this, I guess.
You say
sed="sed 's/mon.tex/"$1$tex"/g'"
Which creates a variable sed containing the string sed 's/mon.text/foo.tex/g', presuming that $1 is foo for the sake of example.
You then expand $sed unquoted and it becomes
sed ''\''s/mon.tex/foo.tex/g'\'''
Which includes a literal ' at the beginning of your expression, as if you had said:
sed -e \''s///'
EDIT: To reiterate, your problem is that you're needlessly quoting your sed expression inside the variable assignment. Use
sed="sed s/mon.tex/$1$tex/g Makefile > Makefile"
And it will work as expected.
The error message is caused by the '(single quote).
And, the ' is not a valid sed command.
Example:
$ echo="echo 'hello, world'"
$ $echo
'hello, world'
You can use eval command to Quote Removal further more:
sed="sed 's/mon.tex/"$1$tex"/g' Makefile > Makefile.new"
eval $sed
Note:
>Makefile will make your orginal Makefile empty! So change it to Makefile.new.
eval is evil. Try to use sed directly!