First of all i apologise in case this has been answered before but i couldn't solve my problem.
I need to search a pattern and then replace it with a line of text comprising of both text and variable.Btw i am using bash..
say
$var = "stacko.ver/rulz=" **Note: $var contain double quotes & = & a dot and /**
i want to so the follow
1.Search for ;te.xt = Note: The value to be search contain ; & = and a dot
2.Replace it with
textnum=$var
Of course $var should be replaced with its actual value
My attempts
sed -i "s/;te.xt =/textnum=$var/" file
sed -i "s/;te.xt =/textnum="$var"/" file
sed -i "s/";te.xt ="/"textnum=$var"/" file
None of these actually worked , either sed giving me an error or the value of $var not shown in file
Thanks for the help
Regards
Quoting doesn't help since this is a sed issue, not a bash issue. Just pick a sed s-expression delimiter that doesn't appear in your text:
sed -i "s|;te.xt =|textnum=$var|" file
You can pick any delimiter for s that doesn't appear in your input. sed -e 'streetlight' is a perfectly valid sed command.
I can see the error:
$ var="stacko.ver/rulz="
$ data="foo ;te.xt = bar"
$ sed "s/;te.xt =/textnum=$var/" <<< "$data"
sed: -e expression #1, char 31: unknown option to `s'
The problem is that $var contains a slash, so sed's s/// command is breaking. You need to pick a character that does not appear in $var
$ sed "s#;te.xt =#textnum=$var#" <<< "$data"
foo textnum=stacko.ver/rulz= bar
This can be hard -- what if slash and hash are in $var? Using bash, you can use ANSI-C quoting to use a control character that is unlikely to appear in your data, e.g.
$ sed $'s\037;te.xt =\037textnum=$var\037' <<< "$data"
foo textnum=stacko.ver/rulz= bar
Related
If I run these commands from a script:
#my.sh
PWD=bla
sed 's/xxx/'$PWD'/'
...
$ ./my.sh
xxx
bla
it is fine.
But, if I run:
#my.sh
sed 's/xxx/'$PWD'/'
...
$ ./my.sh
$ sed: -e expression #1, char 8: Unknown option to `s'
I read in tutorials that to substitute environment variables from shell you need to stop, and 'out quote' the $varname part so that it is not substituted directly, which is what I did, and which works only if the variable is defined immediately before.
How can I get sed to recognize a $var as an environment variable as it is defined in the shell?
Your two examples look identical, which makes problems hard to diagnose. Potential problems:
You may need double quotes, as in sed 's/xxx/'"$PWD"'/'
$PWD may contain a slash, in which case you need to find a character not contained in $PWD to use as a delimiter.
To nail both issues at once, perhaps
sed 's#xxx#'"$PWD"'#'
In addition to Norman Ramsey's answer, I'd like to add that you can double-quote the entire string (which may make the statement more readable and less error prone).
So if you want to search for 'foo' and replace it with the content of $BAR, you can enclose the sed command in double-quotes.
sed 's/foo/$BAR/g'
sed "s/foo/$BAR/g"
In the first, $BAR will not expand correctly while in the second $BAR will expand correctly.
Another easy alternative:
Since $PWD will usually contain a slash /, use | instead of / for the sed statement:
sed -e "s|xxx|$PWD|"
You can use other characters besides "/" in substitution:
sed "s#$1#$2#g" -i FILE
一. bad way: change delimiter
sed 's/xxx/'"$PWD"'/'
sed 's:xxx:'"$PWD"':'
sed 's#xxx#'"$PWD"'#'
maybe those not the final answer,
you can not known what character will occur in $PWD, / : OR #.
if delimiter char in $PWD, they will break the expression
the good way is replace(escape) the special character in $PWD.
二. good way: escape delimiter
for example:
try to replace URL as $url (has : / in content)
x.com:80/aa/bb/aa.js
in string $tmp
URL
A. use / as delimiter
escape / as \/ in var (before use in sed expression)
## step 1: try escape
echo ${url//\//\\/}
x.com:80\/aa\/bb\/aa.js #escape fine
echo ${url//\//\/}
x.com:80/aa/bb/aa.js #escape not success
echo "${url//\//\/}"
x.com:80\/aa\/bb\/aa.js #escape fine, notice `"`
## step 2: do sed
echo $tmp | sed "s/URL/${url//\//\\/}/"
URL
echo $tmp | sed "s/URL/${url//\//\/}/"
URL
OR
B. use : as delimiter (more readable than /)
escape : as \: in var (before use in sed expression)
## step 1: try escape
echo ${url//:/\:}
x.com:80/aa/bb/aa.js #escape not success
echo "${url//:/\:}"
x.com\:80/aa/bb/aa.js #escape fine, notice `"`
## step 2: do sed
echo $tmp | sed "s:URL:${url//:/\:}:g"
x.com:80/aa/bb/aa.js
With your question edit, I see your problem. Let's say the current directory is /home/yourname ... in this case, your command below:
sed 's/xxx/'$PWD'/'
will be expanded to
sed `s/xxx//home/yourname//
which is not valid. You need to put a \ character in front of each / in your $PWD if you want to do this.
Actually, the simplest thing (in GNU sed, at least) is to use a different separator for the sed substitution (s) command. So, instead of s/pattern/'$mypath'/ being expanded to s/pattern//my/path/, which will of course confuse the s command, use s!pattern!'$mypath'!, which will be expanded to s!pattern!/my/path!. I’ve used the bang (!) character (or use anything you like) which avoids the usual, but-by-no-means-your-only-choice forward slash as the separator.
Dealing with VARIABLES within sed
[root#gislab00207 ldom]# echo domainname: None > /tmp/1.txt
[root#gislab00207 ldom]# cat /tmp/1.txt
domainname: None
[root#gislab00207 ldom]# echo ${DOMAIN_NAME}
dcsw-79-98vm.us.oracle.com
[root#gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: ${DOMAIN_NAME}/g'
--- Below is the result -- very funny.
domainname: ${DOMAIN_NAME}
--- You need to single quote your variable like this ...
[root#gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: '${DOMAIN_NAME}'/g'
--- The right result is below
domainname: dcsw-79-98vm.us.oracle.com
VAR=8675309
echo "abcde:jhdfj$jhbsfiy/.hghi$jh:12345:dgve::" |\
sed 's/:[0-9]*:/:'$VAR':/1'
where VAR contains what you want to replace the field with
I had similar problem, I had a list and I have to build a SQL script based on template (that contained #INPUT# as element to replace):
for i in LIST
do
awk "sub(/\#INPUT\#/,\"${i}\");" template.sql >> output
done
If your replacement string may contain other sed control characters, then a two-step substitution (first escaping the replacement string) may be what you want:
PWD='/a\1&b$_' # these are problematic for sed
PWD_ESC=$(printf '%s\n' "$PWD" | sed -e 's/[\/&]/\\&/g')
echo 'xxx' | sed "s/xxx/$PWD_ESC/" # now this works as expected
for me to replace some text against the value of an environment variable in a file with sed works only with quota as the following:
sed -i 's/original_value/'"$MY_ENVIRNONMENT_VARIABLE"'/g' myfile.txt
BUT when the value of MY_ENVIRONMENT_VARIABLE contains a URL (ie https://andreas.gr) then the above was not working.
THEN use different delimiter:
sed -i "s|original_value|$MY_ENVIRNONMENT_VARIABLE|g" myfile.txt
This question already has answers here:
Sed Insert Multiple Lines
(3 answers)
Closed 1 year ago.
I have a multiline variable that I captured from STDOUT.
I want to insert an echo command using this multiline variable to line 15 in another script (target).
#!/bin/bash
TEST=`cat foo`
echo "$TEST"
sed -i "15i echo \"$TEST\" > someotherfile" target
Contents of foo :
apples
oranges
bananas
carrots
I thought the sed command read in line feeds, which I confirmed my foo has:
user#test$ cat foo | tr -cd '\n' | wc -c
4
When I run my test.sh script, I see what's in $TEST, but am getting an error for the sed command:
user#test$ ./test.sh
apples
oranges
bananas
carrots
sed: -e expression #1, char 18: unknown command: `o'
What am I doing wrong?
Thanks in advance.
GNU sed is assumed, as implied by the syntax used in the question.
#!/bin/bash
# Read contents of file 'foo' into shell variable $test.
test=$(<foo)
# \-escape the newlines in $test for use in Sed.
testEscapedForSed=${test//$'\n'/\\$'\n'}
sed -i "15i echo \"$testEscapedForSed\" > someotherfile" target
Your problem was that passing multi-line strings to sed functions such as i (insert) requires the newlines embedded in those strings to be \-escaped, so that sed knows where the string ends and additional commands, if any, start.
A (nonstandard) parameter expansion is used to replace all newlines in $test with themselves prefixed by \, using ANSI C-quoted string $'\n' to generate actual newline chars.
Also note:
I've renamed TEST to test, because all-uppercase shell-variable names should be avoided.
I've used modern command-substitution syntax $(..) in lieu of legacy syntax `...`.
$(<foo) is a slightly more efficient - although nonstandard - way of reading the content of a file at once.
Try:
Solution1:
awk 'NR==15{print;system("cat foo");next} 1' Input_file
No need to get the complete file into a variable, we could simply print it whichever line of Input_file you want to print it.
Solution2:
line=15; sed -e "${line}r foo" target
Or (in script mode)
cat script.ksh
line=15;
sed -e "${line}r foo" target
Where you could change the number of line where you want to insert the lines from another file.
The i command in sed inserts the lines of text that end with a newline, up until a line that doesn't end with a backslash. The a and c commands are similar. Classic sed doesn't like the first line to appear on the same line as the i command; GNU sed isn't as fussy.
If you were writing the command manually, you'd need to write:
15i\
echo "apples\
oranges\
bananas\
carrots" > someotherfile
At issue now is "how do you want to create this given the file foo contains the list of names?". Sometimes, using sed to generate the sed script is useful. However, it can also be intricate if you need to get backslashes at the ends of lines which are subject to an i (or a or c) command, and it is simpler to circumvent the problem.
{
echo "15i\\"
sed -e '1s/^/echo "/' -e 's/$/\\/' -e '$s/\\$/" > someotherfile/' foo
} | sed -f /dev/stdin target
GNU sed can read its script from standard input using -f -; BSD (macOS) sed doesn't like that, but you can use -f /dev/stdin instead (which also works with GNU sed), at least on systems where there is a /dev/stdin.
Interesting issue.
As already mentioned the whole story for sed to be able to insert multiline text in another file is that this new multiline text must have actually literral \n for line breaks.
So we can use sed to convert real new line chars to literal \n:
$ a=$(tr '\n' '\\' <file3 |sed 's#[\]$##' |sed "s#[\]#\0n#g")
#Alternative: a=$(sed "s#[\]#\0n#g" <(sed 's#[\]$##' <(tr '\n' '\\' <file3)))
$ echo "$a"
apples\noranges\nbananas\ncarrots
How this translation works:
* First we replace all new lines with a single backslash using tr
* Then we remove the backslash from the end of the string
* Then we replace all other backslashes with backaslash and n char.
Since now variable $a contains literal \n between lines, sed will translate them back to actuall new lines:
$ cat file4
Line1
line2
line3
$ sed "2i $a" file4
Line1
apples
oranges
bananas
carrots
line2
line3
Result:
Mutliline replacement can be done with two commands:
$ a=$(tr '\n' '\\' <file3 |sed 's#[\]$##' |sed "s#[\]#\0n#g")
$ sed "2i $a" file4
sed 2i means insert a text before line2. 2a can be used in order to insert something after line2.
Remark:
According to this post which seems to be a duplicate, translation of new lines to literal \n seems that can be done with just :
a=$(echo ${a} | tr '\n' "\\n")
But this method never worked in my system.
Remark2:
The sed operation sed "2i $a" = insert variable $a before line 2 , can be also expressed as sed "1 s/.*/\0\n$a/" = replace all chars of first line with the same chars \0 plus a new line \n plus the contents of variable $a => insert $a after line1 = insert $a before line2.
I have a file pattern.txt which is composed of one very long line of complicated code (~8200 chars).
This code can be found in multiple files inside multiple directories.
I can easily identify a list of these files using
grep -rli 'uniquepartofthecode' *
My concern is how do I replace it with the exact text from within the file ?
I tried to do:
var=$(cat pattern.txt)
sed -i "s/$var//g" targetfile.txt
but I got the following error :
sed: -e expression #1, char 96: unknown option to `s'
sed is interpreting my $var content as a regular expression, I would like it to just match the exact text.
The pattern.txt content could be more or less any combination of characters so I'm afraid I cannot escape every characters efficiently.
Is there a solution using sed ? Or should I use another tool for that ?
EDIT:
I tried using this solution to make a proper regex pattern from my text file.
Is it possible to escape regex metacharacters reliably with sed
the overall process is:
var=$(cat pattern.txt)
searchEscaped=$(sed 's/[^^]/[&]/g; s/\^/\\^/g' <<<"$var")
sed -n "s/$searchEscaped/foo/p" <<<"$var" # if ok, echoes 'foo'
This last command displays "foo". $searchEscaped seems to be properly escaped.
Though, this is not returning anything (it should display foo + the rest of the file without the matched part):
sed -n "s/$searchEscaped/foo/p" targetfile.txt
I think that the best solution is to not use regular expressions at all and resort to string replacement.
One way to do this is using perl:
$ echo "$string_to_replace"
some other stuff abc$^%!# some more
$ echo "$search"
abc$^%!#
$ perl -spe '$len = length $search;
while (($pos = index($_, $search, $n)) > -1) {
substr($_, $pos, $len) = "replacement";
$n = $pos + $len;
}' <<<"$string_to_replace" -- -search="$search"
some other stuff replacement some more
The -p switch tells perl to loop through each line of the variable $string_to_replace (which could easily be replaced by a file). -s allows options to be passed to the script - in this case, I've passed a shell variable containing the search string.
For each line of the file, the while loop runs through all of the matches of the search string. substr is used on the left hand of the assignment to replace a substring of $_, which refers to the current line being processed.
Using bash, how would one replace all unquoted characters from a file?
I have a system that I can't modify that spits out CSV files such as:
code;prop1;prop2;prop3;prop4;prop5;prop6
0,1000,89,"a1,a2,a3",33,,
1,,,"a55,a10",1,1 L,87
2,25,1001,a4,,"1,5 L",
I need this to become, for a new system being added
code;prop1;prop2;prop3;prop4;prop5;prop6
0;1000;89;a1,a2,a3;33;;
1;;;a55,a10;1;1 L;87
2;25;1001;a4;1,5 L;
If the quotes can be removed after this substitution happens in one command it would be nice :) But I prefer clarity to complicated one-liners for future maintenance.
Thank you
With sed:
sed -e 's/,/;/g' -e ':loop; s/\("\)\([^;]*\);\([^"]*"\)/\1\2,\3/; t loop'
Test:
$ sed -e 's/,/;/g' -e ':loop; s/\("\)\([^;]*\);\([^"]*"\)/\1\2,\3/; t loop' yourfile
code;prop1;prop2;prop3;prop4;prop5;prop6
0;1000;89;"a1,a2,a3";33;;
1;;;"a55,a10";1;1 L;87
2;25;1001;a4;;"1,5 L";
You want to use a csv parser. Parsing csv with shell tools is hard (you will encounter regular expressions soon, and they rarely get all cases).
There is one in almost every language. I recommend python.
You can also do this using excel/openoffice variants by opening the file and then saving with ; as the separator.
You can used sed:
echo '0,1000,89,"a1,a2,a3",33,,' | sed -e "s|\"||g"
This will replace " with the empty string (deletes it), and you can pipe another sed to replace the , with ;:
sed -e "s|,|;|g"
$ echo '0,1000,89,"a1,a2,a3",33,,' | sed -e "s|\"||g" | sed -e "s|,|;|g"
>> 0;1000;89;a1;a2;a3;33;;
Note that you can use any separator you want instead of | inside the sed command. For example, you can rewrite the first sed as:
sed -e "s-\"--g"
I need to remove escape character from the string in bash. I get a data structure, which contains url paths with / escaped so I receive the regular link:
http://stackoverflow.com/questions/ask
as one with escaped /:
http:\/\/stackoverflow.com\/questions\/ask
Now I need to remove \ from the second link. For this purpose I tried using sed
`echo '"'${paths[$index]}'"' | sed "s#\\##g"`
But I get an error:
sed: -e expression #1, char 6: unterminated `s' command
If I replace \\ with ie. _ it works like a charm and removes all occurrences of _ in a string. How do I get rid of escape characters in a string using sed?
try this:
.......|sed 's#\\##g'
or:
.......|sed "s#\\\\##g"
EDIT add a test output:
kent$ echo "http:\/\/stackoverflow.com\/questions\/ask"|sed "s#\\\\##g"
http://stackoverflow.com/questions/ask
kent$ echo "http:\/\/stackoverflow.com\/questions\/ask"|sed 's#\\##g'
http://stackoverflow.com/questions/ask
Your question isn't clear about which way round you want so here is both ways:
$ sed 's#/#\\/#g' <<< "http://stackoverflow.com/questions/ask"
http:\/\/stackoverflow.com\/questions\/ask
$ sed 's#\\/#/#g' <<< "http:\/\/stackoverflow.com\/questions\/ask"
http://stackoverflow.com/questions/ask
You don't need to use sed.
paths[index]=${paths[index]//\\/}
or simply
echo ${paths[index]//\\/}
to see the result without modifying the value in-place.
You can use this :
sed 's#\\##g'
But the problem is when you encounter a backslash that you actually want in the string, but is escaped. In that case :
sed 's/\\\\/\x1/' |sed 's/[\]//g' | sed 's/\x1/\\/g'
Replaces the double backslash with with a temp character[SOH], replaces all other backslashes and then restores the backslash that is needed.