I am having a problem with sed. Basically, I want to insert some text after a match a small piece of Java code after a match. It seems to be somehow interpreted as something else and ends up failing.
Consider the following basic example that does what I want:
match="string_I_want_to_match"
str_i_want_above="str I want above"
echo $match | sed -e "/$match/i \\
$str_i_want_above"
outputs what I want:
xyzt
string_I_want_to_match
However, if I use read to create a multiline string, it fails with an obscure error. Consider the following example:
match="string_I_want_to_match"
read -r -d '' str_i_want_above <<- 'EOF'
> I
> Want
> This
> Above the match
EOF
echo $match | sed -e "/$match/i \\
$str_i_want_above"
I expect
I
Want
This
Above the match
string_I_want_to_match
but instead I get
sed: 3: "/string_I_want_to_match ...": invalid command code W
It's somehow failing when trying to interpret the second line. I have isolated this to be a problem with read but if I'm wrong, I'd like to understand why and to fix this.
I'm using BSD sed on mac 12.6. I would like to achieve this with sed and read because I need to do in-place replacing and this is going to be a larger script where I need to do several in-place insertions of multiline strings
You need to insert a \ before the line breaks in replacement, so you may use:
match="string_I_want_to_match"
echo "$match" | sed "/$match/i \\
> ${str_i_want_above//$'\n'/\\$'\n'}
> "
I
Want
This
Above the match
string_I_want_to_match
Here:
${str_i_want_above//$'\n'/\\$'\n'}: Inserts \\ before each line break
Code Demo
Related
The name in the title says it all. However, I'm absolutely the worst with the sed command. So I'm trying to edit the following file:
/var/www/html/phpMyAdmin/config.inc.php
I want to edit the line that says
$cfg['Servers'][$i]['AllowRoot'] = false;
into the following
$cfg['Servers'][$i]['AllowRoot'] = true;
It has so many special characters and whatnot and I have no prior knowledge of how sed works. So here's some commands I've tried to specifically edit that one line.
sed -i "/*.AllowRoot.*/\$cfg['Servers'][\$i]['AllowRoot'] = true;/" /var/www/html/phpMyAdmin/config.inc.php
sed -i "/*.AllowRoot.*/$cfg['Servers'][$i]['AllowRoot'] = true;/" /var/www/html/phpMyAdmin/config.inc.php
# this one finds the line successfully and prints it so I know it's got the right string:
sed -n '/AllowRoot/p' /var/www/html/phpMyAdmin/config.inc.php
sed -i "s/'AllowRoot|false'/'AllowRoot|true'/" /var/www/html/phpMyAdmin/config.inc.php
I have absolutely no idea what I'm doing and I'm not learning a whole lot besides the feeling that the last command splits up 'AllowRoot|false' makes sure that both must be present in the sentence to come back as a result. So to my logic, I thought changing the word false into true would make that happen, but nothing. The other commands return... bizarre results at best, one even emptying the file. Or that's one of the commands I had not written down here, I've lost track after 50 attempts. What is the solution here?
The [ and ] need to be escaped to match literal brackets, instead of inadvertently starting a bracket expression. This should work:
$ sed -i "/\$cfg\['Servers'\]\[\$i\]\['AllowRoot'\]/s/false/true/" /var/www/html/phpMyAdmin/config.inc.php
There is not many things to escape in sed. Main problem in your line is / which you have chosen as delimiter (most common, but not required). I suggest you use # and the following will work:
sed -i "s#$cfg['Servers'][$i]['AllowRoot'] = false;<br />#$cfg['Servers'][$i]['AllowRoot'] = true;<br />#g" input.txt
however you need to think about bash interpreter as well. $i and $cfg will be interpreted as variables. My suggestion is that when you want to match a string like this to put the sed expression in a text file like this:
cat allow_root_true.sed
s#['Servers'][]['AllowRoot'] = false;<br />#['Servers'][]['AllowRoot'] = true;<br />#g
and run the command using sed -f like this:
sed -i -f allow_root_true.sed input.txt
Warning -i will change the input file
sed can't do literal string matching which is why you need to escape so many characters (see Is it possible to escape regex metacharacters reliably with sed), but awk can:
$ awk -v str="\$cfg['Servers'][\$i]['AllowRoot']" 'index($0,str){sub(/false/,"true")} 1' file
//some text here
$cfg['Servers'][$i]['AllowRoot'] = true;<br />
//some more text here
Run code snippetHide resultsExpand snippet
In the above we only have to escape the $s to protect them from the shell since the string is enclosed in "s to allow it to include 's.
This question already has answers here:
Using different delimiters in sed commands and range addresses
(3 answers)
Closed 1 year ago.
I have a Visual Studio project, which is developed locally. Code files have to be deployed to a remote server. The only problem is the URLs they contain, which are hard-coded.
The project contains URLs such as ?page=one. For the link to be valid on the server, it must be /page/one .
I've decided to replace all URLs in my code files with sed before deployment, but I'm stuck on slashes.
I know this is not a pretty solution, but it's simple and would save me a lot of time. The total number of strings I have to replace is fewer than 10. A total number of files which have to be checked is ~30.
An example describing my situation is below:
The command I'm using:
sed -f replace.txt < a.txt > b.txt
replace.txt which contains all the strings:
s/?page=one&/pageone/g
s/?page=two&/pagetwo/g
s/?page=three&/pagethree/g
a.txt:
?page=one&
?page=two&
?page=three&
Content of b.txt after I run my sed command:
pageone
pagetwo
pagethree
What I want b.txt to contain:
/page/one
/page/two
/page/three
The easiest way would be to use a different delimiter in your search/replace lines, e.g.:
s:?page=one&:pageone:g
You can use any character as a delimiter that's not part of either string. Or, you could escape it with a backslash:
s/\//foo/
Which would replace / with foo. You'd want to use the escaped backslash in cases where you don't know what characters might occur in the replacement strings (if they are shell variables, for example).
The s command can use any character as a delimiter; whatever character comes after the s is used. I was brought up to use a #. Like so:
s#?page=one&#/page/one#g
A very useful but lesser-known fact about sed is that the familiar s/foo/bar/ command can use any punctuation, not only slashes. A common alternative is s#foo#bar#, from which it becomes obvious how to solve your problem.
add \ before special characters:
s/\?page=one&/page\/one\//g
etc.
In a system I am developing, the string to be replaced by sed is input text from a user which is stored in a variable and passed to sed.
As noted earlier on this post, if the string contained within the sed command block contains the actual delimiter used by sed - then sed terminates on syntax error. Consider the following example:
This works:
$ VALUE=12345
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
MyVar=12345
This breaks:
$ VALUE=12345/6
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
sed: -e expression #1, char 21: unknown option to `s'
Replacing the default delimiter is not a robust solution in my case as I did not want to limit the user from entering specific characters used by sed as the delimiter (e.g. "/").
However, escaping any occurrences of the delimiter in the input string would solve the problem.
Consider the below solution of systematically escaping the delimiter character in the input string before having it parsed by sed.
Such escaping can be implemented as a replacement using sed itself, this replacement is safe even if the input string contains the delimiter - this is since the input string is not part of the sed command block:
$ VALUE=$(echo ${VALUE} | sed -e "s#/#\\\/#g")
$ echo "MyVar=%DEF_VALUE%" | sed -e s/%DEF_VALUE%/${VALUE}/g
MyVar=12345/6
I have converted this to a function to be used by various scripts:
escapeForwardSlashes() {
# Validate parameters
if [ -z "$1" ]
then
echo -e "Error - no parameter specified!"
return 1
fi
# Perform replacement
echo ${1} | sed -e "s#/#\\\/#g"
return 0
}
this line should work for your 3 examples:
sed -r 's#\?(page)=([^&]*)&#/\1/\2#g' a.txt
I used -r to save some escaping .
the line should be generic for your one, two three case. you don't have to do the sub 3 times
test with your example (a.txt):
kent$ echo "?page=one&
?page=two&
?page=three&"|sed -r 's#\?(page)=([^&]*)&#/\1/\2#g'
/page/one
/page/two
/page/three
replace.txt should be
s/?page=/\/page\//g
s/&//g
please see this article
http://netjunky.net/sed-replace-path-with-slash-separators/
Just using | instead of /
Great answer from Anonymous. \ solved my problem when I tried to escape quotes in HTML strings.
So if you use sed to return some HTML templates (on a server), use double backslash instead of single:
var htmlTemplate = "<div style=\\"color:green;\\"></div>";
A simplier alternative is using AWK as on this answer:
awk '$0="prefix"$0' file > new_file
You may use an alternative regex delimiter as a search pattern by backs lashing it:
sed '\,{some_path},d'
For the s command:
sed 's,{some_path},{other_path},'
I'm trying to clean a text file.
I want to delete everything start before the first 12 numbers.
1:0:135103079189:0:0:2:0::135103079189:000011:00
A:908529896240:0:10250:2:0:1:
603307102606:0:0:1:0::01000::M
Output desired:
135103079189:0:0:2:0::135103079189:000011:00
908529896240:0:10250:2:0:1:
603307102606:0:0:1:0::01000::M
Here's my command but seems not working.
sed '/:\([0-9]\{12\}\)/d' t.txt
the d command in sed will delete entire line on matching the given regex, you need to use s command to search and replace only part of line... however, for given problem, sed is not suitable as it doesn't support non-greedy regex
you can use perl instead
$ perl -pe's/^.*?(?=\d{12}:)//' ip.txt
135103079189:0:0:2:0::135103079189:000011:00
908529896240:0:10250:2:0:1:
603307102606:0:0:1:0::01000::M
.*? match zero or more characters as minimally as possible
(?=\d{12}:) only if it is followed by 12-digits ending with :
use perl -i -pe for in-place editing
some possible corner cases
$ # this is matching part of field
$ echo 'foo:123:abc135103079189:23:603307102606:1' | perl -pe's/^.*?(?=\d{12}:)//'
135103079189:23:603307102606:1
$ # this is not matching 12-digit field at end of line
$ echo 'foo:123:135103079189' | perl -pe's/^.*?(?=\d{12}:)//'
foo:123:135103079189
$ # so, add start/end of line matching cases and restrict 12-digits to whole field
$ echo 'foo:123:abc135103079189:23:603307102606:1' | perl -pe 's/^(?:.*?:)?(?=\d{12}(:|$))//'
603307102606:1
$ echo 'foo:123:135103079189' | perl -pe's/^(?:.*?:)?(?=\d{12}(:|$))//'
135103079189
Could you please try following.
awk --re-interval 'match($0,/[0-9]{12}/){print substr($0,RSTART)}' Input_file
Since I have OLD version of awk so I am using --re-interval you could remove it in case you have new version of it.
This might work for you (GNU sed):
sed -n 's/[0-9]\{12\}/\n&/;s/.*\n//p' file
We only want to print specific lines so use the -n option to turn off automatic printing. If a line contains a 12 digit number, insert a newline before it. Remove any characters before and including a newline and print the result.
If you want to print lines that do not contain a 12 digit number as is, use:
sed 's/[0-9]\{12\}/\n&/;s/.*\n//' file
The crux of the problem is to identify the start of a multi-character string, insert a unique marker and delete all characters before and including the unique marker. As sed uses the newline to delimit lines, only the user can introduce newlines into the pattern space and as a result, newlines will always be unique.
Taking the nice answer from #Sundeep, in case you would like to use grep or pcregrep (macOS/BSD) you could give a try to:
$ grep -oP '^(?:.*?:)?(?=\d{12})\K.*' file
or
$ pcregrep -o '^(?:.*?:)?(?=\d{12})\K.*' file
The \K will ignore everything after the pattern
Alternative thoughts - I almost think your data is too dirty for a quick sed fix but if generally it's all similar to your sample set of data then certainly pick one of the answers with sed etc. However if you wanted to be more particular about it you could build up a set of commands to ensure the values. I like doing this for debugging and when speed isn't urgent.
Take this tiny sample of code, you could do this other ways but I'm getting the value for each part of the string and I know the order because it contiguous. You could then set up controls on which parts to keep and such as it builds out say a new string per line. Overwrought for sure, but sometimes that is a better long term approach.
#!/bin/bash
while IFS= read -r line ;do
IFS=':' read -r -a array <<< "$line"
for ((i=0; i<${#array[#]}; i++)) ;do
echo "part : ${array[$i]}"
done
done < "test_data.txt"
You could then build the data back up how you wanted and more easily understand what's happening every step of the way ..
part : 1
part : 0
part : 135103079189
part : 0
part : 0
part : 2
part : 0
part :
part : 135103079189
part : 000011
part : 00
part : A
part : 908529896240
part : 0
While using Sed to search/ insert a config file, I'm greeted by errors. What's causing them, and how can I fix them?
The Heredoc I'm looking to insert can be defined as follows:
read -d '' APPLICATION_ENV_STATE <<'EOF'
defined('APPLICATION_ENV') || define('APPLICATION_ENV',(getenv('APPLICATION_ENV')
? getenv('APPLICATION_ENV') : 'production'));
EOF
While my Sed command uses the variable like this:
sed -i "/\/\/ \*\* MySQL settings \*\* \/\//i$APPLICATION_ENV_STATE" wp-config.php
Which results in:
sed: -e expression #1, char 1: unknown command: `?'
In addition to an extra characters after command error.
However, the following Heredoc works, but results in some less than pretty formatting in my text file:
read -d '' APPLICATION_ENV_STATE <<'EOF'
defined('APPLICATION_ENV') || define('APPLICATION_ENV', (getenv('APPLICATION_ENV') ? getenv('APPLICATION_ENV') : 'production'));
EOF
How do I get the first example to work?
AIUI, it's not the heredoc that's the problem, it's understanding which process is doing what at various times.
In your script that runs the sed command, Bash is substituting the variable before sed even sees it. Being a multi-line string, it would need escaping for sed). From the man page for sed, under the i command:
i \ Insert text, which has each embedded newline preceded by a back-slash.
Personally, I'd recommend using cat or echo if you can (or a scriping language like Python / Ruby / PHP), having broken the template up into atomic elements, so you can simply concatenate the relevant pieces together.
If you do want to continue with the current method though, you'll at least need to replace the newlines with backslashed newlines - try something like:
echo $APPLICATION_ENV_STATE | sed 's/$/\\/'
You're using the wrong tool. The only constructs you should be using in sed are s, g, and p (with -n). Just use awk and avoid all the quoting/escaping nonsense:
$ cat file
foo
// ** MySQL settings ** //
bar
$ awk -v app="defined('APPLICATION_ENV') || define('APPLICATION_ENV',(getenv('APPLICATION_ENV')
? getenv('APPLICATION_ENV') : 'production'));" '
{print} index($0,"// ** MySQL settings ** //"){print app}' file
foo
// ** MySQL settings ** //
defined('APPLICATION_ENV') || define('APPLICATION_ENV',(getenv('APPLICATION_ENV')
? getenv('APPLICATION_ENV') : 'production'));
bar
Notice that you don't need to escape the RE metachars in the string you want to search for because awk can treat a string as a string and you don't need to escape newlines in the string you're adding and you don't need a here doc with a shell variable, etc.
Your read/sed command as written would fail for various character combinations in your search and/or replacement strings - see Is it possible to escape regex metacharacters reliably with sed for how to robustly search/replace "strings" in sed but then just use awk so you don't have to worry about any of it.
I'm trying to get a bunch of regular expressions for a file (one per line) and then fit those regexps into something like this /$regexp/d . I'm trying it this way:
while read line;do sed "/$line/d" to_delete.file >> output;done < to_delete.txt
But it says me 'unknown command', even if I change the delimiter.
--- EDIT
The to_delete.txt file has slashes but i'm already scraping them and that's where i find the error.
To avoid problem with / in regex sed is allow to use another separator, so you can use e.g. sed "\|$line|d".
Secondary if you put script into double-quotes you shoud add space between address range and action e.g. "\|$line| d"
But I see a general mistake in the script. The loop will print into output all to_delete.file (exept 1 line with regexp) by each loop. I suppose it is not the thing what OP wants.
If you'd like to exclude content of to_delete.txt from to_delete.file it can be easy done by grep
grep -vFf "to_delete.txt" "to_delete.file" > output