ANSWER:
as mlv and sorontar mentioned, my SED version is BRE and doesn't support | (pipe). so in my case is possible use something like:
sed "s/\( [namevlu]*=\"\)BASE\.$str1/\1BASE\.$str2/g"
which match name=" and value=" but not other=" and values=". regex ( [namevlu]) contains only characters what i need. ok, it is not as save, as can be, but i don't expect existence of something like valuenm=" or so. if someone needs exactly specified regex, it needs make two (or more) seds.
ORIGINAL QUESTION:
I need to replace one string with another, but for sure i need check specific context.
for example:
blah val1="BASE.OLD_TEXT_OR_SO" blabla
blah val2="BASE.OLD_SOMETHING" ...
i want change to
blah val1="BASE.NEW_TEXT_OR_SO" blabla
blah val2="BASE.NEW_SOMETHING" ...
this script doesn't change anything:
#!/bin/bash
...
str1="OLD_"
str2="NEW_"
sed "s/\(name=\"|value=\"\)BASE\.$str1/\1BASE\.$str2/g" input.file > output.file
but later similiar sed works ok:
sed "s/\(<Tag>\)[A-Z0-9\-\._|]*\(<\/Tag>\)/\1$otherStr\2/g" input.file > output.file
output file has still BASE.OLD_ :/
also when i try it on console, i get same (none) result. i think there is something wrong in "looking for" pattern, but i havent idea what.
$ str0='blah name="BASE.OLD_TEXT_OR_SO" blabla
> blah value="BASE.OLD_SOMETHING" ...
> blah other="BASE.OLD_SOMETHING" ...
> blah values="BASE.OLD_SOMETHING" ...'
$ echo $str0 | sed "s/\(val1=\"|val2=\"\)BASE\.$str1/\1BASE\.$str2/g"
regex was tested on online tester where it works fine.
(name="|value=")BASE\.OLD_
\1BASE\.NEW_
system:
SunOS 5.11
GNU bash 4.1.11(1)-release
Sed 4.2.1
Thanks in advance.
I'm not sure SunOS sed includes |. That would be added with -r which I don't think SunOS sed supports.
But in this case, you should be able to do:
sed "s/\(val[12]=\"\)BASE.$str1/\1BASE.$str2/"
If you can't use val[12], then I don't think it can be done in sed. But it's not too hard in perl:
perl -pe "if (/val1=BASE.$str1/ || /val2=BASE.$str1/) {s/BASE.$str1/BASE.$str2/;}"
In a basic sed only BRE are supported, (POSIX BRE) and the alternation (|) does not exist.
Your regex must be written as:
sed "s/\(val[12]=\"BASE\.\)${str1}/\1${str2}/g" input.file
If, the version of sed you use supports (extended) ERE (which accepts the |) then you may use:
sed -E "s/(val(1|2)=\"BASE\.)${str1}/\1${str2}/g" input.file
as mlv and sorontar mentioned, my SED version is BRE and doesn't support | (pipe). so in my case is possible use something like:
sed "s/\( [namevlu]*=\"\)BASE\.$str1/\1BASE\.$str2/g"
which match name=" and value=" but not other=" and values=". regex ( [namevlu]) contains only characters what i need. ok, it is not as save, as can be, but i don't expect existence of something like valuenm=" or so. if someone needs exactly specified regex, it needs make two (or more) seds.
Related
I want to parse a file and replace the text between "::" and ":::" with the text already there, just now capitalized.
I've tried using this command:
sed 's/\(::\)\(.*\)\(:::\)/\1\U\2\E\3/' filename
but the output just puts a U in beginning and E at the end of the string I want capitalized
Works for me, which makes me think you may not be on Linux?
echo "This is :: some sample text ::: to test uppercasing" | sed 's/\(::\)\(.*\)\(:::\)/\1\U\2\E\3/'
This is :: SOME SAMPLE TEXT ::: to test uppercasing
If Perl is your option, you can say something like:
echo "This is :: some sample text ::: to test uppercasing" | perl -pe 's/(::)(.*)(:::)/\1\U\2\E\3/'
This is :: SOME SAMPLE TEXT ::: to test uppercasing
gawk '{match($0,/::.*:::/,a) ;gsub(/::.*::/,toupper(a[0]))}1' input
Here ,bit less cryptic solution with gawk:, match is used to find the desired string ,later that string is used by gsub to convert it to upped cause using toupper function.
You are pretty close.
On Mac OS X, you will need to install GNU sed, because the feature you are using - \U - is a GNU extension.
So, start by installing it:
▶ brew install gnu-sed
Then I normally stick in some code like this somewhere:
shopt -s expand_aliases
alias sed='/usr/local/bin/gsed'
And then your GNU sed will work.
Finally, I would simplify that code as:
▶ sed -E 's/(::)(.*)(::)/\1\U\2\E\3/' <<< "foo::bar::baz"
foo::BAR::baz
Noting that -E gives you Extended Regular Expressions, and a cleaner syntax when you are doing captures.
This might work for you (GNU sed):
sed 's/::[^:]*:::/\U&/' file
or perhaps:
sed 's/::[^:]*:::/\n&\n/;h;y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/;G;s/.*\n\(.*\)\n.*\n\(.*\)\n.*\n/\2\1/' file
Using seds y native translate command, pattern matching and a copy held in the hold space.
I have a file with text in it that looks like this:
ssid="sdfsdf" #psk="sdfsfsdf" psk=zzzs93j03r093ur0fjfs39uj }
I'm learning sed and want to use it to extract just the string that starts with zzz.
I though I could just grab everything between " psk=" and " }" but this does not seem to work:
sed 's_ psk=\(.*\) }_\1_' /tmp/myfile
I am also curious why it doesn't work. psk appears in the text twice but I though searching for " psk" would distinguish it from "#psk"
With GNU sed:
sed 's/.* psk=\([^ ]*\).*/\1/' file
Output:
zzzs93j03r093ur0fjfs39uj
See: The Stack Overflow Regular Expressions FAQ
Your command are working just fine, It replaces everything from psk=...$ with zzzs..., what you want to add is .* in the front:
% sed 's_.* psk=\(.*\) }_\1_'
zzzs93j03r093ur0fjfs39uj
You can change your regex a little to handle cases where psk is not at the end:
sed 's/.*[^#]psk=\([[:alnum:]]*\).*/\1/'
For some reason I can't seem to find a straightforward answer to this and I'm on a bit of a time crunch at the moment. How would I go about inserting a choice line of text after the first line matching a specific string using the sed command. I have ...
CLIENTSCRIPT="foo"
CLIENTFILE="bar"
And I want insert a line after the CLIENTSCRIPT= line resulting in ...
CLIENTSCRIPT="foo"
CLIENTSCRIPT2="hello"
CLIENTFILE="bar"
Try doing this using GNU sed:
sed '/CLIENTSCRIPT="foo"/a CLIENTSCRIPT2="hello"' file
if you want to substitute in-place, use
sed -i '/CLIENTSCRIPT="foo"/a CLIENTSCRIPT2="hello"' file
Output
CLIENTSCRIPT="foo"
CLIENTSCRIPT2="hello"
CLIENTFILE="bar"
Doc
see sed doc and search \a (append)
Note the standard sed syntax (as in POSIX, so supported by all conforming sed implementations around (GNU, OS/X, BSD, Solaris...)):
sed '/CLIENTSCRIPT=/a\
CLIENTSCRIPT2="hello"' file
Or on one line:
sed -e '/CLIENTSCRIPT=/a\' -e 'CLIENTSCRIPT2="hello"' file
(-expressions (and the contents of -files) are joined with newlines to make up the sed script sed interprets).
The -i option for in-place editing is also a GNU extension, some other implementations (like FreeBSD's) support -i '' for that.
Alternatively, for portability, you can use perl instead:
perl -pi -e '$_ .= qq(CLIENTSCRIPT2="hello"\n) if /CLIENTSCRIPT=/' file
Or you could use ed or ex:
printf '%s\n' /CLIENTSCRIPT=/a 'CLIENTSCRIPT2="hello"' . w q | ex -s file
Sed command that works on MacOS (at least, OS 10) and Unix alike (ie. doesn't require gnu sed like Gilles' (currently accepted) one does):
sed -e '/CLIENTSCRIPT="foo"/a\'$'\n''CLIENTSCRIPT2="hello"' file
This works in bash and maybe other shells too that know the $'\n' evaluation quote style. Everything can be on one line and work in
older/POSIX sed commands. If there might be multiple lines matching the CLIENTSCRIPT="foo" (or your equivalent) and you wish to only add the extra line the first time, you can rework it as follows:
sed -e '/^ *CLIENTSCRIPT="foo"/b ins' -e b -e ':ins' -e 'a\'$'\n''CLIENTSCRIPT2="hello"' -e ': done' -e 'n;b done' file
(this creates a loop after the line insertion code that just cycles through the rest of the file, never getting back to the first sed command again).
You might notice I added a '^ *' to the matching pattern in case that line shows up in a comment, say, or is indented. Its not 100% perfect but covers some other situations likely to be common. Adjust as required...
These two solutions also get round the problem (for the generic solution to adding a line) that if your new inserted line contains unescaped backslashes or ampersands they will be interpreted by sed and likely not come out the same, just like the \n is - eg. \0 would be the first line matched. Especially handy if you're adding a line that comes from a variable where you'd otherwise have to escape everything first using ${var//} before, or another sed statement etc.
This solution is a little less messy in scripts (that quoting and \n is not easy to read though), when you don't want to put the replacement text for the a command at the start of a line if say, in a function with indented lines. I've taken advantage that $'\n' is evaluated to a newline by the shell, its not in regular '\n' single-quoted values.
Its getting long enough though that I think perl/even awk might win due to being more readable.
A POSIX compliant one using the s command:
sed '/CLIENTSCRIPT="foo"/s/.*/&\
CLIENTSCRIPT2="hello"/' file
Maybe a bit late to post an answer for this, but I found some of the above solutions a bit cumbersome.
I tried simple string replacement in sed and it worked:
sed 's/CLIENTSCRIPT="foo"/&\nCLIENTSCRIPT2="hello"/' file
& sign reflects the matched string, and then you add \n and the new line.
As mentioned, if you want to do it in-place:
sed -i 's/CLIENTSCRIPT="foo"/&\nCLIENTSCRIPT2="hello"/' file
Another thing. You can match using an expression:
sed -i 's/CLIENTSCRIPT=.*/&\nCLIENTSCRIPT2="hello"/' file
Hope this helps someone
The awk variant :
awk '1;/CLIENTSCRIPT=/{print "CLIENTSCRIPT2=\"hello\""}' file
I had a similar task, and was not able to get the above perl solution to work.
Here is my solution:
perl -i -pe "BEGIN{undef $/;} s/^\[mysqld\]$/[mysqld]\n\ncollation-server = utf8_unicode_ci\n/sgm" /etc/mysql/my.cnf
Explanation:
Uses a regular expression to search for a line in my /etc/mysql/my.cnf file that contained only [mysqld] and replaced it with
[mysqld]
collation-server = utf8_unicode_ci
effectively adding the collation-server = utf8_unicode_ci line after the line containing [mysqld].
I had to do this recently as well for both Mac and Linux OS's and after browsing through many posts and trying many things out, in my particular opinion I never got to where I wanted to which is: a simple enough to understand solution using well known and standard commands with simple patterns, one liner, portable, expandable to add in more constraints. Then I tried to looked at it with a different perspective, that's when I realized i could do without the "one liner" option if a "2-liner" met the rest of my criteria. At the end I came up with this solution I like that works in both Ubuntu and Mac which i wanted to share with everyone:
insertLine=$(( $(grep -n "foo" sample.txt | cut -f1 -d: | head -1) + 1 ))
sed -i -e "$insertLine"' i\'$'\n''bar'$'\n' sample.txt
In first command, grep looks for line numbers containing "foo", cut/head selects 1st occurrence, and the arithmetic op increments that first occurrence line number by 1 since I want to insert after the occurrence.
In second command, it's an in-place file edit, "i" for inserting: an ansi-c quoting new line, "bar", then another new line. The result is adding a new line containing "bar" after the "foo" line. Each of these 2 commands can be expanded to more complex operations and matching.
I've been searching google for ever, and I cannot find an example of how to do this. I also do not grasp the concept of how to construct a regular expression for SED, so I was hoping someone could explain this to me.
I'm running a bash script against a file full of lines of text that look like this: 2222,H,73.82,04,07,2012
and I need to make them all look like this: 2222,H,73.82,04072012
I need to remove the last two commas, which are the 16th and 19th characters in the line.
Can someone tell me how to do that? I was going to use colrm, which is blessedly simple, but i can't seem to get that installed in CYGWIN. Please and thank you!
I'd use awk for this:
awk -F',' -v OFS=',' '{ print $1, $2, $3, $4$5$6 }' inputfile
This takes a CSV file and prints the first, second and third fields, each followed by the output field separator (",") and then the fourth, fifth and sixth fields concatenated.
Personally I find this easier to read and maintain than regular expression-based solutions in sed and it will cope well if any of your columns get wider (or narrower!).
This will work on any string and will remove only the last 2 commas:
sed -e 's/\(.*\),\([^,]*\),\([^,]*\)$/\1\2\3/' infile.txt
Note that in my sed variant I have to escape parenthesis, YMMV.
I also do not grasp the concept of how to construct a regular
expression for SED, so I was hoping someone could explain this to me.
The basic notation that people are telling you here is: s/PATTERN/REPLACEMENT/
Your PATTERN is a regular expression, which may contain parts that are in brackets. Those parts can then be referred to in the REPLACEMENT part of the command. For example:
> echo "aabbcc" | sed 's/\(..\)\(..\)\(..\)/\2\3\1/'
bbccaa
Note that in the version of sed I'm using defaults to the "basic" RE dialect, where the brackets in expressions need to be escaped. You can do the same thing in the "extended" dialect:
> echo "aabbcc" | sed -E 's/(..)(..)(..)/\2\3\1/'
bbccaa
(In GNU sed (which you'd find in Linux), you can get the same results with the -r options instead of -E. I'm using OS X.)
I should say that for your task, I would definitely follow Johnsyweb's advice and use awk instead of sed. Much easier to understand. :)
It should work :
sed -e 's~,~~4g' file.txt
remove 4th and next commas
echo "2222,H,73.82,04,07,2012" | sed -r 's/(.{15}).(..)./\1\2/'
Take 15 chars, drop one, take 2, drop one.
sed -e 's/(..),(..),(....)$/\1\2\3/' myfile.txt
I'm trying to use SED to extract text from a log file. I can do a search-and-replace without too much trouble:
sed 's/foo/bar/' mylog.txt
However, I want to make the search case-insensitive. From what I've googled, it looks like appending i to the end of the command should work:
sed 's/foo/bar/i' mylog.txt
However, this gives me an error message:
sed: 1: "s/foo/bar/i": bad flag in substitute command: 'i'
What's going wrong here, and how do I fix it?
Update: Starting with macOS Big Sur (11.0), sed now does support the I flag for case-insensitive matching, so the command in the question should now work (BSD sed doesn't reporting its version, but you can go by the date at the bottom of the man page, which should be March 27, 2017 or more recent); a simple example:
# BSD sed on macOS Big Sur and above (and GNU sed, the default on Linux)
$ sed 's/ö/#/I' <<<'FÖO'
F#O # `I` matched the uppercase Ö correctly against its lowercase counterpart
Note: I (uppercase) is the documented form of the flag, but i works as well.
Similarly, starting with macOS Big Sur (11.0) awk now is locale-aware (awk --version should report 20200816 or more recent):
# BSD awk on macOS Big Sur and above (and GNU awk, the default on Linux)
$ awk 'tolower($0)' <<<'FÖO'
föo # non-ASCII character Ö was properly lowercased
The following applies to macOS up to Catalina (10.15):
To be clear: On macOS, sed - which is the BSD implementation - does NOT support case-insensitive matching - hard to believe, but true. The formerly accepted answer, which itself shows a GNU sed command, gained that status because of the perl-based solution mentioned in the comments.
To make that Perl solution work with foreign characters as well, via UTF-8, use something like:
perl -C -Mutf8 -pe 's/öœ/oo/i' <<< "FÖŒ" # -> "Foo"
-C turns on UTF-8 support for streams and files, assuming the current locale is UTF-8-based.
-Mutf8 tells Perl to interpret the source code as UTF-8 (in this case, the string passed to -pe) - this is the shorter equivalent of the more verbose -e 'use utf8;'.Thanks, Mark Reed
(Note that using awk is not an option either, as awk on macOS (i.e., BWK awk and BSD awk) appears to be completely unaware of locales altogether - its tolower() and toupper() functions ignore foreign characters (and sub() / gsub() don't have case-insensitivity flags to begin with).)
A note on the relationship of sed and awk to the POSIX standard:
BSD sed and awk limit their functionality mostly to what the POSIX sed and
POSIX awk specs mandate, whereas their GNU counterparts implement many more extensions.
Editor's note: This solution doesn't work on macOS (out of the box), because it only applies to GNU sed, whereas macOS comes with BSD sed.
Capitalize the 'I'.
sed 's/foo/bar/I' file
Another work-around for sed on Mac OS X is to install gsedfrom MacPorts or HomeBrew and then create the alias sed='gsed'.
If you are doing pattern matching first, e.g.,
/pattern/s/xx/yy/g
then you want to put the I after the pattern:
/pattern/Is/xx/yy/g
Example:
echo Fred | sed '/fred/Is//willma/g'
returns willma; without the I, it returns the string untouched (Fred).
The sed FAQ addresses the closely related case-insensitive search. It points out that a) many versions of sed support a flag for it and b) it's awkward to do in sed, you should rather use awk or Perl.
But to do it in POSIX sed, they suggest three options (adapted for substitution here):
Convert to uppercase and store original line in hold space; this won't work for substitutions, though, as the original content will be restored before printing, so it's only good for insert or adding lines based on a case-insensitive match.
Maybe the possibilities are limited to FOO, Foo and foo. These can be covered by
s/FOO/bar/;s/[Ff]oo/bar/
To search for all possible matches, one can use bracket expressions for each character:
s/[Ff][Oo][Oo]/bar/
The Mac version of sed seems a bit limited. One way to work around this is to use a linux container (via Docker) which has a useable version of sed:
cat your_file.txt | docker run -i busybox /bin/sed -r 's/[0-9]{4}/****/Ig'
Use following to replace all occurrences:
sed 's/foo/bar/gI' mylog.txt
I had a similar need, and came up with this:
this command to simply find all the files:
grep -i -l -r foo ./*
this one to exclude this_shell.sh (in case you put the command in a script called this_shell.sh), tee the output to the console to see what happened, and then use sed on each file name found to replace the text foo with bar:
grep -i -l -r --exclude "this_shell.sh" foo ./* | tee /dev/fd/2 | while read -r x; do sed -b -i 's/foo/bar/gi' "$x"; done
I chose this method, as I didn't like having all the timestamps changed for files not modified. feeding the grep result allows only the files with target text to be looked at (thus likely may improve performance / speed as well)
be sure to backup your files & test before using. May not work in some environments for files with embedded spaces. (?)
Following should be fine:
sed -i 's/foo/bar/gi' mylog.txt