I'm feeling really stupid about asking this. I should know why this is failing. The following command works as expected in linux but not on MacOS Sierra:
echo "ENGINE=MyISAM AUTO_INCREMENT=750 DEFAULT" | sed 's/ AUTO_INCREMENT=[0-9]\+//'
On linux the command results in:
ENGINE=MyISAM DEFAULT
I've tried to various iterations of this command and cannot get it to work. I know that MacOS's sed is the BSD version instead of the GNU version but this substitute command should still work. What am I doing wrong?
EDIT:
The suggested duplicate answer does solve my problem but in a densely worded, roundabout way. The TL;DR version is thusly:
MacOS uses BSD-style sed. The solution is to simply use the -E option (extended regular expressions) which doesn't require escaping certain characters (like the + symbol). This method also works just fine with GNU-style sed found on most Linux systems. I hope this helps somebody down the road.
Use a BRE instead of an ERE. This will work using any sed in any shell on any UNIX box:
$ echo "ENGINE=MyISAM AUTO_INCREMENT=750 DEFAULT" | sed 's/ AUTO_INCREMENT=[0-9][0-9]*//'
ENGINE=MyISAM DEFAULT
Related
We receive a file which is essentially an ssh token key.
This upon inception has values, say
Foo\nBarFoo,\nFoo\nBarFoo
Now, I want to replace these with
Foo
BarFoo,
Foo
BarFoo
I have tried sed and tr commands by copying the entire key in a variable.
One thing that seemed to work was : %s/\\n/\r/g, but this is not acceptable since I cannot open the vi editor.
I recently tried echo -e 'Foo\nBarFoo,\nFoo\nBarFoo, but want to be it more subtle.
You are facing the problem because you are using OSX or BSD probably. With GNU sed, 's/\\n/\n/g’ should have worked.
For POSIX sed, use this,
echo "Foo\nBarFoo,\nFoo\nBarFoo" | sed 's/\\n/\
/g'
I am trying to trouble shoot a problem I am seeing when running bash commands in Cygwin.
I am trying to assign the CLang version from a text file to a variable. If I run this in Cygwin:
$ (sed -n 1p "$CLANGC2_VERSION_FILE" | sed 's/\s//g')
I get this output (which is exactly what I want):
14.10.25903
Now, if I try and assign this to a variable it doesn't work. Here is what I am trying:
$ CLANGC2_VERSION=$(sed -n 1p "$CLANGC2_VERSION_FILE" | sed 's/\s//g')
but when I inspect or print the variable, it is empty.
What am I doing wrong?
Turns out that there is a known 'Big List of Dodgy Apps' (BLODA) which can interfere with Cygwin and bash.
The discussion I found is here: https://cygwin.com/ml/cygwin/2017-07/msg00197.html
The BLODA list is here: https://cygwin.com/faq/faq.html#faq.using.bloda
Turns out my AntiVirus is on the list.
I've removed the AV and now the commands work. There must be some low-level stuff going with the AV that causes it to fail.
You can use backticks to get the desired results.
CLANGC2_VERSION=`(sed -n 1p "$CLANGC2_VERSION_FILE" | sed 's/\s//g')`
I have a file and I need to use sed to process it.
Here is my command: sed -i -e '/.*tour\.html\|.*Thumb[^\/]*\.jpg/!d'.
Now if I execute this command from the terminal, for example, sed -i -e '/.*tour\.html\|.*Thumb[^\/]*\.jpg/!d' myfile.txt, it works well. But if I write a bash script with the same command, it will delete all lines.
#!/bin/bash
sed -i -e '/.*tour\.html\|.*Thumb[^\/]*\.jpg/!d' "$1"
This script will delete all lines in file.
My PC is Mac OS.
As far as I understand getting both sed instances output of sed --help and sed --version showed that you have actually two different sed versions reacting to the two different ways of execuing your code.
Sed is a little inconsistent concerning the syntax, especially when it comes to commandline options.
For example, I know of an important difference for the -i switch, which in some Mac versions requires a file extension for backups being given explicitly. Others allow one optionally. This difference could explain why somethign involving a -i without backup extension works in one case and fails in another.
Anishsane suggested that different "PATH" variables could in turn be part of the mechanism to have two different sed versions executed.
I invite OP to edit the output of --help and --version (where possible, there should be a way to get the version out of both sed instances) here into this answer. I do not have those details actually. Which makes this answer seem a little "guessing".
I'm trying to use SED to extract text from a log file. I can do a search-and-replace without too much trouble:
sed 's/foo/bar/' mylog.txt
However, I want to make the search case-insensitive. From what I've googled, it looks like appending i to the end of the command should work:
sed 's/foo/bar/i' mylog.txt
However, this gives me an error message:
sed: 1: "s/foo/bar/i": bad flag in substitute command: 'i'
What's going wrong here, and how do I fix it?
Update: Starting with macOS Big Sur (11.0), sed now does support the I flag for case-insensitive matching, so the command in the question should now work (BSD sed doesn't reporting its version, but you can go by the date at the bottom of the man page, which should be March 27, 2017 or more recent); a simple example:
# BSD sed on macOS Big Sur and above (and GNU sed, the default on Linux)
$ sed 's/ö/#/I' <<<'FÖO'
F#O # `I` matched the uppercase Ö correctly against its lowercase counterpart
Note: I (uppercase) is the documented form of the flag, but i works as well.
Similarly, starting with macOS Big Sur (11.0) awk now is locale-aware (awk --version should report 20200816 or more recent):
# BSD awk on macOS Big Sur and above (and GNU awk, the default on Linux)
$ awk 'tolower($0)' <<<'FÖO'
föo # non-ASCII character Ö was properly lowercased
The following applies to macOS up to Catalina (10.15):
To be clear: On macOS, sed - which is the BSD implementation - does NOT support case-insensitive matching - hard to believe, but true. The formerly accepted answer, which itself shows a GNU sed command, gained that status because of the perl-based solution mentioned in the comments.
To make that Perl solution work with foreign characters as well, via UTF-8, use something like:
perl -C -Mutf8 -pe 's/öœ/oo/i' <<< "FÖŒ" # -> "Foo"
-C turns on UTF-8 support for streams and files, assuming the current locale is UTF-8-based.
-Mutf8 tells Perl to interpret the source code as UTF-8 (in this case, the string passed to -pe) - this is the shorter equivalent of the more verbose -e 'use utf8;'.Thanks, Mark Reed
(Note that using awk is not an option either, as awk on macOS (i.e., BWK awk and BSD awk) appears to be completely unaware of locales altogether - its tolower() and toupper() functions ignore foreign characters (and sub() / gsub() don't have case-insensitivity flags to begin with).)
A note on the relationship of sed and awk to the POSIX standard:
BSD sed and awk limit their functionality mostly to what the POSIX sed and
POSIX awk specs mandate, whereas their GNU counterparts implement many more extensions.
Editor's note: This solution doesn't work on macOS (out of the box), because it only applies to GNU sed, whereas macOS comes with BSD sed.
Capitalize the 'I'.
sed 's/foo/bar/I' file
Another work-around for sed on Mac OS X is to install gsedfrom MacPorts or HomeBrew and then create the alias sed='gsed'.
If you are doing pattern matching first, e.g.,
/pattern/s/xx/yy/g
then you want to put the I after the pattern:
/pattern/Is/xx/yy/g
Example:
echo Fred | sed '/fred/Is//willma/g'
returns willma; without the I, it returns the string untouched (Fred).
The sed FAQ addresses the closely related case-insensitive search. It points out that a) many versions of sed support a flag for it and b) it's awkward to do in sed, you should rather use awk or Perl.
But to do it in POSIX sed, they suggest three options (adapted for substitution here):
Convert to uppercase and store original line in hold space; this won't work for substitutions, though, as the original content will be restored before printing, so it's only good for insert or adding lines based on a case-insensitive match.
Maybe the possibilities are limited to FOO, Foo and foo. These can be covered by
s/FOO/bar/;s/[Ff]oo/bar/
To search for all possible matches, one can use bracket expressions for each character:
s/[Ff][Oo][Oo]/bar/
The Mac version of sed seems a bit limited. One way to work around this is to use a linux container (via Docker) which has a useable version of sed:
cat your_file.txt | docker run -i busybox /bin/sed -r 's/[0-9]{4}/****/Ig'
Use following to replace all occurrences:
sed 's/foo/bar/gI' mylog.txt
I had a similar need, and came up with this:
this command to simply find all the files:
grep -i -l -r foo ./*
this one to exclude this_shell.sh (in case you put the command in a script called this_shell.sh), tee the output to the console to see what happened, and then use sed on each file name found to replace the text foo with bar:
grep -i -l -r --exclude "this_shell.sh" foo ./* | tee /dev/fd/2 | while read -r x; do sed -b -i 's/foo/bar/gi' "$x"; done
I chose this method, as I didn't like having all the timestamps changed for files not modified. feeding the grep result allows only the files with target text to be looked at (thus likely may improve performance / speed as well)
be sure to backup your files & test before using. May not work in some environments for files with embedded spaces. (?)
Following should be fine:
sed -i 's/foo/bar/gi' mylog.txt
C:\crp\cnp>sed -V
GNU sed version 3.02
Copyright (C) 1998 Free Software Foundation, Inc.......
C:\crp\cnp>type f.f
a a a
a a a
Trying to replace 'a' with spaces.
C:\crp\cnp>type f.f | sed -e s/a/\d032/g
d032 d032 d032
d032 d032 d032
why isn't it working?
I don't mind whether i'm finding or replacing spaces or new lines.. I just want to be able to specify them. It doesn't seem to be working and I don't know why.
(Replacing spaces or a space, with f, doesn't work)
C:\crp\cnp>echo a a | sed s/\d32/f/
a a
Note- it seems it might work in 4.2 , But i'm interested in 3.02 'cos that's the version bundled with unxutils http://unxutils.sourceforge.net/
Update to question-
thanks for paxdiablo's tip.. about gnu32win, I am now using that instead of unxutils. It is more up to date. I can now specify spaces. And tip of ghostdog, and paxdiablo, I see about the double quotes. I am fine specifying spaces with \d(since using 4.2) or with a space.
But, I still can't remove new lines
C:\crp>type f.f | sed -e "s/\r\n/f/g"
a aa
b bb
c cc
C:\crp>type f.f | sed -e "s/\d013\d010/f/g"
a aa
b bb
c cc
C:\crp>type f.f | sed -e "s/\x0D\x0A/f/g"
a aa
b bb
c cc
Note: This question was from 2010. Now it's 2020. Gnuwin32 is out of date(like the last time its Gnuwin32 sed was updated was 2010, with Sed 4.2.1 which was from 2009), Unxutils is even more out of date. So Gnuwin32 as of writing is a decade out of date, and Unxutils is more like 2 decades out of date, as of 2020. Cygwin is still kept up to date and as of writing is on Sed v 4.4 which is from 2017.
Why aren't you just using a space character itself rather than some funny encoding? As in:
sed -e 's/a/ /g'
For what it's worth, the command you gave also fails to work in 4.2.1 but, if you add in the quotes, it does work. So I suggest you change it to:
sed -e 's/a/\d032/g'
Apologies, I've just noticed you're running Windows so you've probably got CygWin or GnuWin32 (or equivalent).
Quotes work differently under Windows so you should try two things. The first is to use " instead of ' quotes:
sed -e "s/a/ /g"
Otherwise, the escape character in Windows is ^ so something like this should be able to escape the space:
sed -e s/a/^ /g
As an aside, I'd be looking to switch to GnuWin32, if possible, which has more recent versions of sed (for example). It doesn't look like UnxUtils has had an update since 2003 based on that web page you link to. You can get individual packages from here. You're looking for coreutils which contains the bulk of the UNIX text processing toolkit.
But, if you're stuck with UnxUtils, I'd just use the actual space rather than a decimal code, and then I'd use tr to get rid of new lines:
tr -d "\n"
assuming of course that the tr in textutils can handle that syntax :-)
I stuck with the same problem on Win XP and double quotes didn't work when trying to print new line "\n".
The solution was to use the new UnxUpdates.zip from http://unxutils.sourceforge.net/
It works correct with "\n".
Sed version states:
GNU sed version 4.0.7
On windows, use double quotes
sed "s/a/\d032/g" file
or just
sed "s/a/ /g" file