Not sure whats going on, I replaced with gnu sed but I am getting backup files somehow. This is exactly what I am doing
mkdir tmp && cd $_
echo 'test' > test.txt
ls
test.txt
sed -ie 's/test/replaced/g' test.txt
ls
test.txt
test.txte
What's going on here and how do I prevent this? It should edit in place, not create a backup
sed version (GNU sed) 4.2.2
As per the fantastic comments from mklement0 the POSIX spec tells us that:
With optional option-arguments, POSIX utility conventions require that (emphasis his) "a conforming application shall place any option-argument for that option directly adjacent to the option in the same argument string, without intervening characters.
Since GNU sed considers the suffix argument to -i to be optional it requires it to be cuddled up against the option argument so when you write -ie GNU sed interprets that as requesting a suffix of e for the -i argument. (BSD sed would interpret it in the same manner for reasons that are explained in the additional info at the bottom.)
What this all means is that you need to use -i -e to get the behavior you want (for GNU sed) instead (for BSD sed you would need -i '' -e).
Additional details about an unfortunate but interesting distinction between GNU sed and BSD sed:
GNU sed and BSD sed (OSX sed) disagree on whether the suffix value for the -i argument is optional or mandatory.
This matters because complementing the POSIX requirement above we find the following in the spec as well (emphasis mkelement0's again):
an option with a mandatory option-argument [...], a conforming application shall use separate arguments for that option and its option-argument. However, a conforming implementation shall also permit applications to specify the option and option-argument in the same argument string without intervening characters.
GNU sed considers the suffix to be optional (and this causes the behavior above) because it accepts the cuddled e for the optional argument but ignores the separated -e (or anything else) as a separate argument.
BSD sed considers the suffix mandatory (even though it may be empty) this then implies that the option should be separated from the flag with a space (e.g. -i .bak or -i '') though as the "However" note indicates, BSD sed also allows any non-empty suffix to be cuddled up against the -i flag.
This disagreement, as mklement0 points out and which comes up on SO every now and then, means that you cannot use an empty in-place edit suffix in a manner that is portable to both GNU and BSD versions of sed.
As always, the man page helps:
-i extension:
Edit files in-place, saving backups with the specified extension. If a zero-length extension is given, no backup will be saved. It is not recommended to give a zero-length extension when in-place edit
ing files, as you risk corruption or partial content in situations where disk space is exhausted, etc.
Your use of -ie adds e to the end. Remove that.
Related
I have a file r. I want to replace the words File and MINvac.pdb in it with nothing. The commands I used are
sed -i 's/File//g' /home/kanika/standard_minimizer_prosee/r
and
sed -i 's/MINvac.pdb//g' /home/kanika/standard_minimizer_prosee/r
I want to combine both sed commands into one, but I don't know the way. Can anyone help?
The file looks like this:
-6174.27 File10MINvac.pdb
-514.451 File11MINvac.pdb
4065.68 File12MINvac.pdb
-4708.64 File13MINvac.pdb
6674.54 File14MINvac.pdb
8563.58 File15MINvac.pdb
sed is a scripting language. You separate commands with semicolon or newline. Many sed dialects also allow you to pass each command as a separate -e option argument.
sed -i 's/File//g;s/MINvac\.pdb//g' /home/kanika/standard_minimizer_prosee/r
I also added a backslash to properly quote the literal dot before pdb, but in this limited context that is probably unimportant.
For completeness, here is the newline variant. Many newcomers are baffled that the shell allows literal newlines in quoted strings, but it can be convenient.
sed -i 's/File//g
s/MINvac\.pdb//g' /home/kanika/standard_minimizer_prosee/r
Of course, in this limited case, you could also combine everything into one regex:
sed -i 's/\(File\|MINvac\.pdb\)//g' /home/kanika/standard_minimizer_prosee/r
(Some sed dialects will want this without backslashes, and/or offer an option to use extended regular expressions, where they should be omitted. BSD sed, and thus also MacOS sed, demands a mandatory argument to sed -i which can however be empty, like sed -i ''.)
Use the -e flag:
sed -i -e 's/File//g' -e 's/MINvac.pdb//g' /home/kanika/standard_minimizer_prosee/r
Once you get more commands than are convenient to define with -es, it is better to store the commands in a separate file and include it with the -f flag.
In this case, you'd make a file containing:
s/File//g
s/MINvac.pdb//g
Let's call that file 'sedcommands'. You'd then use it with sed like this:
sed -i -f sedcommands /home/kanika/standard_minimizer_prosee/r
With only two commands, it's probably not worthwhile using a separate file of commands, but it is quite convenient if you have a lot of transformations to make.
I want to use sed to replace each occurence of a particular text in a full source file tree. I've attempted the following:
$ grep -rlI name2port\(\"Wan1\"\) . --exclude-dir=.svn --exclude=*.vxs | xargs sed -i 's/name2port\(\"Wan1\"\)/T_PORT_ID_WAN1/g' but it doesn't seem to work, I think my sed cmd isn't correct. How do I do this?
The problem is, that the replacements just do not happen.
I tried this: $ sed -i 's/name2port\(\"Wan1\"\)/T_PORT_ID_WAN1/g' ./rtos_core/jpax_switch/api/src/nms/switch_l3_route.c but turns out, the occurences of name2port("Wan1") would not be replaced.
sed uses BREs (basic regular expressions) by default, which, for historical reasons - and surprisingly for someone used to modern regular expressions - require escaping of certain metacharacters in order to be recognized as such.
In BREs, ( and ) are ordinary (literal) characters, and only become special when \-escaped.
Therefore, to match literal name2port("Wan1"), use that literal as-is in a BRE (given that you also don't need to \-escape " instances):
sed -i 's/name2port("Wan1")/T_PORT_ID_WAN1/g' ./rtos_core/jpax_switch/api/src/nms/switch_l3_route.c
If you're not concerned about portability, you can use -r (or -E for limited portability to macOS, though with -i that won't work), which then enables EREs (extended regular expressions), whose syntax and features are more likely to work as you expect:
sed -r -i 's/name2port\("Wan1"\)/T_PORT_ID_WAN1/g' ./rtos_core/jpax_switch/api/src/nms/switch_l3_route.c
Note how literals ( and ) now do need to be \-escaped, lest they be interpreted as enclosing a capture group.
In this particular case, it is the BRE that requires less escaping than the ERE; generally, though, it is the opposite.
Using file structure
foo_11: "Марія"
foo_112: "Superman"
FOOTLONG: "Subway"
foo_13: "Юлія"
I want to remove all strings that don't have at least one character from Ukrainian alphabet.
Script:
for i in *.txt;
do
sed '/[^А-ЯЄЇІа-яєїі]+/d' $i >$i.out
mv $i.out $i
done
doesn't do anything. What is wrong?
Using mac bash.
Assuming that your character class defining Ukrainian letters is correct, the following should work:
sed '/[А-ЯЄЇІа-яєїі]/!d' file
[А-ЯЄЇІа-яєїі] matches a Ukrainian letter anywhere on the line.
Note that even the letters that look like ASCII letters A I a i are actually Ukrainian (Cyrillic) letters with Unicode codepoints U+410 U+406 U+430 U+456.
! negates the match, meaning that only lines not containing at least 1 Ukrainian letter match.
d deletes those lines.
To put it all together:
for f in *.txt; do
sed -i '' '/[А-ЯЄЇІа-яєїі]/!d' "$f" # -i '' is BSD Sed syntax; GNU sed takes just -i
done
As for what you've tried:
As #StefanHegny points out in a comment on the question, + isn't supported when sed is not run with -E in order to enable extended regular expressions; without -E, the cumbersome \{1,\} must be used. (\+ is only supported by GNU sed, not by the BSD version of sed that macOS comes with).
However, even the fixed version of your command, sed '/[^А-ЯЄЇІа-яєїі]\{1,\}/d', doesn't do what you want: it deletes all lines that contain at least one non-Ukrainian-letter character, which eliminates all of your input lines, given that they all have ASCII-based field names and contain :.
You should double-quote variable references such as $i to protect them from shell expansions: "$i"
BSD Sed does support in-place updating with -i, but - unlike GNU Sed - it requires that an empty option-argument (indicating that no backup of the input file should be made) be specified as a separate argument: -i ''.
Your write-to-a-temp-file-first-then-replace-the-original approach works too, but it's generally better to use the following idiom: sed ... file > file.tmp && mv file.tmp file. Separating the mv command with && ensures that the original file is only replaced if the sed command succeeded.
That said, that doesn't help with logic errors as in the case at hand: despite outputting nothing, sed reports success in this case.
This code would achieve what you want (if I understood your question correctly):
grep -i "Я\|Є\|Ї\|І" /folder/file >> /tmp/result
The result is stored on /tmp/result
Note: I don't know Ukranian, so I'm sure I did not included all Ukranian characters, please add/delete Ukranian characters you want to match to the construction above.
Note2: this code is case insensitive thanks to grep -i so you only need to add the character once (lowercase or capital).
To put it on your loop it could be:
for i in *.txt;
do
grep -i "Я\|Є\|Ї\|І" "$i" > "$i".out
mv "$i".out "$i"
done
Edit: I edited this answer to make it simpler, and to add a loop to it.
I have a file like this
parameters:
database_host: 127.0.0.1
database_port: null
database_name: myDb
database_user: root
database_password: null
mailer_transport: smtp
mailer_host: 127.0.0.1
mailer_user: null
mailer_password: null
secret: ea64f518be08e0d5895335990f10d984c22f400c
name_db: newDB
This is a yml file, and it has a particular format
I Want to create a file sh that find and replace the value after name_db with a parameter
Look for the key (name_db:) and replace the value after it. Assuming a sed that supports the -i option:
sed -i.bak "/^[[:space:]]*name_db:/ s/:.*/: $newname/" file.yml
The regular expression /^[[:space:]]*name_db:/ looks for a line that starts with zero or more spaces followed by name_db:. When that line is found, the s/:.*/: $newname/ substitute is executed. It will replace the colon and whatever follows with colon, space, and the value in $newname. This will fail if $newname contains a slash. If that's a possibility, choose a different character (other than /) as the marker. In case of doubt, Control-A is quite useful and unlikely to be part of a valid name_db value.
This doesn't care what the old value is. You can simply wrap this whole expression in double quotes, which interpolates the value in $newname. Nevertheless, using single quotes around sed scripts is generally a good idea.
monkeyUser commented:
sed: 1: "file.yml": extra characters at the end of p command
Note that the command line shown (using -i.bak) works on both Linux (GNU sed) and Mac OS X (BSD sed). GNU sed allows an optional backup suffix which must be attached to the -i option if it is present; BSD sed requires a suffix which must either be attached to the -i option as shown or can be the next argument. If you want no backup with GNU sed, specify just -i …; with BSD sed, specify '' as an empty argument after the -i option: -i '' …. Given the error message sed: 1: "file.yml": extra characters at the end of p command, I'm suspicious that the code was run on a system with BSD sed, though I'm not quite sure why p was mentioned (I suspect some editing of the actual error message). That is: sed -i /something/ file.yml would, with BSD sed, treat /something/ as the backup file suffix and then find problems with file.yml treated as a sed script…except I think the file name began with p rather than f, since f isn't a sed command (but p is).
I'm trying to use SED to extract text from a log file. I can do a search-and-replace without too much trouble:
sed 's/foo/bar/' mylog.txt
However, I want to make the search case-insensitive. From what I've googled, it looks like appending i to the end of the command should work:
sed 's/foo/bar/i' mylog.txt
However, this gives me an error message:
sed: 1: "s/foo/bar/i": bad flag in substitute command: 'i'
What's going wrong here, and how do I fix it?
Update: Starting with macOS Big Sur (11.0), sed now does support the I flag for case-insensitive matching, so the command in the question should now work (BSD sed doesn't reporting its version, but you can go by the date at the bottom of the man page, which should be March 27, 2017 or more recent); a simple example:
# BSD sed on macOS Big Sur and above (and GNU sed, the default on Linux)
$ sed 's/ö/#/I' <<<'FÖO'
F#O # `I` matched the uppercase Ö correctly against its lowercase counterpart
Note: I (uppercase) is the documented form of the flag, but i works as well.
Similarly, starting with macOS Big Sur (11.0) awk now is locale-aware (awk --version should report 20200816 or more recent):
# BSD awk on macOS Big Sur and above (and GNU awk, the default on Linux)
$ awk 'tolower($0)' <<<'FÖO'
föo # non-ASCII character Ö was properly lowercased
The following applies to macOS up to Catalina (10.15):
To be clear: On macOS, sed - which is the BSD implementation - does NOT support case-insensitive matching - hard to believe, but true. The formerly accepted answer, which itself shows a GNU sed command, gained that status because of the perl-based solution mentioned in the comments.
To make that Perl solution work with foreign characters as well, via UTF-8, use something like:
perl -C -Mutf8 -pe 's/öœ/oo/i' <<< "FÖŒ" # -> "Foo"
-C turns on UTF-8 support for streams and files, assuming the current locale is UTF-8-based.
-Mutf8 tells Perl to interpret the source code as UTF-8 (in this case, the string passed to -pe) - this is the shorter equivalent of the more verbose -e 'use utf8;'.Thanks, Mark Reed
(Note that using awk is not an option either, as awk on macOS (i.e., BWK awk and BSD awk) appears to be completely unaware of locales altogether - its tolower() and toupper() functions ignore foreign characters (and sub() / gsub() don't have case-insensitivity flags to begin with).)
A note on the relationship of sed and awk to the POSIX standard:
BSD sed and awk limit their functionality mostly to what the POSIX sed and
POSIX awk specs mandate, whereas their GNU counterparts implement many more extensions.
Editor's note: This solution doesn't work on macOS (out of the box), because it only applies to GNU sed, whereas macOS comes with BSD sed.
Capitalize the 'I'.
sed 's/foo/bar/I' file
Another work-around for sed on Mac OS X is to install gsedfrom MacPorts or HomeBrew and then create the alias sed='gsed'.
If you are doing pattern matching first, e.g.,
/pattern/s/xx/yy/g
then you want to put the I after the pattern:
/pattern/Is/xx/yy/g
Example:
echo Fred | sed '/fred/Is//willma/g'
returns willma; without the I, it returns the string untouched (Fred).
The sed FAQ addresses the closely related case-insensitive search. It points out that a) many versions of sed support a flag for it and b) it's awkward to do in sed, you should rather use awk or Perl.
But to do it in POSIX sed, they suggest three options (adapted for substitution here):
Convert to uppercase and store original line in hold space; this won't work for substitutions, though, as the original content will be restored before printing, so it's only good for insert or adding lines based on a case-insensitive match.
Maybe the possibilities are limited to FOO, Foo and foo. These can be covered by
s/FOO/bar/;s/[Ff]oo/bar/
To search for all possible matches, one can use bracket expressions for each character:
s/[Ff][Oo][Oo]/bar/
The Mac version of sed seems a bit limited. One way to work around this is to use a linux container (via Docker) which has a useable version of sed:
cat your_file.txt | docker run -i busybox /bin/sed -r 's/[0-9]{4}/****/Ig'
Use following to replace all occurrences:
sed 's/foo/bar/gI' mylog.txt
I had a similar need, and came up with this:
this command to simply find all the files:
grep -i -l -r foo ./*
this one to exclude this_shell.sh (in case you put the command in a script called this_shell.sh), tee the output to the console to see what happened, and then use sed on each file name found to replace the text foo with bar:
grep -i -l -r --exclude "this_shell.sh" foo ./* | tee /dev/fd/2 | while read -r x; do sed -b -i 's/foo/bar/gi' "$x"; done
I chose this method, as I didn't like having all the timestamps changed for files not modified. feeding the grep result allows only the files with target text to be looked at (thus likely may improve performance / speed as well)
be sure to backup your files & test before using. May not work in some environments for files with embedded spaces. (?)
Following should be fine:
sed -i 's/foo/bar/gi' mylog.txt