I have a file of strings that are comma separated. I'm trying to replace the commas with a new line. I've tried:
sed 's/,/\n/g' file
but it is not working. What am I missing?
Use tr instead:
tr , '\n' < file
Use an ANSI-C quoted string $'string'
You need a backslash-escaped literal newline to get to sed.
In bash at least, $'' strings will replace \n with a real newline, but then you have to double the backslash that sed will see to escape the newline, e.g.
echo "a,b" | sed -e $'s/,/\\\n/g'
Note this will not work on all shells, but will work on the most common ones.
sed 's/,/\
/g'
works on Mac OS X.
If your sed usage tends to be entirely substitution expressions (as mine tends to be), you can also use perl -pe instead
$ echo 'foo,bar,baz' | perl -pe 's/,/,\n/g'
foo,
bar,
baz
MacOS is different, there is two way to solve this problem with sed in mac
first ,use \'$'\n'' replace \n, it can work in MacOS:
sed 's/,/\'$'\n''/g' file
the second, just use an empty line:
sed 's/,/\
/g' file
Ps. Pay attention the range separated by '
the third, use gnu-sed replace the mac-sed
Apparently \r is the key!
$ sed 's/, /\r/g' file3.txt > file4.txt
Transformed this:
ABFS, AIRM, AMED, BOSC, CALI, ECPG, FRGI, GERN, GTIV, HSON, IQNT, JRCC, LTRE,
MACK, MIDD, NKTR, NPSP, PME, PTIX, REFR, RSOL, UBNT, UPI, YONG, ZEUS
To this:
ABFS
AIRM
AMED
BOSC
CALI
ECPG
FRGI
GERN
GTIV
HSON
IQNT
JRCC
LTRE
MACK
MIDD
NKTR
NPSP
PME
PTIX
REFR
RSOL
UBNT
UPI
YONG
ZEUS
This works on MacOS Mountain Lion (10.8), Solaris 10 (SunOS 5.10) and RHE Linux (Red Hat Enterprise Linux Server release 5.3, Tikanga)...
$ sed 's/{pattern}/\^J/g' foo.txt > foo2.txt
... where the ^J is done by doing ctrl+v+j. Do mind the \ before the ^J.
PS, I know the sed in RHEL is GNU, the MacOS sed is FreeBSD based, and although I'm not sure about the Solaris sed, I believe this will work pretty much with any sed. YMMV tho'...
To make it complete, this also works:
echo "a,b" | sed "s/,/\\$(echo -e '\n\r')/"
Though I am late to this post, just updating my findings. This answer is only for Mac OS X.
$ sed 's/new/
> /g' m1.json > m2.json
sed: 1: "s/new/
/g": unescaped newline inside substitute pattern
In the above command I tried with Shift+Enter to add new line which didn't work. So this time I tried with "escaping" the "unescaped newline" as told by the error.
$ sed 's/new/\
> /g' m1.json > m2.json
Worked! (in Mac OS X 10.9.3)
$ echo $PATH | sed -e $'s/:/\\\n/g'
/usr/local/sbin
/Library/Oracle/instantclient_11_2/sdk
/usr/local/bin
...
Works for me on Mojave
Just to clearify: man-page of sed on OSX (10.8; Darwin Kernel Version 12.4.0) says:
[...]
Sed Regular Expressions
The regular expressions used in sed, by default, are basic regular expressions (BREs, see re_format(7) for more information), but extended
(modern) regular expressions can be used instead if the -E flag is given. In addition, sed has the following two additions to regular
expressions:
1. In a context address, any character other than a backslash (``\'') or newline character may be used to delimit the regular expression.
Also, putting a backslash character before the delimiting character causes the character to be treated literally. For example, in the
context address \xabc\xdefx, the RE delimiter is an ``x'' and the second ``x'' stands for itself, so that the regular expression is
``abcxdef''.
2. The escape sequence \n matches a newline character embedded in the pattern space. You cannot, however, use a literal newline charac-
ter in an address or in the substitute command.
[...]
so I guess one have to use tr - as mentioned above - or the nifty
sed "s/,/^M
/g"
note: you have to type <ctrl>-v,<return> to get '^M' in vi editor
The sed on macOS Mojave was released in 2005, so one solution is to install the gnu-sed,
brew install gnu-sed
then use gsed will do as you wish,
gsed 's/,/\n/g' file
If you prefer sed, just sudo sh -c 'echo /usr/local/opt/gnu-sed/libexec/gnubin > /etc/paths.d/brew', which is suggested by brew info gnu-sed. Restart your term, then your sed in command line is gsed.
FWIW, the following line works in windows and replaces semicolons in my path variables with a newline. I'm using the tools installed under my git bin directory.
echo %path% | sed -e $'s/;/\\n/g' | less
I have found another command that is working also.
find your_filename.txt -type f -exec sed -i 's/,/\n/g' {} \;
Related
I have this line inside a file:
ULNET-PA,client_sgcib,broker_keplersecurities
,KEPLER
I try to get rid of that ^M (carriage return) character so I used:
sed 's/^M//g'
However this does remove everything after ^M:
[root#localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities^M,KEPLER
[root#localhost tmp]# sed 's/^M//g' test
ULNET-PA,client_sgcib,broker_keplersecurities
What I want to obtain is:
[root#localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities,KEPLER
Use tr:
tr -d '^M' < inputfile
(Note that the ^M character can be input using Ctrl+VCtrl+M)
EDIT: As suggested by Glenn Jackman, if you're using bash, you could also say:
tr -d $'\r' < inputfile
still the same line:
sed -i 's/^M//g' file
when you type the command, for ^M you type Ctrl+VCtrl+M
actually if you have already opened the file in vim, you can just in vim do:
:%s/^M//g
same, ^M you type Ctrl-V Ctrl-M
You can simply use dos2unix which is available in most Unix/Linux systems. However I found the following sed command to be better as it removed ^M where dos2unix couldn't:
sed 's/\r//g' < input.txt > output.txt
Hope that helps.
Note: ^M is actually carriage return character which is represented in code as \r
What dos2unix does is most likely equivalent to:
sed 's/\r\n/\n/g' < input.txt > output.txt
It doesn't remove \r when it is not immediately followed by \n and replaces both with just \n. This fails with certain types of files like one I just tested with.
alias dos2unix="sed -i -e 's/'\"\$(printf '\015')\"'//g' "
Usage:
dos2unix file
If Perl is an option:
perl -i -pe 's/\r\n$/\n/g' file
-i makes a .bak version of the input file
\r = carriage return
\n = linefeed
$ = end of line
s/foo/bar/g = globally substitute "foo" with "bar"
In awk:
sub(/\r/,"")
If it is in the end of record, sub(/\r/,"",$NF) should suffice. No need to scan the whole record.
This is the better way to achieve
tr -d '\015' < inputfile_name > outputfile_name
Later rename the file to original file name.
I agree with #twalberg (see accepted answer comments, above), dos2unix on Mac OSX covers this, quoting man dos2unix:
To run in Mac mode use the command-line option "-c mac" or use the
commands "mac2unix" or "unix2mac"
I settled on 'mac2unix', which got rid of my less-cmd-visible '^M' entries, introduced by an Apple 'Messages' transfer of a bash script between 2 Yosemite (OSX 10.10) Macs!
I installed 'dos2unix', trivially, on Mac OSX using the popular Homebrew package installer, I highly recommend it and it's companion command, Cask.
This is clean and simple and it works:
sed -i 's/\r//g' file
where \r of course is the equivalent for ^M.
Simply run the following command:
sed -i -e 's/\r$//' input.file
I verified this as valid in Mac OSX Monterey.
remove any \r :
nawk 'NF+=OFS=_' FS='\r'
gawk 3 ORS= RS='\r'
remove end of line \r :
mawk2 8 RS='\r?\n'
mawk -F'\r$' NF=1
I'm trying to delete all the £ symbols in a text file (.csv) using sed, however it doesn't work and I've no clue as to why. Any suggestions? Basically using:
sed -i 's/£//g' file.csv
Output of locale:
LANG=en_GB.UTF-8
LANGUAGE=
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=
Update:
So, I don't know why sed wasn't working, but
tr -d '\243' < inputfile
Does work! Thanks to #devnull for that!
This might work for you (GNU sed):
sed 's/o\302\o243//g' file
N.B. if you are in doubt how sed sees a particular character use the l command and sed will display the character as it sees it (in this case in octal). e.g.
echo '£' | sed 'l'
\302\243$
The dollar following the octal codes represents the end of string.
For long strings use l0 this will prevent wrapping.
Did you try:
sed "s/\xA3//" inputfile
Alternatively, you can use tr:
tr -d '\243' < inputfile
Use the hex code for the pound sign.
Seems like different versions in different platforms, sed act very differently. I just got the GNU version and works fine.
Before using sed, one should read the section "When should I NOT use sed"
I am trying to run this command on MacOSX terminal , which was initially intended to run on Linux
sed '1 i VISPATH=/mnt/local/gdrive/public/3DVis' init.txt >> ~/.bash_profile
but it gives me the error:
command i expects \ followed by text.
is there any way I could modify the above command to work on MacOSX terminal
Shelter is right but there's another way to do it. You can use the bash $'...' quoting to interpret the escapes before passing the string to sed.
So:
sed -iold '1i\'$'\n''text to prepend'$'\n' file.txt
^^^^^^^^ ^
/ |\|||/ \ |__ No need to reopen
| | \|/ | string to sed
Tells sed to | | | |
escape the next _/ | | +-----------------------------+
char | +-------------+ |
| | |
Close string The special bash Reopen string to
to sed newline char to send to sed
send to sed
This answer on unix.stackexchange.com led me to this solution.
Had the same problem and solved it with brew:
brew install gnu-sed
gsed YOUR_USUAL_SED_COMMAND
If you want to use the sed command, then you can set an alias:
alias sed=gsed
The OSX seds are based on older versions, you need to be much more literal in your directions to sed, AND you're lucky, in this case, sed is telling you exactly what to do. Untested as I don't have OSX, but try
sed '1 i\
VISPATH=/mnt/local/gdrive/public/3DVis
' init.txt >> ~/.bash_profile
Input via the i cmd is terminated by a blank line. Other sed instructions can follow after that. Note, NO chars after the \ char!
Also, #StephenNiedzielski is right. Use the single quote chars to wrap your sed statements. (if you need variable expansion inside your sed and can escape other uses of $, then you can also use dbl-quotes, but it's not recommended as a normal practices.
edit
As I understand now that you're doing this from the command-line, and not in a script or other editor, I have tested the above, and.... all I can say is that famous line from tech support ... "It works for me". If you're getting an error message
sed: -e expression #1, char 8: extra characters after command
then you almost certainly have added some character after the \. I just tested that, and I got the above error message. (I'm using a linux version of sed, so the error messages are exactly the same). You should edit your question to include an exact cut-paste of your command line and the new error message. Using curly-single-quotes will not work.
IHTH
Here's how I worked it out on OS X. In my case, I needed to prepend text to a file. Apparently, modern sed works like this:
sed -i '1i text to prepend' file.txt
But on OS X I had to do the following:
sed -i '' '1i\
text to prepend
' file.txt
It looks like you copied rich text. The single quotes should be straight not curly:
sed '1 i VISPATH=/mnt/local/gdrive/public/3DVis'
This is one for you sed gurus out there. I really don't know enough about sed to take this apart completely. It was written on some standard Linux distro and I need it to run on OSX.
COMPILE_FILES=$(sed -nr '/<script type="text\/javascript"/ { s%^.*src="\{\$baseUrl\}/([^"]+)".*$%\1%p }' templates/common/minifiedScripts.tpl)
The first thing is that the r flag doesn't exist on the OSX version of sed. I thought the equivalent is -E, so changed it. But then I get:
sed: 1: "/<script type="text\/ja ...": bad flag in substitute command: '}'
Thanks!
OS X sed doesn't like multiple commands run together using semicolons or grouped in curly braces (which aren't necessary in the command you have). Try this:
COMPILE_FILES=$(sed -n -E '/<script type="text\/javascript"/ s%^.*src="\{\$baseUrl\}/([^"]+)".*$%\1%p' templates/common/minifiedScripts.tpl)
If you have a sed script that consists of multiple commands, you'll have to break them up using -e:
sed -n -E -e '/match/ {' -e 's/foo/bar/' -e 's/baz/qux/' -e 'p' -e '}'
-r uses the extended regular expression set, and you are using one not supported by the OSX version. Try MacPorts or Fink -- they have ports of gnu sed. OR you can just download and compile source directly.
How to insert a newline before a pattern within a line?
For example, this will insert a newline behind the regex pattern.
sed 's/regex/&\n/g'
How can I do the same but in front of the pattern?
Given this sample input file, the pattern to match on is the phone number.
some text (012)345-6789
Should become
some text
(012)345-6789
This works in bash and zsh, tested on Linux and OS X:
sed 's/regexp/\'$'\n/g'
In general, for $ followed by a string literal in single quotes bash performs C-style backslash substitution, e.g. $'\t' is translated to a literal tab. Plus, sed wants your newline literal to be escaped with a backslash, hence the \ before $. And finally, the dollar sign itself shouldn't be quoted so that it's interpreted by the shell, therefore we close the quote before the $ and then open it again.
Edit: As suggested in the comments by #mklement0, this works as well:
sed $'s/regexp/\\\n/g'
What happens here is: the entire sed command is now a C-style string, which means the backslash that sed requires to be placed before the new line literal should now be escaped with another backslash. Though more readable, in this case you won't be able to do shell string substitutions (without making it ugly again.)
Some of the other answers didn't work for my version of sed.
Switching the position of & and \n did work.
sed 's/regexp/\n&/g'
Edit: This doesn't seem to work on OS X, unless you install gnu-sed.
In sed, you can't add newlines in the output stream easily. You need to use a continuation line, which is awkward, but it works:
$ sed 's/regexp/\
&/'
Example:
$ echo foo | sed 's/.*/\
&/'
foo
See here for details. If you want something slightly less awkward you could try using perl -pe with match groups instead of sed:
$ echo foo | perl -pe 's/(.*)/\n$1/'
foo
$1 refers to the first matched group in the regular expression, where groups are in parentheses.
On my mac, the following inserts a single 'n' instead of newline:
sed 's/regexp/\n&/g'
This replaces with newline:
sed "s/regexp/\\`echo -e '\n\r'`/g"
echo one,two,three | sed 's/,/\
/g'
You can use perl one-liners much like you do with sed, with the advantage of full perl regular expression support (which is much more powerful than what you get with sed). There is also very little variation across *nix platforms - perl is generally perl. So you can stop worrying about how to make your particular system's version of sed do what you want.
In this case, you can do
perl -pe 's/(regex)/\n$1/'
-pe puts perl into a "execute and print" loop, much like sed's normal mode of operation.
' quotes everything else so the shell won't interfere
() surrounding the regex is a grouping operator. $1 on the right side of the substitution prints out whatever was matched inside these parens.
Finally, \n is a newline.
Regardless of whether you are using parentheses as a grouping operator, you have to escape any parentheses you are trying to match. So a regex to match the pattern you list above would be something like
\(\d\d\d\)\d\d\d-\d\d\d\d
\( or \) matches a literal paren, and \d matches a digit.
Better:
\(\d{3}\)\d{3}-\d{4}
I imagine you can figure out what the numbers in braces are doing.
Additionally, you can use delimiters other than / for your regex. So if you need to match / you won't need to escape it. Either of the below is equivalent to the regex at the beginning of my answer. In theory you can substitute any character for the standard /'s.
perl -pe 's#(regex)#\n$1#'
perl -pe 's{(regex)}{\n$1}'
A couple final thoughts.
using -ne instead of -pe acts similarly, but doesn't automatically print at the end. It can be handy if you want to print on your own. E.g., here's a grep-alike (m/foobar/ is a regex match):
perl -ne 'if (m/foobar/) {print}'
If you are finding dealing with newlines troublesome, and you want it to be magically handled for you, add -l. Not useful for the OP, who was working with newlines, though.
Bonus tip - if you have the pcre package installed, it comes with pcregrep, which uses full perl-compatible regexes.
In this case, I do not use sed. I use tr.
cat Somefile |tr ',' '\012'
This takes the comma and replaces it with the carriage return.
To insert a newline to output stream on Linux, I used:
sed -i "s/def/abc\\\ndef/" file1
Where file1 was:
def
Before the sed in-place replacement, and:
abc
def
After the sed in-place replacement. Please note the use of \\\n. If the patterns have a " inside it, escape using \".
Hmm, just escaped newlines seem to work in more recent versions of sed (I have GNU sed 4.2.1),
dev:~/pg/services/places> echo 'foobar' | sed -r 's/(bar)/\n\1/;'
foo
bar
echo pattern | sed -E -e $'s/^(pattern)/\\\n\\1/'
worked fine on El Captitan with () support
In my case the below method works.
sed -i 's/playstation/PS4/' input.txt
Can be written as:
sed -i 's/playstation/PS4\nplaystation/' input.txt
PS4
playstation
Consider using \\n while using it in a string literal.
sed : is stream editor
-i : Allows to edit the source file
+: Is delimiter.
I hope the above information works for you 😃.
in sed you can reference groups in your pattern with "\1", "\2", ....
so if the pattern you're looking for is "PATTERN", and you want to insert "BEFORE" in front of it, you can use, sans escaping
sed 's/(PATTERN)/BEFORE\1/g'
i.e.
sed 's/\(PATTERN\)/BEFORE\1/g'
You can also do this with awk, using -v to provide the pattern:
awk -v patt="pattern" '$0 ~ patt {gsub(patt, "\n"patt)}1' file
This checks if a line contains a given pattern. If so, it appends a new line to the beginning of it.
See a basic example:
$ cat file
hello
this is some pattern and we are going ahead
bye!
$ awk -v patt="pattern" '$0 ~ patt {gsub(patt, "\n"patt)}1' file
hello
this is some
pattern and we are going ahead
bye!
Note it will affect to all patterns in a line:
$ cat file
this pattern is some pattern and we are going ahead
$ awk -v patt="pattern" '$0 ~ patt {gsub(patt, "\n"patt)}1' d
this
pattern is some
pattern and we are going ahead
sed -e 's/regexp/\0\n/g'
\0 is the null, so your expression is replaced with null (nothing) and then...
\n is the new line
On some flavors of Unix doesn't work, but I think it's the solution to your problem.
echo "Hello" | sed -e 's/Hello/\0\ntmow/g'
Hello
tmow
This works in MAC for me
sed -i.bak -e 's/regex/xregex/g' input.txt sed -i.bak -e 's/qregex/\'$'\nregex/g' input.txt
Dono whether its perfect one...
After reading all the answers to this question, it still took me many attempts to get the correct syntax to the following example script:
#!/bin/bash
# script: add_domain
# using fixed values instead of command line parameters $1, $2
# to show typical variable values in this example
ipaddr="127.0.0.1"
domain="example.com"
# no need to escape $ipaddr and $domain values if we use separate quotes.
sudo sed -i '$a \\n'"$ipaddr www.$domain $domain" /etc/hosts
The script appends a newline \n followed by another line of text to the end of a file using a single sed command.
In vi on Red Hat, I was able to insert carriage returns using just the \r character. I believe this internally executes 'ex' instead of 'sed', but it's similar, and vi can be another way to do bulk edits such as code patches. For example. I am surrounding a search term with an if statement that insists on carriage returns after the braces:
:.,$s/\(my_function(.*)\)/if(!skip_option){\r\t\1\r\t}/
Note that I also had it insert some tabs to make things align better.
Just to add to the list of many ways to do this, here is a simple python alternative. You could of course use re.sub() if a regex were needed.
python -c 'print(open("./myfile.txt", "r").read().replace("String to match", "String to match\n"))' > myfile_lines.txt
sed 's/regexp/\'$'\n/g'
works as justified and detailed by mojuba in his answer .
However, this did not work:
sed 's/regexp/\\\n/g'
It added a new line, but at the end of the original line, a \n was added.