I have this line inside a file:
ULNET-PA,client_sgcib,broker_keplersecurities
,KEPLER
I try to get rid of that ^M (carriage return) character so I used:
sed 's/^M//g'
However this does remove everything after ^M:
[root#localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities^M,KEPLER
[root#localhost tmp]# sed 's/^M//g' test
ULNET-PA,client_sgcib,broker_keplersecurities
What I want to obtain is:
[root#localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities,KEPLER
Use tr:
tr -d '^M' < inputfile
(Note that the ^M character can be input using Ctrl+VCtrl+M)
EDIT: As suggested by Glenn Jackman, if you're using bash, you could also say:
tr -d $'\r' < inputfile
still the same line:
sed -i 's/^M//g' file
when you type the command, for ^M you type Ctrl+VCtrl+M
actually if you have already opened the file in vim, you can just in vim do:
:%s/^M//g
same, ^M you type Ctrl-V Ctrl-M
You can simply use dos2unix which is available in most Unix/Linux systems. However I found the following sed command to be better as it removed ^M where dos2unix couldn't:
sed 's/\r//g' < input.txt > output.txt
Hope that helps.
Note: ^M is actually carriage return character which is represented in code as \r
What dos2unix does is most likely equivalent to:
sed 's/\r\n/\n/g' < input.txt > output.txt
It doesn't remove \r when it is not immediately followed by \n and replaces both with just \n. This fails with certain types of files like one I just tested with.
alias dos2unix="sed -i -e 's/'\"\$(printf '\015')\"'//g' "
Usage:
dos2unix file
If Perl is an option:
perl -i -pe 's/\r\n$/\n/g' file
-i makes a .bak version of the input file
\r = carriage return
\n = linefeed
$ = end of line
s/foo/bar/g = globally substitute "foo" with "bar"
In awk:
sub(/\r/,"")
If it is in the end of record, sub(/\r/,"",$NF) should suffice. No need to scan the whole record.
This is the better way to achieve
tr -d '\015' < inputfile_name > outputfile_name
Later rename the file to original file name.
I agree with #twalberg (see accepted answer comments, above), dos2unix on Mac OSX covers this, quoting man dos2unix:
To run in Mac mode use the command-line option "-c mac" or use the
commands "mac2unix" or "unix2mac"
I settled on 'mac2unix', which got rid of my less-cmd-visible '^M' entries, introduced by an Apple 'Messages' transfer of a bash script between 2 Yosemite (OSX 10.10) Macs!
I installed 'dos2unix', trivially, on Mac OSX using the popular Homebrew package installer, I highly recommend it and it's companion command, Cask.
This is clean and simple and it works:
sed -i 's/\r//g' file
where \r of course is the equivalent for ^M.
Simply run the following command:
sed -i -e 's/\r$//' input.file
I verified this as valid in Mac OSX Monterey.
remove any \r :
nawk 'NF+=OFS=_' FS='\r'
gawk 3 ORS= RS='\r'
remove end of line \r :
mawk2 8 RS='\r?\n'
mawk -F'\r$' NF=1
Related
On OSX I need to remove line-ending CR (\r) characters (represented as ^M in the output from cat -v) from my CSV file:
$ cat -v myitems.csv
output:
strPicture,strEmail^M
image1xl.jpg,me#example.com^M
I have tried lots of options with sed and perl but nothing works.
Any ideas?
Solutions with stock utilities:
Note: Except where noted (the sed -i incompatibility), the following solutions work on both OSX (macOS) and Linux.
Use sed as follows, which replaces \r\n with \n:
sed $'s/\r$//' myitems.csv
To update the input file in place, use
sed -i '' $'s/\r$//' myitems.csv
-i '' specifies updating in place, with '' indicating that no backup should be made of the input file; if you specify a extension, e.g., -i'.bak', the original input file will be saved with that extension as a backup.
Caveats:
* With GNU sed (Linux), to not create a backup file, you'd have to use just -i, without the separate '' argument, which is an unfortunate syntactic incompatibility between GNU Sed and the BSD Sed used on OSX (macOS) - see this answer of mine for the full story.
* -i creates a new file with a temporary name and then replaces the original file; the most notably consequence is that if the original file was a symlink, it is replaced with a regular file; for a detailed discussion, see the lower half of this answer.
Note: The above uses an ANSI C-quoted string ($'...') to create the \r character in the sed command, because BSD sed (the one used on OS X), doesn't natively recognize such escape sequences (note that the GNU sed used on Linux distros would).
ANSI C-quoted strings are supported in Bash, Ksh, and Zsh.
If you don't want to rely on such strings, use:
sed 's/'"$(printf '\r')"'$//'
Here, the \r is created via printf and spliced into the sed command with a command substitution ($(...)).
Using perl:
perl -pe 's/\r\n/\n/' myitems.csv | cat -v
To update the input file in place, use
perl -i -ple 's/\r\n/\n/' myitems.csv # -i'.bak' creates backup with suffix '.bak' first
The same caveat as above for sed with regard to in-place updating applies.
Using awk:
awk '{ sub("\r$", ""); print }' myitems.csv # shorter: awk 'sub("\r$", "")+1'
BSD awk offers no in-place updating option, so you'll have to capture the output in a different file; to use a temporary file and have it replace the original afterward, use the following idiom:
awk '{ sub("\r$", ""); print }' myitems.csv > tmpfile && mv tmpfile myitems.csv
GNU awk v4.1 or higher offers -i inplace for in-place updating, to which the same caveat as above for sed applies.
Edge case for all variants above: If the very last char. in the input file happens to be a lone \r without a following \n, it will also be replaced with a \n.
For the sake of completeness: here are additional, possibly suboptimal solutions:
None of them offer in-place updating, but you can employ the > tmpfile && mv tmpfile myitems.csv idiom introduced above
Using tr: a very simple solution that simply removes all \r instances; thus, it can only be used if \r instance only occur as part of \r\n sequences; typically, however, that is the case:
tr -d '\r' < myitems.csv
Using pure bash code: note that this will be slow; like the tr solution, this can only be used if \r instance only occur as part of \r\n sequences.
while IFS=$'\r' read -r line; do
printf '%s\n' "$line"
done < myitems.csv
$IFS is the internal field separator, and setting it to \r causes read to read everything before \r, if present, into variable $line (if there's no \r, the line is read as is). -r prevents read from interpreting \ instances in the input.
Edge case: If the input doesn't end with \n, the last line will not print - you could fix that by using read -r line || [[ -n $line ]].
try this, it will fix your issue.
dos2unix myitems.csv myitems.csv
Try the unix2dos command.
Example: unix2dos infile outfile
http://en.wikipedia.org/wiki/Unix2dos
The wikipedia page has some examples using perl and sed too.
perl -i -p -e 's/\n/\r\n/' file
sed -i -e 's/$/\r/' file
So I have tried to install gsed via macports but this hasnt solved the issue. I was going to uninstall it to reduce clutter, however, before I do so, how would i fix the error below. It is because of the BSD version of sed Mac OS X is running from what I understand, but none of the fixes I seem to have found are helping.
sed: 1: "/\[staging: production\ ...": command i expects \ followed by text
#!/bin/bash
test="lala\nkjdsh"
sed -i -e '/\[staging: production\]/ i '$test'' ./test.txt
You have this problem because of "\n" in the $test. Try to remove \n from it.
POSIX standard sed only accepts \n as part of a search pattern. OS X uses the FreeBSD sed, which is strictly POSIX compliant
So if you need a newline in a variable you need to write something like:
$ test="lala\
> kjdsh"
You can also solve the task with perl:
$ test="lala\nkjdsh"
$ perl -n -i -e 'print "'"$test"'\n" if /\[staging: production\]/; print;' ./test.txt
Example:
$ echo '[staging: production]' > /tmp/test.txt
$ test="lala\nkjdsh"
$ perl -n -i -e 'print "'"$test"'\n" if /\[staging: production\]/; print;' ./test.txt
$ cat ./test.txt
lala
kjdsh
[staging: production]
If the test variable does not contain a line containing only . you can use ed to edit the file:
printf '%s\n' '/\[staging: production\]/i' "$test" . w | ed -s ./test.txt
See http://wiki.bash-hackers.org/howto/edit-ed for more about ed.
EDIT: Oh and I missed that you actually had backslash-followed-by-N and not literal newlines in your variable. If you use literal newlines the above should work.
EDIT2: Given the pastebin given in the comments, try:
#!/usr/bin/env bash
#...
ed -s ./test.txt << EOF
/\[staging: production\]/i
; some comment
someStuffHere[] = "XYZ"
someMoreStuff[] = "$someShellVar"
; another comment
.
w
EOF
The . alone on a line ends the insert command, and w is the write command, which actually saves the changes to the file, (like :w in vim)
I have a file of strings that are comma separated. I'm trying to replace the commas with a new line. I've tried:
sed 's/,/\n/g' file
but it is not working. What am I missing?
Use tr instead:
tr , '\n' < file
Use an ANSI-C quoted string $'string'
You need a backslash-escaped literal newline to get to sed.
In bash at least, $'' strings will replace \n with a real newline, but then you have to double the backslash that sed will see to escape the newline, e.g.
echo "a,b" | sed -e $'s/,/\\\n/g'
Note this will not work on all shells, but will work on the most common ones.
sed 's/,/\
/g'
works on Mac OS X.
If your sed usage tends to be entirely substitution expressions (as mine tends to be), you can also use perl -pe instead
$ echo 'foo,bar,baz' | perl -pe 's/,/,\n/g'
foo,
bar,
baz
MacOS is different, there is two way to solve this problem with sed in mac
first ,use \'$'\n'' replace \n, it can work in MacOS:
sed 's/,/\'$'\n''/g' file
the second, just use an empty line:
sed 's/,/\
/g' file
Ps. Pay attention the range separated by '
the third, use gnu-sed replace the mac-sed
Apparently \r is the key!
$ sed 's/, /\r/g' file3.txt > file4.txt
Transformed this:
ABFS, AIRM, AMED, BOSC, CALI, ECPG, FRGI, GERN, GTIV, HSON, IQNT, JRCC, LTRE,
MACK, MIDD, NKTR, NPSP, PME, PTIX, REFR, RSOL, UBNT, UPI, YONG, ZEUS
To this:
ABFS
AIRM
AMED
BOSC
CALI
ECPG
FRGI
GERN
GTIV
HSON
IQNT
JRCC
LTRE
MACK
MIDD
NKTR
NPSP
PME
PTIX
REFR
RSOL
UBNT
UPI
YONG
ZEUS
This works on MacOS Mountain Lion (10.8), Solaris 10 (SunOS 5.10) and RHE Linux (Red Hat Enterprise Linux Server release 5.3, Tikanga)...
$ sed 's/{pattern}/\^J/g' foo.txt > foo2.txt
... where the ^J is done by doing ctrl+v+j. Do mind the \ before the ^J.
PS, I know the sed in RHEL is GNU, the MacOS sed is FreeBSD based, and although I'm not sure about the Solaris sed, I believe this will work pretty much with any sed. YMMV tho'...
To make it complete, this also works:
echo "a,b" | sed "s/,/\\$(echo -e '\n\r')/"
Though I am late to this post, just updating my findings. This answer is only for Mac OS X.
$ sed 's/new/
> /g' m1.json > m2.json
sed: 1: "s/new/
/g": unescaped newline inside substitute pattern
In the above command I tried with Shift+Enter to add new line which didn't work. So this time I tried with "escaping" the "unescaped newline" as told by the error.
$ sed 's/new/\
> /g' m1.json > m2.json
Worked! (in Mac OS X 10.9.3)
$ echo $PATH | sed -e $'s/:/\\\n/g'
/usr/local/sbin
/Library/Oracle/instantclient_11_2/sdk
/usr/local/bin
...
Works for me on Mojave
Just to clearify: man-page of sed on OSX (10.8; Darwin Kernel Version 12.4.0) says:
[...]
Sed Regular Expressions
The regular expressions used in sed, by default, are basic regular expressions (BREs, see re_format(7) for more information), but extended
(modern) regular expressions can be used instead if the -E flag is given. In addition, sed has the following two additions to regular
expressions:
1. In a context address, any character other than a backslash (``\'') or newline character may be used to delimit the regular expression.
Also, putting a backslash character before the delimiting character causes the character to be treated literally. For example, in the
context address \xabc\xdefx, the RE delimiter is an ``x'' and the second ``x'' stands for itself, so that the regular expression is
``abcxdef''.
2. The escape sequence \n matches a newline character embedded in the pattern space. You cannot, however, use a literal newline charac-
ter in an address or in the substitute command.
[...]
so I guess one have to use tr - as mentioned above - or the nifty
sed "s/,/^M
/g"
note: you have to type <ctrl>-v,<return> to get '^M' in vi editor
The sed on macOS Mojave was released in 2005, so one solution is to install the gnu-sed,
brew install gnu-sed
then use gsed will do as you wish,
gsed 's/,/\n/g' file
If you prefer sed, just sudo sh -c 'echo /usr/local/opt/gnu-sed/libexec/gnubin > /etc/paths.d/brew', which is suggested by brew info gnu-sed. Restart your term, then your sed in command line is gsed.
FWIW, the following line works in windows and replaces semicolons in my path variables with a newline. I'm using the tools installed under my git bin directory.
echo %path% | sed -e $'s/;/\\n/g' | less
I have found another command that is working also.
find your_filename.txt -type f -exec sed -i 's/,/\n/g' {} \;
I'd like edit a file with sed on OS X. I'm using the following command:
sed 's/oldword/newword/' file.txt
The output is sent to the terminal. file.txt is not modified. The changes are saved to file2.txt with this command:
sed 's/oldword/newword/' file1.txt > file2.txt
However I don't want another file. I just want to edit file1.txt. How can I do this?
I've tried the -i flag. This results in the following error:
sed: 1: "file1.txt": invalid command code f
You can use the -i flag correctly by providing it with a suffix to add to the backed-up file. Extending your example:
sed -i.bu 's/oldword/newword/' file1.txt
Will give you two files: one with the name file1.txt that contains the substitution, and one with the name file1.txt.bu that has the original content.
Mildly dangerous
If you want to destructively overwrite the original file, use something like:
sed -i '' 's/oldword/newword/' file1.txt
^ note the space
Because of the way the line gets parsed, a space is required between the option flag and its argument because the argument is zero-length.
Other than possibly trashing your original, I’m not aware of any further dangers of tricking sed this way. It should be noted, however, that if this invocation of sed is part of a script, The Unix Way™ would (IMHO) be to use sed non-destructively, test that it exited cleanly, and only then remove the extraneous file.
I've similar problem with MacOS
sed -i '' 's/oldword/newword/' file1.txt
doesn't works, but
sed -i"any_symbol" 's/oldword/newword/' file1.txt
works well.
The -i flag probably doesn't work for you, because you followed an example for GNU sed while macOS uses BSD sed and they have a slightly different syntax.
All the other answers tell you how to correct the syntax to work with BSD sed. The alternative is to install GNU sed on your macOS with:
brew install gsed
and then use it instead of the sed version shipped with macOS (note the g prefix), e.g:
gsed -i 's/oldword/newword/' file1.txt
If you want GNU sed commands to be always portable to your macOS, you could prepend "gnubin" directory to your path, by adding something like this to your .bashrc/.zshrc file (run brew info gsed to see what exactly you need to do):
export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"
and from then on the GNU sed becomes your default sed and you can simply run:
sed -i 's/oldword/newword/' file1.txt
sed -i -- "s/https/http/g" file.txt
You can use -i'' (--in-place) for sed as already suggested. See: The -i in-place argument, however note that -i option is non-standard FreeBSD extensions and may not be available on other operating systems. Secondly sed is a Stream EDitor, not a file editor.
Alternative way is to use built-in substitution in Vim Ex mode, like:
$ ex +%s/foo/bar/g -scwq file.txt
and for multiple-files:
$ ex +'bufdo!%s/foo/bar/g' -scxa *.*
To edit all files recursively you can use **/*.* if shell supports that (enable by shopt -s globstar).
Another way is to use gawk and its new "inplace" extension such as:
$ gawk -i inplace '{ gsub(/foo/, "bar") }; { print }' file1
This creates backup files. E.g. sed -i -e 's/hello/hello world/' testfile for me, creates a backup file, testfile-e, in the same dir.
You can use:
sed -i -e 's/<string-to-find>/<string-to-replace>/' <your-file-path>
Example:
sed -i -e 's/Hello/Bye/' file.txt
This works flawless in Mac.
If you need to substitute more than one different words:
sed -i '' -e 's/_tools/tools/' -e 's/_static/static/' test.txt
I'm trying to remove the carriage returns (\r) from a file with the following command on AIX, but it's also removing my last line. Any suggestions?
sed -e 's/\r\n/\n/g' ./excprule > ./excprule.tst
Command sequence:
dev1:> sed -e 's/\r\n/\n/g' ./test_file > ./test_file.tst
dev1:> diff test_file.tst test_file
diff: 0653-827 Missing newline at the end of file test_file.
26a27
> Trailer 25
Edit: Found a workaround by doing the same thing with perl, but would like to know why this doesn't work.
You can also use dos2unix if you have it in AIX, however, with sed,
sed '$!{:a;N;s/\r\n/\n/;ta}' file
sed strips off the newline whenever it process current line (see the man page), so you won't really find the \r\n.