I'm trying to remove the carriage returns (\r) from a file with the following command on AIX, but it's also removing my last line. Any suggestions?
sed -e 's/\r\n/\n/g' ./excprule > ./excprule.tst
Command sequence:
dev1:> sed -e 's/\r\n/\n/g' ./test_file > ./test_file.tst
dev1:> diff test_file.tst test_file
diff: 0653-827 Missing newline at the end of file test_file.
26a27
> Trailer 25
Edit: Found a workaround by doing the same thing with perl, but would like to know why this doesn't work.
You can also use dos2unix if you have it in AIX, however, with sed,
sed '$!{:a;N;s/\r\n/\n/;ta}' file
sed strips off the newline whenever it process current line (see the man page), so you won't really find the \r\n.
Related
I wanna replace(delete) the last line of the last control character newline form ls command stdout result.
Which I had try sed command is :
ls | sed '$ s/[[:cntrl:]]$//'
ls | sed '$ s/[[:graph:]]$//'
ls | sed '$ s/[[:print:]]$//'
The result I want like this result ls | head -c -1
Any wrong with my sed command?
Thanks !
This might work for you (GNU sed):
sed -z 's/\n$//' file
sed appears to automatically add a newline if the current line was terminated with a newline, and I don't see how to turn that behaviour off.
You might try a different approach: capture the output of the command (the command substitution syntax strips trailing newlines); and then print it without a newline
printf '%s' "$(ls)"
How can I use a cat and sed to read data from a file and insert it into another file under known line?
For example I have a file named script1.txt that contains a few hundred lines, one of the line has the value "COMMANDS="commands"
If I wanted use sed to insert a line under it, simply I can use sed as the command bellow.
sed -i '/^COMMANDS=.*/a NEW LINE HERE' script1.txt
But if I want to insert a multi lines and these lines inside a file, and these line changes every a few hours.. how can i do that ?
I tried:
DATA=$(cat data.txt)
sed -i '/^COMMANDS=.*/a '$DATA'' script1.txt
I got the error bellow.
sed: -e expression #1, char 1: unknown command: `"'
Is there a way other than sed to insert the data from file under known line with no issues?
This might work for you (GNU sed):
sed -i '/^COMMANDS=/r dataFile' file
This will append the contents of the file dataFile after the line beginning COMMANDS= and update file
If the data you want to append is multi-line, you might want to replace newlines with \n.
#!/bin/sh
DATA="$(awk '{gsub(/[]\/$*.^&[]/, "\\\\&");printf (FNR>1)?"\\n%s":"%s",$0}END{print ""}' data.txt)"
sed -i -e '/^COMMANDS=.*/a\' -e "$DATA" script1.txt
Here the awk command escapes sed special characters (for basic regular expressions), then prints "%s" for the first line, and "\\n%s" for the others. A newline is printed at the end, but it's somewhat pointless as $() strips it anyway.
The sed command is almost the same but multiple expressions are used which is equivalent to a multi-line sed script (The a text sed alternative syntax can act weirdly with leading spaces/backslashes).
I have this line inside a file:
ULNET-PA,client_sgcib,broker_keplersecurities
,KEPLER
I try to get rid of that ^M (carriage return) character so I used:
sed 's/^M//g'
However this does remove everything after ^M:
[root#localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities^M,KEPLER
[root#localhost tmp]# sed 's/^M//g' test
ULNET-PA,client_sgcib,broker_keplersecurities
What I want to obtain is:
[root#localhost tmp]# vi test
ULNET-PA,client_sgcib,broker_keplersecurities,KEPLER
Use tr:
tr -d '^M' < inputfile
(Note that the ^M character can be input using Ctrl+VCtrl+M)
EDIT: As suggested by Glenn Jackman, if you're using bash, you could also say:
tr -d $'\r' < inputfile
still the same line:
sed -i 's/^M//g' file
when you type the command, for ^M you type Ctrl+VCtrl+M
actually if you have already opened the file in vim, you can just in vim do:
:%s/^M//g
same, ^M you type Ctrl-V Ctrl-M
You can simply use dos2unix which is available in most Unix/Linux systems. However I found the following sed command to be better as it removed ^M where dos2unix couldn't:
sed 's/\r//g' < input.txt > output.txt
Hope that helps.
Note: ^M is actually carriage return character which is represented in code as \r
What dos2unix does is most likely equivalent to:
sed 's/\r\n/\n/g' < input.txt > output.txt
It doesn't remove \r when it is not immediately followed by \n and replaces both with just \n. This fails with certain types of files like one I just tested with.
alias dos2unix="sed -i -e 's/'\"\$(printf '\015')\"'//g' "
Usage:
dos2unix file
If Perl is an option:
perl -i -pe 's/\r\n$/\n/g' file
-i makes a .bak version of the input file
\r = carriage return
\n = linefeed
$ = end of line
s/foo/bar/g = globally substitute "foo" with "bar"
In awk:
sub(/\r/,"")
If it is in the end of record, sub(/\r/,"",$NF) should suffice. No need to scan the whole record.
This is the better way to achieve
tr -d '\015' < inputfile_name > outputfile_name
Later rename the file to original file name.
I agree with #twalberg (see accepted answer comments, above), dos2unix on Mac OSX covers this, quoting man dos2unix:
To run in Mac mode use the command-line option "-c mac" or use the
commands "mac2unix" or "unix2mac"
I settled on 'mac2unix', which got rid of my less-cmd-visible '^M' entries, introduced by an Apple 'Messages' transfer of a bash script between 2 Yosemite (OSX 10.10) Macs!
I installed 'dos2unix', trivially, on Mac OSX using the popular Homebrew package installer, I highly recommend it and it's companion command, Cask.
This is clean and simple and it works:
sed -i 's/\r//g' file
where \r of course is the equivalent for ^M.
Simply run the following command:
sed -i -e 's/\r$//' input.file
I verified this as valid in Mac OSX Monterey.
remove any \r :
nawk 'NF+=OFS=_' FS='\r'
gawk 3 ORS= RS='\r'
remove end of line \r :
mawk2 8 RS='\r?\n'
mawk -F'\r$' NF=1
I want to clear the contents of a particular line in a file.
I could delete the line by using sed:
sed -i "${linenumber}d" filename.txt
But I dont want the line to be deleted. I want the line to stay with no content.
Is there a way to do that?
Sure:
sed -i "${linenumber} s/.*//" filename.txt
The ${linenumber} specifies which line to work on, the delete command (d) in your sed script can be replaced by any other command, here the substitution command (s/pattern/replacement/).
I need to write a shell script to re-format a file by inserting line breaks. The condition is that a line break should be inserted when we encounter comma in the file.
For example, if the file delimiter.txt contains:
this, is a file, that should, be added, with a, line break, when we find, a comma.
The output should be:
this
is a file
that should
be added
with a
line break
when we find a
a comma.
Can this be done grep or awk?
Using GNU sed:
sed 's/, /\n/g' your.file
Output:
this
is a file
that should
be added
with a
line break
when we find a
a comma.
Note: the syntax above will work only on system that have the \n as line delimiter as Linux and the most UNIXes.
If you need a portal solution in a a script then use the following expression that uses a literal new line instead of \n:
sed 's/,[[:space:]]/\
/g' your.file
Thanks #EdMorten for this advice.
This is what tr is for
$ tr ',' '\n' <<< 'this, is a file, that should, be added, with a, line break, when we find, a comma.'
this
is a file
that should
be added
with a
line break
when we find
a comma.
Or if you must use awk:
awk '{gsub(", ", "\n", $0)}1' delimiter.txt
Solution using awk:
awk 1 RS=", " file
this
is a file
that should
be added
with a
line break
when we find
a comma.
Here's the solution using perl:
perl -pe 's#,#\n#g'
Here's a sample of it working properly on OpenBSD or OS X:
% echo 'a,b,c,d,e' | perl -pe 's#,#\n#g'
a
b
c
d
e
%
E.g., unlike the sed solutions earlier, this perl works everywhere, because the same search/replace snippet wouldn't work with the BSD sed on OpenBSD or OS X:
% echo 'a,b,c,d,e' | sed -E 's#,#\n#g'
anbncndne
%