replace CR LF in text file using `sed` or `fart` (Find And Replace Text) - windows

I have a 1.5 GB Windows text file with some lines ending with LF and most of lines ending with CR+LF
Can you please help with sed script which
will replace all CR+LF with $|$
replace all LF with CR+LF
replace back all $|$ with CR+LF
I have tried to do all replacements with text editor, but it took very long to perform all replacements in the file (1 percent for half an hour). I've tried to replace it with fart:
fart -c -B -b text.txt "\r\n" "$|$"
with following result
replacement 0 occurence(s) in 0 file(s)..

One with awk:
$ awk '{sub(/(^|[^\r])$/,"&\r")}1' file
Testing it (0x0a is LF, 0x0d is CR):
$ awk 'BEGIN{print "no\nyes\r\n\n\r"}' > foo
$ hexdump -C foo
00000000 6e 6f 0a 79 65 73 0d 0a 0a 0d 0a |no.yes.....|
0000000b
$ awk '{sub(/(^|[^\r])$/,"&\r")}1' foo > bar
$ hexdump -C bar
00000000 6e 6f 0d 0a 79 65 73 0d 0a 0d 0a 0d 0a |no..yes......|
0000000d

I would do this: first remove all \r at the end of the line, then explicitly add a \r to the end of the line.
sed -e 's/\r$//' -e 's/$/\r/' file
Here's a demo:
$ printf "1\r\n2\n3\n4\r\n5\n" > file
$ od -c file
0000000 1 \r \n 2 \n 3 \n 4 \r \n 5 \n
0000014
$ sed -i -e 's/\r$//' -e 's/$/\r/' file
$ od -c file
0000000 1 \r \n 2 \r \n 3 \r \n 4 \r \n 5 \r \n
0000017
This is GNU sed.

It's simpler just to install a util like unix2dos which does it automatically. With unix2dos the proposed intermediate step of converting CR+LF to $|$, (and back), isn't necessary. Demo:
# first dump a file with both *DOS* and *Unix* style line endings:
hexdump -C <({ seq 2 | unix2dos ; seq 3 4; } )
# the same file, run through unix2dos
hexdump -C <({ seq 2 | unix2dos ; seq 3 4; } | unix2dos)
Output:
00000000 31 0d 0a 32 0d 0a 33 0a 34 0a |1..2..3.4.|
0000000a
00000000 31 0d 0a 32 0d 0a 33 0d 0a 34 0d 0a |1..2..3..4..|
0000000c
Or more elaborately, a before/after table, (see man hexdump for details on formatting):
hdf() { hexdump -v -e '/1 "%_ad# "' -e '/1 " _%_u\_\n"' $# ; }
# Note: the `printf` stuff keeps `paste` from misaligning the output.
paste <(hdf <({ seq 2 | unix2dos ; seq 3 4; }) ; printf '\t\n\t\n' ; ) \
<(hdf <({ seq 2 | unix2dos ; seq 3 4; } | unix2dos ))
Output:
0# _1_ 0# _1_
1# _cr_ 1# _cr_
2# _lf_ 2# _lf_
3# _2_ 3# _2_
4# _cr_ 4# _cr_
5# _lf_ 5# _lf_
6# _3_ 6# _3_
7# _lf_ 7# _cr_
8# _4_ 8# _lf_
9# _lf_ 9# _4_
10# _cr_
11# _lf_

Related

how to get substring from

how to get substring from
42 45 47 49 4e 21 40 23 47 68 6a 6b 2c 47 68 6a BEGIN!##Ghjk,Ghj 6b 45 4e 44 23 40 21 kEND##!
to be
BEGIN!##Ghjk,GhjkEND##!
Note: there is whitespaces at end of lines, I tried removing whitespaces at end of lines but I cant.
I tried
#!/bin/bash
s=$(awk '/BEGIN!##/,/END##!/' switch.log )
while IFS= read -r line
do
h=$(echo "$line" | awk '{$1=$1;print}')
for i in {0..100}
do
zzz=$(echo "$h" | awk '{print $(NF-$i)}')
if [ ! -z "$zzz" -a "$zzz" != " " ]; then
hh=$(echo "$h" | awk '{print $(NF-$i)}')
echo "$zzz"
echo -e "$zzz" >> ggg.txt
break
fi
done
done <<< "$s"
I got
BEGIN!##Ghjk,Ghj
Another option is using sed with the normal substitute method storing the text you want to keep as the first two backreferences. For example:
sed -E 's/^.*(BEGIN[^[:space:]]+).*(kEND[^[:space:]]+)/\1\2/' <<< 'your string`
Example Use/Output
(note: updated to handle whitespace at the end)
$ sed -E 's/^.*(BEGIN[^[:space:]]+).*(kEND[^[:space:]]+)/\1\2/' <<< '42 45 47 49 4e 21 40 23 47 68 6a 6b 2c 47 68 6a BEGIN!##Ghjk,Ghj 6b 45 4e 44 23 40 21 kEND##!'
BEGIN!##Ghjk,GhjkEND##!
(note: single-quoting the string is required due to '!')
Using sed
$ sed -E 's/[0-9]+[a-z]? +| +//g' input_file
BEGIN!##Ghjk,GhjkEND##!
UPDATED, to fix an error:
You have not defined precisely in your question, how the string to be extracted looks like in general, but based on your example, this would do:
if [[ $line =~ (BEGIN[^ ]+)\ .*([^ ]+END[^ ]+) ]]
then
substring=${BASH_REMATCH[1]}${BASH_REMATCH[2]}
else
echo Pattern not found in line 1>&2
fi
I would harness GNU AWK for this task following way, let file.txt content be
42 45 47 49 4e 21 40 23 47 68 6a 6b 2c 47 68 6a BEGIN!##Ghjk,Ghj 6b 45 4e 44 23 40 21 kEND##!
then
awk 'BEGIN{FPAT="[^[:space:]]*(BEGIN|END)[^[:space:]]*";OFS=""}{$1=$1;print}' file.txt
gives output
BEGIN!##Ghjk,GhjkEND##!
Explanation: I inform GNU AWK using field pattern (FPAT) that field is BEGIN or (|) END, prefixed and suffixed by zero-or-more (*) non (^)-whitespace ([:space:]) characters and output field separator (OFS) is empty string, then for each line I do $1=$1 to trigger line rebuilt and print it. If you are sure only space characters are used in line you might elect to replace [^[:space:]] using [^ ]
(tested in gawk 4.2.1)
s=$(awk '/BEGIN!##/,/END##!/' switch.log)
echo "$s" > ggg.txt
ss=$(sed -E 's/[0-9]+[a-z]? +| +//g' ggg.txt )
echo "$ss" > ddd.txt
sss=$(awk '{print $1}' ddd.txt)
echo "$sss" > hhhh.txt
ssss=$(awk '/BEGIN!##/,/END##!/' hhhh.txt)
echo "$ssss" > hhh.txt
aaa=$(<hhh.txt)
aaa=$(cat hhh.txt | tr -d '\n' )
By setting the awk record separator RS to " ", awk processes a white-spaced-separated portion of your file at a time (with each record containing only one field). So the two parts that are needed can be extracted with simple awk condition patterns /BEGIN/ and /END/. There can be no white space in any record since this was the delimiter.
If printed, the pattern-filtered records would normally be separated by a new line (the default output record-separator ORS) but this can be changed to an empty string ORS="" to make the two print statements run into one another with no space.
Thus this simple awk command will return the required fields as a concatentated string with no white space:
awk ' BEGIN{RS=" ";ORS=""} /BEGIN/{print} /END/{print}' file.txt
output:
BEGIN!##Ghjk,GhjkEND##!
$ grep -oE '(BEGIN|END)\S*' file | paste -sd'\0'
BEGIN!##Ghjk,GhjEND##!
echo ' 42 45 47 49 4e 21 40 23 47 68 6a 6b 2c 47 68 ' \
'6a BEGIN!##Ghjk,Ghj 6b 45 4e 44 23 40 21 kEND##!' |
{m,g}awk NF=NF FS='[ \t]*([^ \t][^ \t][ \t]+)+[ \t]*' OFS=
BEGIN!##Ghjk,GkEND##!

confusion about ASCII linefeed byte in awk + xxd bash command

I am confused about some 0a (i.e. NL ASCII byte) happening in some bash commands. On the following:
$ echo | sha1sum $1 | awk '{print $1;}' | xxd -r -ps > test.bin
$ echo | sha1sum $1 | awk '{print $1;}' > test.hex
$ xxd test.bin
00000000: adc8 3b19 e793 491b 1c6e a0fd 8b46 cd9f ..;...I..n...F..
00000010: 32e5 92fc 2...
$ xxd test.hex
00000000: 6164 6338 3362 3139 6537 3933 3439 3162 adc83b19e793491b
00000010: 3163 3665 6130 6664 3862 3436 6364 3966 1c6ea0fd8b46cd9f
00000020: 3332 6535 3932 6663 0a 32e592fc.
what is responsible for the 0a byte to be present in test.hex but not in test.bin?
Note 1: this is a question that I have been asking myself following the solution used there:
Dump a ```sha``` checksum output to disk in binary format instead of plaintext hex in bash
Note 2: I am able to suppress the 0a byte, this is not the question, I am just curious of why it is present in one case but not the other:
$ echo | sha1sum $1 | awk '{print $1;}' | head -c-1 > test_2.hex
$ xxd test_2.hex
00000000: 6164 6338 3362 3139 6537 3933 3439 3162 adc83b19e793491b
00000010: 3163 3665 6130 6664 3862 3436 6364 3966 1c6ea0fd8b46cd9f
00000020: 3332 6535 3932 6663 32e592fc
The 0a that you are seeing is coming from awk. By default the output record separator for awk is \n and you can remote it by setting the ORS (e.g. with a BEGIN {ORS=""}).
You lose it when you pipe through xxd -r -ps due to the -r parameter. From the man page: "Additional Whitespace and line-breaks are allowed anywhere."

Converting string using bash

I want to convert the output of command:
dmidecode -s system-serial-number
which is a string looking like this:
VMware-56 4d ad 01 22 5a 73 c2-89 ce 3f d8 ba d6 e4 0c
to:
564dad01-225a-73c2-89ce-3fd8bad6e40c
I suspect I need to first of all extract all letters and numbers after the "VMware-" part at that start and then insert "-" at the known positions after character 10, 14, 18, 22.
To try the first extraction I have tried:
$ echo `dmidecode -s system-serial-number | grep -oE '(VMware-)?[a0-Z9]'`
VMware-5 6 4 d a d 0 1 2 2 5 a 7 3 c 2 8 9 c e 3 f d 8 b a d 6 e 4 0 c
However this isn't going the right way.
EDIT:
This gets me to a single log string however it's not elegant:
$ echo `dmidecode -s system-serial-number | sed -s "s/VMware-//" | sed -s "s/-//" | sed -s "s/ //g"`
564dad01225a73c289ce3fd8bad6e40c
Like this :
dmidecode -s system-serial-number |
sed -E 's/VMware-//;
s/ +//g;
s/(.)/\1-/8;
s/(.)/\1-/13;
s/(.)/\1-/23'
You can use Bash sub string extraction:
$ s="VMware-56 4d ad 01 22 5a 73 c2-89 ce 3f d8 ba d6 e4 0c"
$ s1=$(echo "${s:7}" | tr -d '[:space:]')
$ echo "${s1:0:8}-${s1:8:4}-${s1:12:9}-${s1:21}"
564dad01-225a-73c2-89ce-3fd8bad6e40c
Or, built-ins only (ie, no tr):
$ s1=${s:7}
$ s1="${s1// /}"
$ echo "${s1:0:8}-${s1:8:4}-${s1:12:9}-${s1:21}"

Inconsistencies when packing hex string

I am having some inconsistencies when using hexdump and xxd. When I run the following command:
echo -n "a42d9dfe8f93515d0d5f608a576044ce4c61e61e" \
| sed 's/\(..\)/\1\n/g' \
| awk '/^[a-fA-F0-9]{2}$/ { printf("%c",strtonum("0x" $0)); }' \
| xxd
it returns the following results:
00000000: c2a4 2dc2 9dc3 bec2 8fc2 9351 5d0d 5f60 ..-........Q]._`
00000010: c28a 5760 44c3 8e4c 61c3 a61e ..W`D..La...
Note the "c2" characters. This also happens with I run xxd -p
When I run the same command except with hexdump -C:
echo -n "a42d9dfe8f93515d0d5f608a576044ce4c61e61e" \
| sed 's/\(..\)/\1\n/g' \
| awk '/^[a-fA-F0-9]{2}$/ { printf("%c",strtonum("0x" $0)); }' \
| hexdump -C
I get the same results (as far as including the "c2" character):
00000000 c2 a4 2d c2 9d c3 be c2 8f c2 93 51 5d 0d 5f 60 |..-........Q]._`|
00000010 c2 8a 57 60 44 c3 8e 4c 61 c3 a6 1e |..W`D..La...|
However, when I run hexdump with no arguments:
echo -n "a42d9dfe8f93515d0d5f608a576044ce4c61e61e" \
| sed 's/\(..\)/\1\n/g' \
| awk '/^[a-fA-F0-9]{2}$/ { printf("%c",strtonum("0x" $0)); }' \
| hexdump
I get the following [correct] results:
0000000 a4c2 c22d c39d c2be c28f 5193 0d5d 605f
0000010 8ac2 6057 c344 4c8e c361 1ea6
For the purpose of this script, I'd rather use xxd as opposed to hexdump. Thoughts?
The problem that you observe is due to UTF-8 encoding and little-endiannes.
First, note that when you try to print any Unicode character in AWK, like 0xA4 (CURRENCY SIGN), it actually produces two bytes of output, like the two bytes 0xC2 0xA4 that you see in your output:
$ echo 1 | awk 'BEGIN { printf("%c", 0xA4) }' | hexdump -C
Output:
00000000 c2 a4 |..|
00000002
This holds for any character bigger than 0x7F and it is due to UTF-8 encoding, which is probably the one set in your locale. (Note: some AWK implementations will have different behavior for the above code.)
Secondly, when you use hexdump without argument -C, it displays each pair of bytes in swapped order due to little-endianness of your machine. This is because each pair of bytes is then treated as a single 16-bit word, instead of treating each byte separately, as done by xxd and hexdump -C commands. So the xxd output that you get is actually the correct byte-for-byte representation of input.
Thirdly, if you want to produce the precise byte string that is encoded in the hexadecimal string that you are feeding to sed, you can use this Python solution:
echo -n "a42d9dfe8f93515d0d5f608a576044ce4c61e61e" | sed 's/\(..\)/0x\1,/g' | python3 -c "import sys;[open('tmp','wb').write(bytearray(eval('[' + line + ']'))) for line in sys.stdin]" && cat tmp | xxd
Output:
00000000: a42d 9dfe 8f93 515d 0d5f 608a 5760 44ce .-....Q]._`.W`D.
00000010: 4c61 e61e La..
Why not use xxd with -r and -p?
echo a42d9dfe8f93515d0d5f608a576044ce4c61e61e | xxd -r -p | xxd
output
0000000: a42d 9dfe 8f93 515d 0d5f 608a 5760 44ce .-....Q]._`.W`D.
0000010: 4c61 e61e La..

Easiest way to strip newline character from input string in pasteboard

Hopefully fairly straightforward, to explain the use case when I run the following command (OS X 10.6):
$ pwd | pbcopy
the pasteboard contains a newline character at the end. I'd like to get rid of it.
pwd | tr -d '\n' | pbcopy
printf $(pwd) | pbcopy
or
echo -n $(pwd) | pbcopy
Note that these should really be quoted in case there are whitespace characters in the directory name. For example:
echo -n "$(pwd)" | pbcopy
I wrote a utility called noeol to solve this problem. It pipes stdin to stdout, but leaves out the trailing newline if there is one. E.g.
pwd | noeol | pbcopy
…I aliased copy to noeol | pbcopy.
Check it out here: https://github.com/Sidnicious/noeol
For me I was having issues with the tr -d '\n' approach. On OSX I happened to have the coreutils package installed via brew install coreutils. This provides all the "normal" GNU utilities prefixed with a g in front of their typical names. So head would be ghead for example.
Using this worked more safely IMO:
pwd | ghead -c -1 | pbcopy
You can use od to see what's happening with the output:
$ pwd | ghead -c -1 | /usr/bin/od -h
0000000 552f 6573 7372 732f 696d 676e 6c6f 6c65
0000020 696c
0000022
vs.
$ pwd | /usr/bin/od -h
0000000 552f 6573 7372 732f 696d 676e 6c6f 6c65
0000020 696c 000a
0000023
The difference?
The 00 and 0a are the hexcodes for a nul and newline. The ghead -c -1 merely "chomps" the last character from the output before handing it off to | pbcopy.
$ man ascii | grep -E '\b00\b|\b0a\b'
00 nul 01 soh 02 stx 03 etx 04 eot 05 enq 06 ack 07 bel
08 bs 09 ht 0a nl 0b vt 0c np 0d cr 0e so 0f si
We can first delete the trailing newline if any, and then give it to pbcopy as follows:
your_command | perl -0 -pe 's/\n\Z//' | pbcopy
We can also create an alias of this:
alias pbc="perl -0 -pe 's/\n\Z//' | pbcopy"
Then the command would become:
pwd | pbc

Resources