Bash: capture stdError from looped stream - bash

I have a cpp executable (mycat) that continuously read a looped audio stream from the shared memory and pipes the data to stdOut and the metadata information to stdErr. mycat pipes out 12 lines for every entry of the audio stream, containing the metadata information, looking like this:
0x1 (TimeStamp) 12Bytes:2956 + 6793/(47999+1) (0.141521) delta= 0+ 1536/(47999+1) (0.032000) 2956.151418 -9.898ms 2016.04.04 16:06:37.700
0x4 (ReferenceTime) 12Bytes:2956 + 6156972/(26999999+1) (0.228036) delta= 0+ 1618519/(26999999+1) (0.059944) 2956.151426 76.610ms 2016.04.04 16:06:37.700
0x6 (ProcessDelay) 4Bytes: 64 (0x40)
0x7 (ClockAccuracy) 8Bytes: offset=0.000ppm (+-0.000ppm)
0xb (ClockId) 8Bytes: 01 00 00 00 42 22 01 00
0x20001 (SampleRate) 4Bytes: 48000 (0xbb80)
0x20002 (Channels) 4Bytes: 6 (0x6)
0x20003 (PcmLevel) 24Bytes: -21307 -20348 -31737 -42427 -28786 -26525
0x20004 (PcmPeak) 24Bytes: -14366 -13360 -25203 -39427 -19067 -21307
0x2000e (DolbyDpMetadata) 39352Bytes:
Linear Time: 2956 + 6793/(47999+1) (0.141521) delta= 0+ 1536/(47999+1) (0.032000)
2016.04.04 16:06:37.700 update: slot=0xe2840 validTo=0x3d1dd180 shmT=0x3d195200 (delta=294784) doffset=0xec2c0 msize=39552 dsize=18432 type=0x20001 (PCMS16) data bytes: df f4 f2 fc
What I want is a bash script that:
1) launch mycat eg. ./mycat shm_name > /dev/null.
2) reads stdErr from mycat till the 12th line no matter where it started.
2.1) Eventually store the 12 line into a variable (this is optional)
3) immedialy kills mycat after the 12th line, so that the bash script can continue without being annoyed by the outcoming stdError.
4) Read the value of the line "Channels" (in this case 6) and store it to a variable named "channels"
5) Read the value of the line "SampleRate" (in this case 48000) and store it the a variable named "rate"
is there a way to do it?

You can redirect stderr with mycat shm_name > /dev/null 2>/path/to/file. You can kill mycat with killall mycat when the time comes. For storing the variables, you want export channels =, same pattern with rate. You can find those with grep. I'm not sure how to wait for exactly twelve lines, though.

First, you need to redirect STDERR to STDOUT, and have STDOUT go to null.
mycat 2>&1 >/dev/null | parsing_script
Then you need a parsing script to collect your data
#!/bin/bash
declare -a data=()
count=1
while read line; do
data+=( "$line
")
((count++))
string=$(echo $line|sed 's/.*(\(.[A-Za-z]*\)).*/\1/')
case $string in
SampleRate) rate=$(echo $line|sed 's/.*: \(.[0-9]*\) .*/\1/')
;;
Channels) channels=$(echo $line|sed 's/.*: \(.[0-9]*\) .*/\1/')
;;
*) true;;
esac
if [[ $count -gt 12 ]]; then
break
fi
done <&0
echo $channels
echo $rate
I echo'd the values, but you can redirect them to a file, format them, etc. Also, ${data[#]} contains your metadata. Since you are piping, I'm reading in from STDIN, but it could be more robust if you want to tweak it.

I find out an elegant solution:
while read -r line; # Read mycat line by line
do
if echo "$line" | grep -qi "SampleRate"; then # Search for line containing string 'SampleRate'
rate="${line#*:}"; # Remove all charachters till ':'
rate="${rate%(*}"; # Remove all charachters from '('
rate="$(echo -e "${rate}" | tr -d '[[:space:]]')" # Remove all tralling withespace
elif echo "$line" | grep -qi "Channels"; then
channels="${line#*:}";
channels="${channels%(*}";
channels="$(echo -e "${channels}" | tr -d '[[:space:]]')"
fi
if [ -n "$channels" ] && [ -n "$rate" ]; then # If both variables 'channels' and 'rate' are not empty kill the while loop
break;
fi
done < <(./mycat $shm_name 2>&1 > /dev/null) # Mycat is processed into a while loop, standard Error is redirected to standard Output ( 2>&1 )
echo "Found datarate: $rate"
echo "Found channels: $channels"

Related

Is there a command for substituting a set of characters by a set of strings?

I'm would like to substitute a set of edit: single byte characters with a set of literal strings in a stream, without any constraint on the line size.
#!/bin/bash
for (( i = 1; i <= 0x7FFFFFFFFFFFFFFF; i++ ))
do
printf '\a,\b,\t,\v'
done |
chars_to_strings $'\a\b\t\v' '<bell>' '<backspace>' '<horizontal-tab>' '<vertical-tab>'
The expected output would be:
<bell>,<backspace>,<horizontal-tab>,<vertical-tab><bell>,<backspace>,<horizontal-tab>,<vertical-tab><bell>...
I can think of a bash function that would do that, something like:
chars_to_strings() {
local delim buffer
while true
do
delim=''
IFS='' read -r -d '.' -n 4096 buffer && (( ${#buffer} != 4096 )) && delim='.'
if [[ -n "${delim:+_}" ]] || [[ -n "${buffer:+_}" ]]
then
# Do the replacements in "$buffer"
# ...
printf "%s%s" "$buffer" "$delim"
else
break
fi
done
}
But I'm looking for a more efficient way, any thoughts?
Since you seem to be okay with using ANSI C quoting via $'...' strings, then maybe use sed?
sed $'s/\a/<bell>/g; s/\b/<backspace>/g; s/\t/<horizontal-tab>/g; s/\v/<vertical-tab>/g'
Or, via separate commands:
sed -e $'s/\a/<bell>/g' \
-e $'s/\b/<backspace>/g' \
-e $'s/\t/<horizontal-tab>/g' \
-e $'s/\v/<vertical-tab>/g'
Or, using awk, which replaces newline characters too (by customizing the Output Record Separator, i.e., the ORS variable):
$ printf '\a,\b,\t,\v\n' | awk -vORS='<newline>' '
{
gsub(/\a/, "<bell>")
gsub(/\b/, "<backspace>")
gsub(/\t/, "<horizontal-tab>")
gsub(/\v/, "<vertical-tab>")
print $0
}
'
<bell>,<backspace>,<horizontal-tab>,<vertical-tab><newline>
For a simple one-liner with reasonable portability, try Perl.
for (( i = 1; i <= 0x7FFFFFFFFFFFFFFF; i++ ))
do
printf '\a,\b,\t,\v'
done |
perl -pe 's/\a/<bell>/g;
s/\b/<backspace>/g;s/\t/<horizontal-tab>/g;s/\v/<vertical-tab>/g'
Perl internally does some intelligent optimizations so it's not encumbered by lines which are longer than its input buffer or whatever.
Perl by itself is not POSIX, of course; but it can be expected to be installed on any even remotely modern platform (short of perhaps embedded systems etc).
Assuming the overall objective is to provide the ability to process a stream of data in real time without having to wait for a EOL/End-of-buffer occurrence to trigger processing ...
A few items:
continue to use the while/read -n loop to read a chunk of data from the incoming stream and store in buffer variable
push the conversion code into something that's better suited to string manipulation (ie, something other than bash); for sake of discussion we'll choose awk
within the while/read -n loop printf "%s\n" "${buffer}" and pipe the output from the while loop into awk; NOTE: the key item is to introduce an explicit \n into the stream so as to trigger awk processing for each new 'line' of input; OP can decide if this additional \n must be distinguished from a \n occurring in the original stream of data
awk then parses each line of input as per the replacement logic, making sure to append anything leftover to the front of the next line of input (ie, for when the while/read -n breaks an item in the 'middle')
General idea:
chars_to_strings() {
while read -r -n 15 buffer # using '15' for demo purposes otherwise replace with '4096' or whatever OP wants
do
printf "%s\n" "${buffer}"
done | awk '{print NR,FNR,length($0)}' # replace 'print ...' with OP's replacement logic
}
Take for a test drive:
for (( i = 1; i <= 20; i++ ))
do
printf '\a,\b,\t,\v'
sleep 0.1 # add some delay to data being streamed to chars_to_strings()
done | chars_to_strings
1 1 15 # output starts printing right away
2 2 15 # instead of waiting for the 'for'
3 3 15 # loop to complete
4 4 15
5 5 13
6 6 15
7 7 15
8 8 15
9 9 15
A variation on this idea using a named pipe:
mkfifo /tmp/pipeX
sleep infinity > /tmp/pipeX # keep pipe open so awk does not exit
awk '{print NR,FNR,length($0)}' < /tmp/pipeX &
chars_to_strings() {
while read -r -n 15 buffer
do
printf "%s\n" "${buffer}"
done > /tmp/pipeX
}
Take for a test drive:
for (( i = 1; i <= 20; i++ ))
do
printf '\a,\b,\t,\v'
sleep 0.1
done | chars_to_strings
1 1 15 # output starts printing right away
2 2 15 # instead of waiting for the 'for'
3 3 15 # loop to complete
4 4 15
5 5 13
6 6 15
7 7 15
8 8 15
9 9 15
# kill background 'awk' and/or 'sleep infinity' when no longer needed
don't waste FS/OFS - use the built-in variables to take 2 out of the 5 needed :
echo $' \t abc xyz \t \a \n\n ' |
mawk 'gsub(/\7/, "<bell>", $!(NF = NF)) + gsub(/\10/,"<bs>") +\
gsub(/\11/,"<h-tab>")^_' OFS='<v-tab>' FS='\13' ORS='<newline>'
<h-tab> abc xyz <h-tab> <bell> <newline><newline> <newline>
To have NO constraint on the line length you could do something like this with GNU awk:
awk -v RS='.{1,100}' -v ORS= '{
$0 = RT
gsub(foo,bar)
print
}'
That will read and process the input 100 chars at a time no matter which chars are present, whether it has newlines or not, and even if the input was one multi-terabyte line.
Replace gsub(foo,bar) with whatever substitution(s) you have in mind, e.g.:
$ printf '\a,\b,\t,\v' |
awk -v RS='.{1,100}' -v ORS= '{
$0 = RT
gsub(/\a/,"<bell>")
gsub(/\b/,"<backspace>")
gsub(/\t/,"<horizontal-tab>")
gsub(/\v/,"<vertical-tab>")
print
}'
<bell>,<backspace>,<horizontal-tab>,<vertical-tab>
and of course it'd be trivial to pass a list of old and new strings to awk rather than hardcoding them, you'd just have to sanitize any regexp or backreference metachars before calling gsub().

How do I compare whether two numbers have 1 number of difference between them?

Explaining my algorithm:
I'm trying to find out whether My current job for e.g. Write(W) is the same as my previous job, if my current job (W) is the same as my previous job (W) then check whether there's 1 integer of difference between them, for e.g. if the previous job was W9 and my current job is either W8 or W10, then append 0 to my seek array.
I've tried almost every way I could find on the internet to compare integers but none of them work, I continue to receive an invalid arithmetic syntax error when trying to compare current and previous job.
Any ideas?
# Jobs
lastJob=""
currentJob=""
lastNumber=0
currentNumber=0
# Arrays
seek=()
RW=()
shift 3
# Reads final into array
while IFS= read -r line
do
Job+=($line)
done < ./sim.job
#-----------------------------------
# Single HDD Algorithm
#-----------------------------------
for (( i=0; i<=${#Job[#]}; i++ ));
do
currentString="${Job[$i]}"
currentJob=${currentString:0:1}
currentNumber=${currentString:1:3}
if [[ $currentJob == $lastJob ]]
then
if [[ $currentNumber -eq $lastNumber-1 ]] || [[ $currentNumber -eq $lastNumber+1 ]]
then
seek+=(0)
RW+=(60)
else
seek+=(5)
RW+=(60)
fi
else
seek+=(5)
RW+=(60)
fi
lastString="${Job[$i]}"
lastJob=${lastString:0:1}
lastNumber=${currentString:1:3}
done
This prints output:
#-----------------------------------
# Print Information
#-----------------------------------
for (( i=0; i<${#Job[#]}; i++ ));
do
echo -e "${Job[$i]}:${seek[$i]}:${RW[$i]}"
done
Expected Output:
R9:5:60
W9:5:60
W10:0:60
R11:0:60
R13:5:60
R18:5:60
R19:0:60
R20:0:60
R21:0:60
Actual Output:
") syntax error: invalid arithmetic operator (error token is "
R9:5:60
W9:5:60
W10::
R11::
R13::
R18::
R19::
R20::
R21::
sim.job file (Input):
R9
W9
W10
R11
R13
R18
R19
R20
R21
rogue \r were found in my input file, to solve this I used the commands:
To check if \r are in the file: od -c <filename> | tr ' ' '\n' | grep '\r'
To remove rogue \r use: tr -d '\r' < filename
Thanks again #mark-fuso

Return the read cursor to the start of the file

I'm trying to read into two file (name,number) at the same time and get value of each possible pair.
The two file are like this:
*name1
John
*name2
Paul
*number1
25
*number2
45
What i'm trying to obtain are label and result like:
*name1 *number1 John 25
*name2 *number2 John 45
*name2 *number1 Paul 25
*name2 *number2 Paul 45
Since i come from python i've tried to do it with two loop like this:
name=/home/davide/name.txt
number=/home/davide/number.txt
while read name; do
if [[ ${name:0:1} == "*" ]]; then
n=$(echo $name)
else
while read number; do
if [[ ${number:0:1} == "*" ]]; then
echo $number $n
else
echo $name $number
fi
done < $number
fi
done < $name
I have the first two pair so my guess it's that i need a command to start from the beginning of number again (like seek(0) on python) but i haven't found a similar one for bash.
I also get an "ambiguous redirect" error and i don't understand why.
After setting up your input files:
printf >name.txt '%s\n' '*name1' John '*name2' Paul
printf >number.txt '%s\n' '*number1' 25 '*number2' 45
...the following code:
#!/usr/bin/env bash
name_file=name.txt
number_file=number.txt
while IFS= read -r name1 && IFS= read -r value1; do
while IFS= read -r name2 && IFS= read -r value2; do
printf '%s\n' "$name1 $name2 $value1 $value2"
done <"$number_file"
done <"$name_file"
...properly outputs:
*name1 *number1 John 25
*name1 *number2 John 45
*name2 *number1 Paul 25
*name2 *number2 Paul 45
What changed?
We stopped using name and number both for the filenames and for the values read from them. Because of this, when you ran <$number, it no longer had the filename number.txt in it after the first iteration; likewise for $name.
We started quoting all expansions ("$foo", not $foo). See the http://shellcheck.net/ warning SC2086, and BashPitfalls #14, explaining why even echo $foo is buggy.
Running read with the -r argument and IFS set to an empty value prevents it from consuming literal backslashes or pruning leading and trailing newlines.
Using two reads inside the condition of each while loop lets us read two lines at a time from each file (as is appropriate, given the intent to process content in pairs).
Bash operates more easly on "streams", not like, on the data itself.
first substitute every second newline starting from the first for a tabulation or a space or other separator
then "paste" the files together
Then rearange columns, from *name1 John *number1 25 to *name1 *number1 John 25
cat >name.txt <<EOF
*name1
John
*name2
Paul
EOF
cat <<EOF >number.txt
*number1
25
*number2
45
EOF
paste <(<name.txt sed 'N;s/\n/\t/') <(<number.txt sed 'N;s/\n/\t/') |
awk '{print $1,$3,$2,$4}'
will output:
*name1 *number1 John 25
*name2 *number2 Paul 45
First, in your example you overwrite the variable $number. So you have issues on reading file $number beginning from the second loop-run.
Solution with paste
Command paste can combine multiple files, and with option -d line-by-line.
#!/usr/bin/env bash
name=/home/davide/name.txt
number=/home/davide/number.txt
# combine both files linb-by-line
paste $'-d\n' "$name" "$number" |
while read nam
do
#after reading name to var 'nam', read number to var 'num':
read num
# print both
echo "$nam $num"
done
if you want TABS or any other separator and no other processing, you don't need the while loop. Examples
paste "$name" "$number"
paste -d: "$name" "$number"
paste -d\| "$name" "$number"
$ cat tst.awk
NR==FNR {
if ( NR%2 ) {
tags[++numPairs] = $0
}
else {
vals[numPairs] = $0
}
next
}
!(NR%2) {
for (pairNr=1; pairNr<=numPairs; pairNr++) {
print prev, tags[pairNr], $0, vals[pairNr]
}
}
{ prev = $0 }
$ awk -f tst.awk number.txt name.txt
*name1 *number1 John 25
*name1 *number2 John 45
*name2 *number1 Paul 25
*name2 *number2 Paul 45
In your script, you use the variable name for both the file path and the while-loop variable. This causes the "ambiguous redirect" error. Two lines need fix e.g.:
name_file=/home/davide/name.txt
done < $name_file
No need to for seek(0) in shell scripts. Just process the file again, e.g:
while read line ; do
echo "== $line =="
done < /some/file
while read line ; do
echo "--> ${line:0:1}"
done < /some/file
This is less efficient and less flexible than a more real programming language where you can seek(). But that's about differences, advantages and disadvantages between shell scripting and programming.
By the way, this line:
n=$(echo $name)
... is merely a awkward way of just doing:
n=$name
This can cause your script to behave quite unpredictable when $name contains special character like *. And since $name is read from a text file, this not unlikely to happen. (thanks Charles Duffy for making this point)

Count specific character in each line of at text and remove this character in a specific position until this character has a specific count

Hello i need help with one script that its on Solaris system:
I will explain the script analytically:
i have these files :
i)
cat /tmp/BadTransactions/TRANSACTIONS_DAILY_20180730.txt
201807300000000004
201807300000000005
201807300000000006
201807300000000007
201807300000000008
201807200002056422
201807230003099849
201807230003958306
201806290003097219
201806080001062012
201806110001633519
201806110001675603
ii)
cat /tmp/BadTransactions/test_data_for_validation_script.txt
20180720|201807200002056422||57413620344272|030341-213T |580463|WIRE||EUR|EUR|20180720|20180720|||||||00000000000019.90|00000000000019.90|Debit||||||||||MPA|||574000|129|||||||||||||||||||||||||31313001103712|BFNJKL|K| I P P BONNIER PUBLICATIO|||FI|PERS7
20180723|201807230003099849||57100440165173|140197-216U|593619|WIRE||EUR|EUR|20180723|20180723|||||||00000000000060.00|00000000000060.00|Debit||||||||||MPA|||571004|106|||||||||||||||||||||||||57108320141339|Ura Basket / UraNaiset|||-div|||FI|PERS
20180723|201807230003958306||57206820079775|210489-0788|593619|WIRE||EUR|EUR|20180721|20180723|||||||00000000000046.00|00000000000046.00|Debit||||||||||MPA|||578800|106|||||||||||||||||||||||||18053000009026|IC Kodit||| c/o Newsec Asset Manag|||FI|PERS
20180629|201806290003097219||57206820079775|210489-0788|593619|WIRE||EUR|EUR|20180628|20180629|||||||00000000000856.00|00000000000856.00|Debit||||||||||MPA|||578800|106|||||||||||||||||||||||||18053000009018|IC Kodit||| c/o Newsec Asset Manag|||FI|PERS
20180608|201806080001062012||57206820079441|140197-216S|580463|WIRE||EUR|EUR|20180608|20180608|||||||00000000000019.90|00000000000019.90|Debit||||||||||MPA|||541002|129|||||||||||||||||||||||||57108320141339|N FN|K| IKI I P BONNIER PUBLICATION|||FI|PERS7
20180611|201806110001633519||57206820079525|140197-216B|593619|WIRE||EUR|EUR|20180611|20180611|||||||00000000000242.10|00000000000242.10|Debit||||||||||MPA|||535806|106|||||||||||||||||||||||||57108320141339|As Oy Haikkoonsilta|| mannerheimin|||FI|PERS9
20180611|201806110001675603||57206820079092|140197-216Z|580463|WIRE||EUR|EUR|20180611|20180611|||||||00000000000019.90|00000000000019.90|Debit||||||||||MPA|||536501|129|||||||||||||||||||||||||57108320141339|N ^NLKL|K| I P NJ BONNIER PUBLICAT|||FI|PERS7
The script has to check each line of the
/tmp/BadTransactions/TRANSACTIONS_DAILY_20180730.txt and if the strings are on
the /tmp/BadTransactions/test_data_for_validation_script.txt it will create a
new file `/tmp/BadTransactions/TRANSACTIONS_DAILY_NEW_20180730.txt
From this new file it will count all the " | " in each line and if its more than 64 it will delete the " | " in 61th posistion of the line . This will be continued until its line has 64 pipes.
For example if one line has 67 " | " it will delete the 61th , then it will check it again and now has 66 " | | so it will delete the 61th " | " , etc... until it reach 64 pipes.So all the line have to have 64th " | ".
Here is my code , but in this code i have managed to delete only the 61th pipe in each line , i cannot make the loop so that it will check each line until it reach the 64 pipes.
I will appreciate it if you could help me.
#!/bin/bash
PATH=/usr/xpg4/bin:/bin:/usr/bin
while read line
do
grep "$line" /tmp/BadTransactions/test_data_for_validation_script.txt
awk 'NR==FNR { K[$1]; next } ($2 in K)' /tmp/BadTransactions/TRANSACTIONS_DAILY_20180730.txt FS="|" /opt/NorkomC
onfigS2/inbox/TRANSACTIONS_DAILY_20180730.txt > /tmp/BadTransactions/TRANSACTIONS_DAILY_NEW_20180730.txt
sed '/\([^|]*[|]\)\{65\}/ s/|//61' /tmp/BadTransactions/TRANSACTIONS_DAILY_NEW_20180730.txt
done < /tmp/BadTransactions/TRANSACTIONS_DAILY_20180730.txt > /tmp/BadTransactions/TRANSACTIONS_DAILY_NEW_201807
30.txt
Ok, in this problem you have several pieces of code.
You need to read a file line by line
Check each line against another file
Examine the matching line for the occurrences of "|"
Delete recursively the 61st "|" until the string will remain with 64 of them
You could do something like this
#!/bin/bash
count() { ### We will use this to count how many pipes are there
string="${1}"; shift
char="${1}"
printf "%s" "${string}" | grep -o -e "${char}" | grep -c .
}
file1="/tmp/BadTransactions/TRANSACTIONS_DAILY_20180730.txt" ### File to read
file2="/tmp/BadTransactions/test_data_for_validation_script.txt" ### File to check for duplicates
file3="/tmp/BadTransactions/TRANSACTIONS_DAILY_NEW_20180730.txt" ### File where to save our final work
printf "" > "${file3}" ### Delete (eventual) history
exec 3<"${file1}" ### Put our data in file descriptor 3
while read -r line <&3; do ### read each line and put it in var "$line"
string="$(grep -e "${line}" "${file2}")" ### Check the line against second file
while [ "$(count "${string}" "|")" -gt 64 ]; do ### While we have more than 64 "|"
string="$(printf "%s" "${string}" | sed -e "s/|//61")" ### Delete the 61st occurrence
done
printf "%s" "${string}" >> "${file3}" ### Save the correct line in the third file
done
exec 3>&- ### Clean file descriptor 3
This is not tested, but should work.
N.B. Please note that I am giving for granted that grep will return only one occurrence from second file...
If it is not your case you have to manually check each value with something like:
for value in $(grep -e "${line}" "${file2}"); do
...
done
EDIT:
For systems like Solaris or others that doesn't have GNU grep installed you can substitute the count method as follow:
count() {
string="${1}"; shift
char="${1}"
printf "%s" "${string}" | awk -F"${char}" '{print NF-1}'
}

Replace all control characters in a range of line with awk

I got a file with several lines. Some of these lines contain LFs (0x0A) and CRs (0x0D), which I want to get removed. The point is, that I want to replace them with SPACE them only in a range of characters of every line, eg in a File:
30 30 30 30 30 30 30 30 30 30 **0D 0A** 30 30 0A; 0000000000..00
30 30 30 30 30 30 30 30 **0D 0A** 30 30 30 30 0A; 00000000..0000
I want to remove 0d, 0a from position 0 to 12 in every line of the file.
I got
awk '{l=substr($0,1,12);r=substr($0,13);gsub(/\x00-\1F/," ",l);print l r}' ${f} > ${f}.noLF
but this seems not to work. I guess substr stops at the first 0x0d.
Is there another solution?
awk '/\r$/ && length < 13 {sub(/\r$/,""); printf "%s ", $0; next} {print}' file
Here is something ugly that may work!
Save it as go
#!/bin/bash
while :
do
# Read 13 bytes from stdin, and replace carriage returns and linefeeds with spaces
dd bs=13 count=1 2>/dev/null | tr '\r\n' ' '
# Break out of loop if dd was not successful
[ ${PIPESTATUS[0]} -ne 0 ] && break
# Get rest of conventional line, breaking out of loop if EOF
read rest || break
echo $rest
done
It reads 13 bytes from your file and removes all carriage returns and linefeeds. Then it reads the rest of the conventional line and outputs that.
Use it like this:
chmod +x go
./go < yourfile
Example:
more file
q
wertyuiopqwertyuiop
qwerty
uiopqwertyuiop
./go < file
q wertyuiopqwertyuiop
qwerty uiopqwertyuiop
EDITED TO MATCH FURTHER QUESTIONS
#!/bin/bash
while :
do
# Read 13 bytes from stdin, and replace carriage returns and linefeeds with spaces
first13=$(dd bs=13 count=1 2>/dev/null)
ddexitstatus=$?
if [ echo $first13 | grep -q "^KT" ]; then
echo $first13
else
echo $first13 | tr '\r\n' ' '
fi
# Break out of loop if dd was not successful
[ $ddexitstatus -ne 0 ] && break
# Get rest of conventional line, breaking out of loop if EOF
read rest || break
echo $rest
done

Resources