String together awk commands - bash

I'm writing a script that searches a file, gets info that it then stores into variables, and executes a program that I made using those variables as data. I actually have all of that working, but I need to take it a step further:
What I currently have is
#!/bin/sh
START=0
END=9
LOOP=10
PASS=0
for i in $(seq 0 $LOOP)
do
LEN=$(awk '/Len =/ { print $3; exit;}' ../../Tests/shabittestvectors/SHA1ShortMsg.rsp)
MSG=$(awk '/Msg =/ { print $3; exit; }' ../../Tests/shabittestvectors/SHA1ShortMsg.rsp)
MD=$(awk '/MD =/ { print $3; exit; }' ../../Tests/shabittestvectors/SHA1ShortMsg.rsp)
echo $LEN
echo $MSG
MD=${MD:0:-1}
CIPHER=$(./cyassl hash -sha -i $MSG -l $LEN)
echo $MD
echo $CIPHER
if [ $MD == $CIPHER ]; then
echo "PASSED"
PASS=$[PASS + 1]
echo $PASS
fi
done
if [ $PASS == $[LOOP+1] ]; then
echo "All Tests Successful"
fi
And the input file looks like this:
Len = 0
Msg = 00
MD = da39a3ee5e6b4b0d3255bfef95601890afd80709
Len = 1
Msg = 00
MD = bb6b3e18f0115b57925241676f5b1ae88747b08a
Len = 2
Msg = 40
MD = ec6b39952e1a3ec3ab3507185cf756181c84bbe2
All the program does right now, is read the first instances of the variables and loop around there. I'm hoping to use START and END to determine the lines in which it checks the file, and then increment them every time it loops to obtain the other instances of the variable names, but all of my attempts have been unsuccessful so far. Any ideas?
EDIT: Output should look something like, providing my program "./cyassl" works as it should
0
00
da39a3ee5e6b4b0d3255bfef95601890afd80709
da39a3ee5e6b4b0d3255bfef95601890afd80709
PASSED
1
00
bb6b3e18f0115b57925241676f5b1ae88747b08a
bb6b3e18f0115b57925241676f5b1ae88747b08a
PASSED
2
40
ec6b39952e1a3ec3ab3507185cf756181c84bbe2
ec6b39952e1a3ec3ab3507185cf756181c84bbe2
PASSED
etc.

There's no need to make multiple passes on the input file.
#!/bin/sh
exec < ../../Tests/shabittestvectors/SHA1ShortMsg.rsp
status=pass
awk '{print $3,$6,$9}' RS= | {
while read len msg md; do
if test "$(./cyassl hash -sha -i $msg -l $len)" = "$md"; then
echo passed
else
status=fail
fi
done
test "$status" = pass && echo all tests passed
}
The awk will read from stdin (which the exec redirects from the file; personally I would skip that line and have the caller direct input appropriately) and splits the input into records of one paragraph each. A "paragraph" here means that the records are separated by blank lines (the lines must be truly blank, and cannot contain whitespace). Awk then parses each record and prints the 3rd, 6th, and 9th field on a single line. This is a bit fragile, but for the shown input those fields represent length, message, and md hash, respectively. All the awk is doing is rearranging the input so that it is one record per line. Once the data is in a more readable format, a subshell reads the data one line at a time, parsing it into the variables named "len", "msg", and "md". The do loop processes once per line of input, spewing the rather verbose message "passed" with each test it runs (I would remove that, but retained it here for consistency with the original script), and setting the status if any tests fail. The braces are necessary to ensure that the value of the variable status is retained after the do loop terminates.

The following code,
inputfile="../../Tests/shabittestvectors/SHA1ShortMsg.rsp"
while read -r len msg md
do
echo got: LEN:$len MSG:$msg MD:$md
#cypher=$(./cyassl hash -sha -i $msg -l $len)
#continue as you wish
done < <(perl -00 -F'[\s=]+|\n' -lE 'say qq{$F[1] $F[3] $F[5]}' < "$inputfile")
for your input data, produces:
got: LEN:0 MSG:00 MD:da39a3ee5e6b4b0d3255bfef95601890afd80709
got: LEN:1 MSG:00 MD:bb6b3e18f0115b57925241676f5b1ae88747b08a
got: LEN:2 MSG:40 MD:ec6b39952e1a3ec3ab3507185cf756181c84bbe2

If your input data is in order you can have this with a simplified bash:
#!/bin/bash
LOOP=10
PASS=0
FILE='../../Tests/shabittestvectors/SHA1ShortMsg.rsp'
for (( I = 1; I <= LOOP; ++I )); do
read -r LEN && read -r MSG && read -r MD || break
echo "$LEN"
echo "$MSG"
MD=${MD:0:-1}
CIPHER=$(exec ./cyassl hash -sha -i "$MSG" -l "$LEN")
echo "$MD"
echo "$CIPHER"
if [[ $MD == "$CIPHER" ]]; then
echo "PASSED"
(( ++PASS ))
fi
done < <(exec awk '/Len =/,/Msg =/,/MD =/ { print $3 }' "$FILE")
[[ PASS -eq LOOP ]] && echo "All Tests Successful."
Just make sure you don't run it as sh e.g. sh script.sh. bash script.sh most likely.

Related

Sending the output of a while loop to a bash function

I've created a .bashrc file where I have two functions. One loops through the lines of a file in a while loop. I'm attempting to save the content of the lines if they match certain conditions and then pass all three matches to a second function that will then echo them. However, I've tried exporting variables and I've also tried piping to the second function, but neither works. Piping also acts very strangely as I'll try to illustrate in my code example.
readAndPipe() {
while read -r line || [[ -n "$line" ]]; do
(
if [[ $line == FIRSTNAME=BOB ]]; then
echo $line;
fi;
if [[ $line == LASTNAME=SMITH ]]; then
echo $line;
fi;
if [[ $line == BIRTHMONTH=AUGUST ]]; then
echo $line;
fi;
); done < "file.txt" | printArguments $1 #pass the original command line argument;
}
printArguments() {
#This is where the weirdness happens
echo $# #Prints: only the original command line argument
echo $# #Prints: 1
echo $2 $3 $4 #Prints nothing
varName=$(cat $2 $3 $4)
echo $varName #Prints: FIRSTNAME=BOB
# LASTNAME=SMITH
# BIRTHMONTH=AUGUST
cat $2 $3 $4 #Prints nothing
echo $(cat $2 $3 $4) #Prints nothing
cat $2 $3 $4 | tr "\n" '' #Prints tr: empty string2
}
Obviously I'm not a bash expert so I'm sure there's a lot of mistakes here, but what I'm wondering is
What are these seemingly magical $2 $3 $4 arguments that are not
printed by echo but can be used by cat exactly once.
What is the
correct way to save content during a while loop and pass it to
another function so that I can echo it?
$#, $*, $1, $2, etc are the arguments that are passed to the function. For example, in myfunc foo bar baz, we have $1 == foo, $2 == bar and $3 == baz.
When you pipe data to your function, you have to retrieve it from from stdin:
myfunc() {
data=$(cat)
echo "I received: >$data<"
}
for n in {1..5}; do echo "x=$n"; done | myfunc
produces
I received: >x=1
x=2
x=3
x=4
x=5<
varName=$(cat $2 $3 $4) works because $2 $3 and $4 are empty, so the shell sees this:
varName=$(cat )
The reason cat "only works once" is because you are consuming a stream. Once you consume it, it's gone. "you can't eat your cake and have it too."
The printArguments function can use the readarray command to grab the incoming lines into an array, instead of using cat to grab all the incoming text into a variable:
printArguments() {
readarray -t lines
echo "I have ${#lines[#]} lines"
echo "they are:"
printf ">>%s\n" "${lines[#]}"
}
{ echo foo; echo bar; echo baz; } | printArguments
outputs
I have 3 lines
they are:
>>foo
>>bar
>>baz
Learn more by typing help readarray at an interactive bash prompt.
Imagine a script:
func() {
echo $# # will print 2, func were executed with 2 arguments
echo "$#" # will print `arg1 arg2`, ie. the function arguments
in=$(cat) # will pass stdin to `cat` function and save cat's stdout into a variable
echo "$in" # will print `1 2 3`
}
echo 1 2 3 | func arg1 arg2
# ^^^^^^ function `func` arguments
# ^ passed one command stdout to other command stdin
# ^^^ outputs `1 2 3` on process stdout
cat invoked without any arguments, reads stdin and outputs it on stdout.
Invoking a command in command substitution passes stdin along (ie. in=$(cat) will read stdin as normal cat just saves the output (ie. the cat's stdout) into a variable)
To your script:
readAndPipe() {
# the while read line does not matter, but it outputs something on stdout
while read -r line || [[ -n "$line" ]]; do
echo print something
# the content of `file.txt` is passed as while read input
done < "file.txt" | printArguments $1 # the `print something` (the while loop stdout output) is passed as stdin to the printArguments function
}
printArguments() {
# here $# is equal to 1
# $1 is equal to passed $1 (unless expanded, will get to that)
# $2 $3 $4 expand to nothing
varName=$(cat $2 $3 $4) # this executes varName=$(cat) as $2 $3 $4 expand to nothing
# now after this point stdin has been read (it can be read once, it's a stream or pipe
# is you execute `cat` again it will block (waiting for more input) or fail (will receive EOF - end of file)
echo $varName #Prints: `print something` as it was passed on stdin to this function
}
If the file file.txt contains only just:
FIRSTNAME=BOB
LASTNAME=SMITH
BIRTHMONTH=AUGUST
you can just load the file . file.txt or source file.txt. This will "load" the file, ie. make it part of your script, syntax is bash. So you can:
. file.txt
echo "$FIRSTNAME"
echo "$LASTNAME"
echo "$BIRTHMONTH"
This is a common way of creating configuration files in /etc/ and then they are loaded by scripts. That's why in many /etc/ files comments start with #.
Notes:
Always enclose your variables. echo "$1" printArguments "$1" echo "$#" echo "$#" cat "$2" "$3" "$4" [ "$line" == ... ], a good read is here
Remove newlines with tr -d '\n'
A ( ) creates a subshell, which creates a new shell which has new variables and does not share variable with the parent, see here.

Stuck in an infinite while loop

I am trying to write this code so that if the process reads map finished in the pipe it increments a variable by 1 so that it eventually breaks out of the while loop. Otherwise it will add unique parameters to a keys file. However it goes into an infinite loop and never breaks out of the loop.
while [ $a -le 5 ]; do
read input < map_pipe;
if [ $input = "map finished" ]; then
((a++))
echo $a
else
sort -u map_pipe >> keys.txt;
fi
done
I decided to fix it for you, not sure if this is what you wanted, but I think I am close:
#!/bin/bash
a=0 #Initialize your variable to something
while [ $a -le 5 ]; do
read input < map_pipe;
if [ "$input" = "map finished" ]; then #Put double quotes around variables to allow values with spaces
a=$(($a + 1)) #Your syntax was off, use spaces and do something with the output
else
echo $input >> keys.txt #Don't re-read the pipe, it's empty by now and sort will wait for the next input
sort -u keys.txt > tmpfile #Instead sort your file, don't save directly into the same file it will break
mv tmpfile keys.txt
#sort -u keys.txt | sponge keys.txt #Will also work instead of the other sort and mv, but sponge is not installed on most machines
fi
done

Return an error if input doesn't have exactly 1 line, otherwise pipe input to next step

I have a series of commands chained together with pipes:
should_create_one_line | expects_one_line
The first command should_create_one_line should produce an output that only has one line, but under strange circumstances it is possible for the output to be multiline or empty.
I would like to add a step in between these two, validate_one_line:
should_create_one_line | validate_one_line | expects_one_line
If its input contains exactly 1 line then validate_one_line will simply output its input. If its input contains more than 1 line or is empty then validate_one_line should cause the whole sequence of steps to stop and return an error code.
What command can I use for validate_one_line?
Use read. Here's a shell function that meets your specs:
exactly_one_line() {
local line # Use to echo the line
read -r line || return # Guarantee at least one line is read
read && return 1 # Indicate failure if another line is successfully read
echo "$line"
}
Notes
"One line" assumes a single line followed by a newline. If your input could be like, a file with contents but no newlines, then this will fail.
Given a pipeline like a|b, a cannot prevent b from running. At a minimum, b needs to handle when a produces no output.
Demo:
$ wc -l empty oneline twolines
0 empty
1 oneline
2 twolines
3 total
$ exactly_one_line < empty; echo $?
1
$ exactly_one_line < oneline; echo $?
oneline
0
$ exactly_one_line < twolines; echo $?
1
First off, you should seriously consider adding the validation code to expects_one_line. According to this post, each process starts in its own subshell, meaning that even if validate_one_line fails, you will get an error in expects_one_line because it will try to run with no input (or a blank line). That being said, here is a bash one-liner that you can insert into your pipe to validate:
should_create_one_line.sh | ( var="$(cat)"; [ $(echo "$var" | wc -l) -ne 1 ] && exit 1 || echo "$var") | expects_one_line.sh
The problem here is that when the validation subshell returns in the exit 1 case, expects_one_line.sh will still get a single blank line. If this works for you, then great. If not, it would be better to just put the following into the beginning of expects_one_line.sh:
input="$(cat)"
[ $(echo "$var" | wc -l) -ne 1 ] && exit 1
This would guarantee that expects_one_line.sh fails properly when getting a single line without having to wonder about what the empty line that the validation outputs will do to the script.
You may find this post helpful: How to read mutliline input from stdin into variable and how to print one out in shell(sh,bash)?
You can use a bash script to check the incoming data and call the other command when the input is only 1 line
The following code starts cat when it is ONLY fet in 1 line
sh -c 'while read CMD; do [ ! -z "$LINE" ] && exit 1; LINE=$CMD; done; [ -z "$LINE" ] && exit 1; printf "%s\n" $LINE | "$0" "$#"' cat
How this works
Try reading a line, if failed go to step 5
If variable $LINE is NOT empty, goto step 6
Save line inside variable $LINE
Goto step 1
If $LINE is NOT empty, goto step 7
Exit the program with status code 1
Call our program and pass our $line to it using printf
Example usage:
Printing out only if grep found 1 match:
grep .... | sh -c 'while read CMD; do [ ! -z "$LINE" ] && exit 1; LINE=$CMD; done; [ -z "$LINE" ] && exit 1; printf "%s\n" $LINE | "$0" "$#"' cat
Example of the question poster:
should_create_one_line | sh -c 'while read CMD; do [ ! -z "$LINE" ] && exit 1; LINE=$CMD; done; [ -z "$LINE" ] && exit 1; printf "%s\n" $LINE | "$0" "$#"' expects_one_line

how to parse values from text file and assign it for shell script arguments

I have stored some arguments value in sample.txt
1 >> sample.txt
2 >> sample.txt
3 >> sample.txt
I have tried to parse the sample.txt in a shell script file to collect and assign the values to specific variables.
#!/bin/sh
if [ -f sample.txt ]; then
cat sample.txt | while read Param
do
let count++
if [ "${count}" == 1 ]; then
Var1=`echo ${Param}`
elif [ "${count}" == 2 ]; then
Var2=`echo ${Param}`
else
Var3=`echo ${Param}`
fi
done
fi
echo "$Var1"
echo "$Var2"
echo results prints nothing. I would expect 1 and 2 should be printed. Anyone help?
You are running the while loop in a subshell; use input redirection instead of cat:
while read Param; do
...
done < sample.txt
(Also, Var1=$Param is much simpler than Var1=$(echo $Param).)
However, there's no point in use a while loop if you know ahead of time how many variables you are setting; just use the right number of read commands directly.
{ read Var1; read Var2; read Var3; } < sample.txt

Bash: Reading Output to a String with Special Characters

I'm using TShark to read TCP streams of a PCAP into a file of a set format. My code:
#!/bin/bash
OUT="*/temp/Temp.txt"
NEW="\"REQ:"
i=0
echo "Generating conversations..."
echo "" > $OUT
while [ "$COUNT" != 1 ]
do
BLOCK="$(tshark -r */browser.pcap -q -z follow,tcp,ascii,$i)"
SUB=$(echo "$BLOCK" | sed -n '5p')
PORT=${SUB##*:}
BLOCK="${BLOCK//$'\t'/\"RES:}"
BLOCK=$(echo "$BLOCK" | tail -n +6)
BLOCK=$(echo "$BLOCK" | head -n -1)
COUNT=$(echo "$BLOCK" | wc -l)
BLOCK=$(echo "$BLOCK" | awk '{print $j"\""}')
j=1
while [ $j -lt $(($COUNT+2)) ]
do
CHECK=$(echo "$BLOCK" | sed $j'q;d')
PREF=${CHECK:0:5}
if [ "$PREF" != "\"RES:" ]; then
CHECK=$NEW$CHECK
BLOCK=$(echo "$BLOCK" | sed $j's/.*/'$CHECK'/')
fi
j=$(($j+1))
done
if [ "$COUNT" != 1 ]; then
echo "" >> $OUT
echo "\$" >> $OUT
echo "tag = \"gen."$i"\"" >> $OUT
echo "port = \""$PORT"\"" >> $OUT
echo "base = \"TCP\"" >> $OUT
echo "payloads:" >> $OUT
echo "$BLOCK" >> $OUT
echo "Generated conversation "$i
fi
i=$(($i+1))
done
echo "Generation complete!"
When I run this, I get the following error for each conversation read:
> sed: -e expression #1, char 18: unterminated `s' command
I believe the problem lies in the call to TShark on line 9. Originally I used the "raw" argument for the command, which outputs raw hex data. This worked and output correctly. However, my task requires outputting ASCII data. Changing "raw" to "ascii" (both recognized by TShark) causes the aforementioned errors. I believe this is because the ASCII data in the read packets contains special characters; a small piece of data generated by line 9 in command line is:
..7.<.......Y.|.$.......2...W...v.'#
My question is are the special characters in the ASCII data I'm parsing causing the sed errors? If so, how could I make bash ignore them? Thanks!
Edit- I am ultimately trying to get the output of this TShark command, which looks like this...
===================================================================
Follow: tcp,raw
Filter: tcp.stream eq 4
Node 0: 10.211.55.3:58733
Node 1: 157.127.239.146:80
47455420687474703a2f2f73656d696e617270726f6a656374732e6f72672f6373732e7068703f7374796c6573686565743d393620485454502f312e310d0a486f73743a2073656d696e617270726f6a656374732e6f72670d0a557365722d4167656e743a204d6f7a696c6c612f352e3020285831313b204c696e7578207838365f36343b2072763a33382e3029204765636b6f2f32303130303130312046697265666f782f33382e300d0a4163636570743a20746578742f6373732c2a2f2a3b713d302e310d0a4163636570742d4c616e67756167653a20656e2d55532c656e3b713d302e350d0a4163636570742d456e636f64696e673a20677a69702c206465666c6174650d0a526566657265723a20687474703a2f2f73656d696e617270726f6a656374732e6f72672f632f74736861726b2d666f6c6c6f772d7463702d73747265616d0d0a436f6f6b69653a205f5f6366647569643d646564613432383039663566623634356461663239333963366235336565653764313433373734383236323b206d7962625b6c61737476697369745d3d313433373734383333353b206d7962625b6c6173746163746976655d3d313433373734383333353b207369643d31663739303463373761383761656234363537306131636161316462336161310d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a0d0a
485454502f312e3120323030204f4b0d0a446174653a204672692c203234204a756c20323031352031343a33313a303420474d540d0a436f6e74656e742d547970653a20746578742f6373730d0a582d506f77657265642d42793a205048502f352e342e31360d0a5365727665723a20636c6f7564666c6172652d6e67696e780d0a43462d5241593a20323062303533396434326436313365332d4c41580d0a436f6e74656e742d456e636f64696e673a20677a69700d0a436f6e74656e742d4c656e6774683a203134320d0a4167653a20300d0a5669613a20312e31206e657070737730390d0a0d0a1f8b08000000000000036c8cbd0a03211084ebf52916ac13f2db689bcb6b04bd15919caeac060e42de3d981469325f37df305bcf4ee896436b2e067c2af06ebe47e14721837aba0eac8299171683faf88955e05928c8a6733578a82b365e12a1be9c063fefb977ceff27d511a5120d9eeb6a1564273195efe37e37aa970278030000ffff0300cc348afaa1000000
47455420687474703a2f2f7777772e676f6f676c652d616e616c79746963732e636f6d2f616e616c79746963732e6a7320485454502f312e310d0a486f73743a207777772e676f6f676c652d616e616c79746963732e636f6d0d0a557365722d4167656e743a204d6f7a696c6c612f352e3020285831313b204c696e7578207838365f36343b2072763a33382e3029204765636b6f2f32303130303130312046697265666f782f33382e300d0a4163636570743a202a2f2a0d0a4163636570742d4c616e67756167653a20656e2d55532c656e3b713d302e350d0a4163636570742d456e636f64696e673a20677a69702c206465666c6174650d0a526566657265723a20687474703a2f2f73656d696e617270726f6a656374732e6f72672f632f74736861726b2d666f6c6c6f772d7463702d73747265616d0d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a49662d4d6f6469666965642d53696e63653a205468752c203039204a756c20323031352032333a35303a353620474d540d0a0d0a
485454502f312e3120333034204e6f74204d6f6469666965640d0a446174653a204672692c203234204a756c20323031352031343a33303a353520474d540d0a457870697265733a204672692c203234204a756c20323031352031353a35313a343120474d540d0a43616368652d436f6e74726f6c3a207075626c69632c206d61782d6167653d373230300d0a566172793a204163636570742d456e636f64696e670d0a436f6e6e656374696f6e3a20636c6f73650d0a5669613a20312e31206e657070737730390d0a0d0a
===================================================================
...into a custom format for a program to read. The above output is in the working raw hex data format. The custom format looks like this for the corresponding conversation:
$
tag = "gen.4"
port = "58733"
base = "TCP"
payloads:
"REQ:47455420687474703a2f2f73656d696e617270726f6a656374732e6f72672f6373732e7068703f7374796c6573686565743d393620485454502f312e310d0a486f73743a2073656d696e617270726f6a656374732e6f72670d0a557365722d4167656e743a204d6f7a696c6c612f352e3020285831313b204c696e7578207838365f36343b2072763a33382e3029204765636b6f2f32303130303130312046697265666f782f33382e300d0a4163636570743a20746578742f6373732c2a2f2a3b713d302e310d0a4163636570742d4c616e67756167653a20656e2d55532c656e3b713d302e350d0a4163636570742d456e636f64696e673a20677a69702c206465666c6174650d0a526566657265723a20687474703a2f2f73656d696e617270726f6a656374732e6f72672f632f74736861726b2d666f6c6c6f772d7463702d73747265616d0d0a436f6f6b69653a205f5f6366647569643d646564613432383039663566623634356461663239333963366235336565653764313433373734383236323b206d7962625b6c61737476697369745d3d313433373734383333353b206d7962625b6c6173746163746976655d3d313433373734383333353b207369643d31663739303463373761383761656234363537306131636161316462336161310d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a0d0a"
"RES:485454502f312e3120323030204f4b0d0a446174653a204672692c203234204a756c20323031352031343a33313a303420474d540d0a436f6e74656e742d547970653a20746578742f6373730d0a582d506f77657265642d42793a205048502f352e342e31360d0a5365727665723a20636c6f7564666c6172652d6e67696e780d0a43462d5241593a20323062303533396434326436313365332d4c41580d0a436f6e74656e742d456e636f64696e673a20677a69700d0a436f6e74656e742d4c656e6774683a203134320d0a4167653a20300d0a5669613a20312e31206e657070737730390d0a0d0a1f8b08000000000000036c8cbd0a03211084ebf52916ac13f2db689bcb6b04bd15919caeac060e42de3d981469325f37df305bcf4ee896436b2e067c2af06ebe47e14721837aba0eac8299171683faf88955e05928c8a6733578a82b365e12a1be9c063fefb977ceff27d511a5120d9eeb6a1564273195efe37e37aa970278030000ffff0300cc348afaa1000000"
"REQ:47455420687474703a2f2f7777772e676f6f676c652d616e616c79746963732e636f6d2f616e616c79746963732e6a7320485454502f312e310d0a486f73743a207777772e676f6f676c652d616e616c79746963732e636f6d0d0a557365722d4167656e743a204d6f7a696c6c612f352e3020285831313b204c696e7578207838365f36343b2072763a33382e3029204765636b6f2f32303130303130312046697265666f782f33382e300d0a4163636570743a202a2f2a0d0a4163636570742d4c616e67756167653a20656e2d55532c656e3b713d302e350d0a4163636570742d456e636f64696e673a20677a69702c206465666c6174650d0a526566657265723a20687474703a2f2f73656d696e617270726f6a656374732e6f72672f632f74736861726b2d666f6c6c6f772d7463702d73747265616d0d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a49662d4d6f6469666965642d53696e63653a205468752c203039204a756c20323031352032333a35303a353620474d540d0a0d0a"
"RES:485454502f312e3120333034204e6f74204d6f6469666965640d0a446174653a204672692c203234204a756c20323031352031343a33303a353520474d540d0a457870697265733a204672692c203234204a756c20323031352031353a35313a343120474d540d0a43616368652d436f6e74726f6c3a207075626c69632c206d61782d6167653d373230300d0a566172793a204163636570742d456e636f64696e670d0a436f6e6e656374696f6e3a20636c6f73650d0a5669613a20312e31206e657070737730390d0a0d0a"
You can tell bash to not interpret metacharacters by quoting the variable expansion:
sed $j's/.*/'"$CHECK"'/'
In fact, there is no reason to use single quotes in the above, so you could just double-quote the entire command argument:
sed "${j}s/.*/$CHECK/"
However, neither of the above will tell sed to avoid interpreting special characters in the replacement part of the s command, so if $CHECK contains a /, then that will prematurely terminate the replacement.
So the question really is, is there a better way of accomplishing this:
BLOCK=$(echo "$BLOCK" | sed $j's/.*/'$CHECK'/')
Apparently, the goal is to replace line $j of the value of $BLOCK with the value of $CHECK. One way to do this, using awk:
BLOCK="$(awk -v repl="$CHECK" 'NR==$j{print repl;next}1')"
Notes:
Although I didn't fix it in my example, it is very bad style to use ALL CAPS for shell variables. Normally, shell variables in ALL CAPS are reserved for use as known exported variables by bash or system utilities (eg. $PATH; $IFS; $TERM; etc.). Your own variables should be lower-case to avoid conflicts.
The full loop that the command is excerpted from could probably be all implemented more efficiently and more cleanly (and more understandably) in awk. Based on the sample output, the following would probably work:
echo "Generating conversations..."
i=0
while
tshark -r */browser.pcap -q -z follow,tcp,ascii,$i |
awk -v idx=$i -v '
NR==4 { n = split($0, a, /:/); port = a[n]; }
NR<6 { next; }
/^=========/ { exit port != 0; }
port { print "$"
printf "tag = \"gen.%d\"" idx
print "port = \"%s\"" port
print "base = \"TCP\""
print "payloads:"
port = 0
}
/^\t/ { printf "\"RES:%s\"" substr($0, 2) "\""; next; }
{ printf "\"REQ:%s\"" $0 "\""; }
' >> $OUT;
do
echo "Generated conversation "$i
done
echo "Generation complete!"
I didn't try it. It may well be buggy. I don't understand the termination condition, so I just made a guess. I'm not sure if you really meant to extract the port number from line 5 (as in the code) or line 4 (as in the example.)

Resources