reading hexdump file byte by byte in shell/awk - shell

I am new to bash/awk programming and need your help soon
This is hexadecimal ouptut of an ipv6 tcp packet captured through tcpdump -
6000 0000 0018 0620 0000 0000
0000 0000 0000 0000 0000 0001 0000 0000
0000 0000 0000 0000 0000 0002 *0026 0026
0000 0001 0000 0002 {5}412 0065 0034 0000*
6162 6364
The {5} specify the tcp packet length in 32-bit words and tcp packet is b/w *s.
(* and {} are used to clear the question ,they are not
present actually in output)
Now I need to extract tcp header from this info using bash/awk script.
I think I need to read file byte by byte upto tcp specified length but don't know how to do this.
So any idea how can I extract tcp information from the output specified above assume It has been written in a file.
Thanx in advance ..

Use the awk feature that allows you to set what characters are used for the RecordSeparator (RS) and OutputRecordSeperator (ORS), and use the NumberRecord (NR) var to monitor which part of the rec you print.
awk '{BEGIN RS=" ";ORS="\n";} { if (NR >20 && NR < 31) print $0}' hexFile
If you give more detail about your file (how does repeating sets of this data look?) and expected output, I can probably make this even more efficient for you.
I hope this helps.

Related

Download and start sh.x and .run files with curl or wget

Normally I start the bash programs from the terminal, with this command:
curl https://example.com/assets/file/file.sh | bash
But if i do the same thing with .run or .sh.x file it doesn't work.
I have tried in two different ways:
curl https://example.com/assets/file/file.sh.x | bash
Or
wget -O - https://example.com/assets/file/file.run | bash
I see a series of syntax errors or ascii codes in the terminal that do not allow the script to start.
On the web I have not found anything that can help me solve the problem.
Is it possible to solve this, or does the file extension not allow it?
wget output of file.sh.x file:
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘STDOUT’
- [ <=> ] 11.94K --.-KB/s in 0s
2020-03-28 16:03:19 (43.6 MB/s) - written to stdout [12224]
bash: line 1: syntax error near unexpected token `)'
bash: line 1: `ELF>#)#8 ####888 '
curl output of file.sh.x file:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12224 0 12bash: line 1: syntax error near unexpected token `)':-- 0
2bash: line 1: `ELF>#)#8 ####888 '
24 0 0 39432 0 --:--:-- --:--:-- --:--:-- 39432
(23) Failed writing body
file.sh.x content is something like:
...
7f45 4c46 0201 0100 0000 0000 0000 0000
0300 3e00 0100 0000 d00b 0000 0000 0000
4000 0000 0000 0000 0029 0000 0000 0000
0000 0000 4000 3800 0900 4000 1b00 1a00
0600 0000 0400 0000 4000 0000 0000 0000
4000 0000 0000 0000 4000 0000 0000 0000
f801 0000 0000 0000 f801 0000 0000 0000
0800 0000 0000 0000 0300 0000 0400 0000
3802 0000 0000 0000 3802 0000 0000 0000
3802 0000 0000 0000 1c00 0000 0000 0000
1c00 0000 0000 0000 0100 0000 0000 0000
...
wget output of file.run:
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-makeself]
Saving to: ‘STDOUT’
- [ <=> ] 12.86K --.-KB/s in 0s
2020-03-28 16:08:50 (44.6 MB/s) - written to stdout [13173]
Verifying archive integrity...head: cannot open 'bash' for reading: No such file or directory
main: line 80: bash: No such file or directory
Error in MD5 checksums: d41d8cd98f00b204e9800998ecf8427e is different from 42a127ab90433f4d3fabc166a4e0ca24
curl output of file.run:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 13173 0 13173 0 0 32932 0 --:--:-- --:--:-- --:--:-- 32932
Verifying archive integrity...head: cannot open 'bash' for reading: No such file or directory
main: line 80: bash: No such file or directory
Error in MD5 checksums: d41d8cd98f00b204e9800998ecf8427e is different from 42a127ab90433f4d3fabc166a4e0ca24
the content of the file.run file is quite long, there is all the initialization of the .run file and at the end a long list of ascii codes.
Run directly on the server they all work.
Using wget or curl commands don't work
Based on the response received, is there no way to automatically start sh.x / run files, as is the case with bash scripts by adding "| bash" to the end of the command?
NOTE: You really should be very careful about piping a sequence of unknown commands from a website into your shell!!!
The file you are trying to get bash to run is an ELF binary, i.e. a compiled executable program, not a shell script. You can check like this:
curl https://example.com/assets/file/file.sh.x > /tmp/unknown
file /tmp/unknown

BaseOfCode present in PE+ executable

The MS documentation says that the BaseOfCode value is only present in PE files, not in PE+. Looking at notepad.exe with dotPeek and with PE Viewer seems to indicate that the BaseOfCode is present and consumed.
0 1 2 3 4 5 6 7 8 9 A B C D E F
0x00E0 | 5045 0000 6486 0600 6a98 8957 0000 0000
0x00F0 | 0000 0000 f000 2200 0b02 0e00 0086 0100
0x0100 | 004e 0200 0000 0000 d087 0100 0010 0000
The two bytes at 0x00F8 signify that this is a PE+ header. The BaseOfCode is the four bytes at 0x010C.
Is the documentation (and myself) incorrect or are dotPeek and PE View
incorrect?
The fact that these bytes aren't zeroed out would imply that it the bytes are significant in some way.

How to generate 2 pixel BMP image with Awk?

sorry for the silly question, but I'm trying to create (generate) a simple BMP image 2x1, 24-bit bitmap with pixel format RGB24, using Awk.
Format is "BitmapFileHeader (2+4+4+4=14 bytes) + DIBHeader (4+4+4+2+2+4+4+4+4+4+4=40 bytes) = 54byte", then start of pixel array (bitmap data). Here is my simple script:
BEGIN {
ORS="";
filebmp="Image.bmp"
# BMP Header: 2+4+4+4=14 bytes
printf("%c%c",66,77)>filebmp;
printf("%c%c%c%c",62,0,0,0)>filebmp;
printf("%c%c%c%c",0,0,0,0)>filebmp;
printf("%c%c%c%c",54,0,0,0)>filebpm;
# DIB Header: 4+4+4+2+2+4+4+4+4+4+4=40 bytes
printf("%c%c%c%c",40,0,0,0)>filebmp;
printf("%c%c%c%c",2,0,0,0)>filebmp;
printf("%c%c%c%c",1,0,0,0)>filebmp;
printf("%c%c",1,0)>filebmp;
printf("%c%c",24,0)>filebmp;
printf("%c%c%c%c",0,0,0,0)>filebmp;
printf("%c%c%c%c",8,0,0,0)>filebmp;
printf("%c%c%c%c",19,11,0,0)>filebmp;
printf("%c%c%c%c",19,11,0,0)>filebmp;
printf("%c%c%c%c",0,0,0,0)>filebmp;
printf("%c%c%c%c",0,0,0,0)>filebmp;
# start of pixel array (bitmap data)
# Blu pixel
printf ("%c%c%c",127,0,0)>filebmp;
# Green pixel
printf ("%c%c%c",0,127,0)>filebmp;
# Padding
printf ("%c%c",0,0)>filebmp;
}
it's works (image below): it generates a BMP image 2x1 and 62 byte size.
http://i60.tinypic.com/mlihyt.jpg
But, now, if in the bitmap data I substitute 127 value with 128 (e.g. in blue pixel), same script Awk generates a BMP image 2x1 but 63 byte size and with "other colours" (see image below)!
http://i62.tinypic.com/2ltkjlg.png
I wonder where I am going wrong...
...any ideas?
Thanks & Bye
I was able to reproduce the issue with gawk. It seems that gawk chooses the output encoding based on your LANG environment variable. (Here I've change both instances of 127 to 128.)
$ LANG=en_US.LATIN1 gawk -f bit.awk
$ wc -c Image.bmp
62 Image.bmp
00000000: 424d 3e00 0000 0000 0000 3600 0000 2800 BM>.......6...(.
00000010: 0000 0200 0000 0100 0000 0100 1800 0000 ................
00000020: 0000 0800 0000 130b 0000 130b 0000 0000 ................
00000030: 0000 0000 0000 8000 0000 8000 0000 ..............
$ LANG=en_US.UTF-8 gawk -f bit.awk
$ wc -c Image.bmp
64 Image.bmp
00000000: 424d 3e00 0000 0000 0000 3600 0000 2800 BM>.......6...(.
00000010: 0000 0200 0000 0100 0000 0100 1800 0000 ................
00000020: 0000 0800 0000 130b 0000 130b 0000 0000 ................
00000030: 0000 0000 0000 c280 0000 00c2 8000 0000 ................
Run gawk under a single-byte encoding and you should be fine.

Use bitwise AND with string on ruby on rails

I want use bitwise operator "&" with string like that :
raw_counter_int = raw_counter.to_i
raw_counter_bin = raw_counter_int.to_s(2)
u = (2**62 + 2**63)
k = u.to_s(2)
r = raw_counter_bin & k
#counter_msg = r
but when I run my application I've this error message :
undefined method `&' for "10000000000000000000000000000000000000000000000000000000":String
How I can use this operator "&" with raw_counter_int and u which are converted in binary ?
I try with this: 0000 0000 1000 0000 0000 0000 0000 0000 (64 bits) to
take bytes between the third bytes and the 10th bytes. So I want do a
bitwise "&" with 0000 0000 1000 0000 0000 0000 0000 0000 & 0011 1111
1100 0000 0000 0000 0000 0000 to take just this : 00 0000 10
I try with this: 0000 0000 1000 0000 0000 0000 0000 0000 (64 bits) to
take bytes between the third bytes and the 10th bytes. So I want do a
bitwise "&" with 0000 0000 1000 0000 0000 0000 0000 0000 & 0011 1111
1100 0000 0000 0000 0000 0000 to take just this : 00 0000 10
Let's do it:
("00000000100000000000000000000000".to_i(2) & "00111111110000000000000000000000".to_i(2)).to_s(2)
=> "100000000000000000000000"
Which is exactly what is expected! The number shown in the error ("10000000000000000000000000000000000000000000000000000000") is 2^56, which, when using bitwise AND with it and 2^62+2^63 is expected to give you a zero result...
I suggest you check your input again, and trust ruby's & to do the job...

awk instead of sed is not working (slow, laggy and wrong output)

Why is awk not working for the following command instead of sed:
su -c "stdbuf -i0 -o0 -e0 od --width=144 -x /dev/input/event3 | sed 's%^\([a-z0-9]\+ \)\{11\}%%;s%\(....\).*%\1%'"
This prints at every keypress the USB HID ID number of the key that was pressed.
Example output (0028 is for Return and 00e4 for Right-Ctrl):
0028
0028
0028
0028
0028
00e4
00e4
00e4
This are two lines of output without sed/cut/awk filtering when pressing Return two times. The USB HID ID number is in column 12:
0000000 2d6f 511e 0000 0000 051b 0007 0000 0000 0004 0004 0028 0007 2d6f 511e 0000 0000 051d 0007 0000 0000 0001 001c 0000 0000 2d6f 511e 0000 0000 051e 0007 0000 0000 0000 0000 0000 0000 2d73 511e 0000 0000 a150 0007 0000 0000 0004 0004 0028 0007 2d73 511e 0000 0000 a153 0007 0000 0000 0001 001c 0001 0000 2d73 511e 0000 0000 a154 0007 0000 0000 0000 0000 0000 0000
0000220 2d73 511e 0000 0000 9b5a 0008 0000 0000 0004 0004 0028 0007 2d73 511e 0000 0000 9b5d 0008 0000 0000 0001 001c 0000 0000 2d73 511e 0000 0000 9b5e 0008 0000 0000 0000 0000 0000 0000 2d74 511e 0000 0000 4f90 0005 0000 0000 0004 0004 0028 0007 2d74 511e 0000 0000 4f93 0005 0000 0000 0001 001c 0001 0000 2d74 511e 0000 0000 4f94 0005 0000 0000 0000 0000 0000 0000
I tried it with
su -c "stdbuf -i0 -o0 -e0 od --width=144 -x /dev/input/event3 | cut -d' ' -f12"
and it also works. But it only shows the keys pressed after the next two keypresses. stdbuf did not help here, although it should adjusts standard input/output/error stream buffering to 0.
My mawk command looked like this:
su -c "stdbuf -i0 -o0 -e0 od --width=$((48*3)) -x /dev/input/event3 | mawk '{ print $12 }'"
but it only showed me some twos and then some newlines and again twos. It seems very laggy/slow. When I press a key, nothing happens. After pressing different keys multiple times, I always get a bunch of twos and newlines. No matter which keys I pressed! Example:
2
2
2
2
2
2
2
2
2
2
2
2
How can I fix this? Why is this occurring?
Update
When I use gawk instead of mawk it does not lag any more (unbuffered). But I still don’t see the correct values which should be 0028 or 00e4 and so on for Return and Right-Ctrl as example keys.
With all the comments I came to the solution.
#Olivier Dulac brought me to the idea that there are different awk implementations, which solved the problem of the laggyness (using gawk instead of mawk). I don’t even need the stdbuf.
#Ed Morton’s idea brought me to a problem, because he used quotes " in his example. I already had quotes in my statement around the whole thing, because it was executed using su. So I had to escape them.
My whole statement looks like this (notice the escaped dollar sign in front of the 12):
su -c "od --width=144 -x /dev/input/event3 | awk '{ print \$12 }'"
and now it works as expected.

Resources