I have these 2 arrays representing hexadecimal numbers and I want to write to a file in binary format.
I convert to hexadecimal string like this:
a=["A2","48","04","03","EE","72","B4","6B"]
b=["1A","28","18","06","07","00","11","86","05","01","01","01","A0"]
hex_string1 = a.map{|b| b.to_i(16)}.pack("C*")
hex_string2 = b.map{|b| b.to_i(16)}.pack("C*")
Now I want to write to a file the hex_string2 first and then prepend (with offset "0") the hex_string1 to the file.
I'm proceeding like this but the output is incorrect.
File.binwrite("outfile.bin",hex_string2)
File.binwrite("outfile.bin",hex_string1,0)
The current output is:
A2 48 04 03 EE 72 B4 6B 05 01 01 01 A0
And the correct content within the "output.bin" would be like this:
A2 48 04 03 EE 72 B4 6B 1A 28 18 06 07 00 11 86 05 01 01 01 A0
How would be the way to do this?
You should write second string with the offset by the size of first string:
File.binwrite("outfile.bin",hex_string2,hex_string1.size)
File.binwrite("outfile.bin",hex_string1,0)
In this case you'll get exactly what you want:
A2 48 04 03 EE 72 B4 6B 1A 28 18 06 07 00 11 86 05 01 01 01 A0
Related
I'm triying to find a string in a txt format and each time it's found then look for an specific string to change for another string.
Imagine the next hexa txt:
02 11 86 05 01 01 01 a0 11 60 0f 80 02 07 80 a1
09 06 07 04 00 00 01 00 1d 03 8a 02 01 2a 02 01
b7 09 01 27 30 22 a0 0a 80 08 33 04 03 92 22 14
00 11 86 05 01 01 01 a0 11 60 0f 80 02 07 80 a1
09 06 07 04 00 00 01 00 1d 03 8a 02 01 2a 02 01
b7 09 01 27 30 22 a0 0a 80 08 33 04 03 92 22 14
I need that each time I encounter a 2a sequence to look for 09 01 sequence and replace with 03 02.
Expected output:
02 11 86 05 01 01 01 a0 11 60 0f 80 02 07 80 a1
09 06 07 04 00 00 01 00 1d 03 8a 02 01 2a 02 01
b7 03 02 27 30 22 a0 0a 80 08 33 04 03 92 22 14
00 11 86 05 01 01 01 a0 11 60 0f 80 02 07 80 a1
09 06 07 04 00 00 01 00 1d 03 8a 02 01 2a 02 01
b7 03 02 27 30 22 a0 0a 80 08 33 04 03 92 22 14
Im triying something this:
sed -i 's/09 01\(.*2a\)/03 02/g' packet.txt
I would do this with awk:
$ awk ' { for ( i = 1; i <= NF; ++i ) {
if ( $i == "2a" )
r = 1
if ( r && $i == "09" && $(i+1) == "01" ) {
r = 0
$i = "03"
$++i = "02"
}
}
}
1 ' hexa.txt > hexa.txt.modified
Grep the differences:
$ sdiff hexa.txt hexa.txt.modified
02 11 86 05 01 01 01 a0 11 60 0f 80 02 07 80 a1 02 11 86 05 01 01 01 a0 11 60 0f 80 02 07 80 a1
09 06 07 04 00 00 01 00 1d 03 8a 02 01 2a 02 01 09 06 07 04 00 00 01 00 1d 03 8a 02 01 2a 02 01
b7 09 01 27 30 22 a0 0a 80 08 33 04 03 92 22 14 | b7 03 02 27 30 22 a0 0a 80 08 33 04 03 92 22 14
00 11 86 05 01 01 01 a0 11 60 0f 80 02 07 80 a1 00 11 86 05 01 01 01 a0 11 60 0f 80 02 07 80 a1
09 06 07 04 00 00 01 00 1d 03 8a 02 01 2a 02 01 09 06 07 04 00 00 01 00 1d 03 8a 02 01 2a 02 01
b7 09 01 27 30 22 a0 0a 80 08 33 04 03 92 22 14 | b7 03 02 27 30 22 a0 0a 80 08 33 04 03 92 22 14
Assuming you mean: "only replace if it occurs after 2a", then you can do it by transforming the bytes, so that only one 2a occurs on each line, e.g.:
<hexa.txt tr '\n' ' ' | sed 's/2a/\n&/g'
Now all you need to do is only replace 09 01 when the line starts with 2a, e.g.:
sed -E 's/(^2a.*) 09 01/\1 03 02/'
Now go back to the original formatting, i.e. 16 bytes per line:
tr '\n' ' ' | xargs -n16
All together:
<hexa.txt tr '\n' ' ' | sed 's/2a/\n&/g' |
sed -E 's/(^2a.*) 09 01/\1 03 02/' |
tr '\n' ' ' | xargs -n16
Output:
02 11 86 05 01 01 01 a0 11 60 0f 80 02 07 80 a1
09 06 07 04 00 00 01 00 1d 03 8a 02 01 2a 02 01
b7 03 02 27 30 22 a0 0a 80 08 33 04 03 92 22 14
00 11 86 05 01 01 01 a0 11 60 0f 80 02 07 80 a1
09 06 07 04 00 00 01 00 1d 03 8a 02 01 2a 02 01
b7 03 02 27 30 22 a0 0a 80 08 33 04 03 92 22 14
If this can help,
cat *.txt | sed '/2a/s/09 01/02 03/g'
Alternative awk solution using GNU awk:
awk 'BEGIN { RS="2a" } { ORS=RS } $0 ~ /09 01/ { $0=gensub("09 01","03 02","g",$0)}1' file
Set 2a as the record separator. Check each record for "09 01". If it exists, replace "09 01" with "03 02" with the gensub function and set this as $0. Use short hand 1 to print the record after setting the output record separator the same as the record separator.
This might work for you (GNU sed):
sed -zE 's/^/\x00/ # introduce a unique delimiter
:a;/\x00$/{s///;b} # remove delimiter at end-of-file
/\x002a/!{s/\x00(.)/\1\x00/;ba} # if not 2a pass over next char
s//2a\x00/ # next char is 2a prep for next string
:b;/\x00$/ba # is it end of file
/\x0009(\s)01/{s//03\102\x00/;ba} # replace string and prep for 2a again
s/\x00(.)/\1\x00/;bb' file # not desired string so pass over char
Since the desired string (in this case09 01) may occur on another line or within in the same line twice or more, line processing is not feasible. Processing must be at character level and in this solution the entire file is processed as one string (see -z option).
Two cases are identified:
The key (in this case 2a), processing within the place holder :a.
The string to be replaced (09 01 with 03 02), processing within the place holder :b.
Once the key is identified, processing passes to the next case. Once the desired string has been replaced, processing is passed back the first case. Either case can terminate processing when the end-of-file is encountered.
N.B. The solution relies on the file not containing the null character hex 00.
I am currently using C# to retrieve frames from a borescope (via the FFMPEG library). However, I came across a problem weeks ago and I can't solve it.
The images are returned in JPEG format (since the borescope stream is MJPEG).
Some images come without quality problems, but others come with a strange line in the middle
followed by random staining. (At the end of the question there is an example of a normal image and one with problems).
Analyzing the structure of the files, I realized that there are some differences, but I don't really understand JPEG's binary structure very well, and I can't tell what is corrupted.
Getting to know what is corrupted in the image, which culminates in the quality problem, is very important to me because, through this, I can discard the frame using C#. However, without understanding this problem, I can't even discard the frame, much less fix it.
So, having the image without quality problems as a reference, what is the problem with the binary structure of the image with quality problems?
Examples:
JPEG 1: Image without quality problems
Image's preview (just to see the quality, do not download from here)
JPEG 2: Image with quality problems
Image's preview (just to see the quality, do not download from here)
It's possible to look into binary structure of images through online HEX editors like: Online hex editor, Hexed or Hex-works.
Thank you for reading and have a nice day.
There are at least 2 issues with the file.
The first I can detect with ImageMagick by running this command:
magick identify -verbose image.jpg
and it tells me that the data segment ends prematurely.
Image: outExemplo0169.jpeg
Format: JPEG (Joint Photographic Experts Group JFIF format)
Mime type: image/jpeg
Class: DirectClass
Geometry: 640x480+0+0
Units: Undefined
Colorspace: sRGB
Type: TrueColor
Base type: Undefined
Endianess: Undefined
Depth: 8-bit
Channel depth:
Red: 8-bit
Green: 8-bit
Blue: 8-bit
Channel statistics:
Pixels: 307200
Red:
min: 0 (0)
max: 255 (1)
mean: 107.234 (0.420527)
standard deviation: 66.7721 (0.261851)
kurtosis: -0.67934
skewness: 0.577494
entropy: 0.92876
Green:
min: 0 (0)
:2020-02-26T18:59:19+00:00 0:00.057 0.070u 7.0.9 Resource identify[80956]: resource.c/RelinquishMagickResource/1067/Resource
Memory: 3686400B/0B/32GiB
identify: Corrupt JPEG data: premature end of data segment `outExemplo0169.jpeg' # warning/jpeg.c/JPEGWarningHandler/399.
The second I can see with exiftool when I run this command:
exiftool -v -v -v outExemplo0169.jpeg
ExifToolVersion = 11.11
FileName = outExemplo0169.jpeg
Directory = .
FileSize = 66214
FileModifyDate = 1582743337
FileAccessDate = 1582743559
FileInodeChangeDate = 1582743337
FilePermissions = 33188
FileType = JPEG
FileTypeExtension = JPG
MIMEType = image/jpeg
JPEG APP0 (14 bytes):
0006: 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 [JFIF..........]
+ [BinaryData directory, 9 bytes]
| JFIFVersion = 1 1
| - Tag 0x0000 (2 bytes, int8u[2]):
| 000b: 01 01 [..]
| ResolutionUnit = 0
| - Tag 0x0002 (1 bytes, int8u[1]):
| 000d: 00 [.]
| XResolution = 1
| - Tag 0x0003 (2 bytes, int16u[1]):
| 000e: 00 01 [..]
| YResolution = 1
| - Tag 0x0005 (2 bytes, int16u[1]):
| 0010: 00 01 [..]
| ThumbnailWidth = 0
| - Tag 0x0007 (1 bytes, int8u[1]):
| 0012: 00 [.]
| ThumbnailHeight = 0
| - Tag 0x0008 (1 bytes, int8u[1]):
| 0013: 00 [.]
JPEG SOF0 (15 bytes):
0018: 08 01 e0 02 80 03 01 21 00 02 11 01 03 11 01 [.......!.......]
ImageWidth = 640
ImageHeight = 480
EncodingProcess = 0
BitsPerSample = 8
ColorComponents = 3
JPEG DQT (130 bytes):
002b: 00 03 03 03 03 03 03 04 03 03 03 04 04 04 05 06 [................]
003b: 09 06 06 05 05 06 0c 08 09 07 09 0e 0c 0e 0e 0d [................]
004b: 0c 0d 0d 0f 11 15 12 0f 10 14 10 0d 0d 13 19 13 [................]
005b: 14 16 17 18 18 18 0f 12 1a 1c 1a 17 1c 15 17 18 [................]
006b: 17 01 04 04 04 06 05 06 0b 06 06 0b 17 0f 0d 0f [................]
007b: 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 [................]
008b: 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 [................]
[snip 18 bytes]
JPEG DHT (416 bytes):
00b1: 00 00 01 05 01 01 01 01 01 01 00 00 00 00 00 00 [................]
00c1: 00 00 01 02 03 04 05 06 07 08 09 0a 0b 10 00 02 [................]
00d1: 01 03 03 02 04 03 05 05 04 04 00 00 01 7d 01 02 [.............}..]
00e1: 03 00 04 11 05 12 21 31 41 06 13 51 61 07 22 71 [......!1A..Qa."q]
00f1: 14 32 81 91 a1 08 23 42 b1 c1 15 52 d1 f0 24 33 [.2....#B...R..$3]
0101: 62 72 82 09 0a 16 17 18 19 1a 25 26 27 28 29 2a [br........%&'()*]
0111: 34 35 36 37 38 39 3a 43 44 45 46 47 48 49 4a 53 [456789:CDEFGHIJS]
[snip 304 bytes]
JPEG SOS
JPEG EOI
Unknown trailer (50 bytes at offset 0x10274):
10274: 42 6f 75 6e 64 61 72 79 45 42 6f 75 6e 64 61 72 [BoundaryEBoundar]
10284: 79 53 00 00 01 00 90 08 01 00 fb 4b db 6a 2a 22 [yS.........K.j*"]
10294: 00 00 2a 22 00 00 01 00 01 00 80 02 00 00 e0 01 [..*"............]
102a4: 00 00
So there are 50 extraneous bytes at the end including the text string "BoundaryEBoundaryS" which may be recognisable to you as coming from somewhere else in your processing chain?
One test you could do for JPEG quality is check the last 2 bytes are a valid EOI which means it should end in FF D9 - see here.
In proto stream when decoding the file size and data block length in varint, the data block length is bigger than the file size? Please Help
the data block length is bigger than the file size
That sounds unlikely, and is most likely the result of (any one of):
a broken encoder (very unlikely if you're using any of the established implementations, but not impossible)
corruption of the data-stream in transit (the most common variant of this being: using a text-encoding such as UTF-8 backwards to try to get a string - when if a string is really needed, something like base-16 (hex) or base-64 should be used)
accidentally truncating a stream prematurely
a broken decoder
It sounds like you're parsing the stream manually, so frankly my guess is the last option. If you can post the bytes you're decoding we can probably advise on whether your interpretation is correct. Alternatively, you can try pushing your data through https://protogen.marcgravell.com/decode which will pull apart the data-stream and show how it has interpreted the bytes.
00 E4 02 00 A5 0A F0 3E 1E 08 C9 3F 12 19 08 05 12 03 01 00 05 18 75 2A 0E AE 44 93 42 AF 44 BF 3F B2 44 B1 44 B0 44 0A 03 08 93 42 22 33 12 31 42 2F 0A 0A 54 72 61 6E 73 69 74 69 6F 6E 12 04 6E 6F 6E 65 19 00 00 01 02 08 F0 3F 29 01 07 01 01 54 30 00 58 EB D4 FF AA 07 80 01 00 2A 03 08 AF 44 32 03 08 AE 44 3A 0D 05 78 AF 44 8A 01 03 08 BF 3F 98 01 01 A2 01 03 08 B0 44 DA 01 03 08 B2 44 A2 02 03 08 B1 44 D2 02 01 26 01 06 28 AF 44 1F 08 AE 44 12 1A 08 07 12 05 94 A0 C3 01 22 08 0A 04 0A 02 01 06 18 01 2A 04 B3 44 DC 3B 0A BE 01 0A AF 01 0A 3E 0A 1F 0A 0A 0D 00 00 C8 42 15 00 00 5A 43 12 01 0C E0 4E 44 15 00 00 E1 43 18 03 25 00 00 00 00 12 03 08 C9 3F 1A 12 08 04 10 02 18 01 25 00 00 40 41 2D 00 00 00 3F 30 00 28 00 38 00 12 03 08 DC 3B 1A 68 08 00 10 00 2A 62 12 1D 3D 18 1A 54 0A 0E 08 01 12 01 12 08 00 00 15 01 48 0C 0A 0E 08 02 05 10 05 5F 00 00 3A 10 00 04 E1 43 15 20 05 30 14 E1 43 0A 02 08 05 3E 44 00 14 12 03 08 B3 44 22 01 05 30 30 01 10 03 15 08 B3 44 12 10 08 D1 0F 09 E4 F0 4A 6C 2A 04 83 42 C7 41 12 03 08 A8 3F 1A 1F 41 66 74 65 72 20 4B 65 79 6E 6F 74 65 20 38 3F 29 0A 62 75 6C 6C 65 74 20 77 6F 6D 62 61 74 2A 0D 0A 07 08 00 12 03 08 83 42 0A 02 08 12 32 08 0A 06 08 00 10 00 18 00 3A 09 0A 07 08 01 69 10 C7 41 50 01 72 15 17 04 9A 01 05 0B 10 12 02 65 6E C2 09 0B 18 10 00 18 00 1F 08 AF 62 65 01 0C C0 41 B5 44 4E 65 01 04 A0 41 39 06 00 40 9E 65 01 08 C0 41 1A 42 65 01 00 40 C2 65 01 00 40 3A 65 01 00 40 5E 65 01 00 B5 21 65 00 B5 21 65 0C 02 15 08 B5 2E 65 01 18 4F 2A 04 AE 3B A2 40 29 65 08 06 57 6F 25 4C 31 33 08 A2 40 32 25 26 21 1B 00 3A 11 15 04 AE 3B 86 48 01 18 12 08 B2 44 12 0D 08 2D C9 0C 05 2A 02 B4 61 35 10 B4 44 15 08 B4 2E 7D 00 00 3E 01 7D 0C AC 3B 08 04 05 7F 15 77 04 AC 3B 82 77 00 3D B4 20 0F 08 B1 44 12 0A 08 E7 17 49 36 0C 00 1F 08 B0 62 C4 01 08 DB 40 B6 3E C4 01 18 00 44 15 00 80 3D 44 25 C4 20 48 42 15 00 00 E8 41 18 00 92 29 03 04 DB 40 32 C4 01 0D 3D 6A 29 03 05 22 7D 29 05 10 04 E8 41 36 29 03 04 E8 41 5A 29 03 00 B6 21 C4 00 B6 21 C4 18 01 17 08 B6 44 12 12 75 29 20 4C 2A 06 AE 3B C0 40 B7 44 25 47 10 1A 03 EF BF BC 35 4C 00 C0 56 C3 01 00 4A 31 CE 04 B7 44 2E CE 01 32 57 01 00 B7 21 57 00 FB 2D C4 44 11 0A 04 0A 00 10 00 10 00 22 07 64 65 63 69 6D 61 6C
Decoding the first six bytes
I am writing some code to send electronic identification cards using ISO-7816:
If I send a "SELECT FILE" command:
INFO: Send command PC -> SAM: 00 a4 04 00 0f a0 00 00 00 18 43 4d 08 09 0a 0b 0c 00 00 00
INFO: Receive from SAM -> PC: 6F 62 84 0F A0 00 00 00 18 43 4D 08 09 0A 0B 0C 00 00 00 A5 4F 73 49 06 07 2A 86 48
86 FC 6B 01 60 0B 06 09 2A 86 48 86 FC 6B 02 02 02 63 09 06 07 2A 86 48 86 FC 6B 03
64 0B 06 09 2A 86 48 86 FC 6B 04 02 55 64 0B 06 09 2A 86 48 86 FC 6B 04 80 00 66 0C
06 0A 2B 06 01 04 01 2A 02 6E 01 03 9F 65 01 FF
After that, I send an "INITIALIZE UPDATE" command
-> 80 50 20 00 08 81 C3 21 A7 9D 7A DE 3E
And the response is
<- 69 82
[ERR] Smartcard::Iso::ApduError: ISO-7816 response APDU has error status 0x6982
I don't understand why I'm getting that response.
Well, 6982 means "access condition not fulfilled", depending on the how the card was created this could mean that that you need to verify with CHV or ADM key.
I've successfully been able to retrieve the card number and expiry date from a contactless debit/credit card. However, the cardholder name is not being returned in the READ RECORD command response. Am I missing a something?
- Select Application
# IN_DATA_EXCHANGE
>> D4 40 01 00 A4 04 00 07 A0 00 00 00 03 10 10 00
<< D5 41 00 6F 43 84 07 A0 00 00 00 03 10 10 A5 38 50 10 56 69 73 61 20 20 20 20 20 20 20 20 20 20 20 20 9F 38 18 9F 66 04 9F 02 06 9F 03 06 9F 1A 02 95 05 5F 2A 02 9A 03 9C 01 9F 37 04 BF 0C 08 9F 5A 05 31 08 26 08 26 90 00
- Read the card
# IN_DATA_EXCHANGE
>> D4 40 01 00 B2 01 0C 00
<< D5 41 00 70 12 57 10 XX XX XX XX XX XX XX XX D1 50 52 01 00 00 00 01 90 00
It's not uncommon for an EMV payment card to not reveal the cardholder name over the contactless interface. In fact, all major brands have introduced this as a privacy feature. On many cards the cardholder name field (tag 5F20) is present but filled with a string like " /" to indicate that the cardholder name is not to be revealed. At least for Visa cards (like the one you have above) the cardholder name field is not mandatory (and if its not present, its value should be assumed as " /"). You might want to also check other records/files on the card. Some cards also provide this field in response to the GET PROCESSING OPTIONS command only.