I am working on unpacking a binary file for the first time in Ruby. Already found the unpack method which works pretty nice. Which according to the docs works perfect for 8(1 byte),16(2 byte),32(4 byte) and 64 bit(8 byte).
But now I have to unpack 5 bytes. How do I do this?
Thx in advance!
To literally unpack five bytes: str.unpack 'C5'
That gives you five byte values as unsigned ints. The question is how to reinterpret those ints as a single data type. Pack/unpack only recognize the standard power of two sizes, so you'll have to do that part manually.
For example, to get a little endian unsigned 40-bit int
bytes = str.unpack 'C5'
int = bytes.map.with_index { |byte, i| byte << (i * 8) }.reduce(:+)
If you need to do something more sophisticated like a signed type or a float... good luck.
Related
I am developing a software for using a RFID reader with ruby on rails and, after open the socket and get the tags, I convert data to hexadecimal with:
while line = s.gets
puts line.unpack('H*').to_s
end
Then I get "a55a0019833400393939393939303030303232fd6f02080d0a" for one tag.
The RFID reader user manual tells:
Remark:RSSI express as complement code, total 16 bits，which is 10 times the real value. For example, the real value is -65.7dBm，then RSSI=fd6f
I have found online calculators (mathsinfun and calc.penjee.com) where I am able to convert the fd6f in -675.
I would like to know how can I get this conversion in Ruby 2.3.1 to continue with my project.
Any help will be appreciated.
s> is the correct unpack symbol for a 16-bit unsigned big endian number, so:
"\xfd\x6f".unpack('s>')[0] / 10.0
Result is:
-65.7
I have a blob in postgresql database. Have inserted a C structure into it.
struct temp {
uint64_t a,
uint64_t b,
uint64_t c
};
Now when I write q query in shell for retrieving it.
select resource,.....,blob_column from rtable where rId is=1
I got the result as a blob from database. the result is
x00911ee3561ac801cb0783462586cf01af00000000000000
But now in shell script I need to iterate on this and display the result on console. Tried different things like awe,split , convert_from ,convert function but nothing is helping me.
Can someone tell me how can I read this hex string and get back the integers?
Is this some kind of exersise in programmer-torture? I can't imagine why you'd possibly do this. Not least because your struct-as-a-blob could be subject to padding and alignment that will vary from compiler to compiler and platform to platform. Even then, it'll vary between architectures because of endianness differences. At least you used fixed-width types.
Assuming you only care about little-endian and your compilers don't add any padding or alignment (likely for a struct that's just 3 64-bit fields) it's possible. That doesn't make a great idea.
My preferred approach would be to use some Python code with struct, e.g.
python - "x00911ee3561ac801cb0783462586cf01af00000000000000" <<__END__
import sys
import struct
print "{} {} {}".format(*struct.unpack('#QQQ', sys.argv[1][1:].decode("hex")))
__END__
as this can even handle endianness and packing using appropriate modifiers, and you can easily consume the output in a shell script.
If that's not convenient/suitable, it's also possible in bash, just absolutely horrible. For little-endian, unpadded/packed-unaligned:
To decode each value (adapted from https://stackoverflow.com/a/3678208/398670):
$ x=00911ee3561ac801
$ echo $(( 16#${x:14:2}${x:12:2}${x:10:2}${x:8:2}${x:6:2}${x:4:2}${x:2:2}${x:0:2} ))
so, for the full deal:
x=x00911ee3561ac801cb0783462586cf01af00000000000000
uint64_dec() {
echo $(( 16#${1:14:2}${1:12:2}${1:10:2}${1:8:2}${1:6:2}${1:4:2}${1:2:2}${1:0:2} ))
}
uint64_dec ${x:1:16}
uint64_dec ${x:17:16}
uint64_dec ${x:33:16}
produces:
128381549860000000
130470408871937995
175
Now, I feel dirty and need to go wash. I strongly suggest the following:
CREATE TYPE my_struct AS (a numeric, b numeric, c numeric);
then using my_struct instead of a bytea field. Or just use three numeric columns. You can't use bigint because Pg doesn't have a 64-bit unsigned integer.
I would like to be able to take an arbitrary string, run it through a hashing function (like MD5), and then interpret the resulting digest in base-36.
I know there already exists a Digest library in Ruby, but as far as I can tell I can't get at the raw bytes of a digest; the to_s function is mapped to hexdigest, which is, of course, base-16.
Fixnum#to_s accepts a base as the argument. So does string#to_i. Because of this, you can convert from the base-16 string to an int, then to base 36 string:
i = hexstring.to_i(16)
base_36 = i.to_s(36)
You can access the raw digest bytes using Digest::Class#digest:
Digest::SHA1.digest("test")
# => "\xA9J\x8F\xE5\xCC\xB1\x9B\xA6\x1CL\bs\xD3\x91\xE9\x87\x98/\xBB\xD3"
Unfortunately from that point I'm not sure how to get it into base36 without first going via another number base like in Sammy Larbi's answer..
bytes = Digest::SHA1.digest("test")
Digest.hexencode(bytes).to_i(16).to_s(36)
Hopefully you can find a better way to go from raw bytes to base36.
I am attempting to decrypt a number encrypted by another program that uses the BouncyCastle library for Java.
In Java, I can set the key like this: key = Hex.decode("5F3B603AFCE22359");
I am trying to figure out how to represent that same step in Ruby.
To get Integer — just str.hex. You may get byte array in several ways:
str.scan(/../).map(&:hex)
[str].pack('H*').unpack('C*')
[str].pack('H*').bytes.to_a
See other options for pack/unpack and examples (by codeweblog).
For a string str:
"".tap {|binary| str.scan(/../) {|hn| binary << hn.to_i(16).chr}}
I'm trying to read unsigned integers from a file (stored as consecutive byte) and convert them to Integers. I've tried this:
file = File.new(filename,"r")
num = file.read(2).unpack("S") #read an unsigned short
puts num #value will be less than expected
What am I doing wrong here?
You're not reading enough bytes. As you say in the comment to tadman's answer, you get 202 instead of 3405691582
Notice that the first 2 bytes of 0xCAFEBABE is 0xCA = 202
If you really want all 8 bytes in a single number, then you need to read more than the unsigned short
try
num = file.read(8).unpack("L_")
The underscore is assuming that the native long is going to be 8 bytes, which definitely is not guaranteed.
How about looking in The Pickaxe? (Ruby 1.9, p. 44)
File.open("testfile")
do |file|
file.each_byte {|ch| print "#{ch.chr}:#{ch} " }
end
each_byte iterates over a file byte by byte.
There are a couple of libraries that help with parsing binary data in Ruby, by letting you declare the data format in a simple high-level declarative DSL and then figure out all the packing, unpacking, bit-twiddling, shifting and endian-conversions by themselves.
I have never used one of these, but here's two examples. (There are more, but I don't know them):
BitStruct
BinData
Ok, I got it to work:
num = file.read(8).unpack("N")
Thanks for all of your help.
What format are the numbers stored in the file? Is it in hex? Your code looks correct to me.
When dealing with binary data you need to be sure you're opening the file in binary mode if you're on Windows. This goes for both reading and writing.
open(filename, "rb") do |file|
num = file.read(2).unpack("S")
puts num
end
There may also be issues with "endian" encoding depending on the source platform. For instance, PowerPC-based machines, which include old Mac systems, IBM Power servers, PS3 clusters, or Sun Sparc servers.
Can you post an example of how it's "less"? Usually there's an obvious pattern to the data.
For example, if you want 0x1234 but you get 0x3412 it's an endian problem.