This problem has bugged me for a while.
I have a jpeg file that is 34.6 kilobytes. Let's call it Image A. Using Ruby, when I copy each line of Image A to a newly created file, called Image B, it is copied exactly. It is exactly the same size as Image A and is accessible.
Here is the code I used:
image_a = File.open('image_a.jpg', 'r')
image_b = File.open('image_b.jpg', 'w+')
image_a.each_line do |l|
image_b.write(l)
end
image_a.close
image_b.close
This code generates a perfect copy of image_a into image_b.
When I try to copy Image A into Image B, byte by byte, it copies successfully but the file size is 88.9 kilobytes rather than the 34.6 kilobytes. I can't access Image B. My mac system alerted me it may be damaged or is using a file format that isn't recognized.
The related code:
//same as before
image_a.each_byte do |b|
image_b.write(b)
end
//same as before
Why is Image B, when copied into byte by byte, larger than Image A? Why is it also damaged in some way, shape, or form? Why is Image A the same size as B, when copied line by line, and accessible?
My guess is the problem is an encoding issue. If so, Why does encoding format matter when copying byte by byte if they translate into the correct code points? Are code points jumbled up into each other so the parser is unable to differentiate between them?
Do \s and \n matter? It seems like it. I did some more research and I found that Image A had 128 lines of code whereas Image B had only one line.
Thanks for reading!
IO#each_byte iterates over bytes (aka Integers). IO#write, however, takes a string as an argument. So it converts the integer to a string via to_s.
Given the first byte in your image is 2551, you'd write the string "255" into image_b. This is why your image_b gets larger. You write number-strings into it.
Try the following when writing back bytes:
image_a.each_byte do |l|
image_b.write l.chr
end
1 As #stefan pointed out jpeg images start with FF D8. So the first byte is 255.
Related
I have a pretrained embeddings with word2vec format in txt. I loaded it and then saved it to .bin. But I cannot load this embeddings as an EOFError: unexpected end of input; is count incorrect or file otherwise damaged?
My original code is:
model = KeyedVectors.load_word2vec_format(wordfile)
model.save_word2vec_format("file.bin",binary=True,write_header=True)
bin_model = KeyedVectors.load_word2vec_format("file.bin",binary=True)
And I can load this file.bin with a limit arguement: KeyedVectors.load_word2vec_format("file.bin",binary=True, limit=10000).
Is there some other process needed when I save embeddings?
There's a good chance that your .bin file has an incorrect leading-count, or the file has been otherwise been damaged/truncated – because that error means the file declared in its header (1st line) a larger number of word-vectors than were found during attempted-load.
So, if you downloaded it or copied it from somewhere, check the original source, to make sure you've got the full file.
Is there a reason you're performing this conversion? The formats are essentialy equivalent, and result in the exact same object-in-Python after loading.
If there's any tiny on-disk size savings in binary-format, you could probably save more by GZIPping the file (which the .load_word2vec_format() will also happily decompress, if it sees a trailing .gz on the filename).
I just used this code to convert a picture to binary:
import io
tme = input("Name: ")
with io.open(tme, "rb") as se:
print(se.read())
se.close()
Now it looks like this:
5MEMMMMMMMMMMMMM777777777777777777777777777777777\x95\x95\x95\x95\x95\x95\x95\x95\x95\x95\x95\x95\x95MEEMMMMEEMM\x96\x97\x97\x97\x97\x97\x97\x97\x97\x97\x97\x97\x97\x97\x97\x97\x97
And now I want to be able to interpret what this binary code is telling me exactly... I know it ruffly but not enough to be able to do anything on purpose. I searched the web but I didn't find anything that could help me on that point. Can you tell me how is it working or send me a link where I can read how it's done?
You can't just change random bytes in an image. There's a header at the start with the height and width, probably the date and a palette and information about the number of channels and bits per pixel. Then there is the image data which is often padded, and/or compressed.
So you need an imaging library like PIL/Pillow and code something like this:
from PIL import Image
im = Image.open('image.bmp').convert('RGB')
px = im.load()
# Look at pixel[4,4]
print (px[4,4])
# Make it red
px[4,4] = (255,0,0)
# Save to disk
im.save('result.bmp')
Documentation and examples available here.
The output should be printed in hexadecimal format. The first bytes in bitmap file are 'B' and 'M'.
You are attempting to print the content in ASCII. Moreover, it doesn't show the first bytes because the content has scrolled further down. Add print("start\n") to make sure you see the start of the output.
import io
import binascii
tme = 'path.bmp'
print("start") # make sure this line appears in console output
with io.open(tme, "rb") as se:
content = se.read()
print(binascii.hexlify(content))
Now you should see something like
start
b'424d26040100000000003...
42 is hex value for B
4d is hex value for M ...
The first 14 bytes in the file are BITMAPFILEHEADER
The next 40 bytes are BITMAPINFOHEADER
The bytes after that are color table (if any) and finally the actual pixels.
See BITMAPFILEHEADER and BITMAPINFOHEADER
The background of my problem is that I want to extract the video data of Motion Photos (taken by my Samsung S7). Manually it is easy but time consuming. Just open the .jpg file in a HexEditor and extract all data after the line "MotionPhoto_Data". The first part is the image and the second part is the video.
My current code is
im = 'test.jpg'
with open(im, 'rb') as fin:
data = fin.read()
data_latin = data.decode('latin1')
fin.close()
position = data_latin.find('MotionPhoto_Data')
data_pic = data[:position]
data_mpg = data[position:]
My problem now is that I can´t figure out how to save these strings in a way that data_pic is saved as a working jpg and data_mpg as a working video.
I tried
with open('test_pic.jpg', 'a') as fin:
fin.write(str(data_pic))
fin.close()
But this didn´t worked. I think there is a basic issue on how I try to handle/save my data but I can´t figure out how to fix this.
I assume you use python 3 as it is tagged that way.
You should not decode with 'data.decode('latin1'). It is binary data.
data = fin.read()
Then later write it also as binary data:
with open('test_pic.jpg', 'ab') as fout:
fout.write(data_pic)
fout.close()
I have a homework in which i have to convert some images to grayscale and compress them using huffman encoding. I converted them to grayscale and then i tried to compress them but i get an error. I used the code i found here.
Here is the code i'm using:
A=imread('Gray\36.png');
[symbols,p]=hist(A,unique(A))
p=p/sum(p)
[dict,avglen]=huffmandict(symbols,p)
comp=huffmanenco(A,dict)
This is the error i get. It occurs at the second line.
Error using eps
Class must be 'single' or 'double'.
Error in hist (line 90)
bins = xx + eps(xx);
What am i doing wrong?
Thanks.
P.S. how can i find the compression ratio for each image?
The problem is that when you specify the bin locations (the second input argument of 'hist'), they need to be single or double. The vector A itself does not, though. That's nice because sometimes you don't want to convert your whole dataset from an integer type to floating precision. This will fix your code:
[symbols,p]=hist(A,double(unique(A)))
Click here to see this issue is discussed more in detail.
first, try :
whos A
Seems like its type must be single or double. If not, just do A = double(A) after the imread line. Should work that way, however I'm surprised hist is not doing the conversion...
[EDIT] I have just tested it, and I am right, hist won't work in uint8, but it's okay as soon as I convert my image to double.
Is there a Ruby library that will allow me to either calculate the checksum of an MP3 file's audio data (minus the metadata) or allow me to read in an MP3's audio data to calculate the checksum myself?
I'm looking for something like this:
mp3 = Mp3Lib::MP3.new('/path/to/song.mp3')
mp3.audio.sha1sum # => the sha1 checksum of _only_ the audio, minus the metadata
I found Mp3Info, but it seems a bit tedious. When initializing an Mp3Info object, you can get the frames where the actual audio data begins and ends.
Extracting the mp3file without it's metadata is fairly easy done by yourself.
ID3v1
The metadata are the last 128 bytes of the file. The metadata always begins with the 3 bytes "TAG" if it exists. Just ignore this last 128bytes.
ID3v2
The metadata can be stored at the beginning or the end of the file. Most implemantations only support the beginning. ID3v2 has a header where the size is stored. The header is always loacted at the beginning of the metadata. There is an optional footer, which is a copy of the header at the end of the metadata. If the metadata is at the end of the file, the footer is required.
The header has the folloing form
ID3v2/file identifier "ID3"
ID3v2 version $04 00
ID3v2 flags %abcd0000
ID3v2 size 4 * %0xxxxxxx
The footer has the following form
ID3v2/file identifier "3DI"
ID3v2 version $04 00
ID3v2 flags %abcd0000
ID3v2 size 4 * %0xxxxxxx
The d bit says, wheter the footer is present. The size is measured without header and footer. Every byte of the size has always the highest bit set. So only 28 of the acutal 32 bits represent the size.
Just compute, which part of the file is not the metadata, and use it for your hashing.
Be aware, if both ID3v1 and ID3v2 are located at the end of the file, ID3v1 is located behind IDv2
The spec can be found at http://www.id3.org/id3v2.4.0-structure.
Isn't the ID3 tag stored either at the end of the file (ID3 v1) in a 128-byte block, or in a block at the start of the file (ID3v2.3 and v2.4) ? (id3.org)
You could use the audio_content method from Mp3Info, and read that much data from the file, though it's probably not much more complicated to have a look in the file yourself and work out where the headers aren't.