I have a xamarin application and its files are compressed using lz4 I could easily decompress the files it using lz4.block.decompress but I couldn't compress it again, can anyone help me with that?
I used the following script to decompress the file and I could read the content of the DLL file but when I modified something and try to recompress it again to patch the APK file I couldn't compress the file in the right format.
with open(input_filepath, "rb") as xalz_file:
data = xalz_file
header = data.read(8)
if header[:4] != b"XALZ":
sys.exit("The input file does not contain the expected magic bytes, aborting ...")
payload = data.read()
decompressed = lz4.block.decompress(payload)
with open(output_filepath, "wb") as output_file:
output_file.write(decompressed)
output_file.close()
print("result written to file")
Note that I couldn't decompress or compress the DLL file using lz4tools.
Related
I have a file which is a compressed image. Its size on disk on Windows is 125,966,232 bytes. I uploaded it to S3 using the Ruby aws-S3 gem. Its size on S3, from the properties pane, is also 125,966,232 bytes.
When I download it to disk using the web browser and the image's public URL, it downloads fine, and its size is consistent. It also uncompresses fine with my uncompression utility.
When I download the file from the S3 bucket to disk using RestClient (1.6.7), its size on disk after downloading is 126,456,885 bytes, 890,653 bytes bigger. This successful download cannot be uncompressed with my uncompression utility, and running this download repeatedly with the same S3 file gets a downloaded file, always of the size with the same file size of 126,456,885 bytes.
require 'rest_client'
local_file = "C:\\test\\test_download.cap"
s3_bucket = "my-bucket-not"
remote_S3_file_url = "https://s3.amazonaws.com/#{s3_bucket}/test_download.cap"
File.open(local_file, "w") do |f|
f.write RestClient.read remote_S3_file_url
end
What do I have to do to ensure that the downloaded file is exactly the same size and/or decompresses properly?
I'd recommend not saving the file as text but instead as binary.
You're using:
File.open(local_file, "w")
'w' means:
"w" Write-only, truncates existing file
to zero length or creates a new file for writing.
Use the 'wb' mode for saving the file instead. Without 'b', line-ends will be converted to Windows format effectively ballooning the size and corrupting the file's contents:
"b" Binary file mode
Suppresses EOL <-> CRLF conversion on Windows. And
sets external encoding to ASCII-8BIT unless explicitly
specified.
So use:
File.open(local_file, 'wb')
See "IO Open Mode" for more information.
I have a tar.gz file, which holds multiple csv files archived. I need to read the list of the file names or at least the number of files.
This is what I tried:
require 'zlib'
file = Zlib::GzipReader.open('test/data/file_name.tar.gz')
file.each_line do |line|
p line
end
but this only prints each line in the csv files, not the file names. I also tried this:
require 'zlib'
Zlib::GzipReader.open('test/data/file_name.tar.gz') { | f |
p f.read
}
which reads similarly, but character by character instead of line by line.
Any idea how I could get the list of file names or at least the number of files within the archive?
You need to use a tar reader on the uncompressed output.
".tar.gz" means that two processes were applied to generate the file. First a set of files were "tarred" to make a ".tar" file which contains a sequence of (file header block, uncompressed file data) units. Then that was gzipped as a single stream of bytes, to make the ".tar.gz". In reality, the .tar file was very likely never stored anywhere, but generated as a stream of bytes and gzipped on the fly to write out the .tar.gz file directly.
To get the contents, you reverse the process, ungzipping, and then feeding the result of that to a tar reader to interpret the file header blocks and extract the data. Again, you can ungzip and read the tarred file contents on the fly, with no need to store the intermediate .tar file.
I am trying to decompress a file with following Ruby code.
File.open("file_compressed.gz") do |compressed|
File.open("file_decomp","w") do |decompressed|
gz = Zlib::GzipReader.new(compressed)
result = gz.read
decompressed.write(result)
gz.close
end
end
But I am getting following error -
not in gzip format (Zlib::GzipFile::Error)
./features/support/abc/abc_file.rb:44:in `initialize'
When I decompress the same file using gzip command on Mac it produced the correct uncompressed output.
For following command I can see -
$file file_compressed.gz
file_compressed.gz: gzip compressed data, from FAT filesystem (MS-DOS, OS/2, NT)
Do I need to put any header data while I create the compressed file with Zlib? Because when I use the inflate method instead of the GzipReader I get following error -
incorrect header check (Zlib::DataError)
./features/support/abc/abc_file.rb:69:in `inflate'
If you're on a platform that doesn't use LF delimiters, but CR+LF, you may need to open the file in binary mode for reading:
File.open("file_compressed.gz", "rb") do |compressed|
# ...
end
This should also avoid interpreting the input stream as anything but 8-bit binary.
Be sure to open your output file the same way using "wb" as the flag.
I wrote a Logparser Application and now I want to implement decompression of .gz files. I tried it with boost::iostreams and zlib which seems to work, but I don't know how to handle the input I get from compressed files.
Here's what I do:
input.open(p.source_at(i).c_str(), ios_base::in | ios_base::binary);
boost::iostreams::filtering_streambuf<boost::iostreams::input> in;
in.push(boost::iostreams::gzip_decompressor());
in.push(input);
boost::iostreams::copy(in, cout);
This code is run, if my sourcefile has the .gz ending. The last line outputs the decompressed filestream correctly to cout.
But how can i fetch line by line from the decompressed file? My Program uses getline(input, transfer) to read lines from the input stream, if it's not compressed.
Now I want to read from the decompressed file the same way, but how can I get a new line from in?
The boost decumentation didn't help me much with this.
Thanks in advance!
Ok I found it out. I just had to create an std::istream and pass a reference to the buffer:
std::istream incoming(&in);
getline(incoming, transfer);
Is it possible to list the contents of a LZMA file (.7zip) without uncompressing the whole file? Also, can I extract a single file from the LZMA file?
My problem: I have a 30GB .7z file that uncompresses to >5TB. I would like to manipulate the original .7z file without needing to do a full uncompress.
Yes. Start with XZ Utils. There are Perl and Python APIs.
You can find the file you want from the headers. Each file is compressed separately, so you can extract just the one you want.
Download lzma922.tar.bz2 from the LZMA SDK files page on Sourceforge, then extract the files and open up C/Util/7z/7zMain.c. There, you will find routines to extract a specific archive file from a .7z archive. You don't need to extract all the data from all the entries, the example code shows how to extract just the one you are interested in. This same code has logic to list the entries without extracting all the compressed data.
I solved this problem by installing 7zip (https://www.7-zip.org/) and using the parameter l. For example:
7z l file.7z
The output has some descriptive information and the list of files in the compressed files. Then, I call this inside python using the subprocess library:
import subprocess
output = subprocess.Popen(["7z","l", "file.7z"], stdout=subprocess.PIPE)
output = output.stdout.read().decode("utf-8")
Don't forget to make sure the program 7z is accessible in your PATH variable. I had to do this manually in Windows.