I wanted to write a ruby program to unpackage tar.gz files, and I ran into some issues. After reading the documentation on the ruby-doc site, Zlib::GzipReader and Zlib::Inflate. Then I found this Module someone wrote on GitHub, and that didn't work either. So, using the examples from the Ruby page for Zlib::GzipReader, I these were able to run successfully.
irb(main):027:0* File.open('zlib_inflate.tar.gz') do |f|
irb(main):028:1* gz = Zlib::GzipReader.new(f)
irb(main):029:1> print gz.read
irb(main):030:1> gz.close
irb(main):031:1> end
#
#
#
irb(main):023:0* Zlib::GzipReader.open('zlib_inflate.tar.gz') { |gz|
irb(main):024:1* print gz.read
irb(main):025:1> }
Then, when trying to use the Zlib::Inflate options, I kept running into incorrect header check errors.
irb(main):047:0* zstream = Zlib::Inflate.new
=> #<Zlib::Inflate:0x00000002b15790 #dictionaries={}>
irb(main):048:0> buf = zstream.inflate('zlib_inflate.tar.gz')
Zlib::DataError: incorrect header check
from (irb):48:in `inflate'
from (irb):48
from C:/ruby/Ruby200p353/Ruby200-x64/bin/irb:12:in `<main>'
#
#
#
irb(main):049:0> cf = File.open("zlib_inflate.tar.gz")
=> #<File:zlib_inflate.tar.gz>
irb(main):050:0> ucf = File.open("ruby_inflate.rb", "w+")
=> #<File:ruby_inflate.rb>
irb(main):051:0> zi = Zlib::Inflate.new(Zlib::MAX_WBITS)
=> #<Zlib::Inflate:0x00000002c097f0 #dictionaries={}>
irb(main):052:0> ucf << zi.inflate(cf.read)
Zlib::DataError: incorrect header check
from (irb):52:in `inflate'
from (irb):52
from C:/ruby/Ruby200p353/Ruby200-x64/bin/irb:12:in `<main>'
#
#
#
irb(main):057:0* File.open("zlib_inflate.tar.gz") {|cf|
irb(main):058:1* zi = Zlib::Inflate.new
irb(main):059:1> File.open("zlib_inflate.rb", "w+") {|ucf|
irb(main):060:2* ucf << zi.inflate(cf.read)
irb(main):061:2> }
irb(main):062:1> zi.close
irb(main):063:1> }
Zlib::DataError: incorrect header check
from (irb):60:in `inflate'
from (irb):60:in `block (2 levels) in irb_binding'
from (irb):59:in `open'
from (irb):59:in `block in irb_binding'
from (irb):57:in `open'
from (irb):57
from C:/ruby/Ruby200p353/Ruby200-x64/bin/irb:12:in `<main>'
How would I go about taking a tar.gz file, and extract the contents so the files are not null? I did read this other post about Ruby Zlib from here, but that did not work for me as well. What am I doing wrong?
Related
I'm writing code for copy and paste recursively.
But I got an encoding error when calling FileUtils.copy_entry
Error Messages :
C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:1535:in `join': incompatible character encodings: CP949 and UTF-8 (Encoding::CompatibilityError)
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:1535:in `join'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:1218:in `path'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:463:in `block in copy_entry'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:1485:in `call'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:1485:in `wrap_traverse'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:1488:in `block in wrap_traverse'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:1487:in `each'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:1487:in `wrap_traverse'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:1488:in `block in wrap_traverse'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:1487:in `each'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:1487:in `wrap_traverse'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:460:in `copy_entry'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:435:in `block in cp_r'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:1558:in `block in fu_each_src_dest'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:1574:in `fu_each_src_dest0'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:1556:in `fu_each_src_dest'
from C:/Ruby200/lib/ruby/2.0.0/fileutils.rb:434:in `cp_r'
from copy_test.rb:11:in `copy_files'
from copy_test.rb:81:in `block in <main>'
from copy_test.rb:76:in `each'
from copy_test.rb:76:in `<main>'
I'm calling copy_entry like this.
def copy_files(src, dst)
FileUtils.mkdir_p(File.dirname(dst))
FileUtils.copy_entry(src, dst)
end
There are some sub-folders and files named with my local language in src.
So I think (Encoding::CompatibilityError) occurs because of these sub-folders and files with local language (not English).
When I tested with only Enlgish folders and files, It worked.
But, I need non-English folders and files too.
How can I solve this problem?
Should I define a new method replacing copy_entry?
My Codes ADDED:
# -*- encoding: cp949 -*-
require 'fileutils'
LIST_FILE = "backup_target_list.txt"
DEST_FILE = "backup_dest.txt"
def copy_files(src, dst)
FileUtils.mkdir_p(File.dirname(dst))
FileUtils.copy_entry(src, dst)
end
def get_dst_base_name()
cur_date = Time.now.to_s[0..9]
return "backup_#{cur_date}"
end
def get_backup_list(list_file)
if !File.exist?(list_file) then
return nil
end
path_arr = []
File.open(list_file, "r") do |f|
f.each_line { |line|
path_arr.push(make_path(line.gsub("\n", "")))
}
end
return path_arr
end
def get_dest(dest_file)
if !File.exist?(dest_file) then
return nil
end
return File.open(dest_file, "r").readline
end
def make_path(*str)
path_new = nil
str.each do |item|
if item.class == Array then
path_new = (path_new == nil ? File.join(item) : File.join(path_new, item))
else
if item.include?(File::ALT_SEPARATOR) then
path_new = (path_new == nil ? File.join(item.split(File::ALT_SEPARATOR)) : File.join(path_new, item.split(File::ALT_SEPARATOR)))
else
path_new = (path_new == nil ? File.join(item) : File.join(path_new, item))
end
end
end
return path_new
end
get_backup_list(LIST_FILE).each do |path|
src = path
tmp = src.split(File::SEPARATOR)
dst = make_path(get_dest(DEST_FILE), get_dst_base_name, tmp[1..tmp.length])
print "src: #{src}\n=> dst: #{dst}\n"
copy_files(src, dst)
end
I saw you are using cp949 encoding, and the raised error tells CP949 and UTF-8 are incompatible, so why you just using UTF-8 encoding? So replace the shebang with
# coding: UTF-8
And to ensure all the characters read from LIST_FILE are utf-8 encoded, add the following:
line.force_encoding('utf-8')
I have problem with script that makes simple .xls file and writes data to one cell. Here is simple code:
require 'spreadsheet'
class Filter
def filter
#excel = Spreadsheet::Workbook.new
#sheet = #excel.create_worksheet
#sheet[0, 0] = "test"
#excel.write 'test.xls'
end
end
f = Filter.new
f.filter
But it raises error:
C:/Ruby193/lib/ruby/gems/1.9.1/gems/ruby-ole-1.2.11.5/lib/ole/storage/base.rb:62:in
write_nonblock': Bad file descriptor - test.xls (Errno::EBADF)
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/ruby-ole-1.2.11.5/lib/ole/storage/base.rb:62:in
initialize'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/ruby-ole-1.2.11.5/lib/ole/storage/base.rb:78:in
new'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/ruby-ole-1.2.11.5/lib/ole/storage/base.rb:78:in
open'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/spreadsheet-0.7.4/lib/spreadsheet/excel/writer/workbook.rb:4
53:in write_from_scratch'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/spreadsheet-0.7.4/lib/spreadsheet/excel/writer/workbook.rb:6
31:inwrite_workbook'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/spreadsheet-0.7.4/lib/spreadsheet/writer.rb:15:in
block in write'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/spreadsheet-0.7.4/lib/spreadsheet/writer.rb:14:in
open'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/spreadsheet-0.7.4/lib/spreadsheet/writer.rb:14:in
write'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/spreadsheet-0.7.4/lib/spreadsheet/workbook.rb:116:in
write'
from filter.rb:10:in `filter'
from filter.rb:15:in `<main>'
because ruby-ole 1.2.11.5 doesn't support windows platform,
more detail: ruby-ole issue
you can use ruby-ole 1.2.11.4 to avoid this problem.
require 'rubygems'
gem 'ruby-ole','1.2.11.4'
require 'spreadsheet'
I've seen these before. First verify that you can write to that file's location.
My guess is either the file is already open in Excel or your antivirus is blocking the 'threat'.
I am trying to retrieve files (.csv) from an ftp site and save them all locally in the same folder. My code looks like this:
#! /usr/bin/ruby
require 'logger'
require 'fileutils'
require 'net/ftp'
require 'rubygems'
require 'mysql2'
require 'roo'
require 'date'
# logging setup
log = Logger.new("/path_to_logs/ftp_log.log", 10, 1024000)
log.level = Logger::INFO
export_ftp_path = '/Receive/results/'
export_work_path ='/Users/pierce/results_exports/'
Net::FTP.open('host', 'username', 'password') do |ftp|
log.info("Logged into FTP")
ftp.passive = true
ftp.chdir("#{export_ftp_path}")
ftp.list.each do |file|
log.info("Found file #{file}")
new_file = file[56..115] #take part of the file name and remove spaces and periods
new_file = new_file.gsub(/[.]+/, "")
new_file = new_file.gsub(/\s/, "0")
ftp.gettextfile(file,"#{new_file}")
log.info("Downloaded file #{new_file}")
end
end
And here is the error I receive:
/Users/pierce/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/ftp.rb:560:in `initialize': No such file or directory - (Errno::ENOENT)
from /Users/pierce/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/ftp.rb:560:in `open'
from /Users/pierce/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/ftp.rb:560:in `gettextfile'
from ftp_test.rb:44:in `block (2 levels) in <main>'
from ftp_test.rb:33:in `each'
from ftp_test.rb:33:in `block in <main>'
from /Users/pierce/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/ftp.rb:116:in `open'
As suggested, here are the values I have for puts file and puts new_file.
file = -rwxr-xr-x 1 1130419 114727 9546 May 17 08:11 results_Wed. 16 May 2012.csv
new_file = results_Wed0230May02012csv
Any suggestions on what to change in gettextfile or within my script to get the files saved correctly?
You should use nlst instead of list when you just need a list of files in a directory. The output of list needs to be properly parsed otherwise.
When you request the file it has to be the original filename, including all spaces. When you save the file it can be anything you want (including spaces or not). The error was because you were requesting the wrong file. Use nlst in your case instead. It will make it much easier (no conversion or parsing needed).
I'm trying to read local image file, properly encode it and post to Tumbrl. According to the Tumblr API I can pass a parameter data which is Array (URL-encoded binary contents) Limit: 5 MB
I've tested my code with http://api.tumblr.com/v2/blog/#{BLOG}/info request. It is working. But I can't post a photo. Here is my code:
require 'oauth'
require 'oauth/consumer'
require 'open-uri'
require 'active_support'
CONSUMER = 'foo'
SECRET = 'foo'
TOKEN = 'foo'
TOKEN_SECRET = 'foo'
BLOG = 'foo'
consumer=OAuth::Consumer.new(CONSUMER, SECRET, {:site=>"http://tumblr.com"})
access_token = OAuth::AccessToken.new(consumer, TOKEN, TOKEN_SECRET)
# Here I tried one of two lines:
# data = Base64.encode64(IO.binread('./resized')) #first try
data = URI::encode(IO.binread('./resized')) #second try
# response = access_token.get "http://api.tumblr.com/v2/blog/#{BLOG}/info?api_key=#{CONSUMER}"
# puts response
response=access_token.post "http://api.tumblr.com/v2/blog/#{BLOG}/post?api_key=#{CONSUMER}&type=photo&data=#{data}&link=http://ya.ru&"
puts response
1st try:
% ruby ./w_oauth.rb
/usr/lib/ruby/1.9.1/uri/common.rb:176:in `split': bad URI(is not URI?): http://api.tumblr.com/v2/blog/foo/post?api_key=foo&type=photo&data=/9j/4AAQSkZJRgABAQEASABIAAD//gA7Q1JFQVRPUjogZ2QtanBlZyB2MS4w (URI::InvalidURIError)
ICh1c2luZyBJSkcgSlBFRyB2NjIpLCBxdWFsaXR5ID0gODAK/9sAQwAGBAUG
BQQGBgUGBwcGCAoQCgoJCQoUDg8MEBcUGBgXFBYWGh0lHxobIxwWFiAsICMm
(!!!long piece of image data skipped!!!)
FI/16HfTbyHPWurqdE+TGH4wx2js5SKQb+6b4bIj3aurqCrEtcXrf/4yf/dS
DLet/wCzEB6sa6uoomxJN2eaQj5mkYuerQj611dQM7Fx/wDLF/8AbXV1dTA/
/9k=
&link=http://ya.ru&
from /usr/lib/ruby/1.9.1/uri/common.rb:211:in `parse'
from /usr/lib/ruby/1.9.1/uri/common.rb:747:in `parse'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/tokens/access_token.rb:7:in `request'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/tokens/access_token.rb:47:in `post'
from ./w_oauth.rb:23:in `<main>'
2nd try:
% ruby ./w_oauth.rb
/var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:14:in `force_encoding': can't modify frozen String (RuntimeError)
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:14:in `rescue in escape'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:12:in `escape'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:43:in `block (2 levels) in normalize'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:42:in `collect'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:42:in `block in normalize'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:37:in `map'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/helper.rb:37:in `normalize'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/request_proxy/base.rb:98:in `normalized_parameters'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/request_proxy/base.rb:113:in `signature_base_string'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/signature/base.rb:77:in `signature_base_string'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/signature/hmac/base.rb:12:in `digest'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/signature/base.rb:65:in `signature'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/signature.rb:23:in `sign'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/client/helper.rb:45:in `signature'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/client/helper.rb:75:in `header'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/client/net_http.rb:91:in `set_oauth_header'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/client/net_http.rb:30:in `oauth!'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/consumer.rb:224:in `sign!'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/consumer.rb:188:in `create_signed_request'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/consumer.rb:159:in `request'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/tokens/consumer_token.rb:25:in `request'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/tokens/access_token.rb:12:in `request'
from /var/lib/gems/1.9.1/gems/oauth-0.4.6/lib/oauth/tokens/access_token.rb:47:in `post'
from ./w_oauth.rb:23:in `<main>'
UPD: ./resized is a proper JPEG file:
% file ./resized
./resized: JPEG image data, JFIF standard 1.01, comment: "CREATOR: gd-jpeg v1.0 (using IJG JPEG v62), quality = 80"
URI encoding is not enough. You also need to encode: , / ? : # & = + $ #.
Try:
URI.escape(IO.binread('./resized'), Regexp.new("[^#{URI::PATTERN::UNRESERVED}]"))
I'm doing a simple script to check crc of all files...
require "zlib"
exit if Object.const_defined?(:Ocra)
files = Dir.glob("*")
File.open('dir.txt', 'a+') do |file|
file.puts files
end
File.read('dir.txt').each_line { |line|
file = File.read(line) ; nil
file_crc = Zlib.crc32(file,0).to_s(16)
puts line, file_crc
}
The problem is at the line File.read('dir.txt').each_line { |line|
I get this error:
test.rb:13:in `read': Invalid argument - 1.exe (Errno::EINVAL)
from C:/Users/Administrador/Desktop/1.rb:13:in `block in <main>'
from C:/Users/Administrador/Desktop/1.rb:12:in `each_line'
from C:/Users/Administrador/Desktop/1.rb:12:in `<main>'
PD: 1.exe is a file listed in the "dir.txt".
Have you checked that the line doesn't contain extra characters? p line.
IIRC line will contain the newline character, use line.chomp.