Ruby Build Hash from file - ruby

I'm consuming a web-service and using Savon to do +-1000 (paid) requests and parse the requests to a csv file.
I save the xml.hash response in a file if the parsing failed.
How can I initialize an hash that was saved to a file? (or should I save in XML and then let savon make it into a hash it again?
Extra info:
client = Savon.client do
wsdl "url"
end
response = client.call(:read_request) do
message "dat:number" => number
end
I use the response.hash to build/parse my csv data. Ex:
name = response.hash[:description][:name]
If the building failed I'm thinking about saving the response.hash to a file. But the problem is I don't know how to reuse the saved response (XML/Hash) so that an updated version of the building/parsing can be run using the saved response.

You want to serialize the Hash to a file then deserialize it back again.
You can do it in text with YAML or JSON and in a binary via Marshal.
Marshal
def serialize_marshal filepath, object
File.open( filepath, "wb" ) {|f| Marshal.dump object, f }
end
def deserialize_marshal filepath
File.open( filepath, "rb") {|f| Marshal.load(f)}
end
Marshaled data has a major and minor version number written with it, so it's not guaranteed to always load in another Ruby if the Marshal data version changes.
YAML
require 'yaml'
def serialize_yaml filepath, object
File.open( filepath, "w" ) {|f| YAML.dump object, f }
end
def deserialize_yaml filepath
File.open( filepath, "r") {|f| YAML.load(f) }
end
JSON
require 'json'
def serialize_json filepath, object
File.open( filepath, "w" ) {|f| JSON.dump object, f }
end
def deserialize_json filepath
File.open( filepath, "r") {|f| JSON.load(f)}
end
Anecdotally, YAML is slow, Marshal and JSON are quick.

If your code is expecting to use/manipulate a ruby hash as demonstrated above, then if you want to save the Savon response, then use the json gem and do something like:
require 'json'
File.open("responseX.json","w") do |f|
f << response.hash.to_json
end
Then if you need to read that file to recreate your response hash:
File.open('responseX.json').each do |line|
reponseHash = JSON.parse(line)
# do something with responseHash
end

Related

uploading folder to s3 with ruby sdk

I have a script that is supposed to upload a local folder to s3 using aws-sdk and ruby.
As much as I understand from ruby, the files need to be uploaded one by one, so here is the code used:
require 'aws-sdk'
require 'open3'
s3_bucket = ARGV[0]
debug = ARGV[1] || nil
#s3 = Aws::S3::Client.new(region: 'eu-west-1')
files = Dir[ File.join('srv', '**', '*') ].reject { |p| File.directory? p }
files.each do |f|
o, e, s = Open3.capture3("gio info -a standard::content-type #{f}")
abort(e) unless s.to_s.match(/exit 0/)
content_type = o.split('standard::content-type: ')[1].strip
s3_key = f.split('srv/lila/')[1]
puts "Uploading #{f} with content-type #{content_type}" if debug
File.open(f,'rb') do |file|
#s3.put_object({body: file, content_type: content_type, bucket: s3_bucket, key: s3_key})
end
end
My local file name is like this: srv/lila/1.1.1/somename/index.html
Somehow, only the file name is uploaded, and not the content.So when I go to the URL, I can see the name of the file as the content srv/lila/1.1.1/somename/index.html. My ruby knowledge is limited and I am not sure what is wrong in this script. Can you help please?
Your issue is this line:
resp = #s3.put_object({body: f, content_type: content_type, bucket: s3_bucket, key: s3_key})
In this case f is not a File but rather a String that represents the path to a file.
body: accepts a String, StringIO or File object as an argument. In this case you are passing a String and it treats that as the contents of the uploaded file.
Instead I would recommend the following alteration:
File.open(f,'rb') do |file|
#s3.put_object({body: file, content_type: content_type, bucket: s3_bucket, key: s3_key})
end
Now file is an actual File object.
I also removed resp as that local variable did not serve a purpose.

how to save StringIO (pdf) data into file

I want to save pdf file which is located in external remote server with ruby. The pdf file is coming in StringIO. I tried saving the data with File.write but it is not working. I received the below error .
ArgumentError: string contains null byte
How to save now ?
require 'stringio'
sio = StringIO.new("he\x00llo")
File.open('data.txt', 'w') do |f|
f.puts(sio.read)
end
$ cat data.txt
hello
Response to comment:
Okay, try this:
require 'stringio'
sio = StringIO.new("\c2\xb5")
sio.set_encoding('ASCII-8BIT') #Apparently, this is what you have.
File.open('data.txt', 'w:utf-8') do |f|
f.puts(sio.read)
end
--output:--
1.rb:7:in `write': "\xB5" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)
To get rid of that error, you can set the encoding of the StringIO to UTF-8:
require 'stringio'
sio = StringIO.new("\c2\xb5")
sio.set_encoding('ASCII-8BIT') #Apparently, this is what you have.
sio.set_encoding('UTF-8') #Change the encoding to what it should be.
File.open('data.txt', 'w:UTF-8') do |f|
f.puts(sio.read)
end
Or, you can use the File.open modes:
require 'stringio'
sio = StringIO.new("\c2\xb5")
sio.set_encoding('ASCII-8BIT') #Apparently, this is what you have.
File.open('data.txt', 'w:UTF-8:ASCII-8BIT') do |f|
f.puts(sio.read)
end
But, that assumes the data is encoded in UTF-8. If you actually have binary data, i.e. data that isn't encoded because it represents a .jpg file for instance, then that won't work.

How do I test reading a file?

I'm writing a test for one of my classes which has the following constructor:
def initialize(filepath)
#transactions = []
File.open(filepath).each do |line|
next if $. == 1
elements = line.split(/\t/).map { |e| e.strip }
transaction = Transaction.new(elements[0], Integer(1))
#transactions << transaction
end
end
I'd like to test this by using a fake file, not a fixture. So I wrote the following spec:
it "should read a file and create transactions" do
filepath = "path/to/file"
mock_file = double(File)
expect(File).to receive(:open).with(filepath).and_return(mock_file)
expect(mock_file).to receive(:each).with(no_args()).and_yield("phrase\tvalue\n").and_yield("yo\t2\n")
filereader = FileReader.new(filepath)
filereader.transactions.should_not be_nil
end
Unfortunately this fails because I'm relying on $. to equal 1 and increment on every line and for some reason that doesn't happen during the test. How can I ensure that it does?
Global variables make code hard to test. You could use each_with_index:
File.open(filepath) do |file|
file.each_with_index do |line, index|
next if index == 0 # zero based
# ...
end
end
But it looks like you're parsing a CSV file with a header line. Therefore I'd use Ruby's CSV library:
require 'csv'
CSV.foreach(filepath, col_sep: "\t", headers: true, converters: :numeric) do |row|
#transactions << Transaction.new(row['phrase'], row['value'])
end
You can (and should) use IO#each_line together with Enumerable#each_with_index which will look like:
File.open(filepath).each_line.each_with_index do |line, i|
next if i == 1
# …
end
Or you can drop the first line, and work with others:
File.open(filepath).each_line.drop(1).each do |line|
# …
end
If you don't want to mess around with mocking File for each test you can try FakeFS which implements an in memory file system based on StringIO that will clean up automatically after your tests.
This way your test's don't need to change if your implementation changes.
require 'fakefs/spec_helpers'
describe "FileReader" do
include FakeFS::SpecHelpers
def stub_file file, content
FileUtils.mkdir_p File.dirname(file)
File.open( file, 'w' ){|f| f.write( content ); }
end
it "should read a file and create transactions" do
file_path = "path/to/file"
stub_file file_path, "phrase\tvalue\nyo\t2\n"
filereader = FileReader.new(file_path)
expect( filereader.transactions ).to_not be_nil
end
end
Be warned: this is an implementation of most of the file access in Ruby, passing it back onto the original method where possible. If you are doing anything advanced with files you may start running into bugs in the FakeFS implementation. I got stuck with some binary file byte read/write operations which weren't implemented in FakeFS quite how Ruby implemented them.

Manipulating XML files in ruby with XmlSimple

I've got a complex XML file, and I want to extract a content of a specific tag from it.
I use a ruby script with XmlSimple gem. I retrieve an XML file with HTTP request, then strip all the unnecessary tags and pull out necessary info. That's the script itself:
data = XmlSimple.xml_in(response.body)
hash_1 = Hash[*data['results']]
def find_value(hash, value)
hash.each do |key, val|
if val[0].kind_of? Hash then
find_value(val[0], value)
else
if key.to_s.eql? value
puts val
end
end
end
end
hash_1['book'].each do |arg|
find_value(arg, "title")
puts("\n")
end
The problem is, that when I change replace puts val with return val, and then call find_value method with puts find_value (arg, "title"), i get the whole contents of hash_1[book] on the screen.
How to correct the find_value method?
A "complex XML file" and XmlSimple don't mix. Your task would be solved a lot easier with Nokogiri, and be faster as well:
require 'nokogiri'
doc = Nokogiri::XML(response.body)
puts doc.xpath('//book/title/text()')

Saving output of a query onto a text file in Ruby

I'm trying to query a table, fetch all records, and save the result as a CSV file.
This is what I've done so far:
require 'OCI8'
conn = OCI8.new('scott','tiger','020')
file = File.open('output.csv','w') do |f|
conn.exec('select * from emp') do |e|
f.write log.join(',')
end
end
.. And while it does generate a CSV file, the problem is that all records get saved onto a single line. How can I put the data such that each record goes onto a new line ?
Well, you can use f.puts instead of f.write there, but I'd recommend you take a look at CSV module:
http://ruby-doc.org/stdlib/libdoc/csv/rdoc/index.html
outfile = File.open('csvout', 'wb')
CSV::Writer.generate(outfile) do |csv|
csv << ['c1', nil, '', '"', "\r\n", 'c2']
...
end
outfile.close
PS: Actually, there is another CSV library called FasterCSV, which became CSV in standard library in Ruby 1.9. But in general, any should be better than writing it yourself.

Resources