I'd like to skip parsing some json returned by my DB when I just turn it back into json again immediately in the response from my padrino app.
ie I have
get :data, provides: :json do
Record.order(:day).map do |r|
{r.day.to_s => JSON.parse!(r.data)}
end.reduce({}, :merge!).to_json
end
and I'd like something like the following (inspired by String#html_safe):
get :data, provides: :json do
Record.order(:day).map do |r|
{r.day.to_s => r.data.json_literal}
end.reduce({}, :merge!).to_json
end
I know I can move the hash creation to the model with #as_json but that doesn't address the unnecessary performance hit from parsing and re-encoding the json.
Example output:
{
"2010-01-01":{"linux64":12186213,"mac":24131170},
"2010-01-02":{"linux":10650417,"mac":24139611,"win":12210218},
"2010-01-03":{"linux":10628353,"linux64":12184435,"win":12229263}
}
where the object that is the value of each key/value pair is available as a json-string in r.data eg '{"linux":10650417,"mac":24139611,"win":12210218}' which is why i want to avoid parsing r.data and just inline it.
I tried bypassing the JSON parse/dump altogether with the following:
get :data, provides: :json do
"{"+Record.order(:day).map do |r|
"\"#{r.day}\":#{r.data},"
end.reduce(&:+).delete(' ').chop+"}"
end
but performance was even worse than the version with the unnecessary parsing. (Which is weird, I'm not sure if string concatenation is slow or string interpolation...)
It turns out that the best way to do this is actually at the DB layer, as postgres can inline json columns into json objects and aggregate strings far faster than ruby can. So I did that and dropped my latency from 30ms to 8ms. The body of the method is below:
Sequel::Model.db.fetch("select json_object_agg(day,data) as jsobj from records;").first[:jsobj]
References:
Create json with column values as object keys
How to run raw SQL queries with Sequel
Related
I am trying to convert a CSV file to JSON using Ruby. I am very, very, green when it comes to working with Ruby (or any language for that matter) so the answers may need to be dumbed down for me. Putting it in JSON seems like the most reasonable solution to me because I understand how to work with JSON when assigning variables equal to the attributes that come in the response. If there is a better way to do it, feel free to teach me.
My CSV is in the following format:
Header1,Header,Header3
ValueX,ValueY,ValueZ
I would like to be able to use the data to say something along the lines of this:
For each ValueX in Row 1 after the headers, check if valueZ is > ValueY. If yes, do this, if no do that. I understand how to do the if statement, just now how to parse out my information into variables/arrays.
Any ideas here?
require 'csv'
require 'json'
rows = []
CSV.foreach('a.csv', headers: true, converters: :all) do |row|
rows << row.to_hash
end
puts rows.to_json
# => [{"Header1":"ValueX","Header":"ValueY","Header3":"ValueZ"}]
Here is a first pointer:
require 'csv'
data = CSV.read('your_file.csv', { :col_sep => ',' }
Now you should have the data in data; you can test in irb.
I don't entirely understand the question:
if z > y
# do this
else
# do that
end
For JSON, you should be able to do JSON.parse().
I am not sure what target format JSON requires, probably a Hash.
You can populate your hash with the dataset from the CVS:
hash = Hash.new
hash[key_goes_here] = value_here
My ExtJS frontend sends such a param hash to my Sinatra backend:
{"_dc"=>"1365959782607", "page"=>"6", "start"=>"250", "limit"=>"50", "sort"=>"[{\"property\":\"port\",\"direction\":\"ASC\"}]"}
How to get the params 'property' and 'direction'?
You could do it the following way:
require 'json'
a = {"_dc"=>"1365959782607", "page"=>"6", "start"=>"250", "limit"=>"50", "sort"=>"[{\"property\":\"port\",\"direction\":\"ASC\"}]"}
sort = JSON.parse a["sort"]
p sort[0]["property"] # "port"
p sort[0]["direction"] # "ASC"
Your question has nothing to do Sinatra, it's a basic question of how to extract values from a hash and deal with JSON:
require 'json'
hash = {"_dc"=>"1365959782607", "page"=>"6", "start"=>"250", "limit"=>"50", "sort"=>"[{\"property\":\"port\",\"direction\":\"ASC\"}]"}
JSON[hash['sort']].first.values_at('property', 'direction')
=> ["port", "ASC"]
Parsing the serialized object using JSON[hash['sort']] returns an array containing a single hash. first will return that hash. At that point it's just the usual methods to get at the values. I used values_at to return them as an array.
Pass JSON[] a string and JSON will try to parse it, expecting a JSON encoded object. Pass JSON[] another object, like an array or hash, and JSON will encode it into its serialized format.
I am trying to parse a SOAP response using Savon. The response is XML but is being returned as one long string. If I use #to_hash the entire XML object is still a string, now stored in
hash[:response][:return]
which means it is still a huge unusable mess.
My code looks like
response = soapClient.request(:get_sites_user_can_access) do
soap.body = { :sessionid => session[:login_response][:login_return],
:eid => user }
end
rep = response.to_hash
pp rep[:get_sites_user_can_access_response][:get_sites_user_can_access_return]
What step am I missing to get useful information out of the response? Note: Unfortunately I can't post the XML response because of the info it contains, but it looks like an entire XML document stored as a string. It's class is Nori::StringWithAttributes
I was able to get the desired results but parsing the Nori string(?) using this documentation. This seems like a less than ideal method, but I realized the last element is an array of hashes. So it's hash, of hashes, with an array of hashes. Anyway, here is what worked for me. Advice on how to make this less ugly and clunky would be appreciated.
response = soapClient.request(:get_sites_user_can_access) do
soap.body = { :sessionid => session[:login_response][:login_return],
:eid => user }
end
rep = response.to_hash[:get_sites_user_can_access_response][:get_sites_user_can_access_return]
hrep = Nori.parse(rep)
hrep[:list][:item].each { |item| pp item[:site_id] }
I am trying 'mongo-ruby-driver' for some project. It's working fine except when I call to_json on mongo object. It gives well formed json but it's not converting BSON::Code into readable value i resulting JSON.
Instead of showing code text, it show something like
#<BSON::Code:0x00000100af6fa8>
Did anyone tried it. Any help id highly appreciated.
UPDATE
here is some code snippet:
#records is variable that contains Array of MongoDB documents in hash.
#records.to_json
When I call to_json on it it gives everything as expected. except for the key that contain BSON::Code (means javascript code). for example consider following doc:
{
"_id" : "contains",
"value" : function( obj, target ) { return obj.indexOf(target) != -1; };
}
Querying same doc from ruby gives output like:
{
"_id"=>"contains",
"value"=><BSON::Code:2160165280 #data="function( obj, target ) { return obj.indexOf(target) != -1; };" #scope="{}">
}
and calling to_json on this gives following:
{"_id":"contains","value":"#<BSON::Code:0x00000100b54658>"}
this is what the problem is. Instead of getting actual code for 'value' key I am getting ruby object as string.
Calling code method on BSON::Code we can get it converted into code. But for that I need to loop mongo docs, check values for each key, calling code on it if it is an object of BSON::Code and then assigning it back to key. And at last we can call to_json on it. But I dont want this much overhead. I need to_json itself should take care of it.
The serializer that is iterating over the attributes of the objects in the array is probably calling the to_s method which would cause the output you are seeing.
You can either monkey patch the BSON::Code class to include a to_s method which calls inspect (the method that produces the output you want) or modify the serializer to detect when it encounters a BSON::Code instance and call inspect on it rather than to_s.
The code to mokey patch the BSON::Code class would look like this:
module BSON
class Code
def to_s
inspect
end
end
end
This would have the same behavior as inspect. If you just wanted the code you could monkey patch this in:
module BSON
class Code
def to_s
#code
end
end
end
I might make this the default behavior for to_s in the driver but for now just include that in your code and it should work like a champ.
Which Mongo object, do you mean the module? If you could, please post the code and what you are trying to do.
The only to_json I see in the driver is BSON::ObjectId#to_json which (from the docs) is described as a method that does the following:
Convert to MongoDB extended JSON format. Since JSON includes type information, but lacks an ObjectId type, this JSON format encodes the type using an $oid key.
I get the same results when I use it:
1.9.3-p0 :001 > require 'mongo'
=> true
1.9.3-p0 :002 > BSON::ObjectId
=> BSON::ObjectId
1.9.3-p0 :003 > BSON::ObjectId.new()
=> BSON::ObjectId('4f17350eadd361e91d000001')
1.9.3-p0 :004 > BSON::ObjectId.new().to_json
=> "{\"$oid\": \"4f173512add361e91d000002\"}"
BSON ( http://bsonspec.org/ ) is a binary representation of JSON ( http://www.json.org/ ) . JSON is not meant to be used to describe functions, as it is meant to be portable. Functions in this manner are not portable to other systems. So there is no way to serialize it. There is some hacks defined here that may get you what you need, but ultimately, BSON/JSON may be a weird tool for the job if your trying to serialize functions.
I am learning Ruby and I have written the following code to find out how to consume SOAP services:
require 'soap/wsdlDriver'
wsdl="http://www.abundanttech.com/webservices/deadoralive/deadoralive.wsdl"
service=SOAP::WSDLDriverFactory.new(wsdl).create_rpc_driver
weather=service.getTodaysBirthdays('1/26/2010')
The response that I get back is:
#<SOAP::Mapping::Object:0x80ac3714
{http://www.abundanttech.com/webservices/deadoralive} getTodaysBirthdaysResult=#<SOAP::Mapping::Object:0x80ac34a8
{http://www.w3.org/2001/XMLSchema}schema=#<SOAP::Mapping::Object:0x80ac3214
{http://www.w3.org/2001/XMLSchema}element=#<SOAP::Mapping::Object:0x80ac2f6c
{http://www.w3.org/2001/XMLSchema}complexType=#<SOAP::Mapping::Object:0x80ac2cc4
{http://www.w3.org/2001/XMLSchema}choice=#<SOAP::Mapping::Object:0x80ac2a1c
{http://www.w3.org/2001/XMLSchema}element=#<SOAP::Mapping::Object:0x80ac2774
{http://www.w3.org/2001/XMLSchema}complexType=#<SOAP::Mapping::Object:0x80ac24cc
{http://www.w3.org/2001/XMLSchema}sequence=#<SOAP::Mapping::Object:0x80ac2224
{http://www.w3.org/2001/XMLSchema}element=[#<SOAP::Mapping::Object:0x80ac1f7c>,
#<SOAP::Mapping::Object:0x80ac13ec>,
#<SOAP::Mapping::Object:0x80ac0a28>,
#<SOAP::Mapping::Object:0x80ac0078>,
#<SOAP::Mapping::Object:0x80abf6c8>,
#<SOAP::Mapping::Object:0x80abed18>]
>>>>>>> {urn:schemas-microsoft-com:xml-diffgram-v1}diffgram=#<SOAP::Mapping::Object:0x80abe6c4
{}NewDataSet=#<SOAP::Mapping::Object:0x80ac1220
{}Table=[#<SOAP::Mapping::Object:0x80ac75e4
{}FullName="Cully, Zara"
{}BirthDate="01/26/1892"
{}DeathDate="02/28/1979"
{}Age="(87)"
{}KnownFor="The Jeffersons"
{}DeadOrAlive="Dead">,
#<SOAP::Mapping::Object:0x80b778f4
{}FullName="Feiffer, Jules"
{}BirthDate="01/26/1929"
{}DeathDate=#<SOAP::Mapping::Object:0x80c7eaf4>
{}Age="81"
{}KnownFor="Cartoonists"
{}DeadOrAlive="Alive">]>>>>
I am having a great deal of difficulty figuring out how to parse and show the returned information in a nice table, or even just how to loop through the records and have access to each element (ie. FullName,Age,etc). I went through the whole "getTodaysBirthdaysResult.methods - Object.new.methods" and kept working down to try and work out how to access the elements, but then I get to the array and I got lost.
Any help that can be offered would be appreciated.
If you're going to parse the XML anyway, you might as well skip SOAP4r and go with Handsoap. Disclaimer: I'm one of the authors of Handsoap.
An example implementation:
# wsdl: http://www.abundanttech.com/webservices/deadoralive/deadoralive.wsdl
DEADORALIVE_SERVICE_ENDPOINT = {
:uri => 'http://www.abundanttech.com/WebServices/DeadOrAlive/DeadOrAlive.asmx',
:version => 1
}
class DeadoraliveService < Handsoap::Service
endpoint DEADORALIVE_SERVICE_ENDPOINT
def on_create_document(doc)
# register namespaces for the request
doc.alias 'tns', 'http://www.abundanttech.com/webservices/deadoralive'
end
def on_response_document(doc)
# register namespaces for the response
doc.add_namespace 'ns', 'http://www.abundanttech.com/webservices/deadoralive'
end
# public methods
def get_todays_birthdays
soap_action = 'http://www.abundanttech.com/webservices/deadoralive/getTodaysBirthdays'
response = invoke('tns:getTodaysBirthdays', soap_action)
(response/"//NewDataSet/Table").map do |table|
{
:full_name => (table/"FullName").to_s,
:birth_date => Date.strptime((table/"BirthDate").to_s, "%m/%d/%Y"),
:death_date => Date.strptime((table/"DeathDate").to_s, "%m/%d/%Y"),
:age => (table/"Age").to_s.gsub(/^\(([\d]+)\)$/, '\1').to_i,
:known_for => (table/"KnownFor").to_s,
:alive? => (table/"DeadOrAlive").to_s == "Alive"
}
end
end
end
Usage:
DeadoraliveService.get_todays_birthdays
SOAP4R always returns a SOAP::Mapping::Object which is sometimes a bit difficult to work with unless you are just getting the hash values that you can access using hash notation like so
weather['fullName']
However, it does not work when you have an array of hashes. A work around is to get the result in xml format instead of SOAP::Mapping::Object. To do that I will modify your code as
require 'soap/wsdlDriver'
wsdl="http://www.abundanttech.com/webservices/deadoralive/deadoralive.wsdl"
service=SOAP::WSDLDriverFactory.new(wsdl).create_rpc_driver
service.return_response_as_xml = true
weather=service.getTodaysBirthdays('1/26/2010')
Now the above would give you an xml response which you can parse using nokogiri or REXML. Here is the example using REXML
require 'rexml/document'
rexml = REXML::Document.new(weather)
birthdays = nil
rexml.each_recursive {|element| birthdays = element if element.name == 'getTodaysBirthdaysResult'}
birthdays.each_recursive{|element| puts "#{element.name} = #{element.text}" if element.text}
This will print out all elements that have any text.
So once you have created an xml document you can pretty much do anything depending upon the methods the library you choose has ie. REXML or Nokogiri
Well, Here's my suggestion.
The issue is, you have to snag the right part of the result, one that is something you can actually iterator over. Unfortunately, all the inspecting in the world won't help you because it's a huge blob of unreadable text.
What I do is this:
File.open('myresult.yaml', 'w') {|f| f.write(result.to_yaml) }
This will be a much more human readable format. What you are probably looking for is something like this:
--- !ruby/object:SOAP::Mapping::Object
__xmlattr: {}
__xmlele:
- - &id024 !ruby/object:XSD::QName
name: ListAddressBooksResult <-- Hash name, so it's resul["ListAddressBooksResult"]
namespace: http://apiconnector.com
source:
- !ruby/object:SOAP::Mapping::Object
__xmlattr: {}
__xmlele:
- - &id023 !ruby/object:XSD::QName
name: APIAddressBook <-- this bastard is enumerable :) YAY! so it's result["ListAddressBooksResult"]["APIAddressBook"].each
namespace: http://apiconnector.com
source:
- - !ruby/object:SOAP::Mapping::Object
The above is a result from DotMailer's API, which I spent the last hour trying to figure out how to enumerate over the results. The above is the technique I used to figure out what the heck is going on. I think it beats using REXML etc this way, I could do something like this:
result['ListAddressBooksResult']['APIAddressBook'].each {|book| puts book["Name"]}
Well, I hope this helps anyone else who is looking.
/jason