Related
I am writing a ruby script to accept an input file that is a CSV. I want to make a pretty JSON file. I feel that I am so close but I can't seem to get there. My original project was in JS but the requirements have changed to make it a Ruby file.
My input file looks like this
item id,description,price,cost,price_type,quantity_on_hand,size_1_name,size_1_price,size_2_name,size_2_price,size_3_name,size_3_price
one thing to note is that some of the values in the CSV file maybe missing because it doesn't exist.
require 'csv'
require 'json'
def is_int(str)
return !!(str =~ /^[-+]?[1-9]([0-9]*)?$/)
end
lines = CSV.open(ARGV[0],{:col_sep => ","}).readlines
# remove first entry of the lines array
keys = lines.shift
lines.each do |values|
# convert the line into a hash and transform string into int
#hash=Hash[keys.zip(values.map{|val| is_int(val) ? val.to_i : val}) ]
hash = keys.zip(values.map{ val})
# Write a file with the hash results
File.open("#{hash['NAME']}.json", "w") do |f|
f.write JSON.pretty_generate [hash]
end
end
The output I am trying to get is
[
{
id: 111010,
description: 'Coffee',
price: 1.25,
cost: 0.80,
price_type: 'system',
quantity_on_hand: 100000,
modifiers: [
{
name: 'Small',
price: -0.25
},{
name: 'Medium',
price: 0.00
},{
name: 'Large',
price: 0.30
}
]
the version of Ruby I'm using is 2.0.0p481
Error was
usr/lib/ruby/2.0.0/csv.rb:1254:in `initialize': no implicit conversion of nil into String (TypeError)
from /usr/lib/ruby/2.0.0/csv.rb:1254:in `open'
from /usr/lib/ruby/2.0.0/csv.rb:1254:in `open'
from stockimporter.rb:8:in `<main>'
csv.csv:
id,description,price,cost,price_type,quantity_on_hand,size_1_name,size_1_price,size_2_name,size_2_price,size_3_name,size_3_price
111010,Coffee,1.25,0.80,system,10000,Small,-0.25,Medium,0.00,Large,0.30
111011,Tea,1.00,0.50,system,100,Small,-0.10,Medium,0.00,Large,0.10
111012,MissingInfo,1.00,0.50,,100,,,Medium,,Large,0.10
...
require 'csv'
require 'pp'
require 'json'
csv_options = {
headers: true, #skip first line of csv file
converters: [:numeric] #convert strings that look like integers or floats to integers or floats
}
results = []
CSV.foreach('csv.csv', csv_options) do |row|
record = Hash[row.take(6)]
modifiers = [
{
name: row["size_1_name"],
price: row["size_1_price"]
},
{
name: row["size_2_name"],
price: row["size_2_price"],
},
{
name: row["size_3_name"],
price: row["size_3_price"]
}
]
record['modifiers'] = modifiers
results << record
end
pp results
--output:--
[{"id"=>111010,
"description"=>"Coffee",
"price"=>1.25,
"cost"=>0.8,
"price_type"=>"system",
"quantity_on_hand"=>10000,
"modifiers"=>
[{:name=>"Small", :price=>-0.25},
{:name=>"Medium", :price=>0.0},
{:name=>"Large", :price=>0.3}]},
{"id"=>111011,
"description"=>"Tea",
"price"=>1.0,
"cost"=>0.5,
"price_type"=>"system",
"quantity_on_hand"=>100,
"modifiers"=>
[{:name=>"Small", :price=>-0.1},
{:name=>"Medium", :price=>0.0},
{:name=>"Large", :price=>0.1}]},
{"id"=>111012,
"description"=>"MissingInfo",
"price"=>1.0,
"cost"=>0.5,
"price_type"=>nil,
"quantity_on_hand"=>100,
"modifiers"=>
[{:name=>nil, :price=>nil},
{:name=>"Medium", :price=>nil},
{:name=>"Large", :price=>0.1}]}]
json = JSON.pretty_generate(results)
puts json
--output:--
[
{
"id": 111010,
"description": "Coffee",
"price": 1.25,
"cost": 0.8,
"price_type": "system",
"quantity_on_hand": 10000,
"modifiers": [
{
"name": "Small",
"price": -0.25
},
{
"name": "Medium",
"price": 0.0
},
{
"name": "Large",
"price": 0.3
}
]
},
{
"id": 111011,
"description": "Tea",
"price": 1.0,
"cost": 0.5,
"price_type": "system",
"quantity_on_hand": 100,
"modifiers": [
{
"name": "Small",
"price": -0.1
},
{
"name": "Medium",
"price": 0.0
},
{
"name": "Large",
"price": 0.1
}
]
},
{
"id": 111012,
"description": "MissingInfo",
"price": 1.0,
"cost": 0.5,
"price_type": null,
"quantity_on_hand": 100,
"modifiers": [
{
"name": null,
"price": null
},
{
"name": "Medium",
"price": null
},
{
"name": "Large",
"price": 0.1
}
]
}
]
You could also do it like this:
CSV.foreach('csv.csv', csv_options) do |row|
record = Hash[row.take(6)]
price_adjustments = row.drop(6)
# [["size_1_name", "Small"], ["size_1_price", -0.25]]
# |---------------------------------------------------|
# ^
# |
modifiers = price_adjustments.each_slice(2).map do |size_price|
size_price.first[0] = 'name'
size_price.last[0] = 'price'
Hash[size_price]
end
p modifiers #e.g. [{"name"=>"Small", "price"=>-0.25}, {"name"=>"Medium", "price"=>0.0}, {"name"=>"Large", "price"=>0.3}]
record['modifiers'] = modifiers
results << record
end
pp results
I have the following array:
[ { "attributes": {
"id": "usdeur",
"code": 4
},
"name": "USD/EUR"
},
{ "attributes": {
"id": "eurgbp",
"code": 5
},
"name": "EUR/GBP"
}
]
How can I get both ids for futher processing as output?
I tried a lot but no success. My problem is I always get only one id as output:
Market.all.select.each do |market|
present market.id
end
Or:
Market.all.each{|attributes| present attributes[:id]}
which gives me only "eurgbp" as a result while I need both ids.
JSON#parse should help you with this
require 'json'
json = '[ { "attributes": {
"id": "usdeur",
"code": 4
},
"name": "USD/EUR"
},
{ "attributes": {
"id": "eurgbp",
"code": 5
},
"name": "EUR/GBP"
}]'
ids = JSON.parse(json).map{|hash| hash['attributes']['id'] }
#=> ["usdeur", "eurgbp"]
JSON#parse turns a jSON response into a Hash then just use standard Hash methods for access.
I'm going to assume that the data is JSON that you're parsing (with JSON.parse) into a Ruby Array of Hashes, which would look like this:
hashes = [ { "attributes" => { "id" => "usdeur", "code" => 4 },
"name" => "USD/EUR"
},
{ "attributes" => { "id" => "eurgbp", "code" => 5 },
"name" => "EUR/GBP"
} ]
If you wanted to get just the first "id" value, you'd do this:
first_hash = hashes[0]
first_hash_attributes = first_hash["attributes"]
p first_hash_attributes["id"]
# => "usdeur"
Or just:
p hashes[0]["attributes"]["id"]
# => "usdeur"
To get them all, you'll do this:
all_attributes = hashes.map {|hash| hash["attributes"] }
# => [ { "id" => "usdeur", "code" => 4 },
# { "id" => "eurgbp", "code" => 5 } ]
all_ids = all_attributes.map {|attrs| attrs["id"] }
# => [ "usdeur", "eurgbp" ]
Or just:
p hashes.map {|hash| hash["attributes"]["id"] }
# => [ "usdeur", "eurgbp" ]
JSON library what using Rails is very slowly...
I prefer to use:
gem 'oj'
from https://github.com/ohler55/oj
fast and simple! LET'S GO!
CSV
id,modifier1_name,modifier2_price,modifier2_name,modifier2_price,modifier2_status
1,'Small',10,'Large',20,'YYY'
2,'Small',20,'Large',30,'YYY'
JSON
[
{
id: 1,
modifier: [
{name: 'Small', price: 10},
{name: 'Large', price: 20, status: 'YYY'}]
},
{
id: 2,
modifier: [
{name: 'Small', price: 20},
{name: 'Large', price: 30, status: 'YYY'}],
}
]
How to convert CSV to Json in this case when modifiers can be different ?
You will need to map the modifiers yourself, as there is no built-in method to map hash values into an array from your logic:
JSON.pretty_generate(CSV.open('filename.csv', headers: true).map do |row|
modifier = {}
row.each do |k, v|
if k =~ /modifier(.)_(.*)$/
(modifier[$1] ||= {})[$2] = v
end
end
{ id: row['id'],
modifier: modifier.sort_by { |k, v| k }.map {|k, v| v }
}
end)
For the file*
id,modifier1_name,modifier1_price,modifier2_name,modifier2_price,modifier2_status
1,Small,10,Large,20,YYY
2,Small,20,Large,30,YYY
*I made some changes to the file you show, since it will not give you the required result - you state modifier2_price twice, for example
You will get:
[
{
"id": "1",
"modifier": [
{
"name": "Small",
"price": "10"
},
{
"name": "Large",
"price": "20",
"status": "YYY"
}
]
},
{
"id": "2",
"modifier": [
{
"name": "Small",
"price": "20"
},
{
"name": "Large",
"price": "30",
"status": "YYY"
}
]
}
]
require 'csv'
require 'json'
CSV.open('filename.csv', :headers => true).map { |x| x.to_h }.to_json
I have a simple dataset in a MongoDB (version 2.2.2):
{
"_id": "527d60865593622ba17643e6",
"siteid": "100",
"date": 1383940800,
"visits": 1
},
{
"_id": "527d60865593622ba17643e7",
"siteid": "200",
"date": 1383940801,
"visits": 1
},
{
"_id": "527d60865593622ba17643e8",
"siteid": "200",
"date": 1383940802,
"visits": 1
}
I have a Mongoid (version 3.0.1) collection where I'm trying to use the aggregate method:
query = {
"$project" => { _id: 0 },
"$match" => {"$and" => [{:date=>{"$gte"=>1383940800}}, {:date=>{"$lt"=>1384027200}}]},
"$group" => {"_id" => "$siteid", "visits" => {"$sum" => "$visits"}},
"$project" => { "siteid" => "$_id" }
}
result = #object.collection.aggregate(query)
The result that I get back is this:
[{"_id"=>200, "visits"=>2}, {"_id"=>100, "visits"=>1}]
What I'd like the result to be is this:
[{"siteid"=>200, "visits"=>2}, {"siteid"=>100, "visits"=>1}]
I thought the last $project operation would do this for me but it seems to be completely ignored. Any suggestions?
The argument to the aggregate method is a pipeline which is an array of hashes where each hash is a stage.
Your example query is a hash with values that are hashes; too bad that it reads well, but this obscures the misuse.
Unfortunately, Moped somehow allows this misuse even though the mongo shell catches it and returns the server error message
"exception: A pipeline stage specification object must contain exactly one field."
I hope that you like the following, I think that it is what you want.
test/unit/visit_test.rb
require 'test_helper'
require 'json'
class VisitTest < ActiveSupport::TestCase
def setup
Visit.delete_all
puts
end
test "0. mongoid version" do
puts "Mongoid::VERSION:#{Mongoid::VERSION}\nMoped::VERSION:#{Moped::VERSION}"
end
test "project after group" do
data = JSON.parse(<<-EOD
[
{"_id": "527d60865593622ba17643e6", "siteid": "100", "date": 1383940800, "visits": 1},
{"_id": "527d60865593622ba17643e7", "siteid": "200", "date": 1383940801, "visits": 1},
{"_id": "527d60865593622ba17643e8", "siteid": "200", "date": 1383940802, "visits": 1}
]
EOD
)
Visit.create(data)
assert_equal 3, Visit.count
pipeline = [
{"$match" => {"$and" => [{:date => {"$gte" => 1383940800}}, {:date => {"$lt" => 1384027200}}]}},
{"$group" => {"_id" => "$siteid", "visits" => {"$sum" => "$visits"}}},
{"$project" => {"_id" => 0, "siteid" => "$_id", "visits" => 1}}
]
p Visit.collection.aggregate(pipeline)
end
end
rake test
Run options:
# Running tests:
[1/2] VisitTest#test_0._mongoid_version
Mongoid::VERSION:3.1.5
Moped::VERSION:1.5.1
[2/2] VisitTest#test_project_after_group
[{"visits"=>2, "siteid"=>"200"}, {"visits"=>1, "siteid"=>"100"}]
Finished tests in 0.038172s, 52.3944 tests/s, 26.1972 assertions/s.
2 tests, 1 assertions, 0 failures, 0 errors, 0 skips
I have two json files that I'm trying to merge. The JSONs have different formatting (see below). I'd like to merge records, so [0] from file one and [0] from file two would become one record [0] in the new merged file.
The first JSON (file_a.json), appears like so:
{
"query": {
"count": 4,
"created": "2012-11-21T23:07:00Z",
"lang": "en-US",
"results": {
"quote": [
{
"Name": "Bill",
"Age": "46",
"Number": "3.55"
},
{
"Name": "Jane",
"Age": "33",
"Number": nil
},
{
"Name": "Jack",
"Age": "55",
"Number": nil
},
{
"Name": "Xavier",
"Age": nil,
"Number": "153353535"
}
]
}
}
}
The second JSON (file_b.json) appears like so:
[
{
"Number2": 25253,
"Number3": 435574,
"NAME": "Bill"
},
{
"Number2": 345353,
"Number3": 5566,
"NAME": "Jane"
},
{
"Number2": 56756,
"Number3": 232435,
"NAME": "Jack"
},
{
"Number2": 7457,
"Number3": 45425,
"NAME": "Xavier"
}
]
None of the keys are the same in both JSONs (well, actually "Name" is a key in both, but in the first the key is "Name" and in the second its "NAME" - just so I can check that the merge works correctly - so I want "Name" and "NAME" in the final JSON), the first record in the first file matches with the first record in the second file, and so on.
So far, I tried merging like this:
merged = %w[a b].inject([]) { |m,f| m << JSON.parse(File.read("file_#{f}.json")) }.flatten
But this of course merged them, but not how I wanted them merged (they are merged sucessively, and because of the different formatting, it gets quite ugly).
I also tried merging like this:
a = JSON.parse(File.read("file_a.json"))
b = JSON.parse(File.read("file_b.json"))
merged = a.zip(b)
Came closer but still not correct and the formatting was still horrendous.
In the end, what I want is this (formatting of second JSON - headers from first JSON can be junked):
[
{
"Name": "Bill",
"Age": 46,
"Number": 3.55,
"Number2": 25253,
"Number3": 435574,
"NAME": "Bill"
},
{
"Name": "Jane",
"Age": 33,
"Number": nil,
"Number2": 345353,
"Number3": 5566,
"NAME": "Jane"
},
{
"Name": "Jack",
"Age": 55,
"Number": nil,
"Number2": 56756,
"Number3": 232435,
"NAME": "Jack"
},
{
"Name": "Xavier",
"Age": nil,
"Number": 153353535,
"Number2": 7457,
"Number3": 45425,
"NAME": "Xavier"
}
]
Any help is appreciated. Thanks a lot.
Hеllo, seems format changed from last time :)
UPDATE: more readable version that also convert corresponding values to integers/floats:
require 'json'
require 'ap'
a = JSON.parse(File.read('./a.json'))['query']['results']['quote'] rescue []
b = JSON.parse(File.read('./b.json'))
final = []
a.each_with_index do |ah,i|
unless bh = b[i]
bh = {}
puts "seems b has no #{i} key, merging skipped"
end
final << ah.merge(bh).inject({}) do |f, (k,v)|
if v.is_a?(String)
if v =~ /\A\d+\.\d+\Z/
v = v.to_f
elsif v =~ /\A\d+\Z/
v = v.to_i
end
end
f.update k => v
end
end
ap final
will display:
[
[0] {
"Name" => "Bill",
"Age" => 46,
"Number" => 3.55,
"Number2" => 25253,
"Number3" => 435574,
"NAME" => "Bill"
},
[1] {
"Name" => "Jane",
"Age" => 33,
"Number" => nil,
"Number2" => 345353,
"Number3" => 5566,
"NAME" => "Jane"
},
[2] {
"Name" => "Jack",
"Age" => 55,
"Number" => nil,
"Number2" => 56756,
"Number3" => 232435,
"NAME" => "Jack"
},
[3] {
"Name" => "Xavier",
"Age" => nil,
"Number" => 153353535,
"Number2" => 7457,
"Number3" => 45425,
"NAME" => "Xavier"
}
]
Here is a working demo
Btw, your json is a bit wrong in both files.
See the fixed versions here and here