How to generate JSON data incremental? - ruby

Question

I have a data file about 1000000 lines, 300+ Mb. I want to convert it to a JSON file.
JSON.generate and Hash.merge can help generate JSON, but I need to wait for the program to generate one whole hash. That costs too much time.
I want to write JSON to file incrementally.
Here is my code:
require 'yajl/json_gem'
my_hash = {}
fd1 = File.open("foo.json", "w")
fd2 = File.open("foo.log")
fd2.each_line do | line |
l = fd2.lineno
remote_addr = line.split(" ")[0]
time_local = line.split("]")[0].split("[")[1]
item = {l => {:remote_addr => remote_addr, :time_local => time_local}}
# {
# 1: {
# remote_addr: "1.2.3.4",
# time_local: ""
# },
# 2: {
# ...
# },
# ...
# }
my_hash.merge!(item)
end
fd2.close
fd1.puts JSON.generate(my_hash)
fd1.close
Here is my data:
// access.log
1.2.3.4 - - [02/Apr/2014:03:23:06 +0800] "GET /index" 200 1 "http://foo" "Mozilla/5.0" "-"
Any ideas? Thanks in advance.
--
edit:
Actually, I'd better convert raw data like this:
[
{
id: "1",
remote_addr: "1.2.3.4",
time_local: "02/Apr/2014:03:23:06 +0800"
},
{
...
},
{
...
}
]

Answer 1

This is basic code to generate a valid JSON output based on the sample line:
require 'json'
lines = [
'1.2.3.4 - - [02/Apr/2014:03:23:06 +0800] "GET /index" 200 1 "http://foo" "Mozilla/5.0" "-"',
'1.2.3.5 - - [03/Apr/2014:03:23:06 +0800] "GET /index" 200 1 "http://foo" "Mozilla/5.0" "-"'
]
lines.each_with_index do |line, l|
puts '{' if l == 0
remote_addr, time_local = /^(\S+) .+ \[(.+)\]/.match(line).captures
print '"%d":{"remote_addr":"%s","time_local":"%s"}' % [l + 1, remote_addr, time_local]
puts ',' if l + 1 < lines.size
end
puts "\n}"
# >> {
# >> "1":{"remote_addr":"1.2.3.4","time_local":"02/Apr/2014:03:23:06 +0800"},
# >> "2":{"remote_addr":"1.2.3.5","time_local":"03/Apr/2014:03:23:06 +0800"}
# >> }
You'll need to convert the code to read and write files, or maybe read a single file and redirect the output to capture it. You'll also need to figure out how to determine the number of lines in a file to make it possible to find the places to output the commas. It's easy, and I know examples of doing that exist here on Stack Overflow.
In your original code you need to use the block form for open, rather than assigning to a variable and then explicitly closing the files; Using the block form of open is the Ruby-way. Also, use File.foreach to read through a file line-by-line.

Answer 2

#theTinMan posted a good answer, I will just add that a little more generic solution for a list of JSON object is to construct the list yourself, but let JSON generate each item:
fd1.puts '{'
fd1 = File.open("foo.json", "w")
fd2 = File.open("foo.log")
first_line = true
fd2.each_line do | line |
fd1.puts(',') unless first_line
first_line = false
l = fd2.lineno
remote_addr = line.split(" ")[0]
time_local = line.split("]")[0].split("[")[1]
fd1.print "\"#{l}\": #{JSON.generate(:remote_addr => remote_addr, :time_local => time_local)}"
end
fd1.puts "\n}"