I have a log file that appears to have a time stamp followed by a tab, then a string of JSON.
10/28/2014 00:04:51 {"servers":[{"id":833495,"account_id":39033,"name":"USTSMASCMSP113","host":"USTSMASCMSP113","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":28.7,"cpu_stolen":0,"disk_io":0.17,"memory":60.8,"memory_used":10444865536,"memory_total":17176723456,"fullest_disk":84.0,"fullest_disk_free":8171000000},"links":{"alert_policy":28370}},{"id":831246,"account_id":39033,"name":"USTSMASCMSP118","host":"USTSMASCMSP118","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":30.7,"cpu_stolen":0,"disk_io":2.57,"memory":54.3,"memory_used":9329180672,"memory_total":17176723456,"fullest_disk":83.1,"fullest_disk_free":8653000000},"links":{"alert_policy":28370}},{"id":833455,"account_id":39033,"name":"USTSMASCMSP119","host":"USTSMASCMSP119","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":36.3,"cpu_stolen":0,"disk_io":0.38,"memory":54.5,"memory_used":9362735104,"memory_total":17176723456,"fullest_disk":73.7,"fullest_disk_free":17912000000},"links":{"alert_policy":28370}},{"id":838342,"account_id":39033,"name":"USTSMASCMSP120","host":"USTSMASCMSP120","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":29.6,"cpu_stolen":0,"disk_io":0,"memory":55.6,"memory_used":9550430208,"memory_total":17176723456,"fullest_disk":84.9,"fullest_disk_free":10259000000},"links":{"alert_policy":28370}},{"id":833500,"account_id":39033,"name":"USTSMASCMSP121","host":"USTSMASCMSP121","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":38.9,"cpu_stolen":0,"disk_io":5.47,"memory":56.7,"memory_used":9743368192,"memory_total":17176723456,"fullest_disk":84.9,"fullest_disk_free":10288000000},"links":{"alert_policy":28370}},{"id":831221,"account_id":39033,"name":"USTSMASCMSP123","host":"USTSMASCMSP123","health_status":"orange","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":32.4,"cpu_stolen":0,"disk_io":0.42,"memory":53.8,"memory_used":9232711680,"memory_total":17176723456,"fullest_disk":87.2,"fullest_disk_free":8705000000},"links":{"alert_policy":28370}},{"id":833466,"account_id":39033,"name":"USTSMASCMSP124","host":"USTSMASCMSP124","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":25.1,"cpu_stolen":0,"disk_io":4.28,"memory":58.6,"memory_used":10065281024,"memory_total":17176723456,"fullest_disk":74.2,"fullest_disk_free":19387000000},"links":{"alert_policy":28370}},{"id":838346,"account_id":39033,"name":"USTSMASCMSP125","host":"USTSMASCMSP125","health_status":"orange","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":32.2,"cpu_stolen":0,"disk_io":0.13,"memory":56.5,"memory_used":9703522304,"memory_total":17176723456,"fullest_disk":89.1,"fullest_disk_free":7443000000},"links":{"alert_policy":28370}},{"id":833504,"account_id":39033,"name":"USTSMASCMSP126","host":"USTSMASCMSP126","health_status":"orange","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":29.6,"cpu_stolen":0,"disk_io":0,"memory":56.9,"memory_used":9776922624,"memory_total":17176723456,"fullest_disk":85.7,"fullest_disk_free":9761000000},"links":{"alert_policy":28370}},{"id":831212,"account_id":39033,"name":"USTSMASCMSP127","host":"USTSMASCMSP127","health_status":"orange","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":30.5,"cpu_stolen":0,"disk_io":0,"memory":50.9,"memory_used":8735686656,"memory_total":17176723456,"fullest_disk":85.0,"fullest_disk_free":10222000000},"links":{"alert_policy":28370}},{"id":833510,"account_id":39033,"name":"USTSMASCMSP128","host":"USTSMASCMSP128","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":26.8,"cpu_stolen":0,"disk_io":0,"memory":57.8,"memory_used":9936306176,"memory_total":17176723456,"fullest_disk":79.7,"fullest_disk_free":10393000000},"links":{"alert_policy":28370}},{"id":833473,"account_id":39033,"name":"USTSMASCMSP129","host":"USTSMASCMSP129","health_status":"orange","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":31.9,"cpu_stolen":0,"disk_io":0.17,"memory":56.4,"memory_used":9693036544,"memory_total":17176723456,"fullest_disk":86.7,"fullest_disk_free":9095000000},"links":{"alert_policy":28370}},{"id":3922351,"account_id":39033,"name":"USTSMASCMSP136","host":"USTSMASCMSP136","health_status":"orange","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":16.2,"cpu_stolen":0,"disk_io":0.28,"memory":13.3,"memory_used":4552916992,"memory_total":34356592640,"fullest_disk":73.3,"fullest_disk_free":56043000000},"links":{"alert_policy":28369}},{"id":831184,"account_id":39033,"name":"USTSMASCMSP144","host":"USTSMASCMSP144","health_status":"orange","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":30.8,"cpu_stolen":0,"disk_io":0.88,"memory":61.8,"memory_used":10617880576,"memory_total":17176723456,"fullest_disk":90.9,"fullest_disk_free":8237000000},"links":{"alert_policy":28370}},{"id":833478,"account_id":39033,"name":"USTSMASCMSP145","host":"USTSMASCMSP145","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":30.5,"cpu_stolen":0,"disk_io":0,"memory":50.3,"memory_used":8631877632,"memory_total":17176723456,"fullest_disk":81.5,"fullest_disk_free":9712000000},"links":{"alert_policy":28370}},{"id":838357,"account_id":39033,"name":"USTSMASCMSP146","host":"USTSMASCMSP146","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":29.7,"cpu_stolen":0,"disk_io":0,"memory":52.8,"memory_used":9069133824,"memory_total":17176723456,"fullest_disk":80.6,"fullest_disk_free":9934000000},"links":{"alert_policy":28370}},{"id":838360,"account_id":39033,"name":"USTSMASCMSP147","host":"USTSMASCMSP147","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":28.7,"cpu_stolen":0,"disk_io":0,"memory":73.6,"memory_used":12640583680,"memory_total":17176723456,"fullest_disk":80.8,"fullest_disk_free":9825000000},"links":{"alert_policy":28370}},{"id":3964940,"account_id":39033,"name":"USTSMASCMSP148","host":"USTSMASCMSP148","health_status":"orange","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":12.6,"cpu_stolen":0,"disk_io":0.08,"memory":10.6,"memory_used":3636461568,"memory_total":34356592640,"fullest_disk":73.2,"fullest_disk_free":56213000000},"links":{"alert_policy":28369}},{"id":832099,"account_id":39033,"name":"USTSMASCMSP196","host":"USTSMASCMSP196","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":24.8,"cpu_stolen":0,"disk_io":0,"memory":30.2,"memory_used":5187305472,"memory_total":17176723456,"fullest_disk":77.2,"fullest_disk_free":11962000000},"links":{"alert_policy":28370}},{"id":832093,"account_id":39033,"name":"USTSMASCMSP197","host":"USTSMASCMSP197","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":24.8,"cpu_stolen":0,"disk_io":0.34,"memory":30.1,"memory_used":5169479680,"memory_total":17176723456,"fullest_disk":74.5,"fullest_disk_free":13362000000},"links":{"alert_policy":28370}},{"id":832082,"account_id":39033,"name":"USTSMASCMSP198","host":"USTSMASCMSP198","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":23.5,"cpu_stolen":0,"disk_io":0,"memory":26.8,"memory_used":4604297216,"memory_total":17176723456,"fullest_disk":72.8,"fullest_disk_free":14268000000},"links":{"alert_policy":28370}},{"id":832077,"account_id":39033,"name":"USTSMASCMSP199","host":"USTSMASCMSP199","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":21.0,"cpu_stolen":0,"disk_io":0,"memory":31.3,"memory_used":5379194880,"memory_total":17176723456,"fullest_disk":71.5,"fullest_disk_free":14950000000},"links":{"alert_policy":28370}},{"id":832832,"account_id":39033,"name":"USTSMASCMSP200","host":"USTSMASCMSP200","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":50.3,"cpu_stolen":0,"disk_io":0,"memory":28.9,"memory_used":4966055936,"memory_total":17176723456,"fullest_disk":74.3,"fullest_disk_free":13477000000},"links":{"alert_policy":28370}},{"id":831194,"account_id":39033,"name":"USTSMASCMSP201","host":"USTSMASCMSP201","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":27.0,"cpu_stolen":0,"disk_io":0,"memory":28.6,"memory_used":4918870016,"memory_total":17176723456,"fullest_disk":74.4,"fullest_disk_free":13397000000},"links":{"alert_policy":28370}},{"id":832059,"account_id":39033,"name":"USTSMASCMSP202","host":"USTSMASCMSP202","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":25.1,"cpu_stolen":0,"disk_io":0,"memory":26.6,"memory_used":4564451328,"memory_total":17176723456,"fullest_disk":68.3,"fullest_disk_free":16645000000},"links":{"alert_policy":28370}},{"id":832043,"account_id":39033,"name":"USTSMASCMSP203","host":"USTSMASCMSP203","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":24.7,"cpu_stolen":0,"disk_io":1.17,"memory":25.2,"memory_used":4326424576,"memory_total":17176723456,"fullest_disk":68.2,"fullest_disk_free":16646000000},"links":{"alert_policy":28370}},{"id":832034,"account_id":39033,"name":"USTSMASCMSP204","host":"USTSMASCMSP204","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":26.5,"cpu_stolen":0,"disk_io":0,"memory":27.1,"memory_used":4653580288,"memory_total":17176723456,"fullest_disk":68.5,"fullest_disk_free":16510000000},"links":{"alert_policy":28370}},{"id":832031,"account_id":39033,"name":"USTSMASCMSP205","host":"USTSMASCMSP205","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":27.8,"cpu_stolen":0,"disk_io":0,"memory":24.9,"memory_used":4285530112,"memory_total":17176723456,"fullest_disk":68.8,"fullest_disk_free":16376000000},"links":{"alert_policy":28370}},{"id":832024,"account_id":39033,"name":"USTSMASCMSP206","host":"USTSMASCMSP206","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":30.2,"cpu_stolen":0,"disk_io":0,"memory":26.5,"memory_used":4548722688,"memory_total":17176723456,"fullest_disk":67.2,"fullest_disk_free":17181000000},"links":{"alert_policy":28370}},{"id":831202,"account_id":39033,"name":"USTSMASCMSP207","host":"USTSMASCMSP207","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":23.9,"cpu_stolen":0,"disk_io":0,"memory":25.6,"memory_used":4394582016,"memory_total":17176723456,"fullest_disk":67.3,"fullest_disk_free":17146000000},"links":{"alert_policy":28370}},{"id":3979556,"account_id":39033,"name":"USTSMASCMSP227","host":"USTSMASCMSP227","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":8.25,"cpu_stolen":0,"disk_io":5.96,"memory":18.1,"memory_used":3116367872,"memory_total":17174626304,"fullest_disk":65.1,"fullest_disk_free":18301000000},"links":{"alert_policy":28369}},{"id":3965125,"account_id":39033,"name":"USTSMASCMSP247","host":"USTSMASCMSP247","health_status":"green","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":18.2,"cpu_stolen":0,"disk_io":34.0,"memory":32.3,"memory_used":5543821312,"memory_total":17174626304,"fullest_disk":59.9,"fullest_disk_free":28765000000},"links":{"alert_policy":28369}},{"id":3965022,"account_id":39033,"name":"USTSMASCMSP248","host":"USTSMASCMSP248","health_status":"orange","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":43.9,"cpu_stolen":0,"disk_io":8.61,"memory":28.1,"memory_used":4821352448,"memory_total":17143169024,"fullest_disk":86.5,"fullest_disk_free":9645000000},"links":{"alert_policy":28369}},{"id":3851100,"account_id":39033,"name":"ustsmvscmsp888","host":"ustsmvscmsp888","health_status":"orange","reporting":true,"last_reported_at":"2014-10-28T04:04:17+00:00","summary":{"cpu":19.7,"cpu_stolen":0,"disk_io":0,"memory":42.9,"memory_used":1841299456,"memory_total":4293918720,"fullest_disk":72.9,"fullest_disk_free":14243000000},"links":{"alert_policy":28369}}],"links":{"server.alert_policy":"/v2/alert_policies/{alert_policy_id}"}}
I have code that reads the input file and I am attempting to get values from the string like so.
require 'json'
input = File.open("input.txt", 'r')
input.each do |line|
line = line.split("\t")
my_json = line[1].to_json
puts my_json["cpu"]
end
The end result looks like this:
C:\>ruby test.rb
cpu
I am not entirely sure if the string itself is not formatted correctly or if I'm not calling on the value correctly.
Run this code inside console and check the return but i'm pretty sure you can't call
my_json["cpu"]
You will need to call something like:
my_json["servers"].first["cpu"]
my_json = line[1].to_json
Will take an object and convert it to JSON, not parse an existing JSON string into a ruby object. You're going to wind up with an escaped string within a JSON string.
You need something like
my_json = JSON.parse(line[1])
you can verify this has worked correctly by outputting my_json to stdout:
$stdout << my_json.class.name
You should have an instance of Hash or Openstruct.
Consider this:
require 'json'
line = '10/28/2014 00:04:51 {"servers":[{"id":833495,"account_id":39033}]}'
JSON[line[/\{.+}/]]
# => {"servers"=>[{"id"=>833495, "account_id"=>39033}]}
That means you can reduce your parsing code to something like:
File.foreach("input.txt") do |line|
data = JSON[line[/\{.+}/]]
# do something with the data
end
Regular expressions are greedy, so /\{.+}/ will start at the first { and look until it finds the last }, and return the starting and ending braces, and everything in between.
The JSON [] method is smart enough to know that if its parameter is a string it should try to parse it and return a Ruby array or hash. If the parameter is an Array or Hash, it will serialize it.
Note: It isn't necessary, and definitely not desirable to include huge data samples like you did. Reduce it to the bare minimum necessary to demonstrate the problem.