How to store a collection of floats in postgres? - ruby

I'm building a script that saves tweets to a postgres database using ruby/pg/ActiveRecord/TweetStream (gem).
This script works fine..
...
TweetStream::Client.new.track(SEARCH_TERMS) do |t|
puts "#{t.text}"
attributes = {
tweetid: t[:id],
text: t.text,
in_reply_to_user_id: t.in_reply_to_user_id,
in_reply_to_status_id: t.in_reply_to_status_id,
received_at: t.created_at,
user_statuses_count: t.user.statuses_count,
user_followers_count: t.user.followers_count,
user_profile_image_url: t.user.profile_image_url,
user_screen_name: t.user.screen_name,
user_timezone: t.user.time_zone,
user_location: t.user.location,
user_lang: t.lang,
user_id_str: t.user.id,
user_name: t.user.name,
user_url: t.user.url,
user_created_at: t.user.created_at,
user_geo_enabled: t.user.geo_enabled,
}
if StoreTweet.create(attributes)
puts "saved"
else
puts "error"
end
end
until I add
user_geo_enabled: t.user.geo_enabled,
coordinates: t.coordinates.coordinates,}
I also tried
t.coordinates
t.coordiantes[:coordinates]
t[:coordinates] #this allows me to save but is always blank if when geo_enabled is 'true'
Twitter dev center (https://dev.twitter.com/docs/platform-objects/tweets) tells me that 'coordinates' is a collection of floats like:
"coordinates":
{
"coordinates":
[
-75.14310264,
40.05701649
],
"type":"Point"
}
for the moment i use a 'text' field. Which type should i give to the field in order to store both values together?

Related

Generic restructure of ruby hash (with depth of 4)

Is it possible to convert this HASH into an array of arrays based solely on the position of the key (rather than it's value). ie: I know ahead of time that the first Key will always be PROD/ALPHA, and the second Key will always be a country (that I would like to be able to change in the future at will)
The idea would be to group all servers of the same type (webservers) that are also in the same environment (production) but are located in different farms (UK, USA)
While any suggestions on how to do this are welcome, I'll be happy to just know that I'm not walking into a dead-end I won't be able to solve.
Here are some visuals to aid in my explanation:
{
"PROD": {
"USA": {
"generic": [
"nginx-240"
],
"WEB": [
"nginx-210",
"nginx-241",
"nginx-211",
"nginx-209"
],
"APP": [
"tomcat-269",
"tomcat-255",
"tomcat-119",
"tomcat-124"
]
},
"UK": {
"WEB": [
"nginx-249",
"nginx-250",
"nginx-246",
"nginx-247",
"nginx-248"
],
"generic": [
"tomcat-302"
],
"APP": [
"tomcat-396",
"tomcat-156",
"tomcat-157"
]
}
},
"ALPHA": {
"USA": {
"WEB": [
"nginx-144",
"nginx-146",
"nginx-145",
"nginx-175",
"nginx-173"
],
"APP": [
"tomcat-204",
"tomcat-206"
]
}
}
}
The expectation is that data from the lowest level in the hash would be grouped together.
Again the idea is that all Production app servers (both from UK and USA) are grouped together in the following kind of pattern:
PROD_UK_APP would be represented by
["tomcat-396","tomcat-156","tomcat-157"] as these are the lowest branches of the tree PROD->UK->applicationserver
[
[
[PROD_UK_APP],[PROD_USA_APP]
],
[
[PROD_UK_WEB],[PROD_USA_WEB]
]
]
New list..
[
[
[ALPHA_USA_WEB]
],
[
[ALPHA_USA_APP],
[
[
Again the idea is to keep this generic. Is this something that is practically achievable or am I likely to require some degree of hardcoding to ensure it always works? The idea is that if tomorrow UK becomes JAPAN, it will still work in exactly the same way, comparing between the APP and WEB tier of UK, and JAPAN (separating ALPHA from PROD).
EDIT: my attempt to try and sort it:
def walk
a = []
myhash.each do |env, data|
data.each do |dc, tier|
tier.each do |x, y|
a << y
end
end
end
p a
end
[["nginx240"], ["nginx210", "nginx241", "nginx211", "nginx209"], ["tomcat269", "tomcat255", "tomcat119", "tomcat124"], ["nginx249", "nginx250", "nginx246", "nginx247", "nginx248"], ["tomcat302"], ["tomcat396", "tomcat156", "tomcat157"], ["nginx144", "nginx146", "nginx145", "nginx175", "nginx173"], ["tomcat204", "tomcat206"]]
Thanks,
I think I follow what you're looking for and you should get what you're after with:
myhash.values.each_with_object([]) do |by_country, out_arr|
by_country.values.each do |by_type|
out_arr << by_type.values
end
end
which would return:
[
[
[
"nginx-240"
],
[
"nginx-210",
"nginx-241",
"nginx-211",
"nginx-209"
],
[
"tomcat-269",
"tomcat-255",
"tomcat-119",
"tomcat-124"
]
],
[
[
"nginx-249",
"nginx-250",
"nginx-246",
"nginx-247",
"nginx-248"
],
[
"tomcat-302"
],
[
"tomcat-396",
"tomcat-156",
"tomcat-157"
]
],
[
[
"nginx-144",
"nginx-146",
"nginx-145",
"nginx-175",
"nginx-173"
],
[
"tomcat-204",
"tomcat-206"
]
]
]
Piece by piece
Take your hash, disgard the keys and just create an array of values.
iterate over the values (array of hashes by country) and initialize an array to return.
for each hash that by_country points to, again take the values, to drop into the by type(?) hashes
iterate over your by_type hashes and again take the values of each
push each return array into the array you want to return

How to save and display Dashing historical values?

Currently to setup a graph widget, the job should pass all values to be displayed:
data = [
{ "x" => 1980, "y" => 1323 },
{ "x" => 1981, "y" => 53234 },
{ "x" => 1982, "y" => 2344 }
]
I would like to read just current (the latest) value from my server, but previous values should be also displayed.
It looks like I could create a job, which will read the current value from the server, but remaining values to be read from the Redis (or sqlite database, but I would prefer Redis). The current value after that should be saved to the database.
I never worked with Ruby and Dashing before, so the first question I have - is it possible? If I will use Redis, then the question is how to store the data since this is key-value database. I can keep it as widget-id-1, widget-id-2, widget-id-3 ... widget-id-N etc., but in this case I will have to store N value (like widget-id=N). Or, is there any better way?
I came to the following solution:
require 'redis' # https://github.com/redis/redis-rb
redis_uri = URI.parse(ENV["REDISTOGO_URL"])
redis = Redis.new(:host => redis_uri.host, :port => redis_uri.port, :password => redis_uri.password)
if redis.exists('values_x') && redis.exists('values_y')
values_x = redis.lrange('values_x', 0, 9) # get latest 10 records
values_y = redis.lrange('values_y', 0, 9) # get latest 10 records
else
values_x = []
values_y = []
end
SCHEDULER.every '10s', :first_in => 0 do |job|
rand_data = (Date.today-rand(10000)).strftime("%d-%b") # replace this line with the code to get your data
rand_value = rand(50) # replace this line with the code to get your data
values_x << rand_data
values_y << rand_value
redis.multi do # execute as a single transaction
redis.lpush('values_x', rand_data)
redis.lpush('values_y', rand_value)
# feel free to add more datasets values here, if required
end
data = [
{
label: 'dataset-label',
fillColor: 'rgba(220,220,220,0.5)',
strokeColor: 'rgba(220,220,220,0.8)',
highlightFill: 'rgba(220,220,220,0.75)',
highlightStroke: 'rgba(220,220,220,1)',
data: values_y.last(10) # display last 10 values only
}
]
options = { scaleFontColor: '#fff' }
send_event('barchart', { labels: values_x.last(10), datasets: data, options: options })
end
Not sure if everything is implemented correctly here, but it works.

How could I parse the XML into hash

I hope I can get the list of hashed like that.
Is there any gem can do me a favor ?
Expected result
[
{
"prog_name": "TAIWAN CTA Index",
"prog_id": 9
},
{
"prog_name": "CTO CTA Index",
"prog_id": 12
},
]
Original input file source.xml
<prog>
<prog_name>TAIWAN CTA Index</prog_name>
<prog_id>9</prog_id>
</prog>
<prog>
<prog_name>CTO CTA Index</prog_name>
<prog_id>12</prog_id>
</prog>
...
You should have a look at Nokogiri. Something like:
#doc = Nokogiri::XML(<IO thing here>)
#doc.xpath('prog').map do |prog_element|
{
'prog_name' => prog_element.xpath('prog_name').content,
'prog_id' => prog_element.xpath('prog_id').content
}
end
would do it for you.

How to I reference an array member in Ruby?

Given this array in Ruby:
myarray = [name: "John", age: 35]
How do I refer to the age?
I tried myarray[:age] but got an error can't convert Symbol into Integer
Update:
I was trying to simplify my question by extracting what I thought my problem is. I may not understand completely.
I'm experimenting with Dashing and trying to send a number to a meter widget. I've created a variable, 'response_raw' and am trying to send it in the third send event. Here's my code:
SCHEDULER.every '1m', :first_in => 0 do
# Get checks
url = "https://#{CGI::escape user}:#{CGI::escape password}#api.pingdom.com/api/2.0/checks"
`enter code here`response = RestClient.get(url, {"App-Key" => api_key})
response = JSON.parse(response.body, :symbolize_names => true)
if response[:checks]
checks = response[:checks].map { |check|
if check[:status] == 'up'
state = 'up'
last_response_time = "#{check[:lastresponsetime]}ms"
response_raw = check[:lastresponsetime]
else
state = 'down'
last_response_time = "DOWN"
response_raw = 0
end
{ name: check[:name], state: state, lastRepsonseTime: last_response_time, pt: response_raw }
}
else
checks = [name: "pingdom", state: "down", lastRepsonseTime: "-", pt: 0]
end
checks.sort_by { |check| check['name'] }
send_event('pingdom', { checks: checks })
send_event('pingdom-meter', { value: checks[:pt] })
end
In CoffeeScript [name: "John", age: 35] is an array containing single object with two properties (name and age).
Here is how it'll look in plain JavaScript:
myarray = [
{
name: "John",
age: 35
}
];
So, answering your question, to access an age you should take the first element of an array and then reference an age property:
myarray[0].age
or
myarray[0]['age']
But, judging from your question, your're probably using wrong data structure. Why don't you want to use a plain object instead of an array?
person = name: "John", age: 35
console.log "#{person.name}'s age is #{person.age}"
Update
It looks like your question is actually about Ruby and not about CoffeeScript. Though, my answer will remain the same.
To access an age you should take the first element of an array and then reference an age property:
myarray[0][:age]
Since myarray is an array, Ruby expects an integer index, but you're giving it symbol :age instead.
I finally figured it out with Leonid's help. Thank you.
I changed:
send_event('pingdom-meter', { value: checks[:pt] })
to
send_event('pingdom-meter', { value: checks[0][:pt] })

JSON to CSV via FasterCSV

I'm new to Ruby and had a question. I'm trying to create a .rb file that converts JSON to CSV.
I came across some disparate sources that got me to make:
require "rubygems"
require 'fastercsv'
require 'json'
csv_string = FasterCSV.generate({}) do |csv|
JSON.parse(File.open("small.json").read).each do |hash|
csv << hash
end
end
puts csv_string
Now, it does in fact output text but they are all squashed together without spaces, commas etc. How do I make it more customised, clear for a CSV file so I can export that file?
The JSON would look like:
{
"results": [
{
"reportingId": "s",
"listingType": "Business",
"hasExposureProducts": false,
"name": "Medeco Medical Centre World Square",
"primaryAddress": {
"geoCodeGranularity": "PROPERTY",
"addressLine": "Shop 9.01 World Sq Shopng Cntr 644 George St",
"longitude": "151.206172",
"suburb": "Sydney",
"state": "NSW",
"postcode": "2000",
"latitude": "-33.876416",
"type": "VANITY"
},
"primaryContacts": [
{
"type": "PHONE",
"value": "(02) 9264 8500"
}
]
},xxx
}
The CSV to just have something like:
reportingId, s, listingType, Business, name, Medeco Medical...., addressLine, xxxxx, longitude, xxxx, latitude, xxxx, state, NSW, postcode, 2000, type, phone, value, (02) 92648544
Since your JSON structure is a mix of hashes and lists, and also has levels of different heights, it is not as trivial as the code you show. However (assuming your input files always look the same) it shouldn't be hard to write an appropriate converter. On the lowest level, you can transform a hash to CSV by
hash.to_a.flatten
E.g.
input = JSON.parse(File.open("small_file.json").read)
writer = FasterCSV.open("out.csv", "w")
writer << input["results"][0]["primaryAddress"].to_a.flatten
will give you
type,VANITY,latitude,-33.876416,postcode,2000,state,NSW,suburb,Sydney,longitude,151.206172,addressLine,Shop 9.01 World Sq Shopng Cntr 644 George St,geoCodeGranularity,PROPERTY
Hope that guides you the direction.
Btw, your JSON looks invalid. You should change the },xxx line to }].

Resources