issue with yahoofinance ruby api return value - ruby

I know joining arrays and strings but this issue is very specific to an
api I am working on :
All I want is all hq.close value in an array. EG:
[
{'GOOG' => [744.75,751.48,744.56,744.09,757.84]},
{'MSFT' => {value1,value2, .....}}
]
Reason I am not able to get the above array of hash is because when I do
puts hq.close its printing individual value and I am not sure how to get
all hq.close into one array
Based on code below my output is line seperated : (but i want it in above format)
GOOG -> 744.75
GOOG -> 751.48
GOOG -> 744.56
GOOG -> 744.09
GOOG -> 757.84
MSFT -> 29.2
MSFT -> 28.95
MSFT -> 28.98
MSFT -> 29.28
MSFT -> 29.78
+++++++++++++++++++++++++++
Code:
require 'yahoofinance'
#Stock symbols
user_input = ['GOOG','MSFT']
user_input.each do|symb|
YahooFinance::get_HistoricalQuotes( symb,
Date.parse('2012-10-06'),
Date.today()) do |hq|
puts "#{symb} -> #{hq.close}"
end
end

You could inject over each value to build your hash:
require 'yahoofinance'
company_symbols = ['GOOG','MSFT']
start_date = Date.parse('2012-10-06')
stop_date = Date.today()
company_symbols.inject({}) do |memo, value|
memo[value] ||= []
YahooFinance.get_HistoricalQuotes(value, start_date, stop_date) do |hq|
memo[value] << hq.close
end
memo
end
# => {"GOOG"=>[744.75, 751.48, 744.56, 744.09, 757.84], "MSFT"=>[29.2, 28.95, 28.98, 29.28, 29.78]}

Related

Parsing recursive hashes into data records in Ruby

I've struggled with this problem for a while, and I'm finally going to ask here for help.
Take a very straightforward hash that represents some event:
{
:eventkey=>"someeventkey",
:web_id=>"77d5f434-5a40-4582-88e8-9667b7774c7d",
:apikey=>"eaf3b6e1-b020-41b6-b67f-98f1cc0a9590",
:details=> {
:phone=>"1-936-774-6886",
:email=>"dasia_schuster#wisokytrantow.com",
:pageUrl=>"http://ortiz.info/joe"
}
}
My goal is to create a 'master record' for the entire hash, with the fields in the record being all the keys that do not contain values that are also hashes. When I run into a value that is a hash (in this case 'details'), I need to create a separate record for each k/v pair in that hash bearing the same record id as the parent master record.
I'm not getting the recursion right somehow. Ideally I would get back a single primary record:
{
:recordid=>"some-generated-record-id",
:web_id=>"77d5f434-5a40-4582-88e8-9667b7774c7d",
:apikey=>"eaf3b6e1-b020-41b6-b67f-98f1cc0a9590",
:details=>nil
}
And a distinct entry for each key in the nested hash:
{
:recordid=>"some-generated-detail-record-id",
:parentid=>"the-parent-id-from-the-master-record",
:phone=>"1-936-774-6886"
}
{
:recordid=>"another-generated-detail-record-id",
:parentid=>"the-same-parent-id-from-the-master-record",
:email=>"dasia_schuster#wisokytrantow.com"
}
And so on. I'm trying to get this set of records back as an array of hashes.
I've gotten as far as being able to generate the master record, as well as a detail record, but the detail record contains all the keys in the detail.
def eventToBreakout(eventhash,sequenceid = -1, parentrecordid = nil, records = [])
recordid = SecureRandom.uuid
sequenceid += 1
recordstruc = {:record_id => recordid, :parent_record_id => parentrecordid, :record_processed_ts => Time.now, :sequence_id => sequenceid}
eventhash.each_pair do |k,v|
if recurse?(v)
eventToBreakout(v,sequenceid,recordid,records)
else
if !recordstruc.keys.include?(k)
recordstruc[k]=v
end
end
end
records << recordstruc
records
end
I've included my code and here is the output I'm currently getting from it.
[{:record_id=>"ed98be89-4c1f-496e-beb4-ede5f38dd549",
:parent_record_id=>"fa77299b-95b0-429d-ad8a-f5d365f2f357",
:record_processed_ts=>2016-04-25 16:46:10 -0500,
:sequence_id=>1,
:phone=>"1-756-608-8114",
:email=>"hipolito_wilderman#morar.co",
:pageUrl=>"http://haag.net/alexie.marvin"},
{:record_id=>"fa77299b-95b0-429d-ad8a-f5d365f2f357",
:parent_record_id=>nil,
:record_processed_ts=>2016-04-25 16:46:10 -0500,
:sequence_id=>0,
:eventts=>2016-04-25 22:10:32 -0500,
:web_id=>"a61c57ae-3a01-4994-8803-8d8292df3338",
:apikey=>"9adbc7a4-03ff-4fcc-ac81-ae8d0ee01ef0"}]
Maybe you want something along these lines?
input = { id: 'parent', value: 'parent value', child: { child_value: 1}}
record = {}
input.each do |k,v|
if v.is_a? Hash
v[:parent_id] = input[:id]
(record[:children] ||= []) << v
else
record[k] = v
end
end
puts record
# {:id=>"parent", :value=>"parent value", :children=>[{:child_value=>1, :parent_id=>"parent"}]}
By the way this is a good example to get started with "spec" or "test" frameworks like minitest or rspec (both can be used for both). You have defined input and expected output already and "just" need to code until all test/spec-runs are green.

How to generate all possible combinations

I have a string array like this:
[
"productRankingOct12:LessPopularityEPC",
"productRankingOct12:CtrEpcJob",
"productRankingOct12:DeviceSpecificLPE",
"guidedSearch:false",
"guidedSearch:true",
"spotfront4:true",
"spotfront4:false",
"postClick4:ipsa",
"postClick4:false",
"trendingModule:false"
]
I have to build URLs based on the given key:value pairs. I want the combinations of the elements:
[
"productRankingOct12:LessPopularityEPC",
"productRankingOct12:CtrEpcJob",
"productRankingOct12:DeviceSpecificLPE",
"guidedSearch:false",
"guidedSearch:true",
"productRankingOct12:LessPopularityEPC & guidedSearch:false"
"productRankingOct12:CtrEpcJob & guidedSearch:false"
"productRankingOct12:DeviceSpecificLPE & guidedSearch:false"
"productRankingOct12:LessPopularityEPC & guidedSearch:true"
"productRankingOct12:CtrEpcJob & guidedSearch:true"
"productRankingOct12:DeviceSpecificLPE & guidedSearch:true"
"productRankingOct12:LessPopularityEPC & guidedSearch:false & spotfront4:true"
"productRankingOct12:CtrEpcJob & guidedSearch:false & "spotfront4:true""
"productRankingOct12:DeviceSpecificLPE & guidedSearch:false & "spotfront4:true""
"productRankingOct12:LessPopularityEPC & guidedSearch:true & "spotfront4:true""
]
I tried to generate permutations using this:
def Automate_AB_tests.permutation(arg_keyVal)
return [arg_keyVal] if arg_keyVal.size < 2
ch = arg_keyVal[0]
permutation(arg_keyVal[1..-1]).each_with_object([]) do |perm,result|
(0..perm.size).each {|i| result << perm.dup.insert(i,ch)}
end
end
but it seems it is taking all the elements, and it crashed. I don't want to permute all the elements. I want the desired output as above. Please anyone can help me with that.
First thing first, I cant' figure out why in your output "trendingModule:false" is missing everywhere.
Why the rows "spotfront4:true", "spotfront4:false", "postClick4:ipsa", "postClick4:ipsa" etc.. are missing in the output? What's the logic behind the inclusion or the exclusion of each combination?
Shure you want to exclude rows like "key:true&key:false" because of the duplicated key.
Second, this program produces every possible combination, but that is not what you want, right? 215 elements, way more than what you listed in the example
opts = [
"productRankingOct12:LessPopularityEPC",
"productRankingOct12:CtrEpcJob",
"productRankingOct12:DeviceSpecificLPE",
"guidedSearch:false",
"guidedSearch:true",
"spotfront4:true",
"spotfront4:false",
"postClick4:ipsa",
"postClick4:false",
"trendingModule:false"
]
normal_opts = Hash.new
opts.each do |e|
k, v = e.split(":")
(normal_opts[k] ||= []) << v
end
# normal_ops looks like this
#normal_opts = {
# "productRankingOct12" => ["LessPopularityEPC", "CtrEpcJob", "DeviceSpecificLPE"],
# "guidedSearch" => ["false", "true"],
# "spotfornt4" => ["true", "false"],
# "postClick4" => ["ipsa", "false"],
# "trendingModule" => ["false"]
#}
# nobody is ever going to mantain this
result = normal_opts.keys.length.times.map do |i|
normal_opts.keys.combination(i + 1).map do |keys|
combs = keys.map { |k|
normal_opts[k].map { |e|
"#{k}:#{e}"
}
}
# good luck understanding this!
combs.first.product(*combs[1..-1]).map{|e| e.join("&")}
end
end
puts result.flatten
If you explain yourself better, I can improve my answer

Scraping issue with Nokogiri

I am trying to write a simple script that will tell me when the next episode of x show will be released.
here is what I have so far:
require 'rubygems'
require 'nokogiri'
require 'open-uri'
url = "http://www.tv.com/shows/game-of-thrones/episodes/"
doc = Nokogiri::HTML(open(url))
puts doc.at_css('h1').text
airdate = doc.at_css('.highlight_date span , h1').text
date = /\W/.match(airdate)
puts date
when i run this all it returns is:
Game of thrones
The css selector I use there gives the line airdate is /xx/xx/xx, however I only want to the date so thats why I have used the /\W/ although I could be completely wrong here.
So basically I want it to just print the show title and the date of the next episode.
You can do as below :-
require 'nokogiri'
require 'open-uri'
url = "http://www.tv.com/shows/game-of-thrones/episodes/"
doc = Nokogiri::HTML(open(url))
# under season4 currently 7 episodes present, which may change later.
doc.css('#season-4-eps > li').size # => 7
# collect season4 episodes and then their dates and titles
doc.css('#season-4-eps > li').collect { |node| [node.css('.title').text,node.css('.date').text] }
# => [["Mockingbird", "5/18/14"],
# ["The Laws of God and Men", "5/11/14"],
# ["First of His Name", "5/4/14"],
# ["Oathkeeper", "4/27/14"],
# ["Breaker of Chains", "4/20/14"],
# ["The Lion and the Rose", "4/13/14"],
# ["Two Swords", "4/6/14"]]
Looking at the webpage again, I can see, that it always open with latest season's data. Thus the above code can be modified as below :-
# how many sessions are present
latest_session = doc.css(".filters > li[data-season]").size # => 4
# collect season4 episodes and then their dates and titles
doc.css("#season-#{latest_session}-eps > li").collect do |node|
p [node.css('.title').text,node.css('.date').text]
end
# >> ["The Mountain and the Viper", "6/1/14"]
# >> ["Mockingbird", "5/18/14"]
# >> ["The Laws of God and Men", "5/11/14"]
# >> ["First of His Name", "5/4/14"]
# >> ["Oathkeeper", "4/27/14"]
# >> ["Breaker of Chains", "4/20/14"]
# >> ["The Lion and the Rose", "4/13/14"]
# >> ["Two Swords", "4/6/14"]
As per the comment, it seems OP may interested to get the data out from NEXT EPISODE box of the webpage. Here is a way to do the same :
require 'nokogiri'
require 'open-uri'
url = "http://www.tv.com/shows/game-of-thrones/episodes/"
doc = Nokogiri::HTML(open(url))
hash = {}
doc.css('div[class ~= next_episode] div.highlight_info').tap do |node|
hash['date'] = node.css('p.highlight_date > span').text[/\d{1,2}\/\d{1,2}\/\d{4}/]
hash['title'] = node.css('div.highlight_name > a').text
end
hash # => {"date"=>"5/18/2014", "title"=>"Mockingbird"}
Worth to read tap{|x|...} → obj
Yields x to the block, and then returns x. The primary purpose of this method is to “tap into” a method chain, in order to perform operations on intermediate results within the chain.
and str[regexp] → new_str or nil.
Also read CSS selectors to understand how the selectors are with the method #css.

cassandra Ruby : multiple values for a block parameter (2 for 1)

I am trying to follow a tutorial on big data, it wants to reads data from a keyspace defined with cqlsh.
I have compiled this piece of code successfully:
require 'rubygems'
require 'cassandra'
db = Cassandra.new('big_data', '127.0.0.1:9160')
# get a specific user's tags
row = db.get(:user_tags,"paul")
###
def tag_counts_from_row(row)
tags = {}
row.each_pair do |pair|
column, tag_count = pair
#tag_name = column.parts.first
tag_name = column
tags[tag_name] = tag_count
end
tags
end
###
# insert a new user
db.add(:user_tags, "todd", 3, "postgres")
db.add(:user_tags, "lili", 4, "win")
tags = tag_counts_from_row(row)
puts "paul - #{tags.inspect}"
but when I write this part to output everyone's tags I get an error.
user_ids = []
db.get_range(:user_tags, :batch_size => 10000) do |id|
# user_ids << id
end
rows_with_ids = db.multi_get(:user_tags, user_ids)
rows_with_ids.each do |row_with_id|
name, row = row_with_id
tags = tag_counts_from_row(row)
puts "#{name} - #{tags.inspect}"
end
the Error is:
line 33: warning: multiple values for a block parameter (2 for 1)
I think the error may have came from incompatible versions of Cassandra and Ruby. How to fix it?
Its a little hard to tell which line is 33, but it looks like the problem is that get_range yields two values, but your block is only taking the first one. If you only care about the row keys and not the columns then you should use get_range_keys.
It looks like you do in fact care about the column values because you fetch them out again using db.multi_get. This is an unnecessary additional query. You can update your code to something like:
db.get_range(:user_tags, :batch_size => 10000) do |id, columns|
tags = tag_counts_from_row(columns)
puts "#{id} - #{tags.inspect}"
end

How can I get the absolute URL when extracting links using Nokogiri?

I'm using Nokogiri to extract links from a page but I would like to get the absolute path even though the one on the page is a relative one. How can I accomplish this?
Nokogiri is unrelated, other than the fact that it gives you the link anchor to begin with. Use Ruby's URI library to manage paths:
absolute_uri = URI.join( page_url, href ).to_s
Seen in action:
require 'uri'
# The URL of the page with the links
page_url = 'http://foo.com/zee/zaw/zoom.html'
# A variety of links to test.
hrefs = %w[
http://zork.com/ http://zork.com/#id
http://zork.com/bar http://zork.com/bar#id
http://zork.com/bar/ http://zork.com/bar/#id
http://zork.com/bar/jim.html http://zork.com/bar/jim.html#id
/bar /bar#id
/bar/ /bar/#id
/bar/jim.html /bar/jim.html#id
jim.html jim.html#id
../jim.html ../jim.html#id
../ ../#id
#id
]
hrefs.each do |href|
root_href = URI.join(page_url,href).to_s
puts "%-32s -> %s" % [ href, root_href ]
end
#=> http://zork.com/ -> http://zork.com/
#=> http://zork.com/#id -> http://zork.com/#id
#=> http://zork.com/bar -> http://zork.com/bar
#=> http://zork.com/bar#id -> http://zork.com/bar#id
#=> http://zork.com/bar/ -> http://zork.com/bar/
#=> http://zork.com/bar/#id -> http://zork.com/bar/#id
#=> http://zork.com/bar/jim.html -> http://zork.com/bar/jim.html
#=> http://zork.com/bar/jim.html#id -> http://zork.com/bar/jim.html#id
#=> /bar -> http://foo.com/bar
#=> /bar#id -> http://foo.com/bar#id
#=> /bar/ -> http://foo.com/bar/
#=> /bar/#id -> http://foo.com/bar/#id
#=> /bar/jim.html -> http://foo.com/bar/jim.html
#=> /bar/jim.html#id -> http://foo.com/bar/jim.html#id
#=> jim.html -> http://foo.com/zee/zaw/jim.html
#=> jim.html#id -> http://foo.com/zee/zaw/jim.html#id
#=> ../jim.html -> http://foo.com/zee/jim.html
#=> ../jim.html#id -> http://foo.com/zee/jim.html#id
#=> ../ -> http://foo.com/zee/
#=> ../#id -> http://foo.com/zee/#id
#=> #id -> http://foo.com/zee/zaw/zoom.html#id
The more convoluted answer here previously used URI.parse(root).merge(URI.parse(href)).to_s.
Thanks to #pguardiario for the improvement.
Phrogz' answer is fine but more simply:
URI.join(base, url).to_s
You need check if the URL is absolute or relative with check if begin by http: If the URL is relative you need add the host to this URL. You can't do that by nokogiri. You need process all url inside to render like absolute.

Resources