I've been trying to use autovivification in ruby to do simple record consolidation on this:
2009-08-21|09:30:01|A1|EGLE|Eagle Bulk Shpg|BUY|6000|5.03
2009-08-21|09:30:35|A2|JOYG|Joy Global Inc|BUY|4000|39.76
2009-08-21|09:30:35|A2|LEAP|Leap Wireless|BUY|2100|16.36
2009-08-21|09:30:36|A1|AINV|Apollo Inv Cp|BUY|2300|9.15
2009-08-21|09:30:36|A1|CTAS|Cintas Corp|SELL|9800|27.83
2009-08-21|09:30:38|A1|KRE|SPDR KBW Regional Banking ETF|BUY|9200|21.70
2009-08-21|09:30:39|A1|APA|APACHE CORPORATION|BUY|5700|87.18
2009-08-21|09:30:40|A1|FITB|Fifth Third Bancorp|BUY|9900|10.86
2009-08-21|09:30:40|A1|ICO|INTERNATIONAL COAL GROUP, INC.|SELL|7100|3.45
2009-08-21|09:30:41|A1|NLY|ANNALY CAPITAL MANAGEMENT. INC.|BUY|3000|17.31
2009-08-21|09:30:42|A2|GAZ|iPath Dow Jones - AIG Natural Gas Total Return Sub-Index ETN|SELL|6600|14.09
2009-08-21|09:30:44|A2|CVBF|Cvb Finl|BUY|1100|7.64
2009-08-21|09:30:44|A2|JCP|PENNEY COMPANY, INC.|BUY|300|31.05
2009-08-21|09:30:36|A1|AINV|Apollo Inv Cp|BUY|4500|9.15
so for example I want the record for A1 AINV BUY 9.15 to have a total of 6800. This is a perfect problem to use autovivification on. So heres my code:
#!/usr/bin/ruby
require 'facets'
h = Hash.autonew
File.open('trades_long.dat','r').each do |line|
#date,#time,#account,#ticker,#desc,#type,amount,#price = line.chomp.split('|')
if #account != "account"
puts "#{amount}"
h[#account][#ticker][#type][#price] += amount
end
#puts sum.to_s
end
The problem is no matter how I try to sum up the value in h[#account][#ticker][#type][#price] it gives me this error:
6000
/usr/local/lib/ruby/gems/1.9.1/gems/facets-2.7.0/lib/core/facets/hash/op_add.rb:8:in `merge': can't convert String into Hash (TypeError)
from /usr/local/lib/ruby/gems/1.9.1/gems/facets-2.7.0/lib/core/facets/hash/op_add.rb:8:in `+'
from ./trades_consolidaton.rb:13
from ./trades_consolidaton.rb:8:in `each'
from ./trades_consolidaton.rb:8
I've tried using different "autovivification" methods with no result. This wouldn't happen in perl! The autofvivification would know what you are trying to do. ruby doesn't seem to have this feature.
So my question really is, how do I perform simply "consolidation" of records in ruby. Specifically, how do I get the total for something like:
h[#account][#ticker][#type][#price]
Many thanks for your help!!
Just to clarify on glenn's solution. That would be perfect except it gives (with a few modifications to use the standard CSV library in ruby 1.9:
CSV.foreach("trades_long.dat", :col_sep => "|") do |row|
date,time,account,ticker,desc,type,amount,price = *row
records[[account,ticker,type,price]] += amount
end
gives the following error:
TypeError: String can't be coerced into Fixnum
from (irb):64:in `+'
from (irb):64:in `block in irb_binding'
from /usr/local/lib/ruby/1.9.1/csv.rb:1761:in `each'
from /usr/local/lib/ruby/1.9.1/csv.rb:1197:in `block in foreach'
from /usr/local/lib/ruby/1.9.1/csv.rb:1335:in `open'
from /usr/local/lib/ruby/1.9.1/csv.rb:1196:in `foreach'
from (irb):62
from /usr/local/bin/irb:12:in `<main>'
I agree with Jonas that you (and Sam) are making this more complicated than it needs to be, but I think even his version is too complicated. I'd just do this:
require 'fastercsv'
records = Hash.new(0)
FasterCSV.foreach("trades_long.dat", :col_sep => "|") do |row|
date,time,account,ticker,desc,type,amount,price = row.fields
records[[account,ticker,type,price]] += amount.to_f
end
Now you have a hash with total amounts for each unique combination of account, ticker, type and price.
If you want a hash builder that works that way, you are going to have to redefine the + semantics.
For example, this works fine:
class HashBuilder
def initialize
#hash = {}
end
def []=(k,v)
#hash[k] = v
end
def [](k)
#hash[k] ||= HashBuilder.new
end
def +(val)
val
end
end
h = HashBuilder.new
h[1][2][3] += 1
h[1][2][3] += 3
p h[1][2][3]
# prints 4
Essentially you are trying to apply the + operator to a Hash.
>> {} + {}
NoMethodError: undefined method `+' for {}:Hash
from (irb):1
However in facets{
>> require 'facets'
>> {1 => 10} + {2 => 20}
=> {1 => 10, 2 => 20}
>> {} + 100
TypeError: can't convert Fixnum into Hash
from /usr/lib/ruby/gems/1.8/gems/facets-2.7.0/lib/core/facets/hash/op_add.rb:8:in `merge'
from /usr/lib/ruby/gems/1.8/gems/facets-2.7.0/lib/core/facets/hash/op_add.rb:8:in `+'
from (irb):6
>> {} += {1 => 2}
=> {1=>2}
>>
If you want to redefine the + semantics for your hash in this occasion you can do:
class Hash; def +(v); v; end; end
Place this snippet before your original sample and all should be well. Keep in mind that you are changing the defined behavior for + (note + is not defined on Hash its pulled in with facets)
It looks like you are making it more complicated than it has to be. I would use the FasterCSV gem and Enumerable#inject something like this:
require 'fastercsv'
records=FasterCSV.read("trades_long.dat", :col_sep => "|")
records.sort_by {|r| r[3]}.inject(nil) {|before, curr|
if !before.nil? && curr[3]==before[3]
curr[6]=(curr[6].to_i+before[6].to_i).to_s
records.delete(before)
end
before=curr
}
For others that find their way here, there is now also another option:
require 'xkeys' # on rubygems.org
h = {}.extend XKeys::Hash
...
# Start with 0.0 (instead of nil) and add the amount
h[#account, #ticker, #type, #price, :else => 0.0] += amount.to_f
This will generate a navigable structure. (Traditional keying with arrays of [#account, #ticker, #type, #price] as suggested earlier may be better this particular application). XKeys auto-vivifies on write rather than read, so querying the structure about elements that don't exist won't change the structure.
Related
I am getting an error when executing my test.
Failure/Error: expect(industry_sic_code).to include page.sic_code
TypeError:
no implicit conversion of Array into String
# ./spec/os/bal/company/company_filter_clean_harbors_industries_stub.rb:62:in `block (2 levels) in <top (required)>'
The Method:
def sic_code
subtables = #b.table(:class => 'industry-codes').tables(:class => 'industry-code-table')
subtables.each do |subtable|
if subtable.tbody.h4.text == "US SIC 1987:"
subtable.tr.next_siblings.each do |tr|
codes = tr.cell
puts codes.text.to_s
end
end
end
end
The Test:
it 'Given I search for a random Clean Harbors Industry' do
#Pick a random clean industry from the file
data = CSV.foreach(file_path, headers: true).map{ |row| row.to_h }
random = data.sample
random_industry = random["Class"]
industry_sic_code = random["SIC Code"]
end
it 'Then the result has the expected SIC code' do
page = DetailPage.new(#b)
page.view
expect(industry_sic_code).to include page.sic_code
end
I have tried to implicitly change each variable to a string but it still complain about the array issue.
When I include some puts statments, I get some really wonky responses. The method itself returns the expected result.
When I used the method in the test I end up with the code gibberish below.
here are the sic codes from the method
5511
Here are the codes from the test
#<Watir::Table:0x00007fa3cb23f020>
#<Watir::Table:0x00007fa3cb23ee40>
#<Watir::Table:0x00007fa3cb23ec88>
#<Watir::Table:0x00007fa3cb23ead0>
#<Watir::Table:0x00007fa3cb23e918>
#<Watir::Table:0x00007fa3cb23e738>
#<Watir::Table:0x00007fa3cb23e580>
Your sic_code method returns subtables array, that's why you have this error. It doesn't matter that the method puts something, every method in ruby implicitly returns result of its last line, in your case it is subtables.each do ... end, so you have an array.
You need to explicitly return needed value. Not sure if I correctly understood what are you doing in your code, but try something like this:
def sic_code
subtables = #b.table(:class => 'industry-codes').tables(:class => 'industry-code-table')
result = [] # you need to collect result somewhere to return it later
subtables.each do |subtable|
if subtable.tbody.h4.text == "US SIC 1987:"
subtable.tr.next_siblings.each do |tr|
codes = tr.cell
result << codes.text.to_s
end
end
end
result.join(', ')
end
What is the correct way to view the output of the puts statements below? My apologies for such a simple question.... Im a little rusty on ruby. github repo
require 'active_support'
require 'active_support/core_ext'
require 'indicators'
my_data = Indicators::Data.new(Securities::Stock.new(:symbol => 'AAPL', :start_date => '2012-08-25', :end_date => '2012-08-30').output)
puts my_data.to_s #expected to see Open,High,Low,Close for AAPL
temp=my_data.calc(:type => :sma, :params => 3)
puts temp.to_s #expected to see an RSI value for each data point from the data above
Maybe check out the awesome_print gem.
It provides the .ai method which can be called on anything.
An example:
my_obj = { a: "b" }
my_obj_as_string = my_obj.ai
puts my_obj_as_string
# ... this will print
# {
# :a => "b"
# }
# except the result is colored.
You can shorten all this into a single step with ap(my_obj).
There's also a way to return objects as HTML. It's the my_obj.ai(html: true) option.
Just use .inspect method instead of .to_s if you want to see internal properties of objects.
I have the following hash:
hash = {'name' => { 'Mike' => { 'age' => 10, 'gender' => 'm' } } }
I can access the age by:
hash['name']['Mike']['age']
What if I used Hash#fetch method? How can I retrieve a key from a nested hash?
As Sergio mentioned, the way to do it (without creating something for myself) would be by a chain of fetch methods:
hash.fetch('name').fetch('Mike').fetch('age')
From Ruby 2.3.0 onward, you can use Hash#dig:
hash.dig('name', 'Mike', 'age')
It also comes with the added bonus that if some of the values along the way turned up to be nil, you will get nil instead of exception.
You can use the ruby_dig gem until you migrate.
EDIT: there is a built-in way now, see this answer.
There is no built-in method that I know of. I have this in my current project
class Hash
def fetch_path(*parts)
parts.reduce(self) do |memo, key|
memo[key.to_s] if memo
end
end
end
# usage
hash.fetch_path('name', 'Mike', 'age')
You can easily modify it to use #fetch instead of #[] (if you so wish).
As of Ruby 2.3.0:
You can also use &. called the "safe navigation operator" as: hash&.[]('name')&.[]('Mike')&.[]('age'). This one is perfectly safe.
Using dig is not safe as hash.dig(:name, :Mike, :age) will fail if hash is nil.
However you may combine the two as: hash&.dig(:name, :Mike, :age).
So either of the following is safe to use:
hash&.[]('name')&.[]('Mike')&.[]('age')
hash&.dig(:name, :Mike, :age)
If your goal is to raise a KeyError when any of the intermediate keys are missing, then you need to write your own method. If instead you're using fetch to provide default values for missing keys, then you can circumvent the use of fetch by constructing the Hashes with a default values.
hash = Hash.new { |h1, k1| h1[k1] = Hash.new { |h2, k2| h2[k2] = Hash.new { |h3, k3| } } }
hash['name']['Mike']
# {}
hash['name']['Steve']['age'] = 20
hash
# {"name"=>{"Mike"=>{}, "Steve"=>{"age"=>20}}}
This won't work for arbitrarily nested Hashes, you need to choose the maximum depth when you construct them.
A version that uses a method instead of adding to the Hash class for others using Ruby 2.2 or lower.
def dig(dict, *args)
key = args.shift
if args.empty?
return dict[key]
else
dig(dict[key], *args)
end
end
And so you can do:
data = { a: 1, b: {c: 2}}
dig(data, :a) == 1
dig(data, :b, :c) == 2
If you don't want to monkey patch the standard Ruby class Hash use .fetch(x, {}) variant. So for the example above will look like that:
hash.fetch('name', {}).fetch('Mike', {}).fetch('age')
The point of fetch is that an explicit error is raised at the point of contract violation instead of having to track down a silent nil running amok in the code that can lead to unpredictable state.
Although dig is elegant and useful when you expect nil to be a default, it doesn't offer the same error reporting guarantees of fetch. OP seems to want the explicit errors of fetch but without the ugly verbosity and chaining.
An example use case is receiving a plain nested hash from YAML.load_file() and requiring explicit errors for missing keys.
One option is to alias [] to fetch as shown here, but this isn't a deep operation on a nested structure.
I ultimately used a recursive function and hash.instance_eval {alias [] fetch} to apply the alias to such a plain hash deeply. A class would work just as well, with the benefit of a distinct subclass separate from Hash.
irb(main):001:1* def deeply_alias_fetch!(x)
irb(main):002:2* if x.instance_of? Hash
irb(main):003:2* x.instance_eval {alias [] fetch}
irb(main):004:2* x.each_value {|v| deeply_alias_fetch!(v)}
irb(main):005:2* elsif x.instance_of? Array
irb(main):006:2* x.each {|e| deeply_alias_fetch!(e)}
irb(main):007:1* end
irb(main):008:0> end
=> :deeply_alias_fetch!
irb(main):009:0> h = {:a => {:b => 42}, :c => [{:d => 1, :e => 2}, {}]}
irb(main):010:0> deeply_alias_fetch!(h)
=> {:a=>{:b=>42}, :c=>[{:d=>1, :e=>2}, {}]}
irb(main):011:0> h[:a][:bb]
Traceback (most recent call last):
5: from /usr/bin/irb:23:in `<main>'
4: from /usr/bin/irb:23:in `load'
3: from /usr/lib/ruby/gems/2.7.0/gems/irb-1.2.1/exe/irb:11:in `<top (required)>'
2: from (irb):11
1: from (irb):11:in `fetch'
KeyError (key not found: :bb)
Did you mean? :b
irb(main):012:0> h[:c][0][:e]
=> 2
irb(main):013:0> h[:c][0][:f]
Traceback (most recent call last):
5: from /usr/bin/irb:23:in `<main>'
4: from /usr/bin/irb:23:in `load'
3: from /usr/lib/ruby/gems/2.7.0/gems/irb-1.2.1/exe/irb:11:in `<top (required)>'
2: from (irb):14
1: from (irb):14:in `fetch'
KeyError (key not found: :f)
if you can
use:
hash[["ayy","bee"]]
instead of:
hash["ayy"]["bee"]
it'll save a lot of annoyances
I am working on a program that uses yahoo finance api to collect the historical close data for the number of stocks entered and then go ahead and calculate simple moving average (SMA) for the data for period of 30 days. I have the following so far:
require 'rubygems'
require 'yahoofinance'
array = []
while line = gets
break if line.chomp =~ /N/ #exit when 'N' is entered
array << line.chomp
end
puts "Values: #{array.join(',')}" #joining all the elements with a comma
array.each do |s|
print "\n______\n"
puts s
YahooFinance::get_HistoricalQuotes( s,
Date.parse( '2012-10-06' ),
Date.today() ) do |hq|
puts "#{hq.close}"
end
end
This code is giving me the close values for stocks for the specified range. I have two questions:
Currently, hq.close is holding values for all stocks. How can I put these values in an array so that I can do a computation on it to calculate a SMA for each stock data?
I tried doing something like this:
"#{hq.close}" my_val = [hq.close]
puts my_val
But this only gives the value of first stock in my_val. I know I have to put a loop here. I tried putting
while(!hq.close.emply?)
my_val = [hq.close]
puts my_val
end
But this gives me an error:
C:/Users/Muktak/workspace/RubySample/sample_program.rb:23:in block (2 levels) in <main>': undefined methodemplty?' for 19.52:Float (NoMethodError) from
C:/Ruby193/lib/ruby/gems/1.9.1/gems/yahoofinance-1.2.2/lib/yahoofinance.rb:491:in block in get_HistoricalQuotes' from
C:/Ruby193/lib/ruby/gems/1.9.1/gems/yahoofinance-1.2.2/lib/yahoofinance.rb:456:inblock in get_historical_quotes' from
C:/Ruby193/lib/ruby/gems/1.9.1/gems/yahoofinance-1.2.2/lib/yahoofinance.rb:456:in each' from
C:/Ruby193/lib/ruby/gems/1.9.1/gems/yahoofinance-1.2.2/lib/yahoofinance.rb:456:inget_historical_quotes' from
C:/Ruby193/lib/ruby/gems/1.9.1/gems/yahoofinance-1.2.2/lib/yahoofinance.rb:489:in get_HistoricalQuotes' from
C:/Users/Muktak/workspace/RubySample/sample_program.rb:19:inblock in ' from
C:/Users/Muktak/workspace/RubySample/sample_program.rb:13:in each' from
C:/Users/Muktak/workspace/RubySample/sample_program.rb:13:in'
Values: FB,GOOG
How can I calculate a SMA in Ruby?
You've asked two questions here, so let's address them one at a time.
First, this code:
require 'rubygems'
require 'yahoofinance'
stock_names = %w{MSFT RHT AAPL}
start = Date.parse '2012-10-06'
finish = Date.today
closes = {}
stock_names.each do |stock_name|
quotes = YahooFinance::get_HistoricalQuotes stock_name, start, finish
closes[stock_name] = quotes.collect { |quote| quote.close }
end
... will produce the following hash in closes, which I understand is in the format you want:
{
"AAPL" => [629.71, 628.1, 640.91, 635.85, 638.17],
"RHT"=> [53.69, 53.77, 53.86, 54.0, 54.41],
"MSFT"=> [29.2, 28.95, 28.98, 29.28, 29.78]
}
Secondly, you want to calculate a simple moving average - which for financial applications is just the mean of the values. There is a Gem called simple_statistics that can do this.
This code:
require 'rubygems'
require 'yahoofinance'
require 'simple_statistics'
stock_names = %w{MSFT RHT AAPL}
start = Date.parse '2012-10-06'
finish = Date.today
averages = {}
stock_names.each do |stock_name|
quotes = YahooFinance::get_HistoricalQuotes stock_name, start, finish
closes = quotes.collect { |quote| quote.close }
averages[stock_name] = closes.mean
end
... produces the following hash in averages:
{ "AAPL" => 634.548, "MSFT" => 29.238, "RHT" => 53.946 }
I've tried and tried, but I can't make this less ugly/more ruby-like. It seems like there just must be a better way. Help me learn.
class Df
attr_accessor :thresh
attr_reader :dfo
def initialize
#dfo = []
#df = '/opt/TWWfsw/bin/gdf'
case RUBY_PLATFORM
when /hpux/i
#fstyp = 'vxfs'
when /solaris/i
# fix: need /tmp too
#fstyp = 'ufs'
when /linux/i
#df = '/bin/df'
#fstyp = 'ext3'
end
#dfo = parsedf
end
def parsedf
ldf = []
[" "," -i"] .each do |arg|
fields = %w{device size used avail capp mount}
fields = %w{device inodes inodesused inodesavail iusep mount} if arg == ' -i'
ldf.push %x{#{#df} -P -t #{#fstyp}#{arg}}.split(/\n/)[1..-1].collect{|line| Hash[*fields.zip(line.split).flatten]}
end
out = []
# surely there must be an easier way
ldf[0].each do |x|
ldf[1].select { |y|
if y['device'] == x['device']
out.push x.merge(y)
end
}
end
out
end
end
In my machine, your ldf array after the df calls yields the following:
irb(main):011:0> ldf
=> [[{"device"=>"/dev/sda5", "size"=>"49399372", "mount"=>"/", "avail"=>"22728988", "used"=>"24161036", "capp"=>"52%"}], [{"device"=>"/dev/sda5", "inodes"=>"3137536", "mount"=>"/", "iusep"=>"13%", "inodesavail"=>"2752040", "inodesused"=>"385496"}]]
The most flexible approach to merging such a structure is probably something along these lines:
irb(main):013:0> ldf.flatten.inject {|a,b| a.merge(b)}
=> {"device"=>"/dev/sda5", "inodes"=>"3137536", "size"=>"49399372", "mount"=>"/", "avail"=>"22728988", "inodesavail"=>"2752040", "iusep"=>"13%", "used"=>"24161036", "capp"=>"52%", "inodesused"=>"385496"}
Some ruby programmers frown on this use of inject, but I like it, so your mileage may vary.
As for helping making your code more ruby like, I suggest you talk to some experienced rubyist you might know over your code to help you rewriting it in a way that follows good style and best practices. Probably that would the preferable than to just have someone rewrite it for you here.
Best of Luck!
Didn't test the code, but here goes:
ARGUMENTS = {
" " => %w{size used avail capp mount},
" -i" => %w{inodes inodesused inodesavail iusep mount}
}
def parsedf
# Store resulting info in a hash:
device_info = Hash.new do |h, dev|
h[dev] = {} # Each value will be a empty hash by default
end
ARGUMENTS.each do |arg, fields|
%x{#{#df} -P -t #{#fstyp}#{arg}}.split(/\n/)[1..-1].each do |line|
device, *data = line.split
device_info[device].merge! Hash[fields.zip(data)]
end
end
device_info
end
Notes: returns something a bit different than what you had:
{ "/dev/sda5" => {"inodes" => "...", ...},
"other device" => {...}
}
Also, I'm assuming Ruby 1.8.7 or better for Hash[key_value_pairs], otherwise you can resort to the Hash[*key_value_pairs.flatten] form you had
Depending on your needs, you should consider switch the fields from string to symbols; they are the best type of keys.