I am working on a program that uses yahoo finance api to collect the historical close data for the number of stocks entered and then go ahead and calculate simple moving average (SMA) for the data for period of 30 days. I have the following so far:
require 'rubygems'
require 'yahoofinance'
array = []
while line = gets
break if line.chomp =~ /N/ #exit when 'N' is entered
array << line.chomp
end
puts "Values: #{array.join(',')}" #joining all the elements with a comma
array.each do |s|
print "\n______\n"
puts s
YahooFinance::get_HistoricalQuotes( s,
Date.parse( '2012-10-06' ),
Date.today() ) do |hq|
puts "#{hq.close}"
end
end
This code is giving me the close values for stocks for the specified range. I have two questions:
Currently, hq.close is holding values for all stocks. How can I put these values in an array so that I can do a computation on it to calculate a SMA for each stock data?
I tried doing something like this:
"#{hq.close}" my_val = [hq.close]
puts my_val
But this only gives the value of first stock in my_val. I know I have to put a loop here. I tried putting
while(!hq.close.emply?)
my_val = [hq.close]
puts my_val
end
But this gives me an error:
C:/Users/Muktak/workspace/RubySample/sample_program.rb:23:in block (2 levels) in <main>': undefined methodemplty?' for 19.52:Float (NoMethodError) from
C:/Ruby193/lib/ruby/gems/1.9.1/gems/yahoofinance-1.2.2/lib/yahoofinance.rb:491:in block in get_HistoricalQuotes' from
C:/Ruby193/lib/ruby/gems/1.9.1/gems/yahoofinance-1.2.2/lib/yahoofinance.rb:456:inblock in get_historical_quotes' from
C:/Ruby193/lib/ruby/gems/1.9.1/gems/yahoofinance-1.2.2/lib/yahoofinance.rb:456:in each' from
C:/Ruby193/lib/ruby/gems/1.9.1/gems/yahoofinance-1.2.2/lib/yahoofinance.rb:456:inget_historical_quotes' from
C:/Ruby193/lib/ruby/gems/1.9.1/gems/yahoofinance-1.2.2/lib/yahoofinance.rb:489:in get_HistoricalQuotes' from
C:/Users/Muktak/workspace/RubySample/sample_program.rb:19:inblock in ' from
C:/Users/Muktak/workspace/RubySample/sample_program.rb:13:in each' from
C:/Users/Muktak/workspace/RubySample/sample_program.rb:13:in'
Values: FB,GOOG
How can I calculate a SMA in Ruby?
You've asked two questions here, so let's address them one at a time.
First, this code:
require 'rubygems'
require 'yahoofinance'
stock_names = %w{MSFT RHT AAPL}
start = Date.parse '2012-10-06'
finish = Date.today
closes = {}
stock_names.each do |stock_name|
quotes = YahooFinance::get_HistoricalQuotes stock_name, start, finish
closes[stock_name] = quotes.collect { |quote| quote.close }
end
... will produce the following hash in closes, which I understand is in the format you want:
{
"AAPL" => [629.71, 628.1, 640.91, 635.85, 638.17],
"RHT"=> [53.69, 53.77, 53.86, 54.0, 54.41],
"MSFT"=> [29.2, 28.95, 28.98, 29.28, 29.78]
}
Secondly, you want to calculate a simple moving average - which for financial applications is just the mean of the values. There is a Gem called simple_statistics that can do this.
This code:
require 'rubygems'
require 'yahoofinance'
require 'simple_statistics'
stock_names = %w{MSFT RHT AAPL}
start = Date.parse '2012-10-06'
finish = Date.today
averages = {}
stock_names.each do |stock_name|
quotes = YahooFinance::get_HistoricalQuotes stock_name, start, finish
closes = quotes.collect { |quote| quote.close }
averages[stock_name] = closes.mean
end
... produces the following hash in averages:
{ "AAPL" => 634.548, "MSFT" => 29.238, "RHT" => 53.946 }
Related
I am getting an error when executing my test.
Failure/Error: expect(industry_sic_code).to include page.sic_code
TypeError:
no implicit conversion of Array into String
# ./spec/os/bal/company/company_filter_clean_harbors_industries_stub.rb:62:in `block (2 levels) in <top (required)>'
The Method:
def sic_code
subtables = #b.table(:class => 'industry-codes').tables(:class => 'industry-code-table')
subtables.each do |subtable|
if subtable.tbody.h4.text == "US SIC 1987:"
subtable.tr.next_siblings.each do |tr|
codes = tr.cell
puts codes.text.to_s
end
end
end
end
The Test:
it 'Given I search for a random Clean Harbors Industry' do
#Pick a random clean industry from the file
data = CSV.foreach(file_path, headers: true).map{ |row| row.to_h }
random = data.sample
random_industry = random["Class"]
industry_sic_code = random["SIC Code"]
end
it 'Then the result has the expected SIC code' do
page = DetailPage.new(#b)
page.view
expect(industry_sic_code).to include page.sic_code
end
I have tried to implicitly change each variable to a string but it still complain about the array issue.
When I include some puts statments, I get some really wonky responses. The method itself returns the expected result.
When I used the method in the test I end up with the code gibberish below.
here are the sic codes from the method
5511
Here are the codes from the test
#<Watir::Table:0x00007fa3cb23f020>
#<Watir::Table:0x00007fa3cb23ee40>
#<Watir::Table:0x00007fa3cb23ec88>
#<Watir::Table:0x00007fa3cb23ead0>
#<Watir::Table:0x00007fa3cb23e918>
#<Watir::Table:0x00007fa3cb23e738>
#<Watir::Table:0x00007fa3cb23e580>
Your sic_code method returns subtables array, that's why you have this error. It doesn't matter that the method puts something, every method in ruby implicitly returns result of its last line, in your case it is subtables.each do ... end, so you have an array.
You need to explicitly return needed value. Not sure if I correctly understood what are you doing in your code, but try something like this:
def sic_code
subtables = #b.table(:class => 'industry-codes').tables(:class => 'industry-code-table')
result = [] # you need to collect result somewhere to return it later
subtables.each do |subtable|
if subtable.tbody.h4.text == "US SIC 1987:"
subtable.tr.next_siblings.each do |tr|
codes = tr.cell
result << codes.text.to_s
end
end
end
result.join(', ')
end
I recently wrote a program to return a bunch of stocks from the stock market that are unhealthy. The basic algorithm is this:
Look up all the quotes of every stock in an exchange (either NYSE or NASDAQ)
Find the ones that are less than 5 dollars from step 1
Find the ones from step 2 that are down 3 days and have large volume (expensive because I have to make a request for each stock, which is like ~700 currently for nasdaq).
Scan the news for the ones returned by step 3.
I had this all in one file:
Original implementation (https://github.com/EdmundMai/minion/blob/aa14bc3234a4953e7273ec502276c6f0073b459d/lib/minion.rb):
require 'bundler/setup'
require "minion/version"
require "yahoo-finance"
require "business_time"
require 'nokogiri'
require 'open-uri'
module Minion
class << self
def query(exchange)
client = YahooFinance::Client.new
all_companies = CSV.read("#{exchange}.csv")
small_caps = []
ticker_symbols = all_companies.map { |row| row[0] }
ticker_symbols.each_slice(200) do |batch|
data = client.quotes(batch, [:symbol, :last_trade_price, :average_daily_volume])
small_caps << data.select { |stock| stock.last_trade_price.to_f < 5.0 }
end
attractive = []
small_caps.flatten!.each_with_index do |small_cap, index|
begin
data = client.historical_quotes(small_cap.symbol, { start_date: 2.business_days.ago, end_date: Time.now })
closing_prices = data.map(&:close).map(&:to_f)
volumes = data.map(&:volume).map(&:to_i)
negative_3_days_in_a_row = closing_prices == closing_prices.sort
larger_than_average_volume = volumes.reduce(:+) / volumes.count > small_cap.average_daily_volume.to_i
if negative_3_days_in_a_row && larger_than_average_volume
attractive << small_cap.symbol
puts "Qualified: #{small_cap.symbol}, finished with #{index} out of #{small_caps.count}"
else
puts "Not qualified: #{small_cap.symbol}, finished with #{index} out of #{small_caps.count}"
end
rescue => e
puts e.inspect
end
end
final_results = []
attractive.each do |symbol|
rss_feed = Nokogiri::HTML(open("http://feeds.finance.yahoo.com/rss/2.0/headline?s=#{symbol}®ion=US&lang=en-US"))
html_body = rss_feed.css('body')[0].text
diluting = false
['warrant', 'cashless exercise'].each do |keyword|
diluting = true if html_body.match(/#{keyword}/i)
end
final_results << symbol if diluting
end
final_results
end
end
end
This was really fast and would finish processing like ~700 stocks in a minute or less.
Then, I tried refactoring and splitting up the algorithm into different classes and files without changing the algorithm at all. I decided on using the decorator pattern since it seems to fit. However when I run the program now, it makes each request really slowly (15+ min). I know this because my puts statements get printed out really slowly.
New and slower implementation (https://github.com/EdmundMai/minion/blob/master/lib/minion.rb)
require 'bundler/setup'
require "minion/version"
require "yahoo-finance"
require "minion/dilution_finder"
require "minion/negative_finder"
require "minion/small_cap_finder"
require "minion/market_fetcher"
module Minion
class << self
def query(exchange)
all_companies = CSV.read("#{exchange}.csv")
all_tickers = all_companies.map { |row| row[0] }
short_finder = DilutionFinder.new(NegativeFinder.new(SmallCapFinder.new(MarketFetcher.new(all_tickers))))
short_finder.results
end
end
end
The part it's lagging at according to my puts:
require "yahoo-finance"
require "business_time"
require_relative "stock_finder"
class NegativeFinder < StockFinder
def results
client = YahooFinance::Client.new
results = []
finder.results.each_with_index do |stock, index|
begin
data = client.historical_quotes(stock.symbol, { start_date: 2.business_days.ago, end_date: Time.now })
closing_prices = data.map(&:close).map(&:to_f)
volumes = data.map(&:volume).map(&:to_i)
negative_3_days_in_a_row = closing_prices == closing_prices.sort
larger_than_average_volume = volumes.reduce(:+) / volumes.count > stock.average_daily_volume.to_i
if negative_3_days_in_a_row && larger_than_average_volume
results << stock
puts "Qualified: #{stock.symbol}, finished with #{index} out of #{finder.results.count}"
else
puts "Not qualified: #{stock.symbol}, finished with #{index} out of #{finder.results.count}"
end
rescue => e
puts e.inspect
end
end
results
end
end
It's lagging on step 3 (making one request for each stock). Not sure what's going on so any advice would be appreciated. If you want to clone the program and run it, just comment in the last line in lib/minion.rb and type ruby lib/minion.rb
After debugging it I figured it out. It was because I was calling finder.results (results being the decorated method) inside of the loop as shown below:
require 'bundler/setup'
require "minion/version"
require "yahoo-finance"
require "minion/dilution_finder"
require "minion/negative_finder"
require "minion/small_cap_finder"
require "minion/market_fetcher"
module Minion
class << self
def query(exchange)
all_companies = CSV.read("#{exchange}.csv")
all_tickers = all_companies.map { |row| row[0] }
short_finder = DilutionFinder.new(NegativeFinder.new(SmallCapFinder.new(MarketFetcher.new(all_tickers))))
short_finder.results
end
end
end
The part it's lagging at according to my puts:
require "yahoo-finance"
require "business_time"
require_relative "stock_finder"
class NegativeFinder < StockFinder
def results
client = YahooFinance::Client.new
results = []
finder.results.each_with_index do |stock, index|
begin
data = client.historical_quotes(stock.symbol, { start_date: 2.business_days.ago, end_date: Time.now })
closing_prices = data.map(&:close).map(&:to_f)
volumes = data.map(&:volume).map(&:to_i)
negative_3_days_in_a_row = closing_prices == closing_prices.sort
larger_than_average_volume = volumes.reduce(:+) / volumes.count > stock.average_daily_volume.to_i
if negative_3_days_in_a_row && larger_than_average_volume
results << stock
// HERE!!!!!!!!!!!!!!!!!!!!!!!!!
puts "Qualified: #{stock.symbol}, finished with #{index} out of #{finder.results.count}" <------------------------------------
else
// AND HERE!!!!!!!!!!!!!!!!!!!!!!!!!
puts "Not qualified: #{stock.symbol}, finished with #{index} out of #{finder.results.count}" <-----------------------------------------------------------
end
rescue => e
puts e.inspect
end
end
results
end
end
This caused a cascade of requests every time I iterated through the loop in NegativeFinder. Removing that call fixed it. Lesson: When using the decorator pattern, either only call the decorated method once, especially when you're doing something expensive in each call. Either that or hold the returned variable in an instance variable so you don't have to calculate it each time.
Also as a side note, I've decided not to go with the decorator pattern because I don't think it applies well here. Something like SmallCapFinder.new(SmallCapFinder.new(MarketFetcher.new(all_tickers))) doesn't add functionality at all (the primary function of using the decorator pattern), so chaining decorators doesn't do anything. Therefore, I'm just going to make them methods instead of adding unnecessary complexity.
There are some thing missing in the code you gave us (Base class StockFinder, MarketFetcher). But I think you are now instantate more than one YahooFinance::Client. Input/Output to other systems is very often the cause for speed problems.
I suggest that you first encapsulate the finance client and access to financial data. This makes it easier when you want to switch your financial data provider, or add another one. Instead of the decorator pattern, I would just use plain old methods for finding small caps, finding negative, etc.
I got Ruby to travel to a web site, iterate through a list of campaigns and scrape the pages for specific data. The problem I have now is getting it from the structure Nokogiri gives me, and outputting it into a readable form.
campaign_list = Array.new
campaign_list.push(1042360, 1042386, 1042365, 992307)
browser = Watir::Browser.new :chrome
browser.goto '<redacted>'
browser.text_field(:id => 'email').set '<redacted>'
browser.text_field(:id => 'password').set '<redacted>'
browser.send_keys :enter
file = File.new('hourlysales.csv', 'w')
data = {}
campaign_list.each do |campaign|
browser.goto "<redacted>"
if browser.text.include? "Application Error"
puts "Error loading page, I recommend restarting script"
# Possibly automatic restart of script
else
hourly_data = Nokogiri::HTML.parse(browser.html).text
# file.write data
puts hourly_data
end
This is the output I get:
{"views":[[17,145],[18,165],[19,99],[20,71],[21,31],[22,26],[23,10],[0,15],[1,1], [2,18],[3,19],[4,35],[5,47],[6,44],[7,67],[8,179],[9,141],[10,112],[11,95],[12,46],[13,82],[14,79],[15,70],[16,103]],"orders":[[17,10],[18,9],[19,5],[20,1],[21,1],[22,0],[23,0],[0,1],[1,0],[2,1],[3,0],[4,1],[5,2],[6,1],[7,5],[8,11],[9,6],[10,5],[11,3],[12,1],[13,2],[14,4],[15,6],[16,7]],"conversion_rates":[0.06870229007633588,0.05442176870748299,0.050505050505050504,0.014084507042253521,0.03225806451612903,0.0,0.0,0.06666666666666667,0.0,0.05555555555555555,0.0,0.02857142857142857,0.0425531914893617,0.022727272727272728,0.07462686567164178,0.06134969325153374,0.0425531914893617,0.044642857142857144,0.031578947368421054,0.021739130434782608,0.024390243902439025,0.05063291139240506,0.08571428571428572,0.06741573033707865]}
The arrays stand for { views [[hour, # of views], [hour, # of views], etc. }. Same with orders. I don't need conversion rates.
I also need to add the values up for each key, so after doing this for 5 pages, I have one key for each hour of the day, and the total number of views for that hour. I tried a couple each loops, but couldn't make any progress.
I appreciate any help you guys can give me.
It looks like the output (which from your code I assume is the content of hourly_data) is JSON. In that case, it's easy to parse and add up the numbers. Something like this:
require "json" # at the top of your script
# ...
def sum_hours_values(data, hours_values=nil)
# Start with an empty hash that automatically initializes missing keys to `0`
hours_values ||= Hash.new {|hsh,hour| hsh[hour] = 0 }
# Iterate through the [hour, value] arrays, adding `value` to the running
# count for that `hour`, and return `hours_values`
data.each_with_object(hours_values) do |(hour, value), hsh|
hsh[hour] += value
end
end
# ... Watir/Nokogiri stuff here...
# Initialize these so they persist outside the loop
hours_views, orders_views = nil
campaign_list.each do |campaign|
browser.goto "<redacted>"
if browser.text.include? "Application Error"
# ...
else
# ...
hourly_data_parsed = JSON.parse(hourly_data)
hours_views = sum_hours_values(hourly_data_parsed["views"], hours_views)
hours_orders = sum_hours_values(hourly_data_parsed["orders"], orders_views)
end
end
puts "Views by hour:"
puts hours_views.sort.map {|hour_views| "%2i\t%4i" % hour_views }
puts "Orders by hour:"
puts hours_orders.sort.map {|hour_orders| "%2i\t%4i" % hour_orders }
P.S. There's a really nice recursive version of sum_hours_values I didn't include since the iterative version is clearer to most Ruby programmers. If you're into recursion I leave it as an exercise for you. ;)
I have written some code out of the Ruby Pickaxe book and I am trying to get it to work.
(around page 62 of "Programming Ruby The Pragmatic Programmer's Guide")
**Edit: More info on the book: (C) 2009, for Ruby 1.9
Given this error message, I am not quite sure how to identify what is going wrong. I appreciate any help in understanding what is going wrong here.
How does one know what to identify and solve?
I am wondering if Ruby's CSV functionality is really just this easy-- no gem/bundle install to run?
I would really like to be able to run my test_code.rb file, but I am unable to figure out this error.
Thank you for your time,
Patrick
Note: all of these files are in the same directory.
IRB command, followed by the error message it generates:
2.1.1 :005 > load "test_code.rb"
LoadError: cannot load such file -- csv-reader
from /Users/patrickmeaney/.rvm/rubies/ruby-2.1.1/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /Users/patrickmeaney/.rvm/rubies/ruby-2.1.1/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from test_code.rb:3:in `<top (required)>'
from (irb):5:in `load'
from (irb):5
from /Users/patrickmeaney/.rvm/rubies/ruby-2.1.1/bin/irb:11:in `<main>'
I don't know how relevant this is, based on the error message, but thought I'd include it.
kernel_require.rb line 55:
if Gem::Specification.unresolved_deps.empty? then
begin
RUBYGEMS_ACTIVATION_MONITOR.exit
return gem_original_require(path)
ensure
RUBYGEMS_ACTIVATION_MONITOR.enter
end
end
line 9-11 of irb:
require "irb"
IRB.start(__FILE__)
First file of program: csv-reader.rb
require 'csv'
require 'book-in-stock'
class CsvReader
def initialize
#books_in_stock = []
end
def read_in_csv_data(csv_file_name)
CSV.foreach(csv_file_name, headers: true) do |row|
#books_in_stock << BookInStock.new(row["ISBN"], row["Amount"])
end
end
def total_value_in_stock
sum = 0.0
#books_in_stock.each {|book| sum += book.price}
end
def number_of_each_isbn
end
end
Second file: book-in-stock.rb
class BookInStock
attr_reader :isbn
attr_accessor :price
def initialize(isbn, price)
#isbn = isbn
#price = Float(price)
end
def price_in_cents
Integer(price*100 + 0.5)
end
def price_in_cents=(cents)
#price = cents / 100.0
end
end
Third file: stock-stats.rb
require 'csv-reader'
reader = CsvReader.new
ARGV.each do |csv_file_name|
STDERR.puts "Processing #{csv_file_name}"
reader.read_in_csv_data(csv_file_name)
end
puts "Total value = #{reader.total_value_in_stock}"
Fourth file: test_code.rb
# this is the test code file
require 'csv-reader'
require 'book-in-stock'
require 'stock-stats'
# code to call
reader = CsvReader.new
reader.read_in_csv_data("file1.csv")
reader.read_in_csv_data("file2.csv")
puts "Total value in stock = #{reader.total_value_in_stock}"
# code to call
book = BookInStock.new("isbn1", 33.80)
puts "Price = #{book.price}"
puts "Price in cents = #{book.price_in_cents}"
book.price_in_cents = 1234
puts "Price = #{book.price}"
puts "Price in cents = #{book.price_in_cents}"
CSV files:
file1.csv
ISBN, Amount
isbn1, 49.00
isbn2, 24.54
isbn3, 33.23
isbn4, 15.55
file2.csv
ISBN, Amount
isbn5-file2, 39.98
isbn6-file2, 14.84
isbn7-file2, 43.63
isbn8-file2, 25.55
Edit
After Frederick Cheung's suggestion to change require to require_relative (for all but the 1st line of csv-reader.rb), the script is running, but a method is not working (see below)
(I did receive an error about this line:
#price = Float(price)
and changed it to #price = price.to_f and it runs just fine. )
3 Questions:
-> I changed the header of my csv files to "ISBN, Amount". Previously Amount was amount (not capitalized). Does this matter (i.e. the capitalizing of the header)?
-> While we're on the subject, what is the "row" keyword doing in the following #read_in_csv_data method?
-> Now that my code runs it appears that the output for "Total value in stock" is not summing up all of the prices in the csv file. Could a Rubyist please help me understand why this is happening?
The method
def read_in_csv_data(csv_file_name)
CSV.foreach(csv_file_name, headers: true) do |row|
#books_in_stock << BookInStock.new(row["ISBN"], row["Amount"])
end
end
and call seem fine to me...
reader = CsvReader.new
reader.read_in_csv_data("file1.csv")
reader.read_in_csv_data("file2.csv")
Here is the current output from terminal:
Total value = []
Price = 33.8
Price in cents = 3380
Price = 12.34
Price in cents = 1234
Total value in stock = [#<BookInStock:0xb8168a60 #isbn="isbn1", #price=0.0>, #<BookInStock:0xb8168740 #isbn="isbn2", #price=0.0>, #<BookInStock:0xb8168358 #isbn="isbn3", #price=0.0>, #<BookInStock:0xb81546f0 #isbn="isbn4", #price=0.0>, #<BookInStock:0xb8156a18 #isbn="isbn5-file2", #price=0.0>, #<BookInStock:0xb8156784 #isbn="isbn6-file2", #price=0.0>, #<BookInStock:0xb81564a0 #isbn="isbn7-file2", #price=0.0>, #<BookInStock:0xb8156248 #isbn="isbn8-file2", #price=0.0>]
Thanks again.
Edit: Big thanks to 7Stud for a very thorough followup answer on every question I had. You have been exceptionally helpful. I have learned several important things thanks to your post.
Edit:
Still not able to get the code to run.
I am not sure how to add to / edit the $LOAD_PATH, so I tried putting all of the files into this directory:
directory: ~MY_RUBY_HOME/lib/ruby/site_ruby/2.1.0/csv-reader
(i.e. /Users/patrickmeaney/.rvm/rubies/ruby-2.1.1/lib/ruby/site_ruby/2.1.0/csv-reader)
However, I still receive the same error message:
✘ ~MY_RUBY_HOME/lib/ruby/site_ruby/2.1.0/csv-reader ruby test_code.rb file1.csv file2.csv
/Users/patrickmeaney/.rvm/rubies/ruby-2.1.1/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- ./csv_reader (LoadError)
from /Users/patrickmeaney/.rvm/rubies/ruby-2.1.1/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from test_code.rb:1:in `<main>'
I have written some code out of the Ruby Pickaxe book
Yeah, but there are many Ruby Pickaxe books.
IRB command, followed by the error message it generates:
NEVER run anything in IRB. Never use IRB for ANYTHING. Instead put your code in a file, and then run the file, e.g:
$ ruby my_prog.rb
LoadError: cannot load such file -- csv-reader
If the files you want to require are not located in the directories ruby searches automatically(to see those directories execute the line `p $LOAD_PATH'), then you can specify the absolute or relative path to the file you want to require in the require statement:
require './book_in_stock'
I did receive an error about this line: #price = Float(price) and
changed it to #price = price.to_f and it runs just fine.
x = 'hello'
p x.to_f
p Float(x)
--output:--
0.0
1.rb:3:in `Float': invalid value for Float(): "hello" (ArgumentError)
from 1.rb:3:in `<main>
The difference between Float() and to_f() is that Float will raise an exception when it is unable to convert the String to a Float, while to_f() will return 0 when it cannot convert the String to a Float. Unless you know what you are doing, it's probably best to use Float(), so that you are alerted to the fact that your data has an error in it.
While we're on the subject, what is the "row" keyword doing in the
following #read_in_csv_data method?
When you loop through the rows of your file(e.g. CSV.foreach), csv converts one row of your file into a thing called a "CSV::Row", and then assigns the "CSV::ROW" object to the loop variable, which you have named "row":
CSV.foreach(csv_file_name, headers: true) do |row|
^
|
So "row" is a variable that refers to a "CSV::Row". A "CSV::Row" acts like a hash, enabling you to write things like row['ISBN'] to retrieve the value in that column.
Spaces are significant in csv files. If your header row is ISBN, Amount, then the column names are "ISBN" and " Amount" (see the leading space?). That means there is no value for
row['Amount']
i.e. it will return nil, but there is a value for
row[' Amount']
^
|
Now that my code runs it appears that the output for "Total value in
stock" is not summing up all of the prices in the csv file. Could a
Rubyist please help me understand why this is happening?
1) A def returns the value of the last statement that was executed in the def.
2) Array#each() returns the array.
Here is your def:
def total_value_in_stock
sum = 0.0
#books_in_stock.each {|book| sum += book.price}
end
That def returns the #books_in_stock array. You need to return the sum:
def total_value_in_stock
sum = 0.0
#books_in_stock.each {|book| sum += book.price}
sum
end
If you want to get tricky, you can have csv automatically convert any data in your file that looks like a number to a number:
CSV.foreach(
csv_file_name,
headers: true,
:converters => :numeric
) do |row| ...
...then your BookInStock class would look like this:
class BookInStock
attr_reader :isbn
attr_accessor :price
def initialize(isbn, price)
#isbn = isbn
#price = price #Float(price)
end
Here are all your files amended so they will run correctly:
csv_reader.rb:
require 'csv'
require './book_in_stock'
class CsvReader
def initialize
#books_in_stock = []
end
def read_in_csv_data(csv_file_name)
CSV.foreach(csv_file_name, headers: true) do |row|
#books_in_stock << BookInStock.new(row["ISBN"], row["Amount"])
end
end
def total_value_in_stock
sum = 0.0
#books_in_stock.each {|book| sum += book.price}
sum
end
def number_of_each_isbn
end
end
stock_stats.rb:
require './csv_reader'
reader = CsvReader.new
ARGV.each do |csv_file_name|
STDERR.puts "Processing #{csv_file_name}"
reader.read_in_csv_data(csv_file_name)
end
puts "Total value = #{reader.total_value_in_stock}"
test_code.rb:
require './csv_reader'
require './book_in_stock'
require './stock_stats'
reader = CsvReader.new
reader.read_in_csv_data("file1.csv")
reader.read_in_csv_data("file2.csv")
puts "Total value in stock = #{reader.total_value_in_stock}"
# code to call
book = BookInStock.new("isbn1", 33.80)
puts "Price = #{book.price}"
puts "Price in cents = #{book.price_in_cents}"
book.price_in_cents = 1234
puts "Price = #{book.price}"
puts "Price in cents = #{book.price_in_cents}"
book_in_stock.rb:
class BookInStock
attr_reader :isbn
attr_accessor :price
def initialize(isbn, price)
#isbn = isbn
#price = Float(price)
end
def price_in_cents
Integer(price*100 + 0.5)
end
def price_in_cents=(cents)
#price = cents / 100.0
end
end
file1.csv:
ISBN,Amount
isbn1,49.00
isbn2,24.54
isbn3,33.23
isbn4,15.55
file2.csv:
ISBN,Amount
isbn5-file2,39.98
isbn6-file2,14.84
isbn7-file2,43.63
isbn8-file2,25.55
Now run the program:
~/ruby_programs$ ruby test_code.rb file1.csv file2.csv
Processing file1.csv
Processing file2.csv
Total value = 246.32
Total value in stock = 246.32
Price = 33.8
Price in cents = 3380
Price = 12.34
Price in cents = 1234
require searches for files in Ruby's load path (this is stored in the global variables $: or $LOAD_PATH)
The current directory is not in the load path by default (it used to be in ruby 1.8 and earlier) which is why ruby says that it can't find csv-reader
You can add to the load path either by manipulating the $: variable (it behaves just like an array) or with the the -I option.
For example if you launch irb by doing
irb -I.
Then your code should run without modification (assuming there are no other problems with it)
Lastly you could switch your require statements to use require_relative - this locates files relative to the current file
I've been trying to use autovivification in ruby to do simple record consolidation on this:
2009-08-21|09:30:01|A1|EGLE|Eagle Bulk Shpg|BUY|6000|5.03
2009-08-21|09:30:35|A2|JOYG|Joy Global Inc|BUY|4000|39.76
2009-08-21|09:30:35|A2|LEAP|Leap Wireless|BUY|2100|16.36
2009-08-21|09:30:36|A1|AINV|Apollo Inv Cp|BUY|2300|9.15
2009-08-21|09:30:36|A1|CTAS|Cintas Corp|SELL|9800|27.83
2009-08-21|09:30:38|A1|KRE|SPDR KBW Regional Banking ETF|BUY|9200|21.70
2009-08-21|09:30:39|A1|APA|APACHE CORPORATION|BUY|5700|87.18
2009-08-21|09:30:40|A1|FITB|Fifth Third Bancorp|BUY|9900|10.86
2009-08-21|09:30:40|A1|ICO|INTERNATIONAL COAL GROUP, INC.|SELL|7100|3.45
2009-08-21|09:30:41|A1|NLY|ANNALY CAPITAL MANAGEMENT. INC.|BUY|3000|17.31
2009-08-21|09:30:42|A2|GAZ|iPath Dow Jones - AIG Natural Gas Total Return Sub-Index ETN|SELL|6600|14.09
2009-08-21|09:30:44|A2|CVBF|Cvb Finl|BUY|1100|7.64
2009-08-21|09:30:44|A2|JCP|PENNEY COMPANY, INC.|BUY|300|31.05
2009-08-21|09:30:36|A1|AINV|Apollo Inv Cp|BUY|4500|9.15
so for example I want the record for A1 AINV BUY 9.15 to have a total of 6800. This is a perfect problem to use autovivification on. So heres my code:
#!/usr/bin/ruby
require 'facets'
h = Hash.autonew
File.open('trades_long.dat','r').each do |line|
#date,#time,#account,#ticker,#desc,#type,amount,#price = line.chomp.split('|')
if #account != "account"
puts "#{amount}"
h[#account][#ticker][#type][#price] += amount
end
#puts sum.to_s
end
The problem is no matter how I try to sum up the value in h[#account][#ticker][#type][#price] it gives me this error:
6000
/usr/local/lib/ruby/gems/1.9.1/gems/facets-2.7.0/lib/core/facets/hash/op_add.rb:8:in `merge': can't convert String into Hash (TypeError)
from /usr/local/lib/ruby/gems/1.9.1/gems/facets-2.7.0/lib/core/facets/hash/op_add.rb:8:in `+'
from ./trades_consolidaton.rb:13
from ./trades_consolidaton.rb:8:in `each'
from ./trades_consolidaton.rb:8
I've tried using different "autovivification" methods with no result. This wouldn't happen in perl! The autofvivification would know what you are trying to do. ruby doesn't seem to have this feature.
So my question really is, how do I perform simply "consolidation" of records in ruby. Specifically, how do I get the total for something like:
h[#account][#ticker][#type][#price]
Many thanks for your help!!
Just to clarify on glenn's solution. That would be perfect except it gives (with a few modifications to use the standard CSV library in ruby 1.9:
CSV.foreach("trades_long.dat", :col_sep => "|") do |row|
date,time,account,ticker,desc,type,amount,price = *row
records[[account,ticker,type,price]] += amount
end
gives the following error:
TypeError: String can't be coerced into Fixnum
from (irb):64:in `+'
from (irb):64:in `block in irb_binding'
from /usr/local/lib/ruby/1.9.1/csv.rb:1761:in `each'
from /usr/local/lib/ruby/1.9.1/csv.rb:1197:in `block in foreach'
from /usr/local/lib/ruby/1.9.1/csv.rb:1335:in `open'
from /usr/local/lib/ruby/1.9.1/csv.rb:1196:in `foreach'
from (irb):62
from /usr/local/bin/irb:12:in `<main>'
I agree with Jonas that you (and Sam) are making this more complicated than it needs to be, but I think even his version is too complicated. I'd just do this:
require 'fastercsv'
records = Hash.new(0)
FasterCSV.foreach("trades_long.dat", :col_sep => "|") do |row|
date,time,account,ticker,desc,type,amount,price = row.fields
records[[account,ticker,type,price]] += amount.to_f
end
Now you have a hash with total amounts for each unique combination of account, ticker, type and price.
If you want a hash builder that works that way, you are going to have to redefine the + semantics.
For example, this works fine:
class HashBuilder
def initialize
#hash = {}
end
def []=(k,v)
#hash[k] = v
end
def [](k)
#hash[k] ||= HashBuilder.new
end
def +(val)
val
end
end
h = HashBuilder.new
h[1][2][3] += 1
h[1][2][3] += 3
p h[1][2][3]
# prints 4
Essentially you are trying to apply the + operator to a Hash.
>> {} + {}
NoMethodError: undefined method `+' for {}:Hash
from (irb):1
However in facets{
>> require 'facets'
>> {1 => 10} + {2 => 20}
=> {1 => 10, 2 => 20}
>> {} + 100
TypeError: can't convert Fixnum into Hash
from /usr/lib/ruby/gems/1.8/gems/facets-2.7.0/lib/core/facets/hash/op_add.rb:8:in `merge'
from /usr/lib/ruby/gems/1.8/gems/facets-2.7.0/lib/core/facets/hash/op_add.rb:8:in `+'
from (irb):6
>> {} += {1 => 2}
=> {1=>2}
>>
If you want to redefine the + semantics for your hash in this occasion you can do:
class Hash; def +(v); v; end; end
Place this snippet before your original sample and all should be well. Keep in mind that you are changing the defined behavior for + (note + is not defined on Hash its pulled in with facets)
It looks like you are making it more complicated than it has to be. I would use the FasterCSV gem and Enumerable#inject something like this:
require 'fastercsv'
records=FasterCSV.read("trades_long.dat", :col_sep => "|")
records.sort_by {|r| r[3]}.inject(nil) {|before, curr|
if !before.nil? && curr[3]==before[3]
curr[6]=(curr[6].to_i+before[6].to_i).to_s
records.delete(before)
end
before=curr
}
For others that find their way here, there is now also another option:
require 'xkeys' # on rubygems.org
h = {}.extend XKeys::Hash
...
# Start with 0.0 (instead of nil) and add the amount
h[#account, #ticker, #type, #price, :else => 0.0] += amount.to_f
This will generate a navigable structure. (Traditional keying with arrays of [#account, #ticker, #type, #price] as suggested earlier may be better this particular application). XKeys auto-vivifies on write rather than read, so querying the structure about elements that don't exist won't change the structure.