I am trying to create an oplog watcher in ruby. So far ive come up with a small script below.
require 'rubygems'
require 'mongo'
db = Mongo::Connection.new("localhost", 5151).db("local")
coll = db.collection('oplog.$main')
loop do
cursor = Mongo::Cursor.new(coll, :tailable => true)
while not cursor.closed?
if doc = cursor.next_document
puts doc
else
sleep 1
end
end
end
The problem with this is, after 5 or 6 seconds when it has spit out a lot of data it times out and i get an error
C:/RailsInstaller/Ruby1.8.7/lib/ruby/gems/1.8/gems/mongo-1.4.0/lib/../lib/mongo/connection.rb
:807:in `check_response_flags': Query response returned CURSOR_NOT_FOUND. Either an invalid c
ursor was specified, or the cursor may have timed out on the server. (Mongo::OperationFailure
)
from C:/RailsInstaller/Ruby1.8.7/lib/ruby/gems/1.8/gems/mongo-1.4.0/lib/../lib/mongo/
connection.rb:800:in `receive_response_header'
from C:/RailsInstaller/Ruby1.8.7/lib/ruby/gems/1.8/gems/mongo-1.4.0/lib/../lib/mongo/
connection.rb:768:in `receive'
from C:/RailsInstaller/Ruby1.8.7/lib/ruby/gems/1.8/gems/mongo-1.4.0/lib/../lib/mongo/
connection.rb:493:in `receive_message'
from C:/RailsInstaller/Ruby1.8.7/lib/ruby/gems/1.8/gems/mongo-1.4.0/lib/../lib/mongo/
connection.rb:491:in `synchronize'
from C:/RailsInstaller/Ruby1.8.7/lib/ruby/gems/1.8/gems/mongo-1.4.0/lib/../lib/mongo/
connection.rb:491:in `receive_message'
from C:/RailsInstaller/Ruby1.8.7/lib/ruby/gems/1.8/gems/mongo-1.4.0/lib/../lib/mongo/
cursor.rb:494:in `send_get_more'
from C:/RailsInstaller/Ruby1.8.7/lib/ruby/gems/1.8/gems/mongo-1.4.0/lib/../lib/mongo/
cursor.rb:456:in `refresh'
from C:/RailsInstaller/Ruby1.8.7/lib/ruby/gems/1.8/gems/mongo-1.4.0/lib/../lib/mongo/
cursor.rb:124:in `next_document'
from n.rb:7
from n.rb:6:in `loop'
from n.rb:6
What i dont understand is when i m able to see the actual data how can it suddenly say cursor not found. Im pretty new to ruby and any ideas on what direction i must take will be useful for me.
The solution is that i need to have an exception handling mechanism to capture the exception which is thrown when the cursor reads the last document in a relatively small oplog with an higher number of writes per second. Since the cursor reaches the end of the oplog it would throw an exception that there are no more records.
require 'rubygems'
require 'mongo'
db = Mongo::Connection.new("localhost",5151).db("local")
coll = db.collection('oplog.$main')
loop do
cursor = Mongo::Cursor.new(coll, :timeout => false, :tailable => true)
while not cursor.closed?
begin
if doc = cursor.next_document
puts "Timestamp"
puts doc["ts"]
puts "Record"
puts doc["o"]
puts "Affected Collection"
puts doc["ns"]
end
rescue
puts ""
break
end
end
end
This now works as the exception is been handled. Thanks to the mongodb-user google group for pointing this out to me.
Related
I’m using Rails 4.2.3 and Nokogiri to get data from a web site. I want to perform an action when I don’t get any response from the server, so I have:
begin
content = open(url).read
if content.lstrip[0] == '<'
doc = Nokogiri::HTML(content)
else
begin
json = JSON.parse(content)
rescue JSON::ParserError => e
content
end
end
rescue Net::OpenTimeout => e
attempts = attempts + 1
if attempts <= max_attempts
sleep(3)
retry
end
end
Note that this is different than getting a 500 from the server. I only want to retry when I get no response at all, either because I get no TCP connection or because the server fails to respond (or some other reason that causes me not to get any response). Is there a more generic way to take account of this situation other than how I have it? I feel like there are a lot of other exception types I’m not thinking of.
This is generic sample how you can define timeout durations for HTTP connection, and perform several retries in case of any error while fetching content (edited)
require 'open-uri'
require 'nokogiri'
url = "http://localhost:3000/r503"
openuri_params = {
# set timeout durations for HTTP connection
# default values for open_timeout and read_timeout is 60 seconds
:open_timeout => 1,
:read_timeout => 1,
}
attempt_count = 0
max_attempts = 3
begin
attempt_count += 1
puts "attempt ##{attempt_count}"
content = open(url, openuri_params).read
rescue OpenURI::HTTPError => e
# it's 404, etc. (do nothing)
rescue SocketError, Net::ReadTimeout => e
# server can't be reached or doesn't send any respones
puts "error: #{e}"
sleep 3
retry if attempt_count < max_attempts
else
# connection was successful,
# content is fetched,
# so here we can parse content with Nokogiri,
# or call a helper method, etc.
doc = Nokogiri::HTML(content)
p doc
end
When it comes to rescuing exceptions, you should aim to have a clear understanding of:
Which lines in your system can raise exceptions
What is going on under the hood when those lines of code run
What specific exceptions could be raised by the underlying code
In your code, the line that's fetching the content is also the one that could see network errors:
content = open(url).read
If you go to the documentation for the OpenURI module you'll see that it uses Net::HTTP & friends to get the content of arbitrary URIs.
Figuring out what Net::HTTP can raise is actually very complicated but, thankfully, others have already done this work for you. Thoughtbot's suspenders project has lists of common network errors that you can use. Notice that some of those errors have to do with different network conditions than what you had in mind, like the connection being reset. I think it's worth rescuing those as well, but feel free to trim the list down to your specific needs.
So here's what your code should look like (skipping the Nokogiri and JSON parts to simplify things a bit):
require 'net/http'
require 'open-uri'
HTTP_ERRORS = [
EOFError,
Errno::ECONNRESET,
Errno::EINVAL,
Net::HTTPBadResponse,
Net::HTTPHeaderSyntaxError,
Net::ProtocolError,
Timeout::Error,
]
MAX_RETRIES = 3
attempts = 0
begin
content = open(url).read
rescue *HTTP_ERRORS => e
if attempts < MAX_RETRIES
attempts += 1
sleep(2)
retry
else
raise e
end
end
I would think about using a Timeout that raises an exception after a short period:
MAX_RESPONSE_TIME = 2 # seconds
begin
content = nil # needs to be defined before the following block
Timeout.timeout(MAX_RESPONSE_TIME) do
content = open(url).read
end
# parsing `content`
rescue Timeout::Error => e
attempts += 1
if attempts <= max_attempts
sleep(3)
retry
end
end
Still new to Ruby - I've had a look at some of the answers to seemingly similar questions but, to be honest, I couldn't get my head around them.
I have some code that reads a .csv file. The data is split into groups of 40-50 rows per user record and validates data in the rows against a database accessed via a website.
A login is required for each record, but once that user has logged in each row in the .csv file can be checked until the next user is reached, at which point the user logs out.
All that's working, however, if an error occurs (e.g. a different result on the website than the expected result on the .csv file) the program stops.
I need something that will
a) at tell me which line on the file the error occurred
b) log the row to be output when it's finished running, and
iii) restart the program from the next line in the .csv file
The code I have so far is below
Thanks in advance,
Peter
require 'csv-mapper'
loginrequired = true
Given(/^I compare the User Details from rows "(.*?)" to "(.*?)"$/) do |firstrow, lastrow|
data = CsvMapper.import('C:/auto_test_data/User Data csv.csv') do
[dln, nino, pcode, endor_cd, ct_cd]
end
#Row number changed because Excel starts at 'row 1' and Ruby starts counting at 'row 0'
(firstrow.to_i-1..lastrow.to_i-1).each do |row|
#licnum1 = data.at(row).dln
#licnum2 = data.at(row+1).dln
#nino = data.at(row).nino
#postcode = data.at(row).pcode
#endor_cd = data.at(row).endor_cd
#ct_cd = data.at(row).ct_cd
#Login only required once for each new user-account
if
loginrequired == true
logon_to_vdr #def for this is in hooks
click_on 'P and D'
loginrequired = false
end
#This is the check against the database and is required for every line in the .csv file
check_ctcd #def for this is in hooks
#Need something in here to log errors and move on to the next line in the .csv file
#Compare the ID for the next record and logout if they're different
if #licnum1 == #licnum2
loginrequired = false
else
loginrequired = true`enter code here`
click_on 'Logout'
end
end
end
It seems like you need some error logging since you apparently don't know what type of error you're receiving or where. If this script is standalone you can redirect $stderr to file so that you can read what went wrong.
# put this line at the top of your script
$stderr = File.open("/path/to/your/logfile.log","a")
When an error occurs, ruby will automatically write the error message, class, and backtrace to the log file you specify so that you can trace back the line where things are not going as expected. (When you run a script from the command line, normally this information will just get blurted back to the terminal when an error happens.)
For example, on my desktop I created a file log_stderr.rb with the following (line numbers included):
1 $stderr = File.open("C:/Users/me/Desktop/my_log.log","w")
2
3 #require a file which will raise an error to see the backtrace
4 require_relative 'raise_error.rb'
5
6 puts "code that will never be reached"
Also on my desktop I created the file raise_error.rb with the following (to deepen the backtrace for better example output):
1 # call raise to generate an error arbitrarily
2 # to halt execution and exit the program.
3 raise RuntimeError, 'the program stopped working!'
When I run ruby log_stderr.rb from the command line, my_log.log is created on my desktop with the following:
C:/Users/me/Desktop/raise_error.rb:3:in `<top (required)>': the program stopped working! (RuntimeError)
from C:/Users/me/Desktop/log_stderr.rb:4:in `require_relative'
from C:/Users/me/Desktop/log_stderr.rb:4:in `<main>'
If you are working in a larger environment where your script is being called amidst other scripts then you probably do not want to redirect $stderr because this would affect everything else running in the environment. ($stderr is global as indicated by the $ variable prefix.) If this is the case you would want to implement a begin; rescue; end structure and also make your own logfile so that you don't affect $stderr.
Again, since you don't know where the error is happening you want to wrap the whole script with begin; end
# at the very top of the script, begin watching for weirdness
begin
logfile = File.open("/path/to/your/logfile.log", "w")
require 'csv-mapper'
#. . .
# rescue and end at the very bottom to capture any errors that have happened
rescue => e
# capture details about the error in your logfile
logfile.puts "ERROR:", e.class, e.message, e.backtrace
# pass the error along since you don't know what it is
# and there may have been a very good reason to stop the program
raise e
end
If you find that your error is happening only in the block (firstrow.to_i-1..lastrow.to_i-1).each do |row| you can place the begin; end inside of this block to have access to the local row variable, or else create a top level variable independent of the block and assign it during each iteration of the block to report to your logfile:
begin
logfile = File.open("/path/to/your/logfile.log", "w")
csv_row = "before csv"
#. . .
(firstrow.to_i-1..lastrow.to_i-1).each do |row|
csv_row = row
#. . .
end
csv_row = "after csv"
rescue => e
logfile.puts "ERROR AT ROW: #{csv_row}", e.class, e.message, e.backtrace
raise e
end
I hope this helps!
It doesn't seem like you need to rescue exception here. But what you could do is in your check_ctcd method, raise error if records doesn't match. Then you can rescue from it. In order to know which line it is, in your iteration, you could use #each_with_index and log the index when things go wrong.
(firstrow.to_i-1..lastrow.to_i-1).each_with_index do |row, i|
#licnum1 = data.at(row).dln
#licnum2 = data.at(row+1).dln
#nino = data.at(row).nino
#postcode = data.at(row).pcode
#endor_cd = data.at(row).endor_cd
#ct_cd = data.at(row).ct_cd
#Login only required once for each new user-account
if
loginrequired == true
logon_to_vdr #def for this is in hooks
click_on 'P and D'
loginrequired = false
end
#This is the check against the database and is required for every line in the .csv file
check_ctcd #def for this is in hooks
rescue => e
# log the error and index here
...
And you can make your own custom error, and rescue only the certain type so that you don't silently rescue other errors.
I think my issue is the same as that in Having problems with Ruby file from Dashing which as to date no answer.
Full problem is when I start dashing I get.
scheduler caught exception:
undefined method `[]' for nil:NilClass
/home/bhladmin/Shopify-dashing-e672d84/dashboard/jobs/twitter_user.rb:19:in `block in <top (required)>'
/usr/lib64/ruby/gems/1.9.1/gems/rufus-scheduler-2.0.23/lib/rufus/sc/jobs.rb:230:in `call'
/usr/lib64/ruby/gems/1.9.1/gems/rufus-scheduler-2.0.23/lib/rufus/sc/jobs.rb:230:in `trigger_block'
/usr/lib64/ruby/gems/1.9.1/gems/rufus-scheduler-2.0.23/lib/rufus/sc/jobs.rb:204:in `block in trigger'
/usr/lib64/ruby/gems/1.9.1/gems/rufus-scheduler-2.0.23/lib/rufus/sc/scheduler.rb:430:in `call'
/usr/lib64/ruby/gems/1.9.1/gems/rufus-scheduler-2.0.23/lib/rufus/sc/scheduler.rb:430:in `block in trigger_job'
Something isn't right on line 19, but I can't work out what...
The full section of code is below...
#!/usr/bin/env ruby
require 'net/http'
# Track public available information of a twitter user like follower, follower
# and tweet count by scraping the user profile page.
# Config
# ------
twitter_username = ENV['TWITTER_USERNAME'] || 'foobugs'
SCHEDULER.every '2m', :first_in => 0 do |job|
http = Net::HTTP.new("twitter.com", Net::HTTP.https_default_port())
http.use_ssl = true
response = http.request(Net::HTTP::Get.new("/#{twitter_username}"))
if response.code != "200"
puts "twitter communication error (status-code: #{response.code})\n#{response.body}"
else
tweets = /profile["']>[\n\t\s]*<strong>([\d.,]+)/.match(response.body)[1].delete('.,').to_i
following = /following["']>[\n\t\s]*<strong>([\d.,]+)/.match(response.body)[1].delete('.,').to_i
followers = /followers["']>[\n\t\s]*<strong>([\d.,]+)/.match(response.body)[1].delete('.,').to_i
send_event('twitter_user_tweets', current: tweets)
send_event('twitter_user_followers', current: followers)
send_event('twitter_user_following', current: following)
end
end
From the previous question it looks like the way of extracting the data from the webpage is the problem, but I don't know Ruby well enough. I've tried removing the ENV['TWITTER_USERNAME'] section to make sure the username I used (not the one above) is being used. If I dump out the raw html data then it contains the info I'm searching for so I know that part is working.
I think I've solved this myself, by going about it a different way. I've changed the code to use the twitter API rather than page scraping. Details below... The auth checking and timeout isn't great so if anyone has hints on making that better they'd be welcome...
#### Get your twitter keys & secrets:
#### https://dev.twitter.com/docs/auth/tokens-devtwittercom
Twitter.configure do |config|
config.consumer_key = 'YOUR_CONSUMER_KEY'
config.consumer_secret = 'YOUR_CONSUMER_SECRET'
config.oauth_token = 'YOUR_OAUTH_TOKEN'
config.oauth_token_secret = 'YOUR_OAUTH_SECRET'
end
twitter_username = 'foobugs'
MAX_USER_ATTEMPTS = 10
user_attempts = 0
SCHEDULER.every '10m', :first_in => 0 do |job|
begin
tw_user = Twitter.user("#{twitter_username}")
if tw_user
tweets = tw_user.statuses_count
followers = tw_user.followers_count
following = tw_user.friends_count
send_event('twitter_user_tweets', current: tweets)
send_event('twitter_user_followers', current: followers)
send_event('twitter_user_following', current: following)
end
rescue Twitter::Error => e
user_attempts = user_attempts +1
puts "Twitter error #{e}"
puts "\e[33mFor the twitter_user widget to work, you need to put in your twitter API keys in the jobs/twitter_user.rb file.\e[0m"
sleep 5
retry if(user_attempts < MAX_USER_ATTEMPTS)
end
end
I'm new to ruby so forgive me in advance if this is a silly question. I've googled for answers but nothing relevant comes up and it seems the answer should be obvious.
I'm attempting to pass a DBI::DatabaseHandle as function argument and I'm getting a wrong "number of arguments" error when I run the function. Here's my code...
require 'rubygems'
require 'dbi'
class CmsTest
def self.get_dbi_connection(hostname, user, password)
connection = DBI.connect("DBI:OCI8:" + hostname, user, password)
return connection
end
def self.query(connection, sql)
puts connection
puts sql
begin
request = connection.prepare("#{query}")
request.execute
fetched = []
request.fetch do |row|
fetched << row.to_h
end
request.finish
return fetched
rescue DBI::DatabaseError => e
log "An error occurred"
log "Error code: #{e.err}"
log "Error message: #{e.errstr}"
ensure
end
end
end
So my code that calls this looks like so...
require 'rubygems'
require 'cms_test'
connection = CmsTest.get_dbi_connection('foo', 'bar', 'fubar')
CmsTest.query(connection, "<some sql query>")
So the first argument is a DBI::DatabaseHandle object and the second is some sql query string. When I run that I get this...
`query': wrong number of arguments (0 for 2) (ArgumentError)
This even though the query signature contains two arguments and and I'm passing the method two arguments. The really weird thing for me is that if I put and exit statement anywhere in the method body after the puts it will show that the method did indeed receive 2 arguments...
#<DBI::DatabaseHandle:0x007fa2a316c9f0>
select licensor_id, licensor_name from cf_licensor
I can't make any sense of this. Please help.
You have a method named query:
def self.query(connection, sql)
and then inside query you try to call query:
request = connection.prepare("#{query}")
# -- method call ---------------^^^^^
You probably want to use sql there and there's no need for string interpolation:
request = connection.prepare(sql)
I am working on a program that uses yahoo finance api to collect the historical close data for the number of stocks entered and then go ahead and calculate simple moving average (SMA) for the data for period of 30 days. I have the following so far:
require 'rubygems'
require 'yahoofinance'
array = []
while line = gets
break if line.chomp =~ /N/ #exit when 'N' is entered
array << line.chomp
end
puts "Values: #{array.join(',')}" #joining all the elements with a comma
array.each do |s|
print "\n______\n"
puts s
YahooFinance::get_HistoricalQuotes( s,
Date.parse( '2012-10-06' ),
Date.today() ) do |hq|
puts "#{hq.close}"
end
end
This code is giving me the close values for stocks for the specified range. I have two questions:
Currently, hq.close is holding values for all stocks. How can I put these values in an array so that I can do a computation on it to calculate a SMA for each stock data?
I tried doing something like this:
"#{hq.close}" my_val = [hq.close]
puts my_val
But this only gives the value of first stock in my_val. I know I have to put a loop here. I tried putting
while(!hq.close.emply?)
my_val = [hq.close]
puts my_val
end
But this gives me an error:
C:/Users/Muktak/workspace/RubySample/sample_program.rb:23:in block (2 levels) in <main>': undefined methodemplty?' for 19.52:Float (NoMethodError) from
C:/Ruby193/lib/ruby/gems/1.9.1/gems/yahoofinance-1.2.2/lib/yahoofinance.rb:491:in block in get_HistoricalQuotes' from
C:/Ruby193/lib/ruby/gems/1.9.1/gems/yahoofinance-1.2.2/lib/yahoofinance.rb:456:inblock in get_historical_quotes' from
C:/Ruby193/lib/ruby/gems/1.9.1/gems/yahoofinance-1.2.2/lib/yahoofinance.rb:456:in each' from
C:/Ruby193/lib/ruby/gems/1.9.1/gems/yahoofinance-1.2.2/lib/yahoofinance.rb:456:inget_historical_quotes' from
C:/Ruby193/lib/ruby/gems/1.9.1/gems/yahoofinance-1.2.2/lib/yahoofinance.rb:489:in get_HistoricalQuotes' from
C:/Users/Muktak/workspace/RubySample/sample_program.rb:19:inblock in ' from
C:/Users/Muktak/workspace/RubySample/sample_program.rb:13:in each' from
C:/Users/Muktak/workspace/RubySample/sample_program.rb:13:in'
Values: FB,GOOG
How can I calculate a SMA in Ruby?
You've asked two questions here, so let's address them one at a time.
First, this code:
require 'rubygems'
require 'yahoofinance'
stock_names = %w{MSFT RHT AAPL}
start = Date.parse '2012-10-06'
finish = Date.today
closes = {}
stock_names.each do |stock_name|
quotes = YahooFinance::get_HistoricalQuotes stock_name, start, finish
closes[stock_name] = quotes.collect { |quote| quote.close }
end
... will produce the following hash in closes, which I understand is in the format you want:
{
"AAPL" => [629.71, 628.1, 640.91, 635.85, 638.17],
"RHT"=> [53.69, 53.77, 53.86, 54.0, 54.41],
"MSFT"=> [29.2, 28.95, 28.98, 29.28, 29.78]
}
Secondly, you want to calculate a simple moving average - which for financial applications is just the mean of the values. There is a Gem called simple_statistics that can do this.
This code:
require 'rubygems'
require 'yahoofinance'
require 'simple_statistics'
stock_names = %w{MSFT RHT AAPL}
start = Date.parse '2012-10-06'
finish = Date.today
averages = {}
stock_names.each do |stock_name|
quotes = YahooFinance::get_HistoricalQuotes stock_name, start, finish
closes = quotes.collect { |quote| quote.close }
averages[stock_name] = closes.mean
end
... produces the following hash in averages:
{ "AAPL" => 634.548, "MSFT" => 29.238, "RHT" => 53.946 }