I'm writing some code that will pulls URLS from a text file and then check to see if they load or not. The code I have is:
require 'rubygems'
require 'watir'
require 'timeout'
Watir::Browser.default = "firefox"
browser = Watir::Browser.new
File.open('pl.txt').each_line do |urls|
begin
Timeout::timeout(10) do
browser.goto(urls.chomp)
if browser.text.include? "server"
puts 'here the page didnt'
else
puts 'here site was found'
File.open('works.txt', 'a') { |f| f.puts urls }
end
end
rescue Timeout::Error => e
puts e
end
end
browser.close
The thing is though I get the error:
execution expired
/Library/Ruby/Gems/1.8/gems/firewatir-1.9.4/lib/firewatir/jssh_socket.rb:19:in `const_get': wrong number of arguments (2 for 1) (ArgumentError)
from /Library/Ruby/Gems/1.8/gems/firewatir-1.9.4/lib/firewatir/jssh_socket.rb:19:in `js_eval'
from /Library/Ruby/Gems/1.8/gems/firewatir-1.9.4/lib/firewatir/firefox.rb:303:in `open_window'
from /Library/Ruby/Gems/1.8/gems/firewatir-1.9.4/lib/firewatir/firefox.rb:94:in `get_window_number'
from /Library/Ruby/Gems/1.8/gems/firewatir-1.9.4/lib/firewatir/firefox.rb:103:in `goto'
from samplecodestack.rb:17
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:62:in `timeout'
from samplecodestack.rb:16
from samplecodestack.rb:13:in `each_line'
from samplecodestack.rb:13
Anyone know how to get it working?
You can use net/http and handle the timeouts too.
require "net/http"
require "uri"
File.open('pl.txt').each_line do |urls|
uri = URI.parse(urls.chomp)
begin
response = Net::HTTP.get_response(uri)
rescue Exception=> e
puts e.message
puts "did not load!"
end
end
I had trouble following your stack trace but it seems to be on your goto statement.
execution expired is the error that occurs when the block for the Timeout::timeout is exceeded. Note that the timeout is checking that its entire block is completed in the specified time. Given the line number errors, I am guessing that the URL being loaded took close to 10 seconds and then the text check timed out.
I assume you really only mean for the timeout to occur if the page takes longer than 10 seconds to load, rather than the entire test taking 10 seconds to finish. So you should move the if statement out of the Timeout block:
File.open('pl.txt').each_line do |urls|
begin
Timeout::timeout(10) do
browser.goto(urls.chomp)
end
if browser.text.include? "server"
puts 'here the page didnt'
else
puts 'here site was found'
File.open('works.txt', 'a') { |f| f.puts urls }
end
rescue Timeout::Error => e
puts 'here the page took too long to load'
puts e
end
end
Related
I have a method that scans a URL for a website that contains an error:
def begin_vulnerability_check
info("Checking if sites are vulnerable.")
IO.read("#{PATH}/temp/SQL_sites_to_check.txt").each_line do |parse|
Timeout::timeout(10) do
parsing = Nokogiri::HTML(RestClient.get("#{parse.chomp}"))
info("Parsing page for SQL syntax error: #{parse.chomp}")
if parsing.css('html')[0].to_s[/You have an error in your SQL syntax/]
successful = parse
success("URL: #{parse.chomp} returned SQL syntax error, dumped to SQL_VULN.txt")
File.open("#{PATH}/lib/SQL_VULN.txt", "a+"){|s| s.puts(parse)}
sleep(1)
else
err("URL: #{parse.chomp} returned and error, dumped to non_exploitable.txt")
File.open("#{PATH}/lib/non_exploitable.txt", "a+"){|s| s.puts(parse)}
sleep(1)
end
end
end
end
During testing I'm scanning through this list of URLs:
http://www.bible.com/subcat.php?id=2'
http://www.cidko.com/pro_con.php?id=3'
http://www.slavsandtars.com/about.php?id=25'
http://www.police.gov/content.php?id=275'
http://www.icdprague.org/index.php?id=10'
http://huawei.com/en/plugin.php?id=hwdownload'
https://huawei.com/en/plugin.php?id=unlock'
https://facebook.com/profile.php?id'
http://www.footballclub.com.au/index.php?id=43'
http://www.mesrs.gouv/index.php?id=1525'
I also have a rescue block that is suppose to catch the exception Timeout::Error and move to the next URL in the list:
begin
begin_vulnerability_check
rescue Timeout::Error
if Timeout::Error
warn("Page timed out, this is usually cause by the page returning a white page, or being non-existent, skipping.")
next
end
end
However while attempting to run this program, I get the following error:
whitewidow.rb:130: Invalid next
whitewidow.rb: compile error (SyntaxError)
Line 130:
rescue Timeout::Error
if Timeout::Error
warn("Page timed out, this is usually cause by the page returning a white page, or being non-existent, skipping.")
next #<= HERE
end
end
My question being, am I using the next in the wrong sense? It seems to me like next would be, if this happens go to the next line, am I wrong for thinking like that? How can I refactor this to work?
You can use next to return from a block. You cannot use it outside a block like you're trying to do.
But you don't even need next, because when you rescue the timeout error the iteration will automatically continue with the next line. You just have to move the rescue inside the each_line iteration.
Your code should be something like this:
def begin_vulnerability_check
IO.read("#{PATH}/temp/SQL_sites_to_check.txt").each_line do |parse|
begin
Timeout::timeout(10) do
...
end
rescue Timeout::Error
# Will automatically continue with next line after this
end
end
end
I am trying to put together a series of custom 404 pages.
require 'uri'
def open(url)
page_content = Net::HTTP.get(URI.parse(url))
puts page_content.content
end
open('http://somesite.com/1ygjah1761')
the following code exits the program with an error. How can I get the page content from a website, regardless of it being 404 or not.
You need to rescue from the error
def open(url)
require 'net/http'
page_content = ""
begin
page_content = Net::HTTP.get(URI.parse(url))
puts page_content
rescue Net::HTTPNotFound
puts "THIS IS 404" + page_content
end
end
You can find more information on something like this here: http://tammersaleh.com/posts/rescuing-net-http-exceptions/
Net::HTTP.get returns the page content directly as a string, so there is no need to call .content on the results:
page_content = Net::HTTP.get(URI.parse(url))
puts page_content
ie.link(:id, "ctl00_ContentPlaceHolder1_BtnSearch").click
rescue Timeout::Error
#sleep(5)
puts "timeout"
ie.close
#sleep(9)
retry #open new browser and go to begin
end`
when .click link gets time out , then output is = timeout, but ie.close does not work.
and the time out error comes
*i want to close the browser when time out error comes*
I do not believe that ie.link(:id, "ctl00_ContentPlaceHolder1_BtnSearch").click will ever throw a Timeout::Error. This would be why the rescue block is never executed.
The likely exceptions that are thrown are:
When you do ie.link(:id, "ctl00_ContentPlaceHolder1_BtnSearch").click and the element is not found, a Watir::Exception::UnknownObjectException will occur.
When you do ie.link(:id, "ctl00_ContentPlaceHolder1_BtnSearch").when_present.click and the element is not found within the required time frame, a Watir::Wait::TimeoutError will occur.
Your rescue likely needs to be catching one of these exceptions instead.
begin
ie = Watir::Browser.new
ie.goto 'www.yourpage.com'
ie.link(:id, "ctl00_ContentPlaceHolder1_BtnSearch").click
rescue Watir::Exception::UnknownObjectException
puts "element not found"
ie.close
retry #open new browser and go to begin
end
Or if you are using when_present on the element:
begin
ie = Watir::Browser.new
ie.goto 'www.yourpage.com'
ie.link(:id, "ctl00_ContentPlaceHolder1_BtnSearch").when_presentclick
rescue Watir::Wait::TimeoutError
puts "element did not appear in time"
ie.close
retry #open new browser and go to begin
end
I know how to catch the exceptions but what we do is to put "rescue" after a suspicious section of a code. what if you had a lot functions sending a query to mysql through mysql2 gem and you want to catch their exceptions. One thing you can do is to put a "rescue" statement in each of them. but i want to do that just by one rescue statement. So I put a "rescue" in end of code and put all of code in a "begin" and "end" but it didn't work.
Here is my code and as you see there is a problem in mysql query and just because of "rescue" being end of file, it doesn't catch the exception but when I put it after that query it works.
require 'mysql2'
require 'colored'
begin
def log(string)
p "["+string.cyan+"]"
end
def err
p "["+"FAIL".red+"]"
end
def done
p "["+"DONE".red+"]"
end
class SqlClient
def initialize()
log "SqlClient/initialize"
puts "Host: \n"
#host = gets.strip
puts "User: \n"
#user = gets.strip
puts "Pass: \n"
#pass = gets.strip
#client = Mysql2::Client.new(host: #host , username: #user , password: #pass)
end
def list_databases()
puts "We are listing your databases(not just projects) \n \n \n "
#client.query("ELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA").each do |row|
p row["SCHEMA_NAME"]
end
puts "\n \n \n"
end
end
rescue Mysql2::Error
err
abort
end
`query': You have an error in your SQL syntax; check the manual that corresponds to your
MySQL server version for the right syntax to use near 'ELECT SCHEMA_NAME FROM
INFORMATION_SCHEMA.SCHEMATA' at line 1 (Mysql2::Error)
I'm not looking for something like:
begin
# my code
rescue # this line is right after the code which is going to have problem and we catch it.
end
I'm looking for something like this:
begin
# first method
# second method
# thrid method
# rest of code and etc ...
# now this is end of file:
rescue
end
but as you saw in my code, it didn't work.
UPDATE: I found a similar question here and it seems there will be no answer :| maybe this is a sort of ruby Weakness.
if you want to see ANY error just use e for example
begin
# your call to a method of Mysql2 gem. for example:
client = Mysql2::Client.new(:host => "localhost", :username => "root", etc...)
rescue => e
puts e.message
puts e.backtrace.inspect
end
In order to catch every exception you'd need to wrap each method call with a begin rescue end. When an exception is raised it bails out of the execution so it wouldn't hit the next methods.
To catch all errors I guess I'd do something like this. Keep in mind, this is ugly and I'd recommend for you NOT to do this, but... if you want to try, maybe try something like this:
all_errors = []
# first method you call
begin
# some error might happen here
first_response = Mysql2::Client.new(:host => "localhost", :username => "root", etc...)
rescue => e
all_errors << e
end
# second method you call
begin
# some error might happen here
second_response = Mysql2::Client.new(:host => "localhost", :username => "root", etc...)
rescue => e
all_errors << e
end
puts all_errors.inspect
After a quick search I've found: (http://coderrr.wordpress.com/2008/11/07/the-simple-way-to-print-exceptions-in-ruby/)
# catch all exceptions (anything that derives from Exception)
begin
...
rescue Exception
puts $!, $#
end
You could use an at_exit handler, which has access to the last exception in $!
like
at_exit {
puts "the exception that killed us is", $!
}
If you want to catch Exceptions "as soon as they occur" (not after they're caught) you could use ruby's "debug mode" (which outputs messages when they occur to the console) or ruby-debug see Is there any way to start the Ruby debugger on exception?
Just wrap all your code in:
begin
#yourcode
#as much as you want
rescue
end
Nobody seemed to notice it but using rescue without a class will catch all StandardError and there are so much more.
If you want to catch ALL exceptions you need to do
begin
# your code where you call SqlClient.new etc
rescue Exception => e
puts "error raised"
puts [e, e.backtrace].flatten.join("\n")
end
List of all error classes:
Exception
NoMemoryError
ScriptError
LoadError
NotImplementedError
SyntaxError
SignalException
Interrupt
StandardError
ArgumentError
IOError
EOFError
IndexError
LocalJumpError
NameError
NoMethodError
RangeError
FloatDomainError
RegexpError
RuntimeError
SecurityError
SystemCallError
SystemStackError
ThreadError
TypeError
ZeroDivisionError
SystemExit
fatal
have you tried adding an at_exit method in your class? This would allow you to do something when ruby is exiting. like in this article.
Ruby at_exit
or
From Ruby 2.0 API Docs
However beware of cleverly rescuing from an exception!
You'll start pulling your hair out down the road (or another dev will) when you try to figure out why your code isn't failing when it should be failing. I prefer to fail massively with a bright shiny sign saying the code failed here! hehe.
Good luck!
My question is related to How to rescue timeout issues (Ruby, Rails).
Here's the common way to rescue from a timeout:
def action
# Post using Net::HTTP
rescue Timeout::Error => e
# Do something
end
I'd like to determine if the exception was raised while trying to connect to the host, or if it was raised while trying to read from the host. Is this possible?
Here's the solution (after Ben's fix):
require "net/http"
http = Net::HTTP.new("example.com")
http.open_timeout = 2
http.read_timeout = 3
begin
http.start
begin
http.request_get("/whatever?") do |res|
res.read_body
end
rescue Timeout::Error
puts "Timeout due to reading"
end
rescue Timeout::Error
puts "Timeout due to connecting"
end
Marc-André Lafortune's solution is still the best if you can't upgrade to ruby 2.x.
Starting from 2.x, a subclass of Timeout::Error will be raised depending on which timeout was triggered:
Net::OpenTimeout
Net::ReadTimeout
However, the read_timeout behavior is strange on 2.x, because it seems to double the value you set. This article explains why.
Here's a test for both timeouts (tested on 1.8.7, 1.9.3, 2.1.2, 2.2.4).
EDIT: The open_timeout test works on Mac, but on Linux, the client gets a "connection refused" error.
require "net/http"
require "socket"
SERVER_HOST = '127.0.0.1'
SERVER_PORT = 9999
def main
puts 'with_nonlistening_server'
with_nonlistening_server do
make_request
end
puts
puts 'with_listening_server'
with_listening_server do
make_request
end
end
def with_listening_server
# This automatically starts listening
serv = TCPServer.new(SERVER_HOST, SERVER_PORT)
begin
yield
ensure
serv.close
end
end
def with_nonlistening_server
raw_serv = Socket.new Socket::AF_INET, Socket::SOCK_STREAM, 0
addr = Socket.pack_sockaddr_in SERVER_PORT, SERVER_HOST
# Bind, but don't listen
raw_serv.bind addr
begin
yield
ensure
raw_serv.close
end
end
def make_request
http = Net::HTTP.new(SERVER_HOST, SERVER_PORT)
http.open_timeout = 1
http.read_timeout = 1 # seems to be doubled on ruby 2.x
start_tm = Time.now
begin
http.start
begin
http.get('/')
rescue Timeout::Error => err
puts "Read timeout: #{err.inspect}"
end
rescue Timeout::Error => err
puts "Open timeout: #{err.inspect}"
end
end_tm = Time.now
puts "Duration (sec): #{end_tm - start_tm}"
end
if __FILE__ == $PROGRAM_NAME
main
end
Example output on 1.9.3:
with_nonlistening_server
Open timeout: #<Timeout::Error: execution expired>
Duration (sec): 1.002477
with_listening_server
Read timeout: #<Timeout::Error: Timeout::Error>
Duration (sec): 1.00599
Example output on 2.1.2:
with_nonlistening_server
Open timeout: #<Net::OpenTimeout: execution expired>
Duration (sec): 1.005923
with_listening_server
Read timeout: #<Net::ReadTimeout: Net::ReadTimeout>
Duration (sec): 2.009582