I have the following, which is repeated four times:
def self.curl_delete(url, response_flag)
url = "#{#baseUrl}#{url}"
curl_response = Curl.delete(url) do |curl|
curl.headers['Content-Type'] = 'text/plain'
curl.headers['Client'] = 'Curb DELETE'
curl.headers['Authorization'] = "Bearer #{#token}"
curl.verbose = false
end
response = JSON.parse(curl_response.body_str)
if response_flag == true
BF.puts_and_file " *response flag set to true for DELETE*\n"
BF.puts_and_file " url: #{url}\n"
BF.puts_and_file " response: #{response}\n\n"
end
return response
end
I would like to turn that if statement into a method.
I tried def response_flag_true(flag) but didn't work...what's the right way to DRY up that code?
Keep it simple:
def bf_puts_and_file(indent, *messages)
messages.each { |s| BF.puts_and_file(' ' * indent + s) }
end
Used as follows:
bf_puts_and_file(5, "*response flag set to true for DELETE*\n",
"url: #{url}\n",
"response: #{response}\n\n") if response_flag
Related
This code from Handle Web Requests with Ruby appears in the video demo and runs in the video, but I don't see how it could work. The cookie_parts "block" ends with an else clause that has no preceding if or rescue.
require 'socket'
server = TCPServer.new 8000
PAGES = {
"/" => "Hi, welcome to the home page!",
"/about" => "About us: we are http hackers",
"/news" => "We haven't made much news yet with this server, but stay tuned"
}
PAGE_NOT_FOUND = "Sorry, there's nothing here."
loop do
session = server.accept
request = []
while (line = session.gets) && (line.chomp.length > 0)
request << line.chomp
end
puts "finished reading"
http_method, path, protocol = request[0].split(' ') # 3 parts to request line
cookie_header = request.detect { |line| line =~ /^Cookie: / }
cookie_parts = cookie_header.sub /^Cookie: /, ''
cookie_parts = cookie_parts.split('; ')
cookie_parts = cookie_parts.map { |cookie| cookie.split('=') }
cookies = Hash[*cookie_parts.flatten]
else
cookies = {}
end
cookies["session_key"] ||= Time.now.to_f
# nil.to_i returns 0, so if cookie isn't set, count will be 0
count = cookies["session_count"].to_i
count += 1
cookies["session_count"] = count
if PAGES.keys.include?(path)
status = "200 OK"
response_body = PAGES[path]
else
status = "404 Not Found"
response_body = PAGE_NOT_FOUND
end
session.puts <<-HEREDOC
HTTP/1.1 #{status}
set-cookie:session_key=#{cookies["session_key"]}
set-cookie:session_count=#{cookies["session_count"]}
#{response_body}
This was visit number #{cookies["session_count"]}!
HEREDOC
session.close
end
Running the same code locally yields the errors I'd expect:
server.rb:28: else without rescue is useless
server.rb:57: syntax error, unexpected `end', expecting end-of-input
What conditional or block is the else supposed to belong to?
Watch the rest of the video, to wit, at ~5:30 you'll see the missing if I mentioned in my comment:
if cookie_header = request.detect { |line| line =~ /^Cookie: / }
cookie_parts = cookie_header.sub /^Cookie: /, ''
cookie_parts = cookie_parts.split('; ')
cookie_parts = cookie_parts.map { |cookie| cookie.split('=') }
cookies = Hash[*cookie_parts.flatten]
else
cookies = {}
end
In the middle of the code in your question, there is this section
cookie_header = request.detect { |line| line =~ /^Cookie: / }
cookie_parts = cookie_header.sub /^Cookie: /, ''
cookie_parts = cookie_parts.split('; ')
cookie_parts = cookie_parts.map { |cookie| cookie.split('=') }
cookies = Hash[*cookie_parts.flatten]
else
cookies = {}
end
There you have an else block but there is no if condition in front of it. Ruby doesn't know how to handle it and therefore the error message.
I agree that the error message is a bit misleading because an else block like that is more likely missing an if condition than a rescue. But a rescue block is also an example in which Ruby allows an else like this:
begin
# code that might raise an error
rescue
# code that is run when there was an error
else
# code that is run when there was _no_ error
end
I'm using faraday with net-http-persistent adapter to make HTTP requests.
I want to optimise my requests by making them execute asynchronously but as I need a persistent connection I keep getting errors such as too many connections reset which I assume is due to the fact that I have multiple threads creating new connections.
I tried changing my adapter to typhoeus but as the connection is not persistent the final result of executing all request is not as expected.
My goal is to add items to a basket by making this HTTP requests. Without the persistent connection items are not added to the basket.
So, my question is:
Is it possible to make persistent HTTP requests reusing the connection between threads? If so, how can this be achieved?
Here is a piece of my code:
Create the connection:
Faraday.new do |c|
c.use :cookie_jar, jar: cookie_jar
c.options.open_timeout = 5
c.options.timeout = 10
c.request :url_encoded
c.response :logger, logger
c.adapter :net_http_persistent do |http| # yields Net::HTTP::Persistent
http.idle_timeout = 2
end
end
Creating threads and getting the result of each one of them
result = []
threads = []
total_items = items.size
items.each_slice(5) do |sliced_items|
# Create a thread for a batch of 5 items and store its result
threads << Thread.new do
Thread.current[:output] = browser.add_all_items(sliced_items)
end
end
# Wait for all threads to finish their work and store their output into result
threads.each do |t|
t.join
result << t[:output]
end
add_all_items and add_to_cart methods:
# Add a batch of items by the key passed (id, gtin, url)
def add_all_items(items_info, key)
results = []
items_info.each do |item|
begin
add_to_cart(item[key], item[:quantity])
item[:message] = nil
rescue => e
item[:message] = e.message
puts "---------- BACKTRACE -------------- \n #{e.backtrace}"
end
puts "\n--------- MESSAGE = #{item[:message]} --------- \n"
results << item
puts "-------- RESULTS #{results}"
end
results
end
def add_to_cart(url, count = 1)
response = connection.get(url) do |req|
req.headers["User-Agent"] = #user_agent
end
doc = Nokogiri::HTML(response.body)
stoken = doc.search('form.js-oxProductForm input[name=stoken]').attr('value').value
empty_json = '""'
product_id = get_item_id(url)
data = { #removed payload for security reasons }
# Using example.com for question purposes
response = connection.post('https://www.example.com/index.php?') do |req|
req.headers["Origin"] = "https://www.example.com"
req.headers["Content-Type"] = "application/x-www-form-urlencoded; charset=UTF-8"
req.headers["Accept"] = "application/json, text/javascript, */*; q=0.01"
req.headers["Referer"] = url
req.headers["Pragma"] = "no-cache"
req.headers["Accept-Language"] = "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"
req.headers["User-Agent"] = #user_agent
req.headers["Cache-Control"] = "no-cache"
req.headers["Connection"] = "keep-alive"
req.headers["DNT"] ="1"
req.headers["Content-Length"] = data.size.to_s
req.headers["Accept"] = "*/*"
req.headers["X-Requested-With"] = "XMLHttpRequest"
req.headers["Connection"] = "keep-alive"
req.body = data
end
begin
json = JSON.parse(response.body)
raise "Could not add item: #{json['message']}" if json['success'] != 1 || json['item'] != product_id
rescue JSON::ParserError => e
puts "JSON Error"
end
end
def get_item_id(url)
response = connection.get(url) do |req|
req.headers["User-Agent"] = #user_agent
end
doc = Nokogiri::HTML(response.body)
doc.search('.js-oxProductForm input[name=aid]').attr('value').value
end
Thanks in advance.
I could be missing the obvious here, but this code is failing unless I use the return clause when call it recursively on Net::HTTPRedirection case.
def fetch_headers(limit = REDIRECT_LIMIT)
# You should choose a better exception.
raise ArgumentError, 'too many HTTP redirects' if limit == 0
http = Net::HTTP.new(#uri.host, #uri.port)
http.use_ssl = true if #uri.scheme == 'https'
request_uri = #uri.request_uri.nil? ? '/' : #uri.request_uri
http.request_head(request_uri) do |response|
case response
when Net::HTTPSuccess then
return response
when Net::HTTPRedirection then
location = response['location']
parsed_location = URI.parse location
#uri = parsed_location.absolute? ? parsed_location : #uri.merge(parsed_location)
fetch_headers(limit - 1)
else
return response.value
end
end
end
The caller method:
def perform(link_id)
link = Link.find(link_id)
url = link.url =~ /^http/ ? link.url : "http://#{link.url}"
#uri = URI.parse url
headers = fetch_headers
case headers.content_type
when /application/
filename = File.basename(#uri.path)
link.update title: filename
when /html/
response = fetch_page
page = Nokogiri::HTML(response)
link.update title: page_title(page), description: page_description(page)
else
logger.warn "URL #{url} with unknow mime-type: #{headers.content_type}"
end
end
Here is the spec I am running:
it 'follows the redirects using relative URL' do
link = create(:link, url: url)
path = '/welcome.html'
stub_request(:head, url).to_return(status: 302, body: '',
headers: { 'Location' => path })
redirect_url = "#{url}#{path}"
stub_request(:head, redirect_url).to_return(status: 200, body: '',
headers: html_header)
stub_request(:get, redirect_url).to_return(status: 200, body: title_html_raw,
headers: html_header)
UrlScrapperJob.perform_now link.id
link.reload
expect(link.title).to match(/page title/)
end
Here are the result of fetch_headers method:
With the return clause: #<Net::HTTPOK 200 readbody=true>
Without the return clause: #<Net::HTTPFound 302 readbody=true>
The result I would expect would be the HTTPOK 200 because it should follow the redirects until a 200 OK.
The difference is the value, returned from fetch_headers function.
return returns it’s argument as a result of a function call.
Without explicit return, the return value is what http.request_head(request_uri, &block) returns, which is apparently causes the infinite recursion.
You might want to try
http.request_head(request_uri) do |response|
case response
when Net::HTTPSuccess then
response
when Net::HTTPRedirection then
location = response['location']
parsed_location = URI.parse location
#uri = parsed_location.absolute? ? parsed_location : #uri.merge(parsed_location)
fetch_headers(limit - 1)
else
response.value
end
end.tap { |result| puts result } # ⇐ here
to examine what is actual result without explicit return.
I'm trying to define methods to parse through an apache log file and pull ip addresses, URLs, requests per hour, and error codes. I've got everything working outside of methods, but when attempting to put that code into the methods I keep getting the error message "Stack level too deep." Here is the code in question.
class CommonLog
def initialize(logfile)
#logfile = logfile
end
def readfile
#readfile = File.readlines(#logfile).map { |line|
line.split()
}
#readfile = #readfile.to_s.split(" ")
end
def ip_histogram
#ip_count = 0
#readfile.each_index { |index|
if (#readfile[index] =~ /\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/ )
puts #readfile[index]
puts #ip_count += 1
end
}
end
def url_histogram
url_count = 0
cleaned_file.each_index { |index|
if (cleaned_file[index] =~ /\/{1}(([a-z]{4,})|(\~{1}))\:{0}\S+/ )
puts cleaned_file[index]
puts url_count += 1
end
}
end
def requests_per_hour
end
def sorted_list
end
end
my_file = CommonLog.new("test_log")
cleaned_file = my_file.readfile
puts cleaned_file.ip_histogram
It looks like the problem lies on you CommonLog#readfile method:
def readfile
#readfile = File.readlines(#logfile).map { |line|
line.split()
}
#readfile = readfile.to_s.split(" ")
end
Notice that inside the implementation of readfile your calling readfile recursively? When it executes it reads the lines from the file, maps them and assign the result the #readfile; then it calls readfile and the method starts to execute again; this goes on forever, until you stack blows up because of too many recursive method calls.
I assume what you actually meant is:
#readfile = #readfile.to_s.split(" ")
I've been working on this code and for some reason the server can't open the index file, and I can't understand why. I've checked other people's code and there doesn't look like there is a difference.
Here is my code:
require 'socket'
class Server
def initialize (base, request, server_name, session, charset)
#base = base
#request = request
#charset = charset
#server_name = server_name
#session = session
serve()
end
def serve ()
access_log()
getAddress()
#contentType = getContentType()
#session.print "HTTP/1.1 200 OK\r\nServer: #{#server_name}\r\nContent-Type: #{#contentType}; charset=#{charset}\r\n\r\n"
getFile()
#base = nil
#request = nil
#server_name = nil
#contentType
#session.close
puts "Session Ended\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
#session = nil
end
def access_log ()
log = File.open(#base + "data/access_log.txt", "w")
log.puts "#{Time.now.localtime.strftime("%Y/%m/%d %H:%M:%S")} #{#session.peeraddr[2]}:#{#session.peeraddr[1]} #{#request}" unless log == nil
log.close
log = nil
end
def getAddress ()
#src = #request
#src.gsub!(/GET /, '').gsub!(/ HTTP.*/, '')
#src.strip
#src = #base + "root" + #src
#src.gsub!('/', '\\')
end
def getContentType ()
ext = nil
ext = File.extname(#src)
return "text/html" if ext == ".html" or ext == ".htm"
return "text/plain" if ext == ".txt"
return "text/css" if ext == ".css"
return "image/jpeg" if ext == ".jpeg" or ext == ".jpg"
return "image/gif" if ext == ".gif"
return "image/bmp" if ext == ".bmp"
return "text/plain" if ext == ".rb"
return "text/xml" if ext == ".xml"
return "text/xml" if ext == ".xsl"
#return "image/x-icon" if ext == ".ico" or ext == ".icon"
return "text/html"
end
def getFile ()
begin
if !File.exist?(#src)
puts "File: #{#src} could not be found"
if #contentType.include?("image")
file = File.open(#base + "root/server_files/broken_image.png", "r").each do |code|
#session.puts code
end
else
file = File.open(#base + "root/server_files/error404.html", "r").each do |code|
#session.puts code
end
end
else
puts "File #{#src} was opened"
file = File.open(#src, "r").each do |code|
#session.puts code
end
end
ensure
file.close unless file == nil
end
end
end
base = "C:\\Base\\"
server_name = "Some Server"
host = "localhost"
port = 80
charset = "UFT-8"
server = TCPServer.new(host, port)
puts "~ Server hosted on #{host}:#{port} ~\n====================================\n"
loop {
Thread.new(server.accept) do |session|
request = session.gets
puts "#{session.peeraddr[2]}:#{session.peeraddr[1]} #{request}"
Server.new(base, request, server_name, session, charset)
end
}
p = gets.chomp
server.close
There are problems with the code. I don't know if these directly cause the problem, but there's enough code smell that the following untested changes might help.
Don't use CamelCase method names in Ruby. We use snake_case.
require 'socket'
class Server
def initialize(base, request, server_name, session, charset)
#base = base
#request = request
#charset = charset
#server_name = server_name
#session = session
serve()
end
def serve
access_log
get_address()
#content_type = get_content_type()
#session.print "HTTP/1.1 200 OK\r\nServer: #{#server_name}\r\nContent-Type: #{#content_type}; charset=#{charset}\r\n\r\n"
get_file()
#content_type
#session.close
puts "Session Ended\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
#base = #request = #server_name = #session = nil
end
I don't know why you have #content_type on its own line prior to #session.close. It's not a method call, nor does it look like it's going to return anything to the server or the outgoing data-stream.
def access_log
File.open(File.join(#base, "data/access_log.txt"), "w") do |log|
log.puts "#{Time.now.localtime.strftime("%Y/%m/%d %H:%M:%S")} #{#session.peeraddr[2]}:#{#session.peeraddr[1]} #{#request}"
end
end
Ruby's File.open takes a block. When the block exits the file will be closed automatically. File.join is the proper way to build a path. Ruby is aware of the proper path delimiters and will use them automatically, helping to make your code more portable.
def get_address
src = #request.gsub(/GET /, '').gsub(/ HTTP.*/, '').strip
#src = (#base + "root" + src).gsub('/', '\\')
end
You're doing a lot of gsub! for no reason. Chain them, combine the string, do a final gsub and move on.
def get_content_type()
ext = File.extname(#src)
content_type = case ext
when /\.html?/
"text/html"
when ".txt"
"text/plain"
when ".css"
"text/css"
when /\.jpe?g/
"image/jpeg"
when ".gif"
"image/gif"
when ".bmp"
"image/bmp"
when ".rb"
"text/plain"
when /\.x[ms]l/
"text/xml"
else
"text/html"
end
content_type
end
It's confusing to have multiple returns in a sub-routine or method, so this cleans it up. The final content_type will be returned as the result of the method call. Using case/when lets you use multiple tests for each when, reducing line-noise.
def get_file()
begin
if !File.exist?(#src)
puts "File: #{#src} could not be found"
if #content_type["image"]
File.open(File.join(#base, "root", "server_files", "broken_image.png"), "rb") do |image|
#session.puts image.read
end
else
File.foreach(File.join(#base, "root", "server_files", "error404.html")) do |li|
#session.puts li
end
end
else
puts "File #{#src} was opened"
File.foreach(#src) do |li|
#session.puts li
end
end
rescue Exception => e
puts e.message
end
end
When you want to read a binary file, such as "broken_image.png", ALWAYS open it in binary mode: "rb". Otherwise, Ruby will assume it's OK to translate line-endings to the native format for the OS, which would corrupt the data. Also, again, use the block-forms of the File methods so they automatically close. And, use File.join for portability. When you're reading a text file, use File.foreach to read it line-by-line, unless you KNOW it will fit into the available memory; "Slurping" your files is bad form because it isn't scalable.
end
base = "C:/Base/"
server_name = "Some Server"
host = "localhost"
port = 80
charset = "UFT-8"
server = TCPServer.new(host, port)
puts "~ Server hosted on #{host}:#{port} ~\n====================================\n"
loop {
Thread.new(server.accept) do |session|
request = session.gets
puts "#{session.peeraddr[2]}:#{session.peeraddr[1]} #{request}"
Server.new(base, request, server_name, session, charset)
end
}
p = gets.chomp
server.close
The code changes are available at: https://gist.github.com/anonymous/6515451