I've created a gist which shows exactly what happens.
https://gist.github.com/4418148
I've tested a version which used ruby's 'net/http' library and 'socksify/http' and it worked perfect but if the EventMachine version returns an unexpected result.
The response in Tor Browser is correct but using EventMachine is not!
It return a response but it's not the same as returned response when you send the request via browser, net/http with or without proxy.
For convenience, I will also paste it here.
require 'em-http-request'
DEL = '-'*40
#results = 0
def run_with_proxy
connection_opts = {:proxy => {:host => '127.0.0.1', :port => 9050, :type => :socks5}}
conn = EM::HttpRequest.new("http://www.apolista.de/tegernsee/kloster-apotheke", connection_opts)
http = conn.get
http.callback {
if http.response.include? "Oops"
puts "#{DEL}failed with proxy#{DEL}", http.response
else
puts "#{DEL}success with proxy#{DEL}", http.response
end
#results+=1
EM.stop_event_loop if #results == 2
}
end
def run_without_proxy
conn = EM::HttpRequest.new("http://www.apolista.de/tegernsee/kloster-apotheke")
http = conn.get
http.callback {
if http.response.include? "Oops"
puts "#{DEL}failed without proxy#{DEL}", http.response
else
puts "#{DEL}success without proxy#{DEL}", http.response
end
#results+=1
EM.stop_event_loop if #results == 2
}
end
EM.run do
run_with_proxy
run_without_proxy
end
Appreciate any clarification.
Related
What can cause while making an HTTP connection to return EBADF (Bad file descriptor).
Here is my following code wherein the HTTP connection is made. Although the error is very less now(happening very less) but before I put those error on rescue I need to understand what is the reason for the EBADF
def make_http_request(url, headers={})
uri = URI(url)
Net::HTTP.start(uri.host, uri.port) do |http|
req = Net::HTTP::Get.new(uri, headers)
resp = http.request(req)
if resp.code.to_i != 200
logger.error "Retrieve #{resp.code} with #{url} and #{headers}"
return false
end
return resp.body
end
rescue SocketError, Net::ReadTimeout, Errno::ECONNREFUSED => e
logger.error "make_http_request #{url} with #{headers} resulted in #{e.message} \n #{e.backtrace}"
return false
end
I have a feeling that connect syscall is receiving an FD which ain't valid at that given point in time. But still unable to understand how can that happens.
If it helps the code is used in an application that operates with multiple threads.
In a nutshell, the definition of the above method looks like this...
module Eval
def make_http_request(url, headers={})
...
...
..
end
def request_local_endpoint(url, headers)
response = make_http_request(url, headers)
response && response.fetch('bravo',nil)
end
def request_external_endpoint(url, headers)
response = make_http_request(url, headers)
response && response.fetch('token',nil)
end
end
class RequestBuilder
include Eval
attr_reader :data
def initialize(data)
#data = data
end
def start
token = request_external_endpoint('http://external.com/endpoint1',{'Content-Type'.freeze => 'application/json', 'Authorization' => 'abcdef'})
return unless token
result = request_local_endpoint('http://internal.com/endpoint1',{'Content-Type'.freeze => 'application/json'})
return result
end
end
10.times {
Thread.new { RequestBuilder.new('sample data').start }
}
This is the code I currently have:
#!/usr/bin/ruby
require 'em-websocket'
$cs = []
EM.run do
EM::WebSocket.run(:host => "::", :port => 8085) do |ws|
ws.onopen do |handshake|
$cs << ws
end
ws.onclose do
$cs.delete ws
end
end
end
I would like to watch a file with rb-inotify and send a message to all connected clients ($cs.each {|c| c.send "File changed"}) when a file changes. The problem is, I do not understand EventMachine, and I can't seem to find a good tutorial.
So if anyone could explain to me where to put the rb-inotify-related code, I would really appreciate it.
Of course! As soon as I post the question, I figure it out!
#!/usr/bin/ruby
require 'em-websocket'
$cs = []
module Handler
def file_modified
$cs.each {|c| c.send "File was modified!" }
end
end
EM.run do
EM.watch_file("/tmp/foo", Handler)
EM::WebSocket.run(:host => "::", :port => 8085) do |ws|
ws.onopen do |handshake|
$cs << ws
end
ws.onclose do
$cs.delete ws
end
end
end
Hi this is my very first Ruby program.
I'm trying to write a simple ruby app to make a request to a URL and see if it's available. If it is, it'll print OK and else it'll print false.
This is what I've got so far, can you please assist, do I need to import any libs?
class WebRequest
def initialize(name)
#name = name.capitalize
end
def makeRequest
puts "Hello #{#name}!"
#uri = URI.parse("https://example.com/some/path")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE # read into this
#data = http.get(uri.request_uri)
end
end
req = WebRequest.new("Archie")
req.makeRequest
Here is sample code to do any request:
require 'net/http'
require 'uri'
url = URI.parse('http://www.example.com/index.html')
req = Net::HTTP::Get.new(url.path)
res = Net::HTTP.start(url.host, url.port) do |http|
http.request(req)
end
puts res.body
gem install httparty
then
require 'httparty'
response = HTTParty.get('https://example.com/some/pathm')
puts response.body
Something simpler:
[1] pry(main)> require 'open-uri'
=> true
[2] pry(main)> payload = open('http://www.google.com')
=> #<File:/var/folders/2p/24pztc5s63d69hhx81002bq80000gn/T/open-uri20131217-84948-ttwnho>
[3] pry(main)> payload.inspect
=> "#<Tempfile:/var/folders/2p/24pztc5s63d69hhx81002bq80000gn/T/open-uri20131217-84948-ttwnho>"
[4] pry(main)> payload.read
payload.read would return the response body and you can easy use payload as File object since it is an instance of Tempfile
This is what I've ended up with
require 'net/http'
class WebRequest
def initialize()
#url_addr = 'http://www.google.com/'
end
def makeRequest
puts ""
begin
url = URI.parse(#url_addr)
req = Net::HTTP::Get.new(url.path)
res = Net::HTTP.start(url.host, url.port) {|http|
http.request(req)
}
puts "OK Connected to #{#url_addr} with status code #{res.code}"
rescue
puts "Failed to connect to #{#url_addr}"
end
end
end
req = WebRequest.new()
req.makeRequest
I'd like to stream data from an HTTP GET request to an HTTP POST or PUT request. I'd prefer to use Ruby and have already made an attempt using EventMachine and EM-HTTP-Request.
Here's my attempt, to be called using:
HttpToS3Stream.new(src_url, dest_bucket, dest_key, aws_access_key_id, aws_secret_access_key)
http_to_s3_stream.rb
require 'em-http-request'
class HttpToS3Stream
def initialize(http_url, s3_bucket, s3_key, s3_access_key_id, s3_secret_access_key)
#http_url = http_url
#s3_bucket = s3_bucket
#s3_key = s3_key
#s3_access_key_id = s3_access_key_id
#s3_secret_access_key = s3_secret_access_key
go
end
private
def go
EM.run {
# initialize get stream, without listener does not start request
#get_stream = HttpGetStream.new(#http_url)
# initialize put stream, send content length, request starts
#put_stream = S3PutStream.new(#s3_bucket, #s3_key, #s3_access_key_id, #s3_secret_access_key, #get_stream.content_length)
# set listener on get stream, starts request, pipes data to put stream
#get_stream.listener = #put_stream
}
end
end
http_get_stream.rb
require 'httparty'
require 'em-http-request'
class HttpGetStream
def initialize(http_url, listener = nil)
#http_url = http_url
self.listener = listener
end
def listener=(listener)
#listener = listener
listen unless #listener.nil?
end
def content_length
response = HTTParty.head(#http_url)
response['Content-length']
end
private
def listen
http = EventMachine::HttpRequest.new(#http_url).get
http.stream do |chunk|
#listener.send_data chunk
end
http.callback do |chunk|
EventMachine.stop
end
end
end
s3_put_stream.rb
require 'em-http-request'
class S3PutStream
def initialize(s3_bucket, s3_key, s3_access_key_id, s3_secret_access_key, content_length = nil)
#s3_bucket = s3_bucket
#s3_key = s3_key
#s3_access_key_id = s3_access_key_id
#s3_secret_access_key = s3_secret_access_key
#content_length = content_length
#bytes_sent = 0
listen
end
def send_data(data)
#bytes_sent += data.length
#http.on_body_data data
end
private
def listen
raise 'ContentLengthRequired' if #content_length.nil?
#http = EventMachine::HttpRequest.new(put_url).put(
:head => {
'Content-Length' => #content_length,
'Date' => Time.now.getutc,
'Authorization' => auth_key
}
)
#http.errback { |error| puts "error: #{error}" }
end
def put_url
"http://#{#s3_bucket}.s3.amazonaws.com/#{#s3_key}"
end
def auth_key
"#{#s3_access_key_id}:#{#s3_secret_access_key}"
end
end
HttpToS3Stream.new(src_url, dest_bucket, dest_key, aws_access_key_id, aws_secret_access_key)
It seems to be working but always stops at 33468 bytes. Not sure what that's about. Now, by passing chunks directly to #listener.send_data, it is processing the entire GET body. However, the upload is not occurring successfully.
How can I get this to work? And is there a name for what I'm trying to do? I'm having trouble searching for more information.
Any help is appreciated.
I'm trying to consume the twitter streaming API with Sinatra and give users real-time updates when they search for a keyword.
require 'sinatra'
require 'eventmachine'
require 'em-http'
require 'json'
STREAMING_URL = 'https://stream.twitter.com/1/statuses/sample.json'
get '/' do
stream(:keep_open) do |out|
http = EM::HttpRequest.new(STREAMING_URL).get :head => { 'Authorization' => [ 'USERNAME', 'PASS' ] }
buffer = ""
http.stream do |chunk|
puts "still chugging"
buffer += chunk
while line = buffer.slice!(/.+\r?\n/)
tweet = JSON.parse(line)
unless tweet.length == 0 or tweet['user'].nil?
out << "<p><b>#{tweet['user']['screen_name']}</b>: #{tweet['text']}</p>"
end
end
end
end
end
I want the processing of the em-http-request stream to stop if the user closes the connection. Does anyone know how to do this?
Eric's answer was close, but what it does is closing the response body (not the client connection, btw) once your twitter stream closes, which normally never happens. This should work:
require 'sinatra/streaming' # gem install sinatra-contrib
# ...
get '/' do
stream(:keep_open) do |out|
# ...
out.callback { http.conn.close_connection }
out.errback { http.conn.close_connection }
end
end
I'm not quite familiar with the Sinatra stream API yet, but did you try this?
http.callback { out.close }