I have a proxy set up and running completely fine on my local host. I can connect to the proxy completely fine running this code.
Net::HTTP::Proxy('http://localhost', 1234).start #do whatever I want after this point
I can connect to it through a browser completely fine, however when I go to run it on mechanize it completely fails. Here's the code.
require 'mechanize'
agent=Mechanize.new
agent.set_proxy('localhost', 1234)
agent.get('http://google.com') #or any website for that matter
Here's the error I get back
Net::HTTP::Persistent::Error: too many connection resets (due to end of file reached - EOFError) after 0 requests on 22249020, last used 1376802493.5352573 seconds ago
I've read that the versions after 1.0.0 have difficulties connecting to http proxies, but I need to and I'm currently running version 2.7.2. Is there anything I can do to connect to a proxy.
Some proxies can't handle persistent connections (keep-alive). You need to make sure to use a HTTP/1.1-aware proxy.
What version of Ruby are you running? I have this issue with Ruby 2.1.5 and Mechanize, but not with Ruby 1.9.3.
Presumably you've resolved your problem, but for anyone else who might end up here, downgrading to Ruby 1.9.3 might be a workaround.
Related
I just setup a custom domain for an AWS API Gateway and set up CNAME entries in Google Domains to redirect to my API Gateway. After maybe 30 minutes of waiting I was able to use Chrome to do a simple GET request to my custom domain that properly forwarded to my API Gateway. I tested in Firefox and it worked fine too.
About 3-4 hours later I came back and tried making the same call using Python requests and it worked the first 3 times then failed.
SSLError: HTTPSConnectionPool(host='ids.references.app', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError("hostname '<my_custom_domain>' doesn't match '*.execute-api.us-east-2.amazonaws.com'")))
At first I thought this was a requests problem, but then I opened up Firefox and it didn't work as well. I tried Edge and the call worked. Then I went back to Python and it worked for a bit, then stopped working. I went back to Firefox and it no longer worked. Then I tried Edge and it no longer worked. Sprinkled in there I've tried Chrome and it has worked every time since it started working. (this order of events is from memory and may be slightly off).
Is this a known issue with updating DNS entries that you get some randomness when things first start until the DNS changes have fully propagated. How would I go about even tracking where the error is occurring? I think that's the most frustrating thing about this, it all seems like magic and there's no obvious point where you get something like server 1.2.3.4 says that cert_1 doesn't go with cert_2 and then later you see something like server 4.5.6.7 says cert_2 is all good (so it works). Would I need to install curl for Windows (Is is possible to make a cURL request and get the route that is taken (similar to traceroute)). Would this even matter though? What if curl was like Chrome, it always worked? Does requests have this functionality (bonus points if someone can show a requests solution)? What about Firefox or Chrome? Or could I use something like wireshark (yikes) that could somehow observe the whole system?
I'm using requests 2.25.1 and Python 3.8.5 on Windows 10 and I believe the latest versions of Edge and Firefox.
Required Sinatra, got '/', did 'hello, world.', requested application root in browser and received nothing.
Literally the barest of bones Sinatra web application will not run as intended on my Mac, after previously working just fine. Page requests do not show up in the console.
I feel this may be due to force quitting an instance of a running application but after searching for a solution on the web I am at a loss and do not know how to rectify the issue.
edit:
require 'sinatra'
get '/' do
'Hello, World.'
end
Problem does persist after a restart
Definitely the correct URL (exactly as I've been working with all day). localhost:4567 as stated in the output when the application is run:
== Sinatra (v1.4.7) has taken the stage on 4567 for development with backup from Thin
Thin web server (v1.6.4 codename Gob Bluth)
Maximum connections set to 1024
Listening on localhost:4567, CTRL+C to stop
edit 2: scorched earth strategy; full reinstall with rvm fixed this issue.
I have a test that uses OAuth2 gem to make make an HTTP request:
token = client.auth_code.get_token( code, redirect_uri:redirect_uri )
Capybara does not seem to be running the Rails app as a server:
Faraday::ConnectionFailed:
Connection refused - connect(2) for "localhost" port 3000
Here is the config:
Capybara.app_host = "http://localhost:3000"
Capybara.run_server = true
Capybara.server_port = 3000
Capybara.register_driver :rack_test do |app|
Capybara::RackTest::Driver.new app,
follow_redirects:false
end
Various docs seem to suggest that the above config will run the app as a server, but it does not seem to be the case.
How to run Capybara against a server so that the server responds to HTTP requests?
EDIT: After digging in the Capybara code, it seems that different drivers have different servers. Working with Poltergeist. So perhaps Poltergeist doesn't support running a server.
As usual, a small detail was overlooked.
The spec must be tagged as :js. Otherwise it uses Rack::Test and the server is not started.
My apologies in advance if this is a noobish doubt: I want to use a proxy in my Ruby code to fetch a few web pages. And I want to be sneaky about it! So I am using Tor.
I have Tor running, and I am able to use Net::HTTP.get(uri) as usual. But I can't figure out how to use Net::HTTP::Proxy to fetch the uri. I also can't figure out how using Tor will help make my fetches anonymous.
Any help is greatly appreciated. Please don't just add a link to the ruby-doc page for Net::HTTP::Proxy. If I had understood that, I would not be asking this here :-)
Another easy way to do this is using SOCKSify, but in this case, I receive the following error:
/usr/lib/ruby/gems/1.9.2-p290/gems/socksify-1.5.0/lib/socksify.rb:189:in 'socks_authenticate': SOCKS version not supported (SOCKSError)
I have never done any network programming before. Any guidance about this will also be very helpful. Thanks :-)
You are using HTTP proxy class, so you must provide IP of HTTP proxy. Tor Browser has not HTTP proxy bundled.
So you can either install some proxy software e.g. Privoxy and configure it to use Tor's SOCKS:
In config.txt
forward-socks4a / 127.0.0.1:9050 .
then use Privoxy's default listen-address in your script:
proxy = Net::HTTP::Proxy('127.0.0.1',8118)
or use SOCKSify.
According to docs:
require 'socksify/http'
uri = URI.parse('http://rubyforge.org/')
Net::HTTP.SOCKSProxy('127.0.0.1', 9050).start(uri.host, uri.port) do |http|
http.get(uri.path)
end
No need for additional software..
Third solution is to use SOCKSify as follows:
$ socksify_ruby localhost 9050 script.rb
which redirect all TCP connections of a Ruby script, which means you don't need to use any Proxy code at all.
For clarification you have to understand that 127.0.0.1:9050 is Tor's SOCKS address and 127.0.0.1:8118 is address of Privoxy.
how can I configure Ruby web client to request web pages using Tor ?
I had to use this Gem http://socksify.rubyforge.org/ then I was able to do something like this
TCPSocket::socks_server = "127.0.0.1"
TCPSocket::socks_port = 9050
reply = Net::HTTP.get URI.parse("www.google.com")
Obviously with the Tor proxy running.
I think it's like specifying proxy server for your HTTP connection. I don't know how it works in Ruby. But it will not be different from configuring browsers. Just set proxy server setting to 127.0.0.1:8118.
Created a Gem, maybe can help others: https://github.com/brunogh/tor_requests
You just need to work with the Proxy class. As Ivan says above, get Tor running then point Net::HTTP.Proxy at the correct localhost address and you're golden.