how can I configure Ruby web client to request web pages using Tor ?
I had to use this Gem http://socksify.rubyforge.org/ then I was able to do something like this
TCPSocket::socks_server = "127.0.0.1"
TCPSocket::socks_port = 9050
reply = Net::HTTP.get URI.parse("www.google.com")
Obviously with the Tor proxy running.
I think it's like specifying proxy server for your HTTP connection. I don't know how it works in Ruby. But it will not be different from configuring browsers. Just set proxy server setting to 127.0.0.1:8118.
Created a Gem, maybe can help others: https://github.com/brunogh/tor_requests
You just need to work with the Proxy class. As Ivan says above, get Tor running then point Net::HTTP.Proxy at the correct localhost address and you're golden.
Related
I'm trying to get some protocols work through my company's firewall. Until now I have been succesfull in masking either http or https data by setting a http proxy on localhost and one on a remote server I own. The communication is done via $_POSTed and received modified .bmp files that contain a header and the encripted serialised request array.
This works fine, but there are a few drawbacks that make me think I might have taken a wrong approach.
Firstly I do not use apache's mod-proxy. instead I just created a local subdomain (proxy.localhost) and use that in browser's proxy settings. the subdomain's index.php does all the work. This creates some problems. I cannot use http and https simultaneously or the server will complain of using either "http on a https enabled port" or "incoresc ssl response length".
The second problem is, well, other protocols. I could make use of some ftp, sftp, remote deskoptop, ssh, nust name another... I need it
there are 2 solutions I can think of: First is if I run a php script in CLI so that it listens on a predefined port and handles the requests differently, or some sort of ssh tunnel. Problem is I haven't had any success with freeSSHd and putty because of my ignorance.
Thanks in advance for any advice.
I used the free version of bitvise SSH Client and server and it seems to work just fine.
My apologies in advance if this is a noobish doubt: I want to use a proxy in my Ruby code to fetch a few web pages. And I want to be sneaky about it! So I am using Tor.
I have Tor running, and I am able to use Net::HTTP.get(uri) as usual. But I can't figure out how to use Net::HTTP::Proxy to fetch the uri. I also can't figure out how using Tor will help make my fetches anonymous.
Any help is greatly appreciated. Please don't just add a link to the ruby-doc page for Net::HTTP::Proxy. If I had understood that, I would not be asking this here :-)
Another easy way to do this is using SOCKSify, but in this case, I receive the following error:
/usr/lib/ruby/gems/1.9.2-p290/gems/socksify-1.5.0/lib/socksify.rb:189:in 'socks_authenticate': SOCKS version not supported (SOCKSError)
I have never done any network programming before. Any guidance about this will also be very helpful. Thanks :-)
You are using HTTP proxy class, so you must provide IP of HTTP proxy. Tor Browser has not HTTP proxy bundled.
So you can either install some proxy software e.g. Privoxy and configure it to use Tor's SOCKS:
In config.txt
forward-socks4a / 127.0.0.1:9050 .
then use Privoxy's default listen-address in your script:
proxy = Net::HTTP::Proxy('127.0.0.1',8118)
or use SOCKSify.
According to docs:
require 'socksify/http'
uri = URI.parse('http://rubyforge.org/')
Net::HTTP.SOCKSProxy('127.0.0.1', 9050).start(uri.host, uri.port) do |http|
http.get(uri.path)
end
No need for additional software..
Third solution is to use SOCKSify as follows:
$ socksify_ruby localhost 9050 script.rb
which redirect all TCP connections of a Ruby script, which means you don't need to use any Proxy code at all.
For clarification you have to understand that 127.0.0.1:9050 is Tor's SOCKS address and 127.0.0.1:8118 is address of Privoxy.
I'm using the bert-rpc gem in Ruby 1.9.3 to make calls to an Ernie server that is not on my local network:
BERTRPC::Service.new("www.someurl.com", 9998)
Now I want that connection to be secured via SSH. I was thinking about using a local unix socket, but that means I need to open up the bert-rpc gem code and replace the TCPSocket calls to UnixSocket calls. Isn't there another way?
Isn't it possible to just forward a localhost port 9998 to www.someurl.com 9998, so I can do this:
BERTRPC::Service.new("localhost", 9998)
I've tried the local-to-remote net/ssh examples, but I can't really wrap my head around them, and I can't find any good documentation. Would anybody be so kind to show me an example of how to do the port forwarding?
Thanks
The solution to this was pretty simple. Create a SSH Gateway:
gateway = Net::SSH::Gateway.new('www.someurl.com', 'myuser', :password => "somepass")
gateway.open('www.someurl.com', 9998, 9998)
This routes localhost:9998 to www.someurl.com:9998. This WILL NOT work on Heroku, as Heroku doesn't allow binding on other ports than the assigned $PORT.
Does anyone have an idea on how to make this work on Heroku with a Unix Socket in /tmp?
HI. in node.js, if it is http request, I can get the remoteAddress at req.connection.remoteAddress,
so, how to get it if https request? I find there is req.socket.remoteAddress but I'm not sure. Please advice. thanks.
It appears something is strange/broken indeed.
As of node 0.4.7, it seems http has remoteAddress available on:
req.connection.remoteAddress
req.socket.remoteAddress
on https, both of these are undefined, but
req.connection.socket.remoteAddress
does work.
That one isn't available on http though, so you need to check carefully.
I cannot imagine this behavior is intentional.
Since googling "express js ip" directly points to here, this is somehow relevant.
Express 3.0.0 alpha now offers a new way of retrieving IP adresses for client requests.
Simply use req.ip. If you're doing some proxy jiggery-pokery you might be interested in app.set("trust proxy", true); and req.ips.
I recommend you to read the whole discussion in the Express Google Group.
var ip = req.headers['x-forwarded-for'] ||
req.connection.remoteAddress ||
req.socket.remoteAddress ||
req.connection.socket.remoteAddress;
Note that sometimes you can get more than one ip address in req.headers['x-forwarded-for'], specially when working with mobile phones accessing your server (wifi and carrier data).
As well req.headers['x-forwarded-for'] is easily manipulated so you need a properly configured proxy server.
Is better to check req.connection.remoteAddress against a list of known proxy servers before to go with req.headers['x-forwarded-for'].
I would like to use the net/imap library in ruby behind a authenticated proxy, I was starting to dig in and I'm wondering if there is a way to do this already or if I need to make my own version of the net/imap library that supports a proxy?
It is possible to tunnel any socket connection through a HTTPS proxy server.
To do this:
open a socket to your proxy server
send "CONNECT hostname : portnumber HTTP/1.0\n\r\n\r\n"
read from the socket until you see the end of the HTTP headers (2 blank lines)
your socket is now connected
Here is a ruby example of such a tunnel.
Reasons this will fail:
most network admins will only allow CONNECT to port 443
proxy server has proxy authentication
The easiest way to hack libraries that don't support proxy information is to replace Net::HTTP with an instance of Net::HTTP::Proxy:
# somewhere before you load net/imap
proxy = Net::HTTP::Proxy(address, host)
Net.class_eval do
remove_const :HTTP
HTTP = proxy
end