I have an application that initiates multiple HTTP connections and I would like to add a proxy to all connections.
The application is using net/HTTP, TCP sockets and open-uri so ideally I would like to be able to patch all connections initiated from those libraries instead of adding it manually to each and every location in the code that initiates a connection.
Is there a way to accomplish that (on Ruby 1.9.2)?
Open URI uses the HTTP_PROXY environment variable
Here is an article on how to use it on both windows and unix variants.
http://kaamka.blogspot.com/2009/06/httpproxy-environment-variable.html
you can also set it directly in ruby using the ENV hash
ENV['HTTP_PROXY'] = 'http://username:password#hostname:port'
the net/http documentation says not to rely on the environment and set it each time
require 'net/http'
require 'uri'
proxy_host = 'your.proxy.host'
proxy_port = 8080
uri = URI.parse(ENV['http_proxy'])
proxy_user, proxy_pass = uri.userinfo.split(/:/) if uri.userinfo
Net::HTTP::Proxy(proxy_host, proxy_port,
proxy_user, proxy_pass).start('www.example.com') {|http|
# always connect to your.proxy.addr:8080 using specified username and password
:
}
from http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/classes/Net/HTTP.html
Yes and mechanize does too (this is for the 1.0.0 verison)
require 'mechanize'
url = 'http://www.example.com'
agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
agent.set_proxy('127.0.0.1', '3128')
#page = agent.get(:url => url)
Related
According to MS documentation https://learn.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service
Azure provide instance metadata service at "http://169.254.169.254/metadata/instance?api-version=2017-08-01" .
But I failed to get the metadata on one of my VMs with ruby. While, I can get the metadata with PowerShell.
OS Environment: Windows Server 2012R2.
Ruby version: ruby 2.5.3p105 (2018-10-18 revision 65156) [x64-mingw32]
PowerShell version: 4.0
PowerShell code:
Invoke-WebRequest -Headers #{"Metadata"="true"} -URI "http://169.254.169.254/metadata/instance?api-version=2017-08-01"
response from powershell code:
StatusCode : 200
StatusDescription : OK
...
Ruby Code:
require 'net/http'
AZURE_METADATA_ADDR = "169.254.169.254".freeze unless defined?(AZURE_METADATA_ADDR)
AZURE_METADATA_URL = "/metadata/instance?api-version=2017-08-01".freeze unless defined?(AZURE_METADATA_URL)
def http_get(uri)
conn = Net::HTTP.start(AZURE_METADATA_ADDR)
conn.read_timeout = 6
conn.get(uri, { "Metadata" => "true" })
end
puts "Fetching metadata from host #{AZURE_METADATA_ADDR} at #{AZURE_METADATA_URL}"
response = http_get(AZURE_METADATA_URL)
puts response
Response from Ruby code:
Fetching metadata from host 169.254.169.254 at /metadata/instance?api-version=2017-08-01
#<Net::HTTPNotFound:0x0000000002372128>
Anyone can advice how to continue troubleshooting this issue? Is the problem inside Ruby?
PS. this is an embedded Ruby provided by chef client
So, this worked for me:
require 'net/http'
require 'json'
http = Net::HTTP.new('169.254.169.254', '80')
request = Net::HTTP::Get.new('/metadata/instance?api-version=2020-06-01')
request['Metadata'] = 'true'
response = http.request(request)
JSON.parse(response.body)
... on Ruby 2.5.5. The only obvious difference I see is the explicit port setting to '80'.
It turned out to be an environment variable "http_proxy" is preventing ruby from accessing the endpoint. The resolution is to set "no_proxy" environment variable and set '169.254.169.254' in the list of "no_proxy".
Ruby honors "http_proxy" while other languages like powershell and java don't honor it.
How do you get the IP address of a domain and the server information of that domain, to tell if the server info is Apache, nginix, etc..?
I have a pretty good idea on how to get the IP, but for some reason it won't output the correct information:
require 'socket'
IPSocket::getaddress('http://www.prairiegraphicdesign.com/phpStuff/week2Shopping/cartAdd.php?action=add&id=4')
#<= irb(main):001:0> require 'socket'
# => true
# irb(main):002:0>IPSocket::getaddress('http://www.prairiegraphicdesign.com/phpStuff/week2Shopping/cartAdd.php?action=add&id=4')
# SocketError: getaddrinfo: No such host is known.
# from (irb):2:in `getaddress'
# from (irb):2
# from C:/Ruby22/bin/irb:11:in `<main>'
Is this because I need to strip the URL down to the original URL?
http://www.prairiegraphicdesign.com
That's a URL, and getaddress has no idea what to do with those. All it can handle is fully-qualified domain names (FQDN) like "example.com".
Instead you need to break it out:
require 'uri'
uri = URI.parse('http://www.prairiegraphicdesign.com/phpStuff/week2Shopping/cartAdd.php?action=add&id=4')
uri.host
#=> "www.prairiegraphicdesign.com"
You can resolve that to an address if you like.
As for what server software it's running, you need to make an HTTP request to the server and extract the right header from the response:
require 'net/http'
res = Net::HTTP.get_response(uri)
res['server']
#=> "Microsoft-IIS/6.0"
I am trying to run a ruby script from my computer and I would like to have the script use a proxy IP address / server that I have setup, as opposed to the default IP address associated with my local machine.
I have been able to get my web browsers to use this proxy IP address by making changes inside network settings. But When I run the ruby script from textmate, it doesn't seem to use the proxy IP address I have put into my network settings. Instead it defaults back to the base ip address of my local machine.
Is there anything I can do in textmate or in the script itself to specify a proxy IP address it should route through?
My script looks like the following:
require "open-uri"
url = "some-url"
pattern = "<img"
page = open(url).read
tags = page.scan(pattern)
puts "The site #{url} has #{tags.length} img tags"
Thanks for your help!
Use :proxy option to let open-uri know your proxy server:
page = open(url, :proxy => "http://#{proxy_host}:#{proxy_port}/").read
You can also set environment variable http_proxy instead. If you do so, give :proxy => true for option.
page = open(url, :proxy => true).read
[ADDED]
If you want to use proxy with basic authentication, you can give :proxy_http_basic_authentication option instead of :proxy as follows:
:proxy_http_basic_authentication => ["http://#{proxy_host}:#{proxy_port}/", login, password]
Note that :proxy_http_basic_authentication can be used in ruby 1.9.2 or later.
I recommend using mechanize, and css instead of regex:
require "mechanize"
url = "http://www.google.com/"
#agent = Mechanize.new{|a| a.set_proxy 'localhost', 8888}
page = #agent.get url
tags = page.search('img')
puts "The site #{url} has #{tags.length} img tags"
I am looking for a ruby HTTP client gem that supports NTLM proxy authentication "natively" - not through cntlm or similar local proxies.
Any hints appreciated.
A little digging unearthed Typhoeus:
require 'typhoeus'
e=Typhoeus::Easy.new
e.url="http://www.google.com/"
e.proxy = {:server => "1.2.3.4:80"}
e.proxy_auth={:username => "user", :password => 'password'}
e.perform
Typhoeus seems to have been repurposed. The libcurl wrapper is now Ethon (https://github.com/typhoeus/ethon).
I've successfully authenticated with an NTLM proxy using Curb (https://github.com/taf2/curb), another libcurl wrapper:
require 'spec_helper'
require 'curl'
describe Curl do
it 'should connect via an ISA proxy' do
c = Curl::Easy.new('http://example.com/') do |curl|
curl.proxy_url = 'http://username:password#localhost:8080'
curl.proxy_auth_types = Curl::CURLAUTH_NTLM
end
c.perform
headers = c.header_str.split("\r\n")
#puts headers.inspect
headers.should include 'X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.19'
end
end
Change your settings and assertion as required.
You can do ntlm with Typhoeus and Ethon - depending how many features you need. Typhoeus has more than Ethon, but Ethon is more powerful as it is more low level.
require 'ethon'
easy = Ethon::Easy.new(
url: "http://www.google.com/",
proxy: "1.2.3.4:80",
proxyuserpwd: "user:password",
proxyauth: "ntlm"
)
easy.perform
Typhoeus accepts the same options:
require 'typhoeus'
request = Typhoeus::Request.new(
"http://www.google.com/",
proxy: "1.2.3.4:80",
proxyuserpwd: "user:password",
proxyauth: "ntlm"
)
request.run
I wrote both code examples without testing them b/c I lack a proxy and with the latest Typhoeus/Ethon versions (which you don't have already according to your example).
I am trying to access an HTTPS web service that uses SSL cert authentication using Ruby EventMachine but I am not getting it to work.
I have written the following simple code block to test it end-to-end:
require 'rubygems'
require 'em-http'
EventMachine.run do
url = 'https://foobar.com/'
ssl_opts = {:private_key_file => '/tmp/private.key',
:cert_chain_file => '/tmp/ca.pem',
:verify_peer => false}
http = EventMachine::HttpRequest.new(url).get :ssl => ssl_opts
http.callback do
p http.response_header.status
p http.response_header
p http.response
EventMachine.stop
end
http.errback do
EventMachine.stop
fail "Request failed"
end
end
Running the above outputs <SSL_incomp> followed by the raised RuntimeError message. I have tried running with :verify_peer set to both true and false and it gives me the same error. Running EventMachine::HttpRequest#get without the :ssl option does the same.
I have also tried sending the request to GMail (https://mail.google.com) without the :ssl option (i.e. plain HTTPS without cert) and that works, outputting status code 200, the headers and the body.
I have tried doing the same request to the web service with curl and that works:
curl --silent --cert /tmp/private.key --cacert /tmp/ca.pem https://foobar.com/
I am thinking that I am either using the em-http-request gem or EventMachine incorrectly or that the SSL files are in a format that works with curl but not EventMachine.
I someone knows how to solve the example above or provide a similar example using EventMachine directly would be much appreciated!
The file passed to curl's --cert contains both the cert and the key (unless you pass in a --key separately). Just use /tmp/private.key as the argument to both :private_key_file and :cert_chain_file
See http://github.com/eventmachine/eventmachine/issues/#issue/115 for more details about the issue and a patch that exposes the underlying error (instead of just printing out SSL_incomp).