I am trying to run a ruby script from my computer and I would like to have the script use a proxy IP address / server that I have setup, as opposed to the default IP address associated with my local machine.
I have been able to get my web browsers to use this proxy IP address by making changes inside network settings. But When I run the ruby script from textmate, it doesn't seem to use the proxy IP address I have put into my network settings. Instead it defaults back to the base ip address of my local machine.
Is there anything I can do in textmate or in the script itself to specify a proxy IP address it should route through?
My script looks like the following:
require "open-uri"
url = "some-url"
pattern = "<img"
page = open(url).read
tags = page.scan(pattern)
puts "The site #{url} has #{tags.length} img tags"
Thanks for your help!
Use :proxy option to let open-uri know your proxy server:
page = open(url, :proxy => "http://#{proxy_host}:#{proxy_port}/").read
You can also set environment variable http_proxy instead. If you do so, give :proxy => true for option.
page = open(url, :proxy => true).read
[ADDED]
If you want to use proxy with basic authentication, you can give :proxy_http_basic_authentication option instead of :proxy as follows:
:proxy_http_basic_authentication => ["http://#{proxy_host}:#{proxy_port}/", login, password]
Note that :proxy_http_basic_authentication can be used in ruby 1.9.2 or later.
I recommend using mechanize, and css instead of regex:
require "mechanize"
url = "http://www.google.com/"
#agent = Mechanize.new{|a| a.set_proxy 'localhost', 8888}
page = #agent.get url
tags = page.search('img')
puts "The site #{url} has #{tags.length} img tags"
Related
I'm new to Ruby and to the BC API so there may be something obvious that I'm missing. When I'm running the following code with the proper details replaced with my store's legacy API credentials:
require 'bigcommerce'
Bigcommerce.configure do |config|
config.auth = 'legacy'
# You will get this url when registering for an API key
config.url = ENV['BC_API_ENDPOINT_LEGACY']
config.username = ENV['BC_USERNAME']
config.api_key = ENV['BC_API_KEY']
end
puts Bigcommerce::System.time
I get the following error:
.../lib/ruby/2.3.0/net/http.rb:882:in `rescue in block in connect': Failed to open TCP connection to :80 (Connection refused - connect(2) for nil port 80) (Faraday::ConnectionFailed)
I appreciate any pointers.
tldr: you're environment variables aren't actually being set :p
Sounds like your environment variables aren't actually being set.
I say this because the error Failed to open TCP connection to :80, does not list any URL, but only the port.
To confirm, within your code, can you simply run:
puts ENV['BC_API_ENDPOINT_LEGACY'], and see if anything is printed?
An alternative is to just hardcode your credentials, and not rely on environment variables.
Bigcommerce.configure do |config|
config.auth = 'legacy'
config.url = 'https://url.com'
config.username = 'api_username'
config.api_key = 'api_key'
end
FYI, the way you set an environment variable would generally be (from the command line):
export BC_API_ENDPOINT_LEGACY="https://url.com"
And then confirm it is set by echo $BC_API_ENDPOINT_LEGACY
You can view this link for other options on setting environment variables for your RoR app.
Currently, I am sending GET requests to S3 using aws-sdk ruby as follow:
#!/usr/bin/ruby
#
require 'aws-sdk'
s3 = Aws::S3::Resource.new(region: 'test', endpoint:'http://10.0.23.45:8081')
my_bucket = s3.bucket('test.bucket-name')
my_bucket.objects.limit(50).each do |obj|
puts " #{obj.key} => #{obj.etag}"
end
But the request is trying to hit this url endpoint(virtual hosting):
http://test.bucket-name.10.0.23.45:8081
I would like to use path style addressing instead. This is what I want the request url endpoint to look like:
http://10.0.23.45:8081/test.bucket-name/
Any idea how to set path style addressing instead of virtual hosting address? Thanks.
I found the answer for my own question after looking at the source code of ruby aws-sdk Source Code
Aws.config[:s3] = { force_path_style: true }
Adding the above line forced to use path style addressing.
You need to set option :virtual_host to true according to documentation.
So in your case something like this should work:
s3.bucket('10.0.23.45:8081').object('test.bucket-name').public_url(virtual_host: true)
#=> "http://10.0.23.45:8081/test.bucket-name/"
How do you get the IP address of a domain and the server information of that domain, to tell if the server info is Apache, nginix, etc..?
I have a pretty good idea on how to get the IP, but for some reason it won't output the correct information:
require 'socket'
IPSocket::getaddress('http://www.prairiegraphicdesign.com/phpStuff/week2Shopping/cartAdd.php?action=add&id=4')
#<= irb(main):001:0> require 'socket'
# => true
# irb(main):002:0>IPSocket::getaddress('http://www.prairiegraphicdesign.com/phpStuff/week2Shopping/cartAdd.php?action=add&id=4')
# SocketError: getaddrinfo: No such host is known.
# from (irb):2:in `getaddress'
# from (irb):2
# from C:/Ruby22/bin/irb:11:in `<main>'
Is this because I need to strip the URL down to the original URL?
http://www.prairiegraphicdesign.com
That's a URL, and getaddress has no idea what to do with those. All it can handle is fully-qualified domain names (FQDN) like "example.com".
Instead you need to break it out:
require 'uri'
uri = URI.parse('http://www.prairiegraphicdesign.com/phpStuff/week2Shopping/cartAdd.php?action=add&id=4')
uri.host
#=> "www.prairiegraphicdesign.com"
You can resolve that to an address if you like.
As for what server software it's running, you need to make an HTTP request to the server and extract the right header from the response:
require 'net/http'
res = Net::HTTP.get_response(uri)
res['server']
#=> "Microsoft-IIS/6.0"
I am working in rails 4 and I am trying to authenticate using github. So in my Github application I have this:
URL: http:// localhost:4000
Callback URL: http:// localhost:4000/auth/github/callback
The callback url is the url that Github will try to reach when the authentication is done right?
So why do I get a Github page 404 error when I click on my link:
<%= link_to 'Sign in with Github', '/auth/github' %>
I am working on a localhost development enviroment so that might be the problem?
Also when i type http:// localhost:4000/auth/github/callback on my browser I get an OmniAuth::Strategies::OAuth2::CallbackError
why? I have this in my routes.rb
post 'auth/:provider/callback' => 'home#index'
Is Rails 4 and Omniauth bugged?
(added the space in localhost so stackoverflow accepts my post)
I have github working with the gem omniauth-github
and a file config/initializers/omniauth.rb containing
Rails.application.config.middleware.use OmniAuth::Builder do
provider :github, ENV['GITHUB_KEY'], ENV['GITHUB_SECRET']
end
However, when I enter http://localhost:3000/auth/github/callback on my browser I also get OmniAuth::Strategies::OAuth2::CallbackError so this shouldn't be the problem.
My config/environment.rb looks like
# Load the rails application
require File.expand_path('../application', __FILE__)
# Load the app's custom environment variables here, so that they are loaded before environments/*.rb
app_environment_variables = File.join(Rails.root, 'config', 'app_environment_variables.rb')
load(app_environment_variables) if File.exists?(app_environment_variables)
...
and my config/app/environment_variables.rb looks like
# OAuth Keys and Secrets
if Rails.env.production?
ENV['GITHUB_KEY'] = 'd1234a3a123a1a3a123c'
ENV['GITHUB_SECRET'] = '1234azer123azer1231209jeunsghezkndaz1234'
else
ENV['GITHUB_KEY'] = 'qsflkjkj685bg554456b'
ENV['GITHUB_SECRET'] = 'qslkfj7757kqfmlsdh675hlfsd587kjfdh687jsd'
end
See Is it possible to set ENV variables for rails development environment in my code? for more details on that.
I have 2 applications registered on github. One app_name-dev with key qsflk..., url http://localhost:3000 and callback url http://localhost:3000/auth/github/callback and one app_name with key d1234a....
Check that you have done that correctly. Maybe try to change localhost to 127.0.0.1.
For me it was Github's new stricter URI matching that was producing a 404 when trying to redirect to http://localhost:3000/auth/github/callback, I solved it by passing the redirect URI as a parameter with Omniauth.
Rails.application.config.middleware.use OmniAuth::Builder do
provider :github, ENV['GITHUB_KEY'], ENV['GITHUB_SECRET'],
:scope => 'user,public_repo',
:redirect_uri => ENV['GITHUB_REDIRECT']
end
If your on Linux/Mac you can add environment variables from the command line.
$ export GITHUB_REDIRECT=http://localhost:3000/auth/github/callback
Alternatively, you could use something like Foreman that will let you add a .env file which you can use to store your variables in.
Just remember to add the appropriate redirect URI to your production environment's variables, and you're good, to go.
I have an application that initiates multiple HTTP connections and I would like to add a proxy to all connections.
The application is using net/HTTP, TCP sockets and open-uri so ideally I would like to be able to patch all connections initiated from those libraries instead of adding it manually to each and every location in the code that initiates a connection.
Is there a way to accomplish that (on Ruby 1.9.2)?
Open URI uses the HTTP_PROXY environment variable
Here is an article on how to use it on both windows and unix variants.
http://kaamka.blogspot.com/2009/06/httpproxy-environment-variable.html
you can also set it directly in ruby using the ENV hash
ENV['HTTP_PROXY'] = 'http://username:password#hostname:port'
the net/http documentation says not to rely on the environment and set it each time
require 'net/http'
require 'uri'
proxy_host = 'your.proxy.host'
proxy_port = 8080
uri = URI.parse(ENV['http_proxy'])
proxy_user, proxy_pass = uri.userinfo.split(/:/) if uri.userinfo
Net::HTTP::Proxy(proxy_host, proxy_port,
proxy_user, proxy_pass).start('www.example.com') {|http|
# always connect to your.proxy.addr:8080 using specified username and password
:
}
from http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/classes/Net/HTTP.html
Yes and mechanize does too (this is for the 1.0.0 verison)
require 'mechanize'
url = 'http://www.example.com'
agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
agent.set_proxy('127.0.0.1', '3128')
#page = agent.get(:url => url)