Im working in a small sinatra app that i want it to fetch a json file for latter use.
Using browser, i can access the json just OK, also on irb:
1.9.2p320 :001 > require 'open-uri'
=> true
1.9.2p320 :002 > metrics = open "http://foo-bar.com:8085/metrics/index.json"
=> #<File:/tmp/open-uri20130529-12715-1upc3bm>
1.9.2p320 :003 > metrics.read
=> "[\"carbon.agents.io-a.avgUpdateTime\", \"carbon.agents.io-a.cache.overflow\", \"carbon.agents.io-a.cache.queries\", \"carbon.agents.io-a.cache.queues\", \"carbon.agents.io-a.cache.size\", \"carbon.agents.io-a.committedPoints\", \"carbon.agents.io-a.cpuUsage\", \"carbon.agents.io-a.creates\", \"carbon.agents.io-a.errors\", \"carbon.agents.io-a.memUsage\" ...
it returns me the desired file.
But when i try to do the same from sinatra_app.rb:
get '/json' do
#all_metrics = open #graphite_all_metrics
erb :json
end
or
get '/json' do
#all_metrics = Net::HTTP.get_response(URI #graphite_all_metrics)
erb :json
end
returns me a 502 bad gateway error.
Any help?
How i get #graphite_all_metrics:
#graphite_base = "http://foo-bar.com:8085/"
#graphite_all_metrics = [#graphite_base, "/metrics/index.json"].join
Your join
#graphite_all_metrics = [#graphite_base, "/metrics/index.json"].join
is adding a double slash to the URL. Use
#graphite_all_metrics = File.join(#graphite_base, "/metrics/index.json")
or just remove one of the slashes from your example.
Most browsers/programs will fix double slash errors, but there could be some issue with your server setup.
Solved. It was a misconfigured multi-domain dns. Nothing with ruby.
Related
2.5.0 :150 > url = 'https://www.online.citibank.co.in/credit-card/apply'
=> "https://www.online.citibank.co.in/credit-card/apply"
2.5.0 :151 > Domainatrix.parse(url)
=> #<Domainatrix::Url:0x00007fd7850df4a8 #scheme="https", #host="www.online.citibank.co.in", #port="", #url="https://www.online.citibank.co.in/credit-card/apply", #public_suffix="co.in", #domain="citibank", #subdomain="www.online", #path="/credit-card/apply", #localhost=false, #ip=false>
2.5.0 :152 > Domainatrix.parse(url).domain_with_public_suffix
=> "citibank.co.in"
its getting "citibank.co.in"
but i required online.citibank.co.in with out using gsub or split thing
can any one help
Make use of URI#parse
While you will get www as well it's better to use this as you won't have to handle various cases manually.
url = 'https://www.online.citibank.co.in/credit-card/apply'
#=> "https://www.online.citibank.co.in/credit-card/apply"
parsed_url = URI.parse(url)
#=> #<URI::HTTPS https://www.online.citibank.co.in/credit-card/apply>
parsed_url.host
#=> "www.online.citibank.co.in"
I need to perform a very specific request on a legacy system and discovered that during a get request the http libraries are changing any %2C back to a ,.
Same issue with net-http, httparty, faraday, and open-uri using different implementations
2.5.0 :001 > require 'net/http'
=> true
2.5.0 :002 > require 'erb'
=> true
2.5.0 :003 > link = "http://example.com?q=" + ERB::Util.url_encode("Hello, World")
=> "http://example.com?q=Hello%2C%20World"
2.5.0 :004 > uri = URI(link)
=> #<URI::HTTP http://example.com?q=Hello%2C%20World>
2.5.0 :005 > res = Net::HTTP.get_response(uri)
=> #<Net::HTTPOK 200 OK readbody=true>
All this looks good until I look at the actual request with VCR
http_interactions:
- request:
method: get
uri: http://example.com/?q=Hello,%20World
body:
encoding: US-ASCII
string: ''
...
How do I keep the request to be http://example.com?q=Hello%2C%20World?
, is a legal character in a query (extended clarification is here https://stackoverflow.com/a/31300627/3820185)
ERB::Util.url_encode instead does replace [^a-zA-Z0-9_\-.]:
# File erb.rb, line 930
def url_encode(s)
s.to_s.dup.force_encoding("ASCII-8BIT").gsub(/[^a-zA-Z0-9_\-.]/n) {
sprintf("%%%02X", $&.unpack("C")[0])
}
end
So when doing the request, most probably the query is re-parsed to conform to the actual standards.
EDIT
Also you don't need to use ERB::Util.url_encode at all, you can just pass your URL to URI, it will propery escape it according to the standards.
irb(main):001:0> require 'net/http'
=> true
irb(main):002:0> link = URI 'http://example.com?q=Hello, World'
=> #<URI::HTTP http://example.com?q=Hello,%20World>
irb(main):003:0>
I posted the same in github of Koala but noone answered to me so I put here.
So when I try to login with Twitter with Omniauth:
I, [2013-11-15T18:57:12.371006 #28412] INFO -- omniauth: (twitter) Request phase initiated.
127.0.0.1 - - [15/Nov/2013 18:57:13] "GET /auth/twitter HTTP/1.1" 500 144366 0.9355
I have also a Koala login to facebook I don't use Omniauth for Facebook I just use Omniauth for twitter, If I don't require Koala is ok, but if I have both it generates:
undefined method `[]=' for #<Koala::Facebook::OAuth:0x00000001b03348>
~>In oauth.rb line 31
I'm using 1.6.0 version of Koala and Sinatra.
My code is:
#Facebook
get '/loginfb' do
session['oauth'] = Koala::Facebook::OAuth.new($APP_ID, $APP_SECRET, "#{request.base_url}/callbackfb")
redirect session['oauth'].url_for_oauth_code(:permissions => ["publish_stream"])
end
get '/callbackfb' do
session['access_token'] = session['oauth'].get_access_token(params[:code])
registerUserFB() #Just register the user function
redirect '/accounts'
end
#Twitter
#By defualt logs in with /auth/twitter
get '/auth/twitter/callback' do
erb "<h1>#{params[:provider]}</h1><pre>#{JSON.pretty_generate(request.env['omniauth.auth'])}</pre>"
p auth['credentials']['token']
end
get '/auth/failure' do
erb "<h1>Authentication Failed:</h1><h3>message:<h3> <pre>#{params}</pre>"
end
Thanks guys in advance.
I used another gem for logging with twitter called twitter_oauth yo can find here
For use with sinatra is quite simple:
#Sinatra stuff
require 'twitter_oauth'
#more sinatra stuff
$CONSUMER_KEY = '32423...'
$CONSUMER_SECRET = '...adads...'
$CALLBACK_URL = 'http://....'
tw_client = TwitterOAuth::Client.new(
:consumer_key => $CONSUMER_KEY,
:consumer_secret => $CONSUMER_SECRET
)
$request_token = tw_client.request_token(:oauth_callback => $CALLBACK_URL)
#sinatra routes
get '/logintw' do
redirect $request_token.authorize_url
end
get '/callbacktw' do
#access_token = $request_token.get_access_token :oauth_verifier => params[:oauth_verifier]
p #access_token.params[:oauth_token]
p #access_token.params[:oauth_token_secret]
p #access_token.params[:screen_name]
p #access_token.params[:user_id]
redirect '/accounts'
end
#more sinatra routes
Is not the best solution but is one and for me works!
Thanks anyway.
I am building a script to parse multiple page titles. Thanks to another question in stack I have now this working bit
curl = %x(curl http://odin.1.ai)
simian = curl.match(/<title>(.*)<\/title>/)[1]
puts simian
but if you try the same where a page has no title for example
curl = %x(curl http://zales.1.ai)
it dies with undefined method for nill class as it has no title ....
I can't check if curl is nil as it is not in this case (it contains another line)
Do you have any solution to have this working even if the title is not present and move to the next page to check ? I would appreciate if we stick to this code as I did try other solutions with nokogiri and uri (Nokogiri::HTML(open("http:/.....") but this is not working either as subdomains like byname_meee.1.ai do not work with the default open-uri so I am thankful if we can stick to this code that uses curl.
UPDATE
I realize that I probably left out some specific cases that are ought to be clarified. This is for parsing 300-400 pages. In the first run I have noticed at least two cases where nokogiri, hpricot but even the more basic open-uri do not work
1) open-uri simply fails in a simple domain with _
like http://levant_alejandro.1.ai this is a valid domain and works with curl but not with open_uri or nokogiri using open_uri
2)The second case if a page has no title like
http://zales.1.ai
3) Third is a page with an image and no valid HTML like http://voldemortas.1.ai/
A fourth case would be a page that has nothing but an internal server error or passenger/rack error.
The first three cases can be sorted with this solution (thanks to Havenwood in #ruby IRC channel)
curl = %x(curl http://voldemortas.1.ai/)
begin
simian = curl.match(/<title>(.*)<\/title>/)[1]
rescue NoMethodError
simian = "" # curl was nil?
rescue ArguementError
simian = "" # not html?
end
puts simian
Now I am aware that this is not elegant nor optimal.
REPHRASED QUESTION
Do you have better way to achieve the same with nokogiri or another gem that includes these cases (no title or no HTML valid page or even 404 page) ? Given that the pages I am parsing have a fairly simple title structure, is the above solution suitable ? For the sake of knowledge it would be useful to know why using an extra gem for the parsing like nokogiri would be better option (note: I try to have few gem dependencies as often and over time they tend to break).
You're making it much to hard on yourself.
Nokogiri doesn't care where you get the HTML, it just wants the body of the document. You can use Curb, Open-URI, a raw Net::HTTP connection, and it'll parse the content returned.
Try Curb:
require 'curb'
require 'nokogiri'
doc = Nokogiri::HTML(Curl.get('http://http://odin.1.ai').body_str)
doc.at('title').text
=> "Welcome to Dotgeek.org * 1.ai"
If you don't know whether you'll have a <title> tag, then don't try to do it all at once:
title = doc.at('title')
next if (!title)
puts title.text
Take a look at "equivalent of curl for Ruby?" for more ideas.
You just need to check for the match before accessing it. If curl.match is nil, the you can't access the grouping:
curl = %x(curl http://odin.1.ai)
simian = curl.match(/<title>(.*)<\/title>/)
simian &&= simian[1] # only access the matched group if available
puts simian
Do heed the Tin Man's advice and use Nokogiri. Your regexp is really only suitable for a brittle solution -- it fails when the title element is spread over multiple lines.
Update
If you really don't want to use an HTML parser and if you promise this is for a quick script, you can use OpenURI (wrapper around net/http) in the standard library. It's at least a little cleaner than parsing curl output.
require 'open-uri'
def extract_title_content(line)
title = line.match(%r{<title>(.*)</title>})
title &&= title[1]
end
def extract_title_from(uri)
title = nil
open(uri) do |page|
page.lines.each do |line|
return title if title = extract_title_content(line)
end
end
rescue OpenURI::HTTPError => e
STDERR.puts "ERROR: Could not download #{uri} (#{e})"
end
puts extract_title_from 'http://odin.1.ai'
What you're really looking for, it seems, is a way to skip non-HTML responses. That's much easier with a curl wrapper like curb, like the Tin Man suggested, than dropping to the shell and using curl there:
1.9.3p125 :001 > require 'curb'
=> true
1.9.3p125 :002 > response = Curl.get('http://odin.1.ai')
=> #<Curl::Easy http://odin.1.ai?>
1.9.3p125 :003 > response.content_type
=> "text/html"
1.9.3p125 :004 > response = Curl.get('http://voldemortas.1.ai')
=> #<Curl::Easy http://voldemortas.1.ai?>
1.9.3p125 :005 > response.content_type
=> "image/png"
1.9.3p125 :006 >
So your code could look something like this:
response = Curl.get(url)
if response.content_type == "text/html" # or more fuzzy: =~ /text/
match = response.body_str.match(/<title>(.*)<\/title>/)
title = match && match[1]
# or use Nokogiri for heavier lifting
end
No more exceptions
puts simian
I'm using the Ruby REST API for Redmine (here: http://www.redmine.org/projects/redmine/wiki/Rest_api_with_ruby). I need to be able to get all issues in a chunk of 100 at a time.
I know there is an options[:offset] and an options[:limit] that the method "api_offset_and_limit" is looking for.
How do I pass those options when I'm doing this? I tried putting them in the URL as GET options, but they didn't come through on the other end. The following gives me the first 25 issues, as I expect it to.
class Issue < ActiveResource::Base
self.site = 'http://redmine.server/'
self.user = 'foo'
self.password = 'bar'
end
# Retrieving issues
issues = Issue.find(:all)
I'm not familiar with the API, but the way you describe it, the following should work:
issues = Issue.find(:all, :params => {:offset => 0, :limit => 100})