I try to learn REST in Ruby using Twitter API.
According https://dev.twitter.com/docs/api/1/get/trends I have to write GET request to http://api.twitter.com/1/trends.json.
My Ruby code is:
require 'rubygems'
require 'rest-client'
require 'json'
url = 'http://api.twitter.com/1/trends.json'
response = RestClient.get(url)
puts response.body
But i'm getting next errors:
/home/danik/.rvm/gems/ruby-1.9.3-p194/gems/rest-client-1.6.7/lib/restclient /abstract_response.rb:48:in `return!': 404 Resource Not Found (RestClient::ResourceNotFound)
from /home/danik/.rvm/gems/ruby-1.9.3-p194/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result'
from /home/danik/.rvm/gems/ruby-1.9.3-p194/gems/rest-client-1.6.7/lib/restclient/request.rb:178:in `block in transmit'
from /home/danik/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:745:in `start'
from /home/danik/.rvm/gems/ruby-1.9.3-p194/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in `transmit'
from /home/danik/.rvm/gems/ruby-1.9.3-p194/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
from /home/danik/.rvm/gems/ruby-1.9.3-p194/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
from /home/danik/.rvm/gems/ruby-1.9.3-p194/gems/rest-client-1.6.7/lib/restclient.rb:68:in `get'
from TwitterTrends.rb:5:in `<main>'
What is wrong?
You are getting that error because the resource you are trying to fetch with http://api.twitter.com/1/trends.json does not exist, as is explained in this doc trends docs
This method is deprecated and has been replaced by GET trends/:woeid.
Please update your applications with the new endpoint.
You want to fetch a URL like this https://api.twitter.com/1/trends/1.json. So, in your code, try doing this:
require 'rubygems'
require 'rest-client'
require 'json'
url = 'https://api.twitter.com/1/trends/1.json'
response = RestClient.get(url)
puts response.body
And you should get a response.
Related
I can't seem to get this RSS feed to work properly. I've tried Nokogiri and now RSS::Parser and neither work:
a = 'https://phys.org/rss-feed/biology-news/biology-other/'
URI.open(a) do |rss|
feed = RSS::Parser.parse(rss)
puts "Title: #{feed.channel.title}"
feed.items.each do |item|
puts "Item: #{item.title}"
end
end
The code is taken directly out of the docs: https://github.com/ruby/rss
The feed is valid, so I'm confused as to why there's a 400 error code.
What am I doing wrong? Anybody have insight as to how to get this RSS parsed?
Here is the error:
/Users/user3/.rbenv/versions/3.1.2/lib/ruby/3.1.0/open-uri.rb:364:in `open_http': 400 Bad request (OpenURI::HTTPError)
from /Users/user3/.rbenv/versions/3.1.2/lib/ruby/3.1.0/open-uri.rb:741:in `buffer_open'
from /Users/user3/.rbenv/versions/3.1.2/lib/ruby/3.1.0/open-uri.rb:212:in `block in open_loop'
from /Users/user3/.rbenv/versions/3.1.2/lib/ruby/3.1.0/open-uri.rb:210:in `catch'
from /Users/user3/.rbenv/versions/3.1.2/lib/ruby/3.1.0/open-uri.rb:210:in `open_loop'
from /Users/user3/.rbenv/versions/3.1.2/lib/ruby/3.1.0/open-uri.rb:151:in `open_uri'
from /Users/user3/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/open_uri_redirections-0.2.1/lib/open-uri/redirections_patch.rb:55:in `open_uri'
from /Users/user3/.rbenv/versions/3.1.2/lib/ruby/3.1.0/open-uri.rb:721:in `open'
from /Users/user3/.rbenv/versions/3.1.2/lib/ruby/3.1.0/open-uri.rb:29:in `open'
from /users/user3/app.rb:1856:in `<main>'
The web server requires the request to have a User-Agent set in the headers. Without such a User-Agent header it returns the 400 error message.
require 'uri'
require 'open-uri'
require 'rss'
uri = URI.parse("https://phys.org/rss-feed/biology-news/biology-other/")
uri.open("User-Agent" => "Ruby/#{RUBY_VERSION}") do |rss|
feed = RSS::Parser.parse(rss)
puts "Title: #{feed.channel.title}"
feed.items.each do |item|
puts "Item: #{item.title}"
end
end
This code work for me.
Thanks for your time. Somewhat new to OOP and Ruby and after synthesizing solutions from a few different stack overflow answers I've got myself turned around.
My goal is to write a script that parses a CSV of URLs using Nokogiri library. After trying and failing to use open-uri and the open-uri-redirections plugin to follow redirects, I settled on Net::HTTP and that got me moving...until I ran into URLs that have a 302 redirect specifically.
Here's the method I'm using to engage the URL:
require 'Nokogiri'
require 'Net/http'
require 'csv'
def fetch(uri_str, limit = 10)
# You should choose better exception.
raise ArgumentError, 'HTTP redirect too deep' if limit == 0
url = URI.parse(uri_str)
#puts "The value of uri_str is: #{ uri_str}"
#puts "The value of URI.parse(uri_str) is #{ url }"
req = Net::HTTP::Get.new(url.path, { 'User-Agent' => 'Mozilla/5.0 (etc...)' })
# puts "THE URL IS #{url.scheme + ":" + url.host + url.path}" # just a reporter so I can see if it's mangled
response = Net::HTTP.start(url.host, url.port, :use_ssl => url.scheme == 'https') { |http| http.request(req) }
case response
when Net::HTTPSuccess then response
when Net::HTTPRedirection then fetch(response['location'], limit - 1)
else
#puts "Problem clause!"
response.error!
end
end
Further down in my script I take an ARGV with the URL csv filename, do CSV.read, encode the URL to a string, then use Nokogiri::HTML.parse to turn it all into something I can use xpath selectors to examine and then write to an output CSV.
Works beautifully...so long as I encounter a 200 response, which unfortunately is not every website. When I run into a 302 I'm getting this:
C:/Ruby24-x64/lib/ruby/2.4.0/Net/http.rb:1570:in `addr_port': undefined method `+' for nil:NilClass (NoMethodError)
from C:/Ruby24-x64/lib/ruby/2.4.0/Net/http.rb:1503:in `begin_transport'
from C:/Ruby24-x64/lib/ruby/2.4.0/Net/http.rb:1442:in `transport_request'
from C:/Ruby24-x64/lib/ruby/2.4.0/Net/http.rb:1416:in `request'
from httpcsv.rb:14:in `block in fetch'
from C:/Ruby24-x64/lib/ruby/2.4.0/Net/http.rb:877:in `start'
from C:/Ruby24-x64/lib/ruby/2.4.0/Net/http.rb:608:in `start'
from httpcsv.rb:14:in `fetch'
from httpcsv.rb:17:in `fetch'
from httpcsv.rb:42:in `block in <main>'
from C:/Ruby24-x64/lib/ruby/2.4.0/csv.rb:866:in `each'
from C:/Ruby24-x64/lib/ruby/2.4.0/csv.rb:866:in `each'
from httpcsv.rb:38:in `<main>'
I know I'm missing something right in front of me but I can't tell what I should puts to see if it is nil. Any help is appreciated, thanks in advance.
I'm behind a proxy and I must get an HTTPS webpage to collect some information, but OpenURI returns an error: "Non-HTTP proxy URI".
This is the issue:
> yadayada#ubuntu:~/Desktop/test/lib$ ruby JenkinsTest.rb
/home/yadayada/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:257:in `open_http': Non-HTTP proxy URI: https://web-proxy.yadayada:8088 (RuntimeError)
from /home/yadayada/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:736:in `buffer_open'
from /home/yadayada/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:211:in `block in open_loop'
from /home/yadayada/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:209:in `catch'
from /home/yadayada/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:209:in `open_loop'
from /home/yadayada/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:150:in `open_uri'
from /home/yadayada/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:716:in `open'
from /home/yadayada/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:34:in `open'
from JenkinsTest.rb:6:in `<main>'
yadayada#ubuntu:~/Desktop/test/lib$
This is the code I'm running:
1 require 'rubygems'
2 require 'nokogiri'
3 require 'open-uri'
4
5 # Request the Jenkins webpage
6 #jenkinsWebPage = Nokogiri::HTML(open("https://yadayad.yada.yada.com:8443"))
7
8 # Prints the received page
9 puts #jenkinsWebPage
The proxy has no login/password.
Any ideas?
Okay, so I've found out how to get the page, but I had to switch open-uri for net/https, also, I set OpenSSL to VERIFY_NONE, since it's a self signed certificate (company server):
require 'rubygems'
require 'nokogiri'
require 'net/https'
require 'openssl'
class JenkinsTest
# Request the Jenkins webpage
def request_jenkins_webpage
uri = URI.parse("https://https://yadayad.yada.yada.com:8443")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Get.new(uri.request_uri)
response = http.request(request)
##page = Nokogiri::HTML(response.body)
end
def print_jenkins_webpage
puts ##page
end
end
It looks ugly, if anybody finds out a better way to put this, please edit this post, but as of now, it's working fine.
One of my Sinatra actions performs another request using net/http and computes something based on the response:
api_response = Net::HTTP.get_response(uri)
It works like a charm, but when I try to profile this:
configure do
use ::Rack::PerftoolsProfiler, default_printer: 'pdf', frequency: 4000, mode: :walltime
end
(and put ?profile=true into the params of the request I get this:
Errno::EADDRINUSE - Address already in use - connect(2):
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/net/http.rb:762:in `initialize'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/net/http.rb:762:in `open'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/net/http.rb:762:in `block in connect'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/timeout.rb:54:in `timeout'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/net/http.rb:762:in `connect'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/net/http.rb:755:in `do_start'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/net/http.rb:744:in `start'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/net/http.rb:454:in `get_response'
Any idea what's happening?
I try to setup these the same way as you.
require 'sinatra'
require 'rack/perftools_profiler'
require 'net/http'
configure do
use ::Rack::PerftoolsProfiler, default_printer: 'pdf', frequency: 4000, mode: :walltime
end
get "/" do
uri = URI('http://google.com')
api_response = Net::HTTP.get_response(uri)
end
And it run with out a problem on ruby 1.9.3-p286 and with rack-perftools_profiler (0.6.1).
Two recommendations:
update your ruby version
update your gems
If these don't help check which program has bound your port with sudo netstat -nlp
If these all don't help please tell us more about your setup.
I have trouble with getting page source.
require 'mechanize'
agent = Mechanize.new
page = agent.get("https://#{ip}/")
end have error:
/home/lord/.gem/ruby/1.9.1/gems/mechanize-2.4/lib/mechanize/http/agent.rb:682:in `response_authenticate': 401 => Net::HTTPUnauthorized for https://82.144.208.6/cgi-bin/welcome.cgi -- no credentials found, provide some with #add_auth -- available realms: r722 (Mechanize::UnauthorizedError)
from /home/lord/.gem/ruby/1.9.1/gems/mechanize-2.4/lib/mechanize/http/agent.rb:288:in `fetch'
from /home/lord/.gem/ruby/1.9.1/gems/mechanize-2.4/lib/mechanize.rb:407:in `get'
from /home/lord/ruby/ruby_backup/backup-done.ru:35:in `block (2 levels) in <main>'
how can I ignore http auth, and get source? thx
The exception contains a page accessor. The documentation describes it: http://mechanize.rubyforge.org/Mechanize/ResponseCodeError.html
Try:
begin
page = agent.get ...
rescue Mechanize::ResponseCodeError => e
page = e.page
end