I'm trying to write a simple program to parse JSON from the results of an API call. Very new to ruby and just can't figure this one out.
Here's all the code:
require "rubygems"
require "json"
require "net/http"
require "uri"
uri = URI.parse("http://api.chartbeat.com/live/recent/v3/?apikey=eaafffb9a735796b6edd50fd31eaab69&host=enactus.org")
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Get.new(uri.request_uri)
response = http.request(request)
if response.code == "200"
result = JSON.parse(response.body)
result.each do |doc|
puts doc["id"] #reference properties like this
puts doc # this is the result in object form
puts ""
puts ""
end
else
puts "ERROR!!!"
end
Here's the output of running the program (chartbeat.rb):
chartbeat.rb:14: syntax error, unexpected end-of-input, expecting keyword_end
The program comes verbatim from here with the url replaced: https://gist.github.com/timsavery/1657351
It doesn't look like what you're doing is taking advantage of any of Net::HTTPs power, so I'd probably do it like this:
require "rubygems"
require "json"
require "open-uri"
response = open("http://api.chartbeat.com/live/recent/v3/?apikey=eaafffb9a735796b6edd50fd31eaab69&host=enactus.org").read
result = JSON.parse(response)
result.each do |doc|
puts doc["id"] #reference properties like this
puts doc # this is the result in object form
puts ""
puts ""
end
OpenURI is the basis of a lot of code that hits URLs, and is a great starting place.
If you want to trap exceptions raised, use something like:
begin
response = open("http://api.chartbeat.com/live/recent/v3/?apikey=eaafffb9a735796b6edd50fd31eaab69&host=enactus.org").read
rescue Exception => e
puts e.message
exit
end
It could even be reduced to:
require "rubygems"
require "json"
require "open-uri"
JSON[
open("http://api.chartbeat.com/live/recent/v3/?apikey=eaafffb9a735796b6edd50fd31eaab69&host=enactus.org").read
].each do |doc|
puts doc["id"] #reference properties like this
puts doc # this is the result in object form
puts ""
puts ""
end
But that might be too drastic.
Related
I've been working for a while to try to use the .uniq method to generate a unique list of URL's from a website (within the /informatics path). No matter what I try I get a method error when trying to generate the list. I'm sure it's a syntax issue, and I was hoping someone could point me in the right direction.
Once I get the list I'm going to need to store these to a database via ActiveRecord, but I need the unique list before I get start to wrap my head around that.
require 'nokogiri'
require 'open-uri'
require 'active_record'
ARGV[0]="https://www.nku.edu/academics/informatics.html"
ARGV.each do |arg|
open(arg) do |f|
# Display connection data
puts "#"*25 + "\nConnection: '#{arg}'\n" + "#"*25
[:base_uri, :meta, :status, :charset, :content_encoding,
:content_type, :last_modified].each do |method|
puts "#{method.to_s}: #{f.send(method)}" if f.respond_to? method
end
# Display the href links
base_url = /^(.*\.nku\.edu)\//.match(f.base_uri.to_s)[1]
puts "base_url: #{base_url}"
Nokogiri::HTML(f).css('a').each do |anchor|
href = anchor['href']
# Make Unique
if href =~ /.*informatics/
puts href
#store stuff to active record
end
end
end
end
Replace the Nokogiri::HTML part to select only those href attributes that matches with /*.informatics/ and then you can use uniq, as it's already an array:
require 'nokogiri'
require 'open-uri'
require 'active_record'
ARGV[0] = 'https://www.nku.edu/academics/informatics.html'
ARGV.each do |arg|
open(arg) do |f|
puts "#{'#' * 25} \nConnection: '#{arg}'\n #{'#' * 25}"
%i[base_uri meta status charset content_encoding, content_type last_modified].each do |method|
puts "#{method.to_s}: #{f.send(method)}" if f.respond_to? method
end
puts "base_url: #{/^(.*\.nku\.edu)\//.match(f.base_uri.to_s)[1]}"
anchors = Nokogiri::HTML(f).css('a').select { |anchor| anchor['href'] =~ /.*informatics/ }
puts anchors.map { |anchor| anchor['href'] }.uniq
end
end
See output.
require "rubygems"
require "json"
require "net/http"
require "uri"
uri = URI.parse("https://api.website.com/top/inside/end")
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Get.new(uri.request_uri)
http.use_ssl = true
response = http.request(request)
if response.code == "200"
result = JSON.parse(response.body)
names = Array.new
i = 0
result.each do |doc|
names.insert(i, doc.name)
puts doc["id"] #reference properties like this
puts doc # this is the result in object form
puts ""
puts ""
end
puts names
else
puts "ERROR!!!"
end
Why am I getting undefined method error on variable names inside the for each loop? I cannot understand why
script.rb:18:in `block in ': undefined method `name' for #<Hash:0x0000000002e04c08> (NoMethodError)
line 18,
names.insert(i, doc.name)
doc is a Hash, doc['name'] should works.
in javascript, you can do
doc.name
I am trying to put together a series of custom 404 pages.
require 'uri'
def open(url)
page_content = Net::HTTP.get(URI.parse(url))
puts page_content.content
end
open('http://somesite.com/1ygjah1761')
the following code exits the program with an error. How can I get the page content from a website, regardless of it being 404 or not.
You need to rescue from the error
def open(url)
require 'net/http'
page_content = ""
begin
page_content = Net::HTTP.get(URI.parse(url))
puts page_content
rescue Net::HTTPNotFound
puts "THIS IS 404" + page_content
end
end
You can find more information on something like this here: http://tammersaleh.com/posts/rescuing-net-http-exceptions/
Net::HTTP.get returns the page content directly as a string, so there is no need to call .content on the results:
page_content = Net::HTTP.get(URI.parse(url))
puts page_content
I'm doing a GET request and getting this response:
"oauth_token=USYS96A708CACBDA9C74322DAB41A53CA_idses-int02.a.fsglobal.net&oauth_token_secret=09c8b05b874fac29b4e542c388cb3f&oauth_callback_confirmed=true"
How can I convert this to JSON in Ruby?
Step 1: Parse the GET response:
require 'cgi'
CGI::parse(MYSTRING)
Returns: {"param1"=>["value1"], "param2"=>["value2"], "param3"=>["value3"]}
Step 2. Convert to JSON:
require 'json'
myObject.to_json
Alternatively, look at this snippet:
https://gist.github.com/timsavery/1657351
require "rubygems"
require "json"
require "net/http"
require "uri"
uri = URI.parse("http://api.sejmometr.pl/posiedzenia/BZfWZ/projekty")
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Get.new(uri.request_uri)
response = http.request(request)
if response.code == "200"
result = JSON.parse(response.body)
result.each do |doc|
puts doc["id"] #reference properties like this
puts doc # this is the result in object form
puts ""
puts ""
end
else
puts "ERROR!!!"
end
How do I take this URL http://t.co/yjgxz5Y and get the destination URL which is http://nickstraffictricks.com/4856_how-to-rank-1-in-google/
require 'net/http'
require 'uri'
Net::HTTP.get_response(URI.parse('http://t.co/yjgxz5Y'))['location']
# => "http://nickstraffictricks.com/4856_how-to-rank-1-in-google/"
I've used open-uri for this, because it's nice and simple. It will retrieve the page, but will also follow multiple redirects:
require 'open-uri'
final_uri = ''
open('http://t.co/yjgxz5Y') do |h|
final_uri = h.base_uri
end
final_uri # => #<URI::HTTP:0x00000100851050 URL:http://nickstraffictricks.com/4856_how-to-rank-1-in-google/>
The docs show a nice example for using the lower-level Net::HTTP to handle redirects.
require 'net/http'
require 'uri'
def fetch(uri_str, limit = 10)
# You should choose better exception.
raise ArgumentError, 'HTTP redirect too deep' if limit == 0
response = Net::HTTP.get_response(URI.parse(uri_str))
case response
when Net::HTTPSuccess then response
when Net::HTTPRedirection then fetch(response['location'], limit - 1)
else
response.error!
end
end
puts fetch('http://www.ruby-lang.org')
Of course this all breaks down if the page isn't using a HTTP redirect. A lot of sites use meta-redirects, which you have to handle by retrieving the URL from the meta tag, but that's a different question.
For resolving redirects you should use a HEAD request to avoid downloading the whole response body (imagine resolving a URL to an audio or video file).
Working example using the Faraday gem:
require 'faraday'
require 'faraday_middleware'
def resolve_redirects(url)
response = fetch_response(url, method: :head)
if response
return response.to_hash[:url].to_s
else
return nil
end
end
def fetch_response(url, method: :get)
conn = Faraday.new do |b|
b.use FaradayMiddleware::FollowRedirects;
b.adapter :net_http
end
return conn.send method, url
rescue Faraday::Error, Faraday::Error::ConnectionFailed => e
return nil
end
puts resolve_redirects("http://cre.fm/feed/m4a") # http://feeds.feedburner.com/cre-podcast
You would have to follow the redirect. I think that would help :
http://shadow-file.blogspot.com/2009/03/handling-http-redirection-in-ruby.html