I'm trying to parse a url but keep having this 500. Any suggestion please?
require 'open-uri'
require 'json'
require 'csv'
url = 'https://gist.githubusercontent.com/gregclermont/ca9e8abdff5dee9ba9db/raw/
7b2318efcf8a7048f720bcaff2031d5467a4a2c8/users.json'
encoded_url = URI.encode(url)
open(encoded_url) do |stream|
quote = JSON.parse(stream.read)
puts quote
end
require 'open-uri'
require 'json'
url = 'https://gist.githubusercontent.com/gregclermont/ca9e8abdff5dee9ba9db/raw/7b2318efcf8a7048f720bcaff2031d5467a4a2c8/users.json'
open(url) { |f| JSON.parse(f.read) }
Works fine for me.
Related
How do I capture a Json data from POST route and save it to file? I have simple ruby sinatra code as below.
#!/usr/bin/env ruby
require 'rubygems'
require 'sinatra'
require 'json'
post '/' do
values = JSON.parse(request.env["rack.input"].read)
# How do I save "values" of JSON to file..
end
Try this
#!/usr/bin/env ruby
require 'rubygems'
require 'sinatra'
require 'json'
post '/' do
values = JSON.parse(request.env["rack.input"].read)
File.open('file.txt', 'w') { |file| file.write(values) }
end
To write file in ruby you can use:
File.open('/your/path/file', 'w') { |file| file.write(values) }
I'm parsing a JSON file in Ruby and want to output the results using pp to a file. How can I do that? Here's the code I'm trying:
require 'rubygems'
require 'json'
require 'pp'
json = File.read('players.json')
plyrs = JSON.parse(json)
File.open('plyrs.txt', 'a') { |fo| pp page, fo }
require "rubygems" is redundant in Ruby >= 1.9.
require "json"
require "pp"
plyrs = JSON.load("players.json")
File.open("plyrs.txt", "a"){|io| io.write(plyrs.pretty_inspect)}
Try this
require 'rubygems'
require 'json'
require 'pp'
json = File.read('players.json')
plyrs = JSON.parse(json)
File.open('plyrs.txt', 'a') { |file| file.write(pp plyrs) }
More info available at the ruby documentation
So far as I can tell the Google::Reader API is working fine, in that it returns an sid successfully. However, lower level interactions with gmail won't run properly:
warning: peer certificate won't be verified in this SSL session
#<Google::Reader::Base:0xb76efa0c
#email="hawat.thufir",
#password="pword",
#sid=
"DQAAAL4AAACq-Wrm1V_anY1sV4r_3kA4EuRax9oTt5z7upD6NNfT0e7bsN-8WA7cQOTt7zypI5fymS9Ux8QTtyu-7xal9c6szb2ZoeBR5dwPH_m7OrBe6ICkKY-dPus0_g5DFW6tckpCZmJIyrP9zfUQKJzGYjnYKJzJEJYFEdvMu756Hl68qeD6AuGKDdFWbyBEvgQGR2oFjkxHYGqwTQ9oHJBfBkMH9hrDl2Q9C_cVE5A-_Bb9RiUy6WuwIbS-pPN56z3XtpA">
#<URI::HTTPS:0xb76e7988 URL:https://hawat.thufir:pword#gmail.com>
#<Net::HTTP gmail.com:443 open=false>
#<Net::HTTP::Get GET>
["Basic aGF3YXQudGh1ZmlyOmRldm90Y2hrYQ=="]
#<Net::HTTP::Get GET>
/usr/lib/ruby/1.8/net/http.rb:1060:in `request': undefined method `closed?' for nil:NilClass (NoMethodError)
from ./req_uri.rb:23
code:
#!/usr/bin/ruby -w
require 'rubygems'
require 'google/reader'
require 'pp'
require 'net/http'
require 'net/https'
require 'uri'
require 'yaml'
yml = YAML.load_file 'login.yml'
user = yml["user"]
pword = yml["pword"]
pp Google::Reader::Base.establish_connection(user, pword)
uri = URI.parse "https://#{user}:#{pword}#gmail.com"
pp uri
pp http = Net::HTTP.new(uri.host, uri.port)
pp request = Net::HTTP::Get.new(uri.request_uri)
pp request.basic_auth(user, pword)
pp request
response = http.request(request)
So, the question is, should the request be basically empty when printed? What's wrong with sending the request to the response? That seems to be correct so far as I can ascertain. What am I missing?
I am trying to scrape a website. I am able to scrape data from that website. I am having trouble saving the data from the scrape to yaml file that I have included
My Code:
require 'rubygems'
require 'open-uri'
require 'hpricot'
article = []
doc = open("http://www.cmegroup.com/trading/interest-rates/cleared-otc/irs.html"{|f| Hpricot(f) }
(doc/"/html/body/div/div/div/div/table/").each do |article|
puts "#{article.inner_html}"
end
File.open('test.yaml', 'w') { |f|
f <<article.to_yaml
}
First you are missing a closing parenthesis for the open call (a ) right before the block starts).
When you add that you'll notice that you'll get a NoMethodError (undefined method 'to_yaml' for []:Array). To fix that you have to require 'yaml', which pulls in the monkey-patches for the Array class. After that you'll notice that your yaml file is empty, because you never put anything into article. Here's a fixed version:
require 'rubygems'
require 'open-uri'
require 'hpricot'
require 'yaml'
articles = []
url = "http://www.cmegroup.com/trading/interest-rates/cleared-otc/irs.html"
doc = open(url) {|f| Hpricot(f) }
(doc/"/html/body/div/div/div/div/table/").each do |article|
articles << article.inner_html
end
File.open('test.yaml', 'w') { |f| f << articles.to_yaml }
How do I take this URL http://t.co/yjgxz5Y and get the destination URL which is http://nickstraffictricks.com/4856_how-to-rank-1-in-google/
require 'net/http'
require 'uri'
Net::HTTP.get_response(URI.parse('http://t.co/yjgxz5Y'))['location']
# => "http://nickstraffictricks.com/4856_how-to-rank-1-in-google/"
I've used open-uri for this, because it's nice and simple. It will retrieve the page, but will also follow multiple redirects:
require 'open-uri'
final_uri = ''
open('http://t.co/yjgxz5Y') do |h|
final_uri = h.base_uri
end
final_uri # => #<URI::HTTP:0x00000100851050 URL:http://nickstraffictricks.com/4856_how-to-rank-1-in-google/>
The docs show a nice example for using the lower-level Net::HTTP to handle redirects.
require 'net/http'
require 'uri'
def fetch(uri_str, limit = 10)
# You should choose better exception.
raise ArgumentError, 'HTTP redirect too deep' if limit == 0
response = Net::HTTP.get_response(URI.parse(uri_str))
case response
when Net::HTTPSuccess then response
when Net::HTTPRedirection then fetch(response['location'], limit - 1)
else
response.error!
end
end
puts fetch('http://www.ruby-lang.org')
Of course this all breaks down if the page isn't using a HTTP redirect. A lot of sites use meta-redirects, which you have to handle by retrieving the URL from the meta tag, but that's a different question.
For resolving redirects you should use a HEAD request to avoid downloading the whole response body (imagine resolving a URL to an audio or video file).
Working example using the Faraday gem:
require 'faraday'
require 'faraday_middleware'
def resolve_redirects(url)
response = fetch_response(url, method: :head)
if response
return response.to_hash[:url].to_s
else
return nil
end
end
def fetch_response(url, method: :get)
conn = Faraday.new do |b|
b.use FaradayMiddleware::FollowRedirects;
b.adapter :net_http
end
return conn.send method, url
rescue Faraday::Error, Faraday::Error::ConnectionFailed => e
return nil
end
puts resolve_redirects("http://cre.fm/feed/m4a") # http://feeds.feedburner.com/cre-podcast
You would have to follow the redirect. I think that would help :
http://shadow-file.blogspot.com/2009/03/handling-http-redirection-in-ruby.html