I'm trying to parse a url but keep having this 500. Any suggestion please?
require 'open-uri'
require 'json'
require 'csv'
url = 'https://gist.githubusercontent.com/gregclermont/ca9e8abdff5dee9ba9db/raw/
7b2318efcf8a7048f720bcaff2031d5467a4a2c8/users.json'
encoded_url = URI.encode(url)
open(encoded_url) do |stream|
quote = JSON.parse(stream.read)
puts quote
end
require 'open-uri'
require 'json'
url = 'https://gist.githubusercontent.com/gregclermont/ca9e8abdff5dee9ba9db/raw/7b2318efcf8a7048f720bcaff2031d5467a4a2c8/users.json'
open(url) { |f| JSON.parse(f.read) }
Works fine for me.
How do I capture a Json data from POST route and save it to file? I have simple ruby sinatra code as below.
#!/usr/bin/env ruby
require 'rubygems'
require 'sinatra'
require 'json'
post '/' do
values = JSON.parse(request.env["rack.input"].read)
# How do I save "values" of JSON to file..
end
Try this
#!/usr/bin/env ruby
require 'rubygems'
require 'sinatra'
require 'json'
post '/' do
values = JSON.parse(request.env["rack.input"].read)
File.open('file.txt', 'w') { |file| file.write(values) }
end
To write file in ruby you can use:
File.open('/your/path/file', 'w') { |file| file.write(values) }
I'm trying to parse a simple html page http://www.brainjar.com/java/host/test.html using the jruby and htmlunit but i encountered the error undefined method 'asText'.
my code:
require 'java';
require 'commons-codec-1.7.jar';
require 'commons-collections-3.2.1.jar';
require 'commons-io-2.4.jar';
require 'commons-lang3-3.1.jar';
require 'commons-logging-1.1.1.jar';
require 'cssparser-0.9.8.jar'
require 'htmlunit-2.11.jar'
require 'htmlunit-core-js-2.11.jar'
require 'httpclient-4.2.2.jar'
require 'httpcore-4.2.2.jar'
require 'httpmime-4.2.2.jar'
require 'jetty-http-8.1.7.v20120910.jar'
require 'jetty-io-8.1.7.v20120910.jar'
require 'jetty-util-8.1.7.v20120910.jar'
require 'jetty-websocket-8.1.7.v20120910.jar'
require 'nekohtml-1.9.17.jar'
require 'sac-1.3.jar'
require 'serializer-2.7.1.jar'
require 'xalan-2.7.1.jar'
require 'xercesImpl-2.10.0.jar'
require 'xml-apis-1.4.01.jar'
java_import 'com.gargoylesoftware.htmlunit.WebClient';
def get_desc
wc = WebClient.new;
page = wc.getPage("http://www.brainjar.com/java/host/test.html");
a = page.getByXPath('/html/body/p')
ss = a
puts ss.asText
end
get_desc
I am trying to scrape a website. I am able to scrape data from that website. I am having trouble saving the data from the scrape to yaml file that I have included
My Code:
require 'rubygems'
require 'open-uri'
require 'hpricot'
article = []
doc = open("http://www.cmegroup.com/trading/interest-rates/cleared-otc/irs.html"{|f| Hpricot(f) }
(doc/"/html/body/div/div/div/div/table/").each do |article|
puts "#{article.inner_html}"
end
File.open('test.yaml', 'w') { |f|
f <<article.to_yaml
}
First you are missing a closing parenthesis for the open call (a ) right before the block starts).
When you add that you'll notice that you'll get a NoMethodError (undefined method 'to_yaml' for []:Array). To fix that you have to require 'yaml', which pulls in the monkey-patches for the Array class. After that you'll notice that your yaml file is empty, because you never put anything into article. Here's a fixed version:
require 'rubygems'
require 'open-uri'
require 'hpricot'
require 'yaml'
articles = []
url = "http://www.cmegroup.com/trading/interest-rates/cleared-otc/irs.html"
doc = open(url) {|f| Hpricot(f) }
(doc/"/html/body/div/div/div/div/table/").each do |article|
articles << article.inner_html
end
File.open('test.yaml', 'w') { |f| f << articles.to_yaml }
For tedious reasons to do with Hpricot, I need to write a function that is passed a URL, and returns the whole contents of the page as a single string.
I'm close. I know I need to use OpenURI, and it should look something like this:
require 'open-uri'
open(url) {
# do something mysterious here to get page_string
}
puts page_string
Can anyone suggest what I need to add?
You can do the same without OpenURI:
require 'net/http'
require 'uri'
def open(url)
Net::HTTP.get(URI.parse(url))
end
page_content = open('http://www.google.com')
puts page_content
Or, more succinctly:
Net::HTTP.get(URI.parse('http://www.google.com'))
The open method passes an IO representation of the resource to your block when it yields. You can read from it using the IO#read method
open([mode [, perm]] [, options]) [{|io| ... }]
open(path) { |io| data = io.read }
require 'open-uri'
open(url) do |f|
page_string = f.read
end
See also the documentation of IO class
I was also very confused what to use for better performance and speedy results. I ran a benchmark for both to make it more clear:
require 'benchmark'
require 'net/http'
require "uri"
require 'open-uri'
url = "http://www.google.com"
Benchmark.bm do |x|
x.report("net-http:") { content = Net::HTTP.get_response(URI.parse(url)).body if url }
x.report("open-uri:") { open(url){|f| content = f.read } if url }
end
Its result is:
user system total real
net-http: 0.000000 0.000000 0.000000 ( 0.097779)
open-uri: 0.030000 0.010000 0.040000 ( 0.864526)
I'd like to say that it depends on what your requirement is and how you want to process.
To make code a little clearer, the OpenURI open method will return the value returned by the block, so you can assign open's return value to your variable. For example:
xml_text = open(url) { |io| io.read }
Starting with Ruby 3.0, calling URI.open via Kernel#open has been removed, so instead call URI.open directly:
require 'open-uri'
page_string = URI.open(url, &:read)
Try the following instead:
require 'open-uri'
content = URI(your_url).read
require 'open-uri'
open(url) {|f| #url must specify the protocol
str = f.read()
}