I'm currently working on a code where I'm requesting a URL with space on request path:
http://example.com/path1/path2/some details.detail?keyword=key word
I'm using rest-client gem.
url = http://example.com/path1/path2/some details.detail?keyword=key word
RestClient::Request.execute(method: :get, url: url)
But it throws an error:
URI::InvalidURIError: bad URI(is not URI?): http://example.com/path1/path2/some details.detail?keyword=key word
The documentation suggests to parse and normalize the URL first:
parsed_url = Addressable::URI.parse(url).normalize.to_str
# parsed_url is now: http://example.com/path1/path2/some%20details.detail?keyword=key%20word
RestClient::Request.execute(method: :get, url: parsed_url)
But the error still exists.
URI::InvalidURIError: bad URI(is not URI?): http://example.com/path1/path2/some details.detail?keyword=key%20word
It looks like it cannot parse the /some details.detail part of the url.
Does anyone encountered the same issue?
Try URI.encode:
$ irb
2.0.0-p247 :001 > require "uri"
=> true
2.0.0-p247 :002 > URI.encode("http://example.com/path1/path2/some details.detail?keyword=key word")
=> "http://example.com/path1/path2/some%20details.detail?keyword=key%20word"
2.0.0-p247 :003 >
Using rest-client 1.6.7 and addressable 2.3.5, on Ruby 2.0.0, this is an irb session trying to replicate your fault:
:001 > require 'rest-client'
=> true
:002 > require "addressable/uri"
=> true
:003 > url = 'http://example.com/path1/path2/some details.detail?keyword=key word'
=> "http://example.com/path1/path2/some details.detail?keyword=key word"
:004 > parsed_url = Addressable::URI.parse(url).normalize.to_str
=> "http://example.com/path1/path2/some%20details.detail?keyword=key%20word"
:005 > RestClient::Request.execute(method: :get, url: parsed_url)
RestClient::ResourceNotFound: 404 Resource Not Found: <!doctype html>
<html>
<head>
<title>Example Domain</title>
It seems to work ok according to your own solution. As far as I can tell you either missed out repeating a step whilst investigating the problem (try again, from the start, just in case), or perhaps it is a bug in an older version of one of the two gems you are using - in which case you could fix it by installing latest versions.
Related
I need to perform a very specific request on a legacy system and discovered that during a get request the http libraries are changing any %2C back to a ,.
Same issue with net-http, httparty, faraday, and open-uri using different implementations
2.5.0 :001 > require 'net/http'
=> true
2.5.0 :002 > require 'erb'
=> true
2.5.0 :003 > link = "http://example.com?q=" + ERB::Util.url_encode("Hello, World")
=> "http://example.com?q=Hello%2C%20World"
2.5.0 :004 > uri = URI(link)
=> #<URI::HTTP http://example.com?q=Hello%2C%20World>
2.5.0 :005 > res = Net::HTTP.get_response(uri)
=> #<Net::HTTPOK 200 OK readbody=true>
All this looks good until I look at the actual request with VCR
http_interactions:
- request:
method: get
uri: http://example.com/?q=Hello,%20World
body:
encoding: US-ASCII
string: ''
...
How do I keep the request to be http://example.com?q=Hello%2C%20World?
, is a legal character in a query (extended clarification is here https://stackoverflow.com/a/31300627/3820185)
ERB::Util.url_encode instead does replace [^a-zA-Z0-9_\-.]:
# File erb.rb, line 930
def url_encode(s)
s.to_s.dup.force_encoding("ASCII-8BIT").gsub(/[^a-zA-Z0-9_\-.]/n) {
sprintf("%%%02X", $&.unpack("C")[0])
}
end
So when doing the request, most probably the query is re-parsed to conform to the actual standards.
EDIT
Also you don't need to use ERB::Util.url_encode at all, you can just pass your URL to URI, it will propery escape it according to the standards.
irb(main):001:0> require 'net/http'
=> true
irb(main):002:0> link = URI 'http://example.com?q=Hello, World'
=> #<URI::HTTP http://example.com?q=Hello,%20World>
irb(main):003:0>
I’m using Rails 4.2.7. I have a request I set up like so:
2.3.0 :001 > url = 'http://www.mydomeina.com/results/browse.cfm?ID=4187141102&Gen=B&Begin=1&End=31&Max=31'
=> "http://www.mydomeina.com/results/browse.cfm?ID=4187141102&Gen=B&Begin=1&End=31&Max=31"
2.3.0 :002 > uri = URI(url)
=> #<URI::HTTP http://www.mydomeina.com/results/browse.cfm?ID=4187141102&Gen=B&Begin=1&End=31&Max=31>
2.3.0 :003 > req = Net::HTTP::Get.new uri
=> #<Net::HTTP::Get GET>
2.3.0 :004 > req["Referer"] = 'http://www.mydomeina.com/results/browse.cfm?ID=4187141102&Gen=B&Begin=1&End=31&Max=31'
=> "http://www.mydomeina.com/results/browse.cfm?ID=4187141102&Gen=B&Begin=1&End=31&Max=31"
I set req["Referer"] as an example of setting a header. There may be other headers in my request object. How do I figure out what they all are? This fails:
2.3.0 :009 > req.keys
NoMethodError: undefined method `keys' for #<Net::HTTP::Get GET>
Did you mean? key?
as does req.headers. How can I iterate over my request object to figure out what all the headers are? It is not an option to construct the request object in a different way then what I am doing now.
try
req.each_key{|k| puts k}
or
req.each{|k,v| puts "#{k} => #{v}"}
to see methods on an object from inside console you can always do
req.methods - Ojbect.methods
We have been using JSON.parse for a while now and we just updated to the latest version.
We are getting the following error.
Has the functionality been updated to return nil instead of throwing a ParserError?
2.3.0 :001 > gem 'json', '2.0.2'
=> true
2.3.0 :002 > require 'json'
=> true
2.3.0 :003 > JSON.parse("null")
=> nil
2.3.0 :001 > gem 'json', '1.8.3'
=> true
2.3.0 :003 > require 'json'
=> true
2.3.0 :004 > JSON.parse("null")
JSON::ParserError: 784: unexpected token at 'null'
from /Users/ryannealmes/.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/json/common.rb:156:in `parse'
from /Users/ryannealmes/.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/json/common.rb:156:in `parse'
from (irb):4
from /Users/ryannealmes/.rvm/rubies/ruby-2.3.0/bin/irb:11:in `<main>'
2.3.0 :005 >
The functionality has been updated to comply with the JSON specification according to RFC 7159, which says:
A JSON value MUST be an object, array, number, or string, or one of
the following three literal names:
false null true
The literal names MUST be lowercase. No other literal names are allowed.
So, JSON.parse("null") returning nil is expected functionality, which you can confirm by checking out the tests for parsing single values.
Im working in a small sinatra app that i want it to fetch a json file for latter use.
Using browser, i can access the json just OK, also on irb:
1.9.2p320 :001 > require 'open-uri'
=> true
1.9.2p320 :002 > metrics = open "http://foo-bar.com:8085/metrics/index.json"
=> #<File:/tmp/open-uri20130529-12715-1upc3bm>
1.9.2p320 :003 > metrics.read
=> "[\"carbon.agents.io-a.avgUpdateTime\", \"carbon.agents.io-a.cache.overflow\", \"carbon.agents.io-a.cache.queries\", \"carbon.agents.io-a.cache.queues\", \"carbon.agents.io-a.cache.size\", \"carbon.agents.io-a.committedPoints\", \"carbon.agents.io-a.cpuUsage\", \"carbon.agents.io-a.creates\", \"carbon.agents.io-a.errors\", \"carbon.agents.io-a.memUsage\" ...
it returns me the desired file.
But when i try to do the same from sinatra_app.rb:
get '/json' do
#all_metrics = open #graphite_all_metrics
erb :json
end
or
get '/json' do
#all_metrics = Net::HTTP.get_response(URI #graphite_all_metrics)
erb :json
end
returns me a 502 bad gateway error.
Any help?
How i get #graphite_all_metrics:
#graphite_base = "http://foo-bar.com:8085/"
#graphite_all_metrics = [#graphite_base, "/metrics/index.json"].join
Your join
#graphite_all_metrics = [#graphite_base, "/metrics/index.json"].join
is adding a double slash to the URL. Use
#graphite_all_metrics = File.join(#graphite_base, "/metrics/index.json")
or just remove one of the slashes from your example.
Most browsers/programs will fix double slash errors, but there could be some issue with your server setup.
Solved. It was a misconfigured multi-domain dns. Nothing with ruby.
I am using JSON implementation for Ruby in my rails project to parse the JSON string sent by ajax, but I found that although the json string is in UTF-8, the result coming out is in ASCII-8BIT by default, see below
jruby-1.6.7 :068 > json_text = '["に到着を待っている"]'
=> "[\"に到着を待っている\"]"
jruby-1.6.7 :069 > json_text.encoding
=> #<Encoding:UTF-8>
jruby-1.6.7 :070 > json_parsed = JSON.parse(json_text)
=> ["\u00E3\u0081\u00AB\u00E5\u0088\u00B0\u00E7\u009D\u0080\u00E3\u0082\u0092\u00E5\u00BE\u0085\u00E3\u0081\u00A3\u00E3\u0081\u00A6\u00E3\u0081\u0084\u00E3\u0082\u008B"]
jruby-1.6.7 :071 > json_parsed.first.encoding
=> #<Encoding:ASCII-8BIT>
I don't want it being escaped, I would like to have a UTF-8 result. Is there a way to set that? I check the documentation of the JSON project, finding not encoding options for the method JSON.parse. Maybe I missed something, how could I do that?
UPDATE:
as notified by #fl00r, this example is working fine in MRI, but not in JRUBY
This looks like a bug, as this actually works when using the pure version:
jruby-1.6-head :001 > require 'json/pure'
=> true
jruby-1.6-head :002 > json_text = '["に到着を待っている"]'
=> "[\"に到着を待っている\"]"
jruby-1.6-head :003 > json_parsed = JSON.parse(json_text)
=> ["に到着を待っている"]
jruby-1.6-head :004 > json_parsed.first.encoding
=> #<Encoding:UTF-8>
jruby-1.6-head :005 >
Edit: Just saw you opened a ticket for this...
Edit 2: This actually seems to have already been fixed by this commit. To install latest code from json:
$ git clone https://github.com/flori/json.git
$ cd json
$ rake jruby_gem
$ jruby -S gem install pkg/json-1.6.6-java.gem