How do you get the full path of the URL given the following
uri = URI("http://foo.com/posts?id=30&limit=5#time=1305298413")
I just want posts?id=30&limit=5#time=1305298413
I tried uri.path and that returns /posts and ui.query returns 'id=30&limit=5'
The method you are looking for is request_uri
uri.request_uri
=> "/posts?id=30&limit=5"
You can use any method you'd like to remove the leading / if needed.
Edit: To get the part after the # sign, use fragment:
[uri.request_uri, uri.fragment].join("#")
=> "/posts?id=30&limit=5#time=1305298413"
You can ask the URI object for its path, query, and fragment like this:
"#{uri.path}?#{uri.query}##{uri.fragment}"
# => "/posts?id=30&limit=5#time=1305298413"
or (a little more consice, but less explicit):
"#{uri.request_uri}##{uri.fragment}"
# => "/posts?id=30&limit=5#time=1305298413"
A bit of an artsy solution, for completeness – others may be more reasonable :)
uri.dup.tap { _1.scheme = _1.user = _1.password = _1.host = _1.port = nil }.to_s
Though a benefit of this solution is that it also handles URI::Generic like you'd get from parsing a standalone path:
uri = URI.parse("/foo?bar#baz") # => #<URI::Generic /foo?bar#baz>
Solutions that rely on URI::HTTP#request_uri won't work with URI::Generic. So if you're writing code that accepts both full URLs and paths, that may be a factor.
File.basename("http://foo.com/posts?id=30&limit=5#time=1305298413")
# => "posts?id=30&limit=5#time=1305298413"
Related
Reading the HTTP API docs. My requests fail though for bad signature. From error message I can see that my string to sign is correct but looks like I can't generate the correct HMAC-SHA1 (seriously why use SHA1 still??).
So I decided to try replicate the signature of the sample inside same document.
[47] pry(main)> to_sign = "GET&%2F&AccessKeyId%3Dtestid&Action%3DDescribeRegions&Format%3DXML&SignatureMethod%3DHMAC-SHA1&SignatureNonce%3D3ee8c1b8-83d3-44af-a94f-4e0ad82fd6cf&SignatureVersion%3D1.0&Timestamp%3D2016-02-23T12%253A46%253A24Z&Version%3D2014-05-26"
[48] pry(main)> Base64.encode64 OpenSSL::HMAC.digest("sha1", "testsecret", to_sign)
=> "MLAxpXej4jJ7TL0smgWpOgynR7s=\n"
[49] pry(main)> Base64.encode64 OpenSSL::HMAC.digest("sha1", "testsecret&", to_sign)
=> "VyBL52idtt+oImX0NZC+2ngk15Q=\n"
[50] pry(main)> Base64.encode64 OpenSSL::HMAC.hexdigest("sha1", "testsecret&", to_sign)
=> "NTcyMDRiZTc2ODlkYjZkZmE4MjI2NWY0MzU5MGJlZGE3ODI0ZDc5NA==\n"
[51] pry(main)> Base64.encode64 OpenSSL::HMAC.hexdigest("sha1", "testsecret", to_sign)
=> "MzBiMDMxYTU3N2EzZTIzMjdiNGNiZDJjOWEwNWE5M2EwY2E3NDdiYg==\n"
[52] pry(main)> OpenSSL::HMAC.hexdigest("sha1", "testsecret&", to_sign)
=> "57204be7689db6dfa82265f43590beda7824d794"
[53] pry(main)> OpenSSL::HMAC.hexdigest("sha1", "testsecret", to_sign)
=> "30b031a577a3e2327b4cbd2c9a05a93a0ca747bb"
As evident none of these matches the example signature of CT9X0VtwR86fNWSnsc6v8YGOjuE=. Any idea what is missing here?
Update: taking tcpdump from the Golang client tool I see that it does a POST request like:
POST /?AccessKeyId=**********&Action=DescribeRegions&Format=JSON&RegionId=cn-qingdao&Signature=aHZVpIMb0%2BFKdoWSIVaFJ7bd2LA%3D&SignatureMethod=HMAC-SHA1&SignatureNonce=c29a0e28964c470a8997aebca4848b57&SignatureType=&SignatureVersion=1.0&Timestamp=2018-07-16T19%3A46%3A33Z&Version=2014-05-26 HTTP/1.1
Host: ecs.aliyuncs.com
User-Agent: Aliyun-CLI-V3.0.3
Content-Length: 0
Content-Type: application/x-www-form-urlencoded
x-sdk-client: golang/1.0.0
x-sdk-core-version: 0.0.1
x-sdk-invoke-type: common
Accept-Encoding: gzip
When I take parameters from the above request and generate signature it does match. So I tried all tree: GET, POST with URL params and POST with params in body. Every time I am getting a signature error. If I redo the request with exact same params as the golang tool, I'm getting nonce already used error (as expected).
Finally got this working. The main issue in my case was that I have been double-percent-encoding the signature parameter thus it turned out invalid. What helped me most was running the aliyun cli utility and capturing traffic, then running a query with exactly the same parameters to compare the exact query string.
But let me list some key points for me:
once hmac-sha1 sig is generated, do not percent-encode it, just add it to the query with normal form www encoding
order of parameters in the HTTP query is not significant; order of parameters in the signing string is significant though
I find all the following types of requests to work: GET, POST with parameters in URL query, POST with parameters in request body form www encoded; I'm using GET per documentation but I see aliyun using POST vs query params and ordered params in the query
you must add & character to the end of the secret key when generating HMAC-SHA1
generate HMAC-SHA1 in binary form, then encode as Base64 (no hex values)
some parameters might be case insensitive, e.g. Format works both as json and JSON
I see aliyun, #wanghq and John using UUID 4 for SignatureNonce but I deferred to plain random (according to docs) because it seems to be only a replay attack protection. So cryptographically secure random number must unnecessary.
The special encoding rules for +, * and ~ seem to only apply to string for signing, not actually to encode data in such a way in the HTTP query.
I decided to not use #wanghq's wrapper as it didn't work for me as well disables certificate validation but maybe it's going to be fixed. Just I thought that queries are simple enough once signature is figured out and an additional layer of indirection is not worth it. +1 to his answer though as it was helpful to get my signature right.
Here's example ruby code to make a simple request:
require 'base64'
require 'cgi'
require 'openssl'
require 'time'
require 'rest-client'
# perform a request against Alibaba Cloud API
# #see https://www.alibabacloud.com/help/doc-detail/25489.htm
def request(action:, params: {})
api_url = "https://ecs.aliyuncs.com/"
# method = "POST"
method = "GET"
process_params!(http: method, action: action, params: params)
RestClient::Request.new(method: method, url: api_url, headers: {params: params})
# RestClient::Request.new(method: method, url: api_url, payload: params)
# RestClient::Request.new(method: method, url: api_url, payload: params.map{|k,v| "#{k}=#{CGI.escape(v)}"}.join("&"))
end
# generates the required common params for a request and adds them to params
# #return undefined
# #see https://www.alibabacloud.com/help/doc-detail/25490.htm
def process_params!(http:, action:, params:)
params.merge!({
"Action" => action,
"AccessKeyId" => config[:auth][:key_id],
"Format" => "JSON",
"Version" => "2014-05-26",
"Timestamp" => Time.now.utc.iso8601
})
sign!(http: http, action: action, params: params)
end
# generate request signature and adds to params
# #return undefined
# #see https://www.alibabacloud.com/help/doc-detail/25492.htm
def sign!(http:, action:, params:)
params.delete "Signature"
params["SignatureMethod"] = "HMAC-SHA1"
params["SignatureVersion"] = "1.0"
params["SignatureNonce"] = "#{rand(1_000_000_000_000)}"
# params["SignatureNonce"] = SecureRandom.uuid.gsub("-", "")
canonicalized_query_string = params.sort.map { |key, value|
"#{key}=#{percent_encode value}"
}.join("&")
string_to_sign = %{#{http}&#{percent_encode("/")}&#{percent_encode(canonicalized_query_string)}}
params["Signature"] = hmac_sha1(string_to_sign)
end
# #param data [String]
# #return [String]
def hmac_sha1(data, secret: config[:auth][:key_secret])
Base64.encode64(OpenSSL::HMAC.digest('sha1', "#{secret}&", data)).strip
end
# encode strings per Alibaba cloud rules for signing
# #return [String] encoded string
# #see https://www.alibabacloud.com/help/doc-detail/25492.htm
def percent_encode(str)
CGI.escape(str).gsub(?+, "%20").gsub(?*, "%2A").gsub("%7E", ?~)
end
## example call
request(action: "DescribeRegions")
Code can be simplified a little but decided to keep it very close to documentation instructions.
P.S. not sure why John deleted his answer but leaving a link above to his web page for any python guys looking for example code
Seems this aliyun ruby sdk (non official, just for reference) works. You may want to check how it's implemented.
Check how its string_to_sign looks like. I did a run and seems it's slightly different than what you provided. The params are concatenated with & instead of %26.
GET&%2F&AccessKeyId%3Dtestid&Action%3DDescribeRegions&Format%3DXML&SignatureMethod%3DHMAC-SHA1&SignatureNonce%3D3ee8c1b8-83d3-44af-a94f-4e0ad82fd6cf&SignatureVersion%3D1.0&Timestamp%3D2016-02-23T12%253A46%253A24Z&Version%3D2014-05-26
require 'rubygems'
require 'aliyun'
$DEBUG = true
options = {
:access_key_id => "k",
:access_key_secret => "s",
:service => :ecs
}
service = Aliyun::Service.new options
puts service.DescribeRegions({})
wanted to share a library I found (Python) that does everything for me w/o the need to sign the request myself.
It can also help those who wants to just copy their functions and still construct the signature on their own
I'm using this:
from aliyunsdkcore.client import AcsClient
from aliyunsdkvpc.request.v20160428.DescribeEipAddressesRequest import DescribeEipAddressesRequest
client = AcsClient(access_key, secret_key, region)
request = DescribeEipAddressesRequest()
request.set_accept_format('json')
response = client.do_action_with_exception(request) # FYI returned as Bytes
print(response)
Each section in Alibaba Cloud has its own library (just like I used: aliyunsdkvpc for EIP addresses)
And they are all listed here:
https://develop.aliyun.com/tools/sdk?#/python
This code work well
Geokit::default_units = :miles #:kms, :nms, :meters
But this code make errors
puts params[:unit] # miles
Geokit::default_units = params[:unit] #:miles, :kms, :nms, :meters
What is wrong with this?
That's because all that goes through the params is an string, if you want a symbol, then consider using .to_sym:
params = { unit: 'miles' }
p params[:unit].class # String
p params[:unit].to_sym.class # Symbol
have you confirmed that params[:unit] is actually a symbol, and not a string?
Geokit::default_units = params[:unit].to_sym
If the above solves your problem, then you didn't have a symbol in there to start with (likely, if params has been read from an HTTP request)
I want to create a ruby script which will take barcodes from a text file, search a webservice for that barcode and download the result.
First I tried to test the webservice download. In a file when I hardcode the query things work fine:
result_download = open('http://webservice.org/api/?query=barcode:78686112327', 'User-Agent' => 'UserAgent email#gmail.com').read
It all works fine.
When I try to take the barcode from a textfile and run the query I run into problems.
IO.foreach(filename) {|barcode| barcode
website = "'http://webservice.org/api/?query=barcode:"+barcode.to_str.chomp + "', 'User-Agent' => 'UserAgent email#gmail.com'"
website = website.to_s
mb_metadata = open(website).read
}
The result of this is:
/home/user/.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/open-uri.rb:37:in `initialize': No such file or directory # rb_sysopen - http://webservice.org/api/?query=barcode:78686112327', 'User-Agent' => 'UserAgent email#gmail.com' (Errno::ENOENT)
I can't figure out if this problem occurs because the string I generate somehow isn't a valid url and ruby is trying to open a non-existent file, or is the issue that I am doing all this in a for loop and the file/url doesn't exist there. I have tried using open(website).write instead of open(website).read but that produces the same error.
Any help would be much appreciated.
The error message you get explicitly states, that there is no such file:
http://webservice.org/api/?query=barcode:78686112327', 'User-Agent' => 'UserAgent email#gmail.com'.
You try to pass all the parameters to open method using 1 big string (website), which is wrong. You should do it like that.
IO.foreach(filename) do |barcode|
website = "http://webservice.org/api/?query=barcode:#{barcode.to_str.chomp}"
mb_metadata = open(website, 'User-Agent' => 'UserAgent email#gmail.com').read
end
I have the following strings:
src = "dav://w.lvh.me:3000/Home/Transit/file"
host = "w.lvh.me:3000"
What I want to obtain is "/Home/Transit/file" using those two strings
I thought of searching for host in src and delete it the first time it appears, and everything before it, but I'm not sure exactly how to do that. Or maybe there's a better way?
Any help would be appreciated!
There is a better way indeed:
require 'uri'
src = "dav://w.lvh.me:3000/Home/Transit/file"
src = URI.parse src
src.path # => "/Home/Transit/file"
When there are spaces in the string, you must pass extra step of escaping/unescaping. Fortunantly, this is simple:
require 'uri'
src = "dav://w.lvh.me:3000/Home/Transit/Folder 144/webdav_put_request"
src = URI.parse(URL.escape src)
URL.unescape(src.path) # => "/Home/Transit/Folder 144/webdav_put_request"
This should do the job:
src = "dav://w.lvh.me:3000/Home/Transit/file"
host = "w.lvh.me:3000"
result = src.sub(/.*#{host}/, '')
#=> "/Home/Transit/file"
Ruby's Net:HTTP needs to be given a full URL in order for it to connect to the server and get the file properly. By "full URL" I mean a URL including the http:// part and the trailing slash if it needs it. For instance, Net:HTTP won't connect to a URL looking like this: example.com, but will connect just fine to http://example.com/. Is there any way to make sure a URL is a full URL, and add the required parts if it isn't?
EDIT: Here is the code I am using:
parsed_url = URI.parse(url)
req = Net::HTTP::Get.new(parsed_url.path)
res = Net::HTTP.start(parsed_url.host, parsed_url.port) {|http|
http.request(req)
}
If this is only doing what the sample code shows, Open-URI would be an easier approach.
require 'open-uri'
res = open(url).read
This would do a simple check for http/https:
if !(url =~ /^https?:/i)
url = "http://" + url
end
This could be a more general one to handle multiple protocols (ftp, etc.)
if !(url =~ /^\w:/i)
url = "http://" + url
end
In order to make sure parsed_url.path gives you a proper value (it should be / when no specific path was provided), you could do something like this:
req = Net::HTTP::Get.new(parsed_url.path.empty? ? '/' : parsed_url.path)