Reading the HTTP API docs. My requests fail though for bad signature. From error message I can see that my string to sign is correct but looks like I can't generate the correct HMAC-SHA1 (seriously why use SHA1 still??).
So I decided to try replicate the signature of the sample inside same document.
[47] pry(main)> to_sign = "GET&%2F&AccessKeyId%3Dtestid&Action%3DDescribeRegions&Format%3DXML&SignatureMethod%3DHMAC-SHA1&SignatureNonce%3D3ee8c1b8-83d3-44af-a94f-4e0ad82fd6cf&SignatureVersion%3D1.0&Timestamp%3D2016-02-23T12%253A46%253A24Z&Version%3D2014-05-26"
[48] pry(main)> Base64.encode64 OpenSSL::HMAC.digest("sha1", "testsecret", to_sign)
=> "MLAxpXej4jJ7TL0smgWpOgynR7s=\n"
[49] pry(main)> Base64.encode64 OpenSSL::HMAC.digest("sha1", "testsecret&", to_sign)
=> "VyBL52idtt+oImX0NZC+2ngk15Q=\n"
[50] pry(main)> Base64.encode64 OpenSSL::HMAC.hexdigest("sha1", "testsecret&", to_sign)
=> "NTcyMDRiZTc2ODlkYjZkZmE4MjI2NWY0MzU5MGJlZGE3ODI0ZDc5NA==\n"
[51] pry(main)> Base64.encode64 OpenSSL::HMAC.hexdigest("sha1", "testsecret", to_sign)
=> "MzBiMDMxYTU3N2EzZTIzMjdiNGNiZDJjOWEwNWE5M2EwY2E3NDdiYg==\n"
[52] pry(main)> OpenSSL::HMAC.hexdigest("sha1", "testsecret&", to_sign)
=> "57204be7689db6dfa82265f43590beda7824d794"
[53] pry(main)> OpenSSL::HMAC.hexdigest("sha1", "testsecret", to_sign)
=> "30b031a577a3e2327b4cbd2c9a05a93a0ca747bb"
As evident none of these matches the example signature of CT9X0VtwR86fNWSnsc6v8YGOjuE=. Any idea what is missing here?
Update: taking tcpdump from the Golang client tool I see that it does a POST request like:
POST /?AccessKeyId=**********&Action=DescribeRegions&Format=JSON&RegionId=cn-qingdao&Signature=aHZVpIMb0%2BFKdoWSIVaFJ7bd2LA%3D&SignatureMethod=HMAC-SHA1&SignatureNonce=c29a0e28964c470a8997aebca4848b57&SignatureType=&SignatureVersion=1.0&Timestamp=2018-07-16T19%3A46%3A33Z&Version=2014-05-26 HTTP/1.1
Host: ecs.aliyuncs.com
User-Agent: Aliyun-CLI-V3.0.3
Content-Length: 0
Content-Type: application/x-www-form-urlencoded
x-sdk-client: golang/1.0.0
x-sdk-core-version: 0.0.1
x-sdk-invoke-type: common
Accept-Encoding: gzip
When I take parameters from the above request and generate signature it does match. So I tried all tree: GET, POST with URL params and POST with params in body. Every time I am getting a signature error. If I redo the request with exact same params as the golang tool, I'm getting nonce already used error (as expected).
Finally got this working. The main issue in my case was that I have been double-percent-encoding the signature parameter thus it turned out invalid. What helped me most was running the aliyun cli utility and capturing traffic, then running a query with exactly the same parameters to compare the exact query string.
But let me list some key points for me:
once hmac-sha1 sig is generated, do not percent-encode it, just add it to the query with normal form www encoding
order of parameters in the HTTP query is not significant; order of parameters in the signing string is significant though
I find all the following types of requests to work: GET, POST with parameters in URL query, POST with parameters in request body form www encoded; I'm using GET per documentation but I see aliyun using POST vs query params and ordered params in the query
you must add & character to the end of the secret key when generating HMAC-SHA1
generate HMAC-SHA1 in binary form, then encode as Base64 (no hex values)
some parameters might be case insensitive, e.g. Format works both as json and JSON
I see aliyun, #wanghq and John using UUID 4 for SignatureNonce but I deferred to plain random (according to docs) because it seems to be only a replay attack protection. So cryptographically secure random number must unnecessary.
The special encoding rules for +, * and ~ seem to only apply to string for signing, not actually to encode data in such a way in the HTTP query.
I decided to not use #wanghq's wrapper as it didn't work for me as well disables certificate validation but maybe it's going to be fixed. Just I thought that queries are simple enough once signature is figured out and an additional layer of indirection is not worth it. +1 to his answer though as it was helpful to get my signature right.
Here's example ruby code to make a simple request:
require 'base64'
require 'cgi'
require 'openssl'
require 'time'
require 'rest-client'
# perform a request against Alibaba Cloud API
# #see https://www.alibabacloud.com/help/doc-detail/25489.htm
def request(action:, params: {})
api_url = "https://ecs.aliyuncs.com/"
# method = "POST"
method = "GET"
process_params!(http: method, action: action, params: params)
RestClient::Request.new(method: method, url: api_url, headers: {params: params})
# RestClient::Request.new(method: method, url: api_url, payload: params)
# RestClient::Request.new(method: method, url: api_url, payload: params.map{|k,v| "#{k}=#{CGI.escape(v)}"}.join("&"))
end
# generates the required common params for a request and adds them to params
# #return undefined
# #see https://www.alibabacloud.com/help/doc-detail/25490.htm
def process_params!(http:, action:, params:)
params.merge!({
"Action" => action,
"AccessKeyId" => config[:auth][:key_id],
"Format" => "JSON",
"Version" => "2014-05-26",
"Timestamp" => Time.now.utc.iso8601
})
sign!(http: http, action: action, params: params)
end
# generate request signature and adds to params
# #return undefined
# #see https://www.alibabacloud.com/help/doc-detail/25492.htm
def sign!(http:, action:, params:)
params.delete "Signature"
params["SignatureMethod"] = "HMAC-SHA1"
params["SignatureVersion"] = "1.0"
params["SignatureNonce"] = "#{rand(1_000_000_000_000)}"
# params["SignatureNonce"] = SecureRandom.uuid.gsub("-", "")
canonicalized_query_string = params.sort.map { |key, value|
"#{key}=#{percent_encode value}"
}.join("&")
string_to_sign = %{#{http}&#{percent_encode("/")}&#{percent_encode(canonicalized_query_string)}}
params["Signature"] = hmac_sha1(string_to_sign)
end
# #param data [String]
# #return [String]
def hmac_sha1(data, secret: config[:auth][:key_secret])
Base64.encode64(OpenSSL::HMAC.digest('sha1', "#{secret}&", data)).strip
end
# encode strings per Alibaba cloud rules for signing
# #return [String] encoded string
# #see https://www.alibabacloud.com/help/doc-detail/25492.htm
def percent_encode(str)
CGI.escape(str).gsub(?+, "%20").gsub(?*, "%2A").gsub("%7E", ?~)
end
## example call
request(action: "DescribeRegions")
Code can be simplified a little but decided to keep it very close to documentation instructions.
P.S. not sure why John deleted his answer but leaving a link above to his web page for any python guys looking for example code
Seems this aliyun ruby sdk (non official, just for reference) works. You may want to check how it's implemented.
Check how its string_to_sign looks like. I did a run and seems it's slightly different than what you provided. The params are concatenated with & instead of %26.
GET&%2F&AccessKeyId%3Dtestid&Action%3DDescribeRegions&Format%3DXML&SignatureMethod%3DHMAC-SHA1&SignatureNonce%3D3ee8c1b8-83d3-44af-a94f-4e0ad82fd6cf&SignatureVersion%3D1.0&Timestamp%3D2016-02-23T12%253A46%253A24Z&Version%3D2014-05-26
require 'rubygems'
require 'aliyun'
$DEBUG = true
options = {
:access_key_id => "k",
:access_key_secret => "s",
:service => :ecs
}
service = Aliyun::Service.new options
puts service.DescribeRegions({})
wanted to share a library I found (Python) that does everything for me w/o the need to sign the request myself.
It can also help those who wants to just copy their functions and still construct the signature on their own
I'm using this:
from aliyunsdkcore.client import AcsClient
from aliyunsdkvpc.request.v20160428.DescribeEipAddressesRequest import DescribeEipAddressesRequest
client = AcsClient(access_key, secret_key, region)
request = DescribeEipAddressesRequest()
request.set_accept_format('json')
response = client.do_action_with_exception(request) # FYI returned as Bytes
print(response)
Each section in Alibaba Cloud has its own library (just like I used: aliyunsdkvpc for EIP addresses)
And they are all listed here:
https://develop.aliyun.com/tools/sdk?#/python
Related
I'm prototyping a Rails application to upload documents to FSCrawler (running the REST interface), to incorporate into an Elasticsearch index. Using their example, this works:
response = `curl -F "file=##{params[:document][:upload].tempfile.path}" "http://127.0.0.1:8080/fscrawler/_upload?debug=true"`
The file gets uploaded, and the content gets indexed. This is an example of what I get:
"{\n \"ok\" : true,\n \"filename\" : \"RackMultipart20200130-91061-16swulg.pdf\",\n \"url\" : \"http://127.0.0.1:9200/local/_doc/d661edecf3e28572676e97a6f0d1d\",\n \"doc\" : {\n \"content\" : \"\\n \\n \\n\\nBasically, what you need to know is that Dante is all IP-based, and makes use of common IT standards. Each Dante device behaves \\n\\nmuch like any other network device you would already find on your network. \\n\\nIn order to make integration into an existing network easy, here are some of the things that Dante does: \\n\\n▪ Dante...
When I run curl at the command line, I get EVERYTHING, like the "filename" being properly set. If I use it as above, in the Rails controller, as you can see, the filename is set to the Tempfile's filename. That's not a workable solution. Trying to use params[:document][:upload].tempfile (without .path) or just params[:document][:upload] both fail entirely.
I'm trying to do this "the right way," but every incarnation of using a proper HTTP client to do this fails. I can't figure out how to invoke an HTTP POST that will submit a file to FSCrawler the way curl (on the command line) does it.
In this example, I'm just trying to send the file by using the Tempfile file object. For some reason, FSCrawler gives me the error in the comment, and get a little metadata, but no content is indexed:
## Failed to extract [100000] characters of text for ...
## org.apache.tika.exception.ZeroByteFileException: InputStream must have > 0 bytes
uri = URI("http://127.0.0.1:8080/fscrawler/_upload?debug=true")
request = Net::HTTP::Post.new(uri)
form_data = [['file', params[:document][:upload].tempfile,
{ filename: params[:document][:upload].original_filename,
content_type: params[:document][:upload].content_type }]]
request.set_form form_data, 'multipart/form-data'
response = Net::HTTP.start(uri.hostname, uri.port) do |http|
http.request(request)
end
If I change the above to use params[:document][:upload].tempfile.path, then I don't get the error about the InputStream, but I also (still) do not get any content indexed. This is an example of what I get:
{"_index":"local","_type":"_doc","_id":"72c9ecf2a83440994eb87d28786e6","_version":3,"_seq_no":26,"_primary_term":1,"found":true,"_source":{"content":"/var/folders/bn/pcc1h8p16tl534pw__fdz2sw0000gn/T/RackMultipart20200130-91061-134tcxn.pdf\n","meta":{},"file":{"extension":"pdf","content_type":"text/plain; charset=ISO-8859-1","indexing_date":"2020-01-30T15:33:45.481+0000","filename":"Similarity in Postgres and Rails using Trigrams · pganalyze.pdf"},"path":{"virtual":"Similarity in Postgres and Rails using Trigrams · pganalyze.pdf","real":"Similarity in Postgres and Rails using Trigrams · pganalyze.pdf"}}}
If I try to use RestClient, and I try send the file by referencing the actual path to the Tempfile, then I get this error message, and I get nothing:
## Unsupported media type
response = RestClient.post 'http://127.0.0.1:8080/fscrawler/_upload?debug=true',
file: params[:document][:upload].tempfile.path,
content_type: params[:document][:upload].content_type
If I try to .read() the file, and submit that, then I break the FSCrawler form:
## Internal server error
request = RestClient::Request.new(
:method => :post,
:url => 'http://127.0.0.1:8080/fscrawler/_upload?debug=true',
:payload => {
:multipart => true,
:file => File.read(params[:document][:upload].tempfile),
:content_type => params[:document][:upload].content_type
})
response = request.execute
Obviously, I've been trying this every way I can, but I can't replicate whatever curl is doing with any known Ruby-based HTTP clients. I'm utterly lost as to how to get Ruby to submit data to FSCrawler in a way that will get the document contents indexed properly. I've been at this far longer than I care to admit. What am I missing here?
I finally tried Faraday, and, based on this answer, came up with the following:
connection = Faraday.new('http://127.0.0.1:8080') do |f|
f.request :multipart
f.request :url_encoded
f.adapter :net_http
end
file = Faraday::UploadIO.new(
params[:document][:upload].tempfile.path,
params[:document][:upload].content_type,
params[:document][:upload].original_filename
)
payload = { :file => file }
response = connection.post('/fscrawler/_upload', payload)
Using Fiddler helped me to see the results of my attempts, as I got closer and closer to the curl request. This snippet posts the request almost exactly as curl does. To route this call through the proxy, I just needed to add , proxy: 'http://localhost:8866' to the end of the connection setup.
So this is related to an earlier post I made on this method. This is essentially what I am using to send files via hipchat:
#!/usr/bin/env ruby
require 'hipchat'
client = HipChat::Client.new('HIPCHAT_TOKEN', :api_version => 'v2', :server_url => 'HIPCHAT_URL')
client.user('some_username').send_file('message', File.open('./output/some-file.csv') )
client['some_hipchat_room'].send_file('some_user', 'message', File.open('./output/some-file.csv') )
Now for some reason the send_file method is invalid:
/path/to/gems/hipchat-1.5.4/lib/hipchat/errors.rb:40:in `response_code_to_exception_for': You requested an invalid method. path:https://hipchat.illum.io/v2/user/myuser#myemail/share/file?auth_token=asdfgibberishasdf method:Net::HTTP::Get (HipChat::MethodNotAllowed)
from /path/to/gems/gems/hipchat-1.5.4/lib/hipchat/user.rb:50:in `send_file'
I think this indicating that you should be using POST instead of GET, but I'm not sure because I haven't used this library nor Hipchat.
Looking at the question you referenced and the source posted by another user they're sending the request using self.class.post, and your debug output shows Net::HTTP::Get
To debug, could you try,
file = Tempfile.new('foo').tap do |f|
f.write("the content")
f.rewind
end
user = client.user(some_username)
user.send_file('some bytes', file)
The issue is that I was attempting to connect to the server via http instead of https. If the following client is causing issues:
client = HipChat::Client.new('HIPCHAT_TOKEN', :api_version => 'v2', :server_url => 'my.company.com')
Then try adding https:// to the beginning of your company's name.
client = HipChat::Client.new('HIPCHAT_TOKEN', :api_version => 'v2', :server_url => 'https://my.company.com')
I decode (secret_key,client_id, path) into signature by following code :
require 'rubygems'
require 'base64'
require 'cgi'
require 'hmac-sha1'
#client_id = "asdkasdlda"
#secret = "3fdsdsfxds"
binary_key = Base64.decode64(#secret)
params.update({"client" => #client_id})
path = uri_path + "?" + params.collect{|k,v| "#{k}=#{v}"}.inject{|initial,cur| initial + "&" + cur}
digest = HMAC::SHA1.new(binary_key).update(path).digest
digest = Base64.encode64(digest).gsub(/[+\/]/, {"+" => "-", "/" => "_"}).delete("=")
return "#{path}&sig=#{digest}"
So, this code generates sig and path. we send request to server in following way:
/api/v1/customers/sign_in.json?user[email]=amit1656789#gmail.com&user[password]=[FILTERED]&client=asdkasdlda&sig=JSdP5xUHhgS8ZbKApBOIlsJKg_Q
Now, on server side, i want to decode this params["sign"] into app_id, secret_key and path means reverse process of above code. But i am not found any reverse process of this. Means
(app_id, secret, path) => "signature"
"signature" => (app_id, secret, path) /* Here i stuck */
First thing you should know:
"signature" => (app_id, secret, path)
This is not possible. It is not how MACs of any kind work. The signature does not contain the data. Signatures are meant to be sent alongside the message that they sign.
For secure HMAC, you should never send the secret with the message that you sign. It is also not possible to figure out a secret from the signature, except by repeatedly guessing what the value might be.
The usual way to confirm a signature is to follow the same process on the server, signing the same message, using the same secret (which the server should already have), and compare the signatures. You have made it difficult for yourself because you have signed the params as you sent them, and then put the signature on the end. You have to re-construct the message.
First, you need to use whatever web server library you can to get the request URI including the query string
signed_uri = "/api/v1/customers/sign_in.json?user[email]=amit1656789#gmail.com&user[password]=[FILTERED]&client=asdkasdlda&sig=JSdP5xUHhgS8ZbKApBOIlsJKg_Q"
Then split it into the message and its signature (I'll leave that to you, but just a regular expression ought to work):
message = "/api/v1/customers/sign_in.json?user[email]=amit1656789#gmail.com&user[password]=[FILTERED]&client=asdkasdlda"
signature = "JSdP5xUHhgS8ZbKApBOIlsJKg_Q"
To decode this signature back to the original digest (for easy comparison), just reverse the replace and encoding you did at the end on the client:
client_digest = Base64.decode64(
signature.gsub(/[-_]/, {"-" => "+", "_" => "/"}) )
Then on the server (where you should already have a value for #secret), calculate what you expect the signature to be:
#secret = '3fdsdsfxds'
binary_key = Base64.decode64(#secret)
server_digest = HMAC::SHA1.new(binary_key).update( message ).digest
if server_digest == client_digest
puts "The message was signed correctly"
else
puts "ERROR: The message or signature is not correct!"
end
Is there any library in Ruby that generates the Signature, 'X-PAYPAL-AUTHORIZATION' header that is required to make calls on behalf of the account holder who has authorized us through the paypal Permissions API.
I am done with the permissions flow and get the required access token, tokenSecret. I feel I am generating the signature incorrectly as all my calls with the the generated 'X-PAYPAL-AUTHORIZATION' fail. They give the following errors:
For NVP call I get:
You do not have permissions to make this API call
And for the GetBasicPersonalData call I get:
Authentication failed. API credentials are incorrect.
Has anyone gone through this in Ruby? What is best way to generate signature. Paypal has just provided some SDK in Paypal, Java, but not the algorithm to generate signature.
Thanks,
Nilesh
Take a look at the PayPal Permissions gem.
https://github.com/moshbit/paypal_permissions
Specifically lib/paypal_permissions/x_pp_authorization.rb
require 'cgi'
require 'openssl'
require 'base64'
class Hash
def to_paypal_permissions_query
collect do |key, value|
"#{key}=#{value}"
end.sort * '&'
end
end
module ActiveMerchant #:nodoc:
module Billing #:nodoc:
module XPPAuthorization
public
def x_pp_authorization_header url, api_user_id, api_password, access_token, access_token_verifier
timestamp = Time.now.to_i.to_s
signature = x_pp_authorization_signature url, api_user_id, api_password, timestamp, access_token, access_token_verifier
{ 'X-PAYPAL-AUTHORIZATION' => "token=#{access_token},signature=#{signature},timestamp=#{timestamp}" }
end
public
def x_pp_authorization_signature url, api_user_id, api_password, timestamp, access_token, access_token_verifier
# no query params, but if there were, this is where they'd go
query_params = {}
key = [
paypal_encode(api_password),
paypal_encode(access_token_verifier),
].join("&")
params = query_params.dup.merge({
"oauth_consumer_key" => api_user_id,
"oauth_version" => "1.0",
"oauth_signature_method" => "HMAC-SHA1",
"oauth_token" => access_token,
"oauth_timestamp" => timestamp,
})
sorted_query_string = params.to_paypal_permissions_query
base = [
"POST",
paypal_encode(url),
paypal_encode(sorted_query_string)
].join("&")
base = base.gsub /%([0-9A-F])([0-9A-F])/ do
"%#{$1.downcase}#{$2.downcase}" # hack to match PayPal Java SDK bit for bit
end
digest = OpenSSL::HMAC.digest('sha1', key, base)
Base64.encode64(digest).chomp
end
# The PayPalURLEncoder java class percent encodes everything other than 'a-zA-Z0-9 _'.
# Then it converts ' ' to '+'.
# Ruby's CGI.encode takes care of the ' ' and '*' to satisfy PayPal
# (but beware, URI.encode percent encodes spaces, and does nothing with '*').
# Finally, CGI.encode does not encode '.-', which we need to do here.
def paypal_encode str
s = str.dup
CGI.escape(s).gsub('.', '%2E').gsub('-', '%2D')
end
end
end
end
Sample parameters:
url = 'https://svcs.sandbox.paypal.com/Permissions/GetBasicPersonalData'
api_user_id = 'caller_1234567890_biz_api1.yourdomain.com'
api_password = '1234567890'
access_token = 'YJGjMOmTUqVPlKOd1234567890-jdQV3eWCOLuCQOyDK1234567890'
access_token_verifier = 'PgUjnwsMhuuUuZlPU1234567890'
The X-PAYPAL-AUTHORIZATION header [is] generated with URL "https://svcs.paypal.com/Permissions/GetBasicPersonalData". (see page 23, and chapter 7, at the link)
NVP stating "You do not have permissions to make this API call" means your API credentials are correct, just that your account does not have permission for the particular API you are trying to call. Something between the two calls you are submitting is not using the same API credentials.
For NVP call I get:
What NVP call?
TransactionSearch (see comments below)
Also, if you haven't already done so, you will want to use the sandbox APP-ID for testing in the sandbox, and you will need to apply for an app-id with Developer Technical Services (DTS) at PayPal to get an App-ID for live.
EDIT:
To use the TransactionSearch API, all you should be submitting is below. You do not need to specify any extra headers.
USER=xxxxxxxxxxxxxxxxxx
PWD=xxxxxxxxxxxxxxxxxx
SIGNATURE=xxxxxxxxxxxxxxxxxx
METHOD=TransactionSearch
VERSION=86.0
STARTDATE=2009-10-11T00:00:00Z
TRANSACTIONID=1234567890
//And for submitting API calls on bob's behalf, if his PayPal email was bob#bob.com:
SUBJECT=bob#bob.com
I just getting started with OAuth, and I tried to make a small client to connect to some webservices... I tried twitter and it worked like a charm, however, I also tried to access Yelp V2 API (following their Python example) but I always get back as an answers:
HTTP 400 Bad Request
Missing parameter: oauth_consumer_key
Here's my code:
require 'rubygems'
require 'oauth'
CONSUMER_KEY = "MY_CONSUMER_KEY"
SECRET = "MY_CONSUMER_SECRET"
TOKEN = "MY_TOKEN"
TOKEN_SECRET = "MY_TOKEN_SECRET"
consumer = OAuth::Consumer.new( CONSUMER_KEY,SECRET, {:site => "http://api.yelp.com", :signature_method => "HMAC-SHA1", :scheme => :header})
access_token = OAuth::AccessToken.new( consumer, TOKEN,TOKEN_SECRET)
p access_token.get("/v2/search?location=new+york").body
Regardless to say, that code works with twitter API without any problem (I actually followed twitter's example code)
Cheers and thanks in advance,
Ze
Use :query_string instead of :header and everything will work (at least for me).
Same code, using Signet:
require 'signet/oauth_1/client'
client = Signet::OAuth1::Client.new(
:consumer_key => 'MY_CONSUMER_KEY',
:consumer_secret => 'MY_CONSUMER_SECRET',
:access_token_key => 'MY_TOKEN_KEY',
:access_token_secret => 'MY_TOKEN_SECRET'
)
response = client.fetch_protected_resource(
:uri => 'http://api.yelp.com/v2/search?location=new+york'
)
# The Rack response format is used here
status, headers, body = response
p body
As per the Yelp documentation, the OAuth parameters do not have to be passed in the query string. The fact that the accepted answer resolved the issue indicates to me that there's probably a bug in the oauth gem causing this.