I am trying to push a hash object to a rabbitmq .
this is how i am pushing:
msg[:gateway] = "gat1"
msg[:params] = {
:status => "2",
:token_id => "SADF3434FG2FG",
:mb_tranaction_id => "1235ABC",
:random_num => "#{2*rand(155)}"
}
AmqpClient.instance.publish("#{RABBIT_MQ_CONFIG['ipn_queue']}", msg.to_json)
on the output it looks like this :
{\"gateway\":\"gat1\",\"params\":{\"status\":\"2\",\"token_id\":\"SADF3434FG2FG\",\"mb_tranaction_id\":\"1235ABC\",\"random_num\":\"62\"}}
when I am pulling, i am trying to decode it back to a hash object like so :
msg = JSON.parse(msg)
p "worker: #{msg.inspect}"
and the output is :
"worker: {\"gateway\"=>\"skrill\", \"params\"=>{\"status\"=>\"2\", \"token_id\"=>\"SADF3434FG2FG\", \"mb_tranaction_id\"=>\"1235ABC\", \"random_num\"=>\"62\"}}"
but when I try to access one of it's values by the key (gateway) msg[:gateway]
I get a blank string...
In JSON format, keys are represented as strings - so msg[:gateway] won't work - but msg["gateway"] would.
Alternatively, you can use Rails's [with_indifferent_access][1] to be able to access the hash via symbols.
Related
Reading the HTTP API docs. My requests fail though for bad signature. From error message I can see that my string to sign is correct but looks like I can't generate the correct HMAC-SHA1 (seriously why use SHA1 still??).
So I decided to try replicate the signature of the sample inside same document.
[47] pry(main)> to_sign = "GET&%2F&AccessKeyId%3Dtestid&Action%3DDescribeRegions&Format%3DXML&SignatureMethod%3DHMAC-SHA1&SignatureNonce%3D3ee8c1b8-83d3-44af-a94f-4e0ad82fd6cf&SignatureVersion%3D1.0&Timestamp%3D2016-02-23T12%253A46%253A24Z&Version%3D2014-05-26"
[48] pry(main)> Base64.encode64 OpenSSL::HMAC.digest("sha1", "testsecret", to_sign)
=> "MLAxpXej4jJ7TL0smgWpOgynR7s=\n"
[49] pry(main)> Base64.encode64 OpenSSL::HMAC.digest("sha1", "testsecret&", to_sign)
=> "VyBL52idtt+oImX0NZC+2ngk15Q=\n"
[50] pry(main)> Base64.encode64 OpenSSL::HMAC.hexdigest("sha1", "testsecret&", to_sign)
=> "NTcyMDRiZTc2ODlkYjZkZmE4MjI2NWY0MzU5MGJlZGE3ODI0ZDc5NA==\n"
[51] pry(main)> Base64.encode64 OpenSSL::HMAC.hexdigest("sha1", "testsecret", to_sign)
=> "MzBiMDMxYTU3N2EzZTIzMjdiNGNiZDJjOWEwNWE5M2EwY2E3NDdiYg==\n"
[52] pry(main)> OpenSSL::HMAC.hexdigest("sha1", "testsecret&", to_sign)
=> "57204be7689db6dfa82265f43590beda7824d794"
[53] pry(main)> OpenSSL::HMAC.hexdigest("sha1", "testsecret", to_sign)
=> "30b031a577a3e2327b4cbd2c9a05a93a0ca747bb"
As evident none of these matches the example signature of CT9X0VtwR86fNWSnsc6v8YGOjuE=. Any idea what is missing here?
Update: taking tcpdump from the Golang client tool I see that it does a POST request like:
POST /?AccessKeyId=**********&Action=DescribeRegions&Format=JSON&RegionId=cn-qingdao&Signature=aHZVpIMb0%2BFKdoWSIVaFJ7bd2LA%3D&SignatureMethod=HMAC-SHA1&SignatureNonce=c29a0e28964c470a8997aebca4848b57&SignatureType=&SignatureVersion=1.0&Timestamp=2018-07-16T19%3A46%3A33Z&Version=2014-05-26 HTTP/1.1
Host: ecs.aliyuncs.com
User-Agent: Aliyun-CLI-V3.0.3
Content-Length: 0
Content-Type: application/x-www-form-urlencoded
x-sdk-client: golang/1.0.0
x-sdk-core-version: 0.0.1
x-sdk-invoke-type: common
Accept-Encoding: gzip
When I take parameters from the above request and generate signature it does match. So I tried all tree: GET, POST with URL params and POST with params in body. Every time I am getting a signature error. If I redo the request with exact same params as the golang tool, I'm getting nonce already used error (as expected).
Finally got this working. The main issue in my case was that I have been double-percent-encoding the signature parameter thus it turned out invalid. What helped me most was running the aliyun cli utility and capturing traffic, then running a query with exactly the same parameters to compare the exact query string.
But let me list some key points for me:
once hmac-sha1 sig is generated, do not percent-encode it, just add it to the query with normal form www encoding
order of parameters in the HTTP query is not significant; order of parameters in the signing string is significant though
I find all the following types of requests to work: GET, POST with parameters in URL query, POST with parameters in request body form www encoded; I'm using GET per documentation but I see aliyun using POST vs query params and ordered params in the query
you must add & character to the end of the secret key when generating HMAC-SHA1
generate HMAC-SHA1 in binary form, then encode as Base64 (no hex values)
some parameters might be case insensitive, e.g. Format works both as json and JSON
I see aliyun, #wanghq and John using UUID 4 for SignatureNonce but I deferred to plain random (according to docs) because it seems to be only a replay attack protection. So cryptographically secure random number must unnecessary.
The special encoding rules for +, * and ~ seem to only apply to string for signing, not actually to encode data in such a way in the HTTP query.
I decided to not use #wanghq's wrapper as it didn't work for me as well disables certificate validation but maybe it's going to be fixed. Just I thought that queries are simple enough once signature is figured out and an additional layer of indirection is not worth it. +1 to his answer though as it was helpful to get my signature right.
Here's example ruby code to make a simple request:
require 'base64'
require 'cgi'
require 'openssl'
require 'time'
require 'rest-client'
# perform a request against Alibaba Cloud API
# #see https://www.alibabacloud.com/help/doc-detail/25489.htm
def request(action:, params: {})
api_url = "https://ecs.aliyuncs.com/"
# method = "POST"
method = "GET"
process_params!(http: method, action: action, params: params)
RestClient::Request.new(method: method, url: api_url, headers: {params: params})
# RestClient::Request.new(method: method, url: api_url, payload: params)
# RestClient::Request.new(method: method, url: api_url, payload: params.map{|k,v| "#{k}=#{CGI.escape(v)}"}.join("&"))
end
# generates the required common params for a request and adds them to params
# #return undefined
# #see https://www.alibabacloud.com/help/doc-detail/25490.htm
def process_params!(http:, action:, params:)
params.merge!({
"Action" => action,
"AccessKeyId" => config[:auth][:key_id],
"Format" => "JSON",
"Version" => "2014-05-26",
"Timestamp" => Time.now.utc.iso8601
})
sign!(http: http, action: action, params: params)
end
# generate request signature and adds to params
# #return undefined
# #see https://www.alibabacloud.com/help/doc-detail/25492.htm
def sign!(http:, action:, params:)
params.delete "Signature"
params["SignatureMethod"] = "HMAC-SHA1"
params["SignatureVersion"] = "1.0"
params["SignatureNonce"] = "#{rand(1_000_000_000_000)}"
# params["SignatureNonce"] = SecureRandom.uuid.gsub("-", "")
canonicalized_query_string = params.sort.map { |key, value|
"#{key}=#{percent_encode value}"
}.join("&")
string_to_sign = %{#{http}&#{percent_encode("/")}&#{percent_encode(canonicalized_query_string)}}
params["Signature"] = hmac_sha1(string_to_sign)
end
# #param data [String]
# #return [String]
def hmac_sha1(data, secret: config[:auth][:key_secret])
Base64.encode64(OpenSSL::HMAC.digest('sha1', "#{secret}&", data)).strip
end
# encode strings per Alibaba cloud rules for signing
# #return [String] encoded string
# #see https://www.alibabacloud.com/help/doc-detail/25492.htm
def percent_encode(str)
CGI.escape(str).gsub(?+, "%20").gsub(?*, "%2A").gsub("%7E", ?~)
end
## example call
request(action: "DescribeRegions")
Code can be simplified a little but decided to keep it very close to documentation instructions.
P.S. not sure why John deleted his answer but leaving a link above to his web page for any python guys looking for example code
Seems this aliyun ruby sdk (non official, just for reference) works. You may want to check how it's implemented.
Check how its string_to_sign looks like. I did a run and seems it's slightly different than what you provided. The params are concatenated with & instead of %26.
GET&%2F&AccessKeyId%3Dtestid&Action%3DDescribeRegions&Format%3DXML&SignatureMethod%3DHMAC-SHA1&SignatureNonce%3D3ee8c1b8-83d3-44af-a94f-4e0ad82fd6cf&SignatureVersion%3D1.0&Timestamp%3D2016-02-23T12%253A46%253A24Z&Version%3D2014-05-26
require 'rubygems'
require 'aliyun'
$DEBUG = true
options = {
:access_key_id => "k",
:access_key_secret => "s",
:service => :ecs
}
service = Aliyun::Service.new options
puts service.DescribeRegions({})
wanted to share a library I found (Python) that does everything for me w/o the need to sign the request myself.
It can also help those who wants to just copy their functions and still construct the signature on their own
I'm using this:
from aliyunsdkcore.client import AcsClient
from aliyunsdkvpc.request.v20160428.DescribeEipAddressesRequest import DescribeEipAddressesRequest
client = AcsClient(access_key, secret_key, region)
request = DescribeEipAddressesRequest()
request.set_accept_format('json')
response = client.do_action_with_exception(request) # FYI returned as Bytes
print(response)
Each section in Alibaba Cloud has its own library (just like I used: aliyunsdkvpc for EIP addresses)
And they are all listed here:
https://develop.aliyun.com/tools/sdk?#/python
I am using a Sinatra app to retrieve an item from DynamoDB. I am receiving # signs where integers are stored. I am using the most recent AWS SDK. hk is my hash key. This may be done with the newer version.
My DynamoDB looks like this:
hk A B
2015-06-13 4 4
My code in my route looks like:
resp = dynamo_client.get_item(
# required
table_name: "test",
# required
key: {
"hk" => Date.today.to_s
})
resp.data[:item]
What I get when hitting the URL is:
["A", #]["hk", "2015-06-13"]["B", #]
Why do I get the '#' signs instead of the values?
I'm calling collection update from ruby driver to mongodb and gets a return code 117.
How do I generally interpret the error codes that I get?
If you are using safe mode, the update method returns a hash containing the output of getLastError. However, when you are not using safe mode, we simply return the number of bytes that were sent to the server.
# setup connection & get handle to collection
connection = Mongo::Connection.new
collection = connection['test']['test']
# remove existing documents
collection.remove
=> true
# insert test document
collection.insert(:_id => 1, :a => 1)
=> 1
collection.find_one
=> {"_id"=>1, "a"=>1}
# we sent a message with 64 bytes to a mongod
collection.update({_id: 1},{a: 2.0})
=> 64 # number of bytes sent to server
# with safe mode we updated one document -- output of getLastError command
collection.update({_id: 1},{a: 3.0}, :safe => true)
=> {"updatedExisting"=>true, "n"=>1, "connectionId"=>19, "err"=>nil, "ok"=>1.0}
This is something that could be made clearer in the documentation. I will update it for the next ruby driver release.
Has anybody succeeded in sending standalone attachments using the standalone attachment API (if possible gziped) to couchDB from ruby?
I know there are a few CURL examples, but my tries with Typhoeus haven't been successful so far. It usually just stops and waits for > 1 minute after the first few documents.
CouchRest doesn't seem to support it, neither do any of the other libraries I've looked at
edit: clarification
I'm NOT looking for regular Base64 encoded attachments. CouchRest does that pretty fine.
Got it working with typhoeus
Typhoeus::Request.put("http://127.0.0.1:5984/db/document/my_attachment_name?rev=#{rev}", "content-type" => "text/html", "Content-Encoding" => "gzip", "Accept-Encoding" => "gzip", :body => my_html_body)
this will store the "my_html_body" string into couchDB as a gziped standalone attachment
For a binary standalone attachment, I just used IO.read("/path/to/my/file") to give a string to the put method as the :body. It looks like it is working but I don't know if it is the right way to do it.
It looks like this:
res = Typhoeus::Request.get("http://localhost:5984/_uuids")
uuid = JSON.parse(res.body)["uuids"].first
doc = {}
doc["name"] = name
...
res = Typhoeus::Request.put("http://localhost:5984/products/#{uuid}", :body => JSON.generate(doc))
res = Typhoeus::Request.put("http://localhost:5984/products/#{uuid}/image.jpg?rev=#{rev}", :headers => {"Content-Type" => "image/jpeg" }, :body => IO.read("output/images/#{image}"))
I simply want to upload an image to a server with POST. As simple as this task sounds, there seems to be no simple solution in Ruby.
In my application I am using WWW::Mechanize for most things so I wanted to use it for this too, and had a source like this:
f = File.new(filename, File::RDWR)
reply = agent.post(
'http://rest-test.heroku.com',
{
:pict => f,
:function => 'picture2',
:username => #username,
:password => #password,
:pict_to => 0,
:pict_type => 0
}
)
f.close
This results in a totally garbage-ready file on the server that looks scrambled all over:
alt text http://imagehub.org/f/1tk8/garbage.png
My next step was to downgrade WWW::Mechanize to version 0.8.5. This worked until I tried to run it, which failed with an error like "Module not found in hpricot_scan.so". Using the Dependency Walker tool I could find out that hpricot_scan.so needed msvcrt-ruby18.dll. Yet after I put that .dll into my Ruby/bin-folder it gave me an empty error box from where on I couldn't debug very much further. So the problem here is that Mechanize 0.8.5 has a dependency on Hpricot instead of Nokogiri (which works flawlessly).
The next idea was to use a different gem, so I tried using Net::HTTP. After short research I could find out that there is no native support for multipart forms in Net::HTTP and instead you have to build a class that encodes etc. for you. The most helpful I could find was the Multipart-class by Stanislav Vitvitskiy. This class looked good so far, but it does not do what I need, because I don't want to post only files, I also want to post normal data, and that is not possible with his class.
My last attempt was to use RestClient. This looked promising, as there have been examples on how to upload files. Yet I can't get it to post the form as multipart.
f = File.new(filename, File::RDWR)
reply = RestClient.post(
'http://rest-test.heroku.com',
:pict => f,
:function => 'picture2',
:username => #username,
:password => #password,
:pict_to => 0,
:pict_type => 0
)
f.close
I am using http://rest-test.heroku.com which sends back the request to debug if it is sent correctly, and I always get this back:
POST http://rest-test.heroku.com/ with a 101 byte payload,
content type application/x-www-form-urlencoded
{
"pict" => "#<File:0x30d30c4>",
"username" => "s1kx",
"pict_to" => "0",
"function" => "picture2",
"pict_type" => "0",
"password" => "password"
}
This clearly shows that it does not use multipart/form-data as content-type but the standard application/x-www-form-urlencoded, although it definitely sees that pict is a file.
How can I upload a file in Ruby to a multipart form without implementing the whole encoding and data aligning myself?
Long problem, short answer: I was missing the binary mode for reading the image under Windows.
f = File.new(filename, File::RDWR)
had to be
f = File.new(filename, "rb")
Another method is to use Bash and Curl. I used this method when I wanted to test multiple file uploads.
bash_command = 'curl -v -F "file=#texas.png,texas_reversed.png"
http://localhost:9292/fog_upload/upload'
command_result = `#{bash_command}` # the backticks are important <br/>
puts command_result