I am using AWS version 4 signing process for creating signature for AWS lambda service with API ListFunctions.I am getting correct URL which on postman worked.But when running that python file I am getting 403 status error which tells "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.".Can anyone help me solve the issue?thanks in advance!!
python file :
# Copyright 2010-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# This file is licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License. A copy of the
# License is located at
#
# http://aws.amazon.com/apache2.0/
#
# This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
# OF ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
#
# ABOUT THIS PYTHON SAMPLE: This sample is part of the AWS General Reference
# Signing AWS API Requests top available at
# https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
#
# AWS Version 4 signing example
# EC2 API (DescribeRegions)
# See: http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
# This version makes a GET request and passes the signature
# in the Authorization header.
import sys, os, base64, datetime, hashlib, hmac
import requests # pip install requests
# ************* REQUEST VALUES *************
method = 'GET'
service = 'lambda'
host = 'lambda.us-east-1.amazonaws.com'
region = 'us-east-1'
endpoint = 'https://lambda.us-east-1.amazonaws.com/2015-03-31/functions'
request_parameters = 'Action=ListFunctions&Version=2015-03-31'
# Key derivation functions. See:
# http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-python
def sign(key, msg):
return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()
def getSignatureKey(key, dateStamp, regionName, serviceName):
kDate = sign(('AWS4' + key).encode('utf-8'), dateStamp)
kRegion = sign(kDate, regionName)
kService = sign(kRegion, serviceName)
kSigning = sign(kService, 'aws4_request')
return kSigning
# Read AWS access key from env. variables or configuration file. Best practice is NOT
# to embed credentials in code.
access_key = ""
secret_key = ""
if access_key is None or secret_key is None:
print('No access key is available.')
sys.exit()
# Create a date for headers and the credential string
t = datetime.datetime.utcnow()
amzdate = t.strftime('%Y%m%dT%H%M%SZ')
datestamp = t.strftime('%Y%m%d') # Date w/o time, used in credential scope
# ************* TASK 1: CREATE A CANONICAL REQUEST *************
# http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
# Step 1 is to define the verb (GET, POST, etc.)--already done.
# Step 2: Create canonical URI--the part of the URI from domain to query
# string (use '/' if no path)
canonical_uri = '/'
# Step 3: Create the canonical query string. In this example (a GET request),
# request parameters are in the query string. Query string values must
# be URL-encoded (space=%20). The parameters must be sorted by name.
# For this example, the query string is pre-formatted in the request_parameters variable.
canonical_querystring = request_parameters
# Step 4: Create the canonical headers and signed headers. Header names
# must be trimmed and lowercase, and sorted in code point order from
# low to high. Note that there is a trailing \n.
canonical_headers = 'host:' + host + '\n' + 'x-amz-date:' + amzdate + '\n'
# Step 5: Create the list of signed headers. This lists the headers
# in the canonical_headers list, delimited with ";" and in alpha order.
# Note: The request can include any headers; canonical_headers and
# signed_headers lists those that you want to be included in the
# hash of the request. "Host" and "x-amz-date" are always required.
signed_headers = 'host;x-amz-date'
# Step 6: Create payload hash (hash of the request body content). For GET
# requests, the payload is an empty string ("").
payload_hash = hashlib.sha256(('').encode('utf-8')).hexdigest()
# Step 7: Combine elements to create canonical request
canonical_request = method + '\n' + canonical_uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash
# ************* TASK 2: CREATE THE STRING TO SIGN*************
# Match the algorithm to the hashing algorithm you use, either SHA-1 or
# SHA-256 (recommended)
algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + service + '/' + 'aws4_request'
string_to_sign = algorithm + '\n' + amzdate + '\n' + credential_scope + '\n' + hashlib.sha256(canonical_request.encode('utf-8')).hexdigest()
# ************* TASK 3: CALCULATE THE SIGNATURE *************
# Create the signing key using the function defined above.
signing_key = getSignatureKey(secret_key, datestamp, region, service)
# Sign the string_to_sign using the signing_key
signature = hmac.new(signing_key, (string_to_sign).encode('utf-8'), hashlib.sha256).hexdigest()
# ************* TASK 4: ADD SIGNING INFORMATION TO THE REQUEST *************
# The signing information can be either in a query string value or in
# a header named Authorization. This code shows how to use a header.
# Create authorization header and add to request headers
authorization_header = algorithm + ' ' + 'Credential=' + access_key + '/' + credential_scope + ', ' + 'SignedHeaders=' + signed_headers + ', ' + 'Signature=' + signature
# The request can include any headers, but MUST include "host", "x-amz-date",
# and (for this scenario) "Authorization". "host" and "x-amz-date" must
# be included in the canonical_headers and signed_headers, as noted
# earlier. Order here is not significant.
# Python note: The 'host' header is added automatically by the Python 'requests' library.
headers = {'x-amz-date':amzdate, 'Authorization':authorization_header}
print(headers)
# ************* SEND THE REQUEST *************
request_url = endpoint + '?' + canonical_querystring
print('\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++')
print('Request URL = ' + request_url)
r = requests.get(request_url, headers=headers)
print('\nRESPONSE++++++++++++++++++++++++++++++++++++')
print('Response code: %d\n' % r.status_code)
print(r.text)
The problem is you didn't set up AWS Secret Access Key properly.
Solution:
Go to IAM section of your AWS account and select user or create one.
Assign specific roles needed for the project.
Navigate to Security credentials tab in your user account.
Select existing key or create new one and find AWS Secret Access Key.
You can setup AWS Secret Access Key as environmental varialbe in your system or in code.
Related
In my pure Ruby app one of the components to create a token for my request authentication to an external API is to create signature which is HMAC value that is created using the api_key and the secret_key. The signature contains the following elements that are each separated by a new line \n (except the last line) and are in the same order as below list:
ts = '1529342939277'
nonce = '883b170c-a768-41a1-ae6d-c626323aa128'
host = 'ws.idms.lexisnexis.com'
resource_path = '/restws/identity/v3/accounts/11111/workflows/rdp.test.workflow/conversations'
body_hash = 'fQoIAs0IO4vNleZVE9tcI3Ni7h+niT+GrrgEHsKZOyM='
API_KEY = '6njQLkz7uCiz1ZeJ1bFCWX4DFVTfKQXa'
SECRET_KEY = 'CcdaZEt7co647iJoEc5G29CHtlo7T9M3'
# create string signature separated by new line
signature = [ts, nonce, host, resource_path, body_hash].join("\n")
# create HMAC for signature
mac = Base64.strict_encode64(OpenSSL::HMAC.hexdigest('SHA256', API_KEY, signature))
2.7.0 :146 > mac
=> "ZDE4NDQxZDdiNmZkODNiODgyODI4Nzc2OTQ3OGFlMjVhZTMyNThhZTZlMTRiMjkxMzI0NmQ5NzljNDJkZWVhZg=="
According to the docs the signature should be Syb6i+sRygAGCgxLQJ4NwwKcT5Mnkh4r3QXgwZ3vmcE= but I'm getting ZDE4NDQxZDdiNmZkODNiODgyODI4Nzc2OTQ3OGFlMjVhZTMyNThhZTZlMTRiMjkxMzI0NmQ5NzljNDJkZWVhZg== instead. Where did I go wrong?
I've got an example how to do it in Java if something will be unclear: https://gist.github.com/mrmuscle1234/20c9d46d163fee66528449c0ea8419a7
I'm trying to create a Data Structure
# Data Structures
## Incorrect Credentials (string)
- `Insufficient privileges.`
This is where it will be used:
+ Response 401 (application/vnd.api+json)
Not allowed.
+ Attributes (object)
+ errors (array[Incorrect Credentials])
This is what it outputs. I'm trying to replace the "Hello, world!" with "Insufficient privileges.".
Just changing the string to enum should solve it. When inheriting from primitive types you can't set a default value.
# API
# A [GET /]
+ Response 401 (application/vnd.api+json)
Not allowed.
+ Attributes (object)
+ errors (array[Incorrect Credentials])
# Data Structures
## Incorrect Credentials (enum)
- Insufficient privileges.
I decode (secret_key,client_id, path) into signature by following code :
require 'rubygems'
require 'base64'
require 'cgi'
require 'hmac-sha1'
#client_id = "asdkasdlda"
#secret = "3fdsdsfxds"
binary_key = Base64.decode64(#secret)
params.update({"client" => #client_id})
path = uri_path + "?" + params.collect{|k,v| "#{k}=#{v}"}.inject{|initial,cur| initial + "&" + cur}
digest = HMAC::SHA1.new(binary_key).update(path).digest
digest = Base64.encode64(digest).gsub(/[+\/]/, {"+" => "-", "/" => "_"}).delete("=")
return "#{path}&sig=#{digest}"
So, this code generates sig and path. we send request to server in following way:
/api/v1/customers/sign_in.json?user[email]=amit1656789#gmail.com&user[password]=[FILTERED]&client=asdkasdlda&sig=JSdP5xUHhgS8ZbKApBOIlsJKg_Q
Now, on server side, i want to decode this params["sign"] into app_id, secret_key and path means reverse process of above code. But i am not found any reverse process of this. Means
(app_id, secret, path) => "signature"
"signature" => (app_id, secret, path) /* Here i stuck */
First thing you should know:
"signature" => (app_id, secret, path)
This is not possible. It is not how MACs of any kind work. The signature does not contain the data. Signatures are meant to be sent alongside the message that they sign.
For secure HMAC, you should never send the secret with the message that you sign. It is also not possible to figure out a secret from the signature, except by repeatedly guessing what the value might be.
The usual way to confirm a signature is to follow the same process on the server, signing the same message, using the same secret (which the server should already have), and compare the signatures. You have made it difficult for yourself because you have signed the params as you sent them, and then put the signature on the end. You have to re-construct the message.
First, you need to use whatever web server library you can to get the request URI including the query string
signed_uri = "/api/v1/customers/sign_in.json?user[email]=amit1656789#gmail.com&user[password]=[FILTERED]&client=asdkasdlda&sig=JSdP5xUHhgS8ZbKApBOIlsJKg_Q"
Then split it into the message and its signature (I'll leave that to you, but just a regular expression ought to work):
message = "/api/v1/customers/sign_in.json?user[email]=amit1656789#gmail.com&user[password]=[FILTERED]&client=asdkasdlda"
signature = "JSdP5xUHhgS8ZbKApBOIlsJKg_Q"
To decode this signature back to the original digest (for easy comparison), just reverse the replace and encoding you did at the end on the client:
client_digest = Base64.decode64(
signature.gsub(/[-_]/, {"-" => "+", "_" => "/"}) )
Then on the server (where you should already have a value for #secret), calculate what you expect the signature to be:
#secret = '3fdsdsfxds'
binary_key = Base64.decode64(#secret)
server_digest = HMAC::SHA1.new(binary_key).update( message ).digest
if server_digest == client_digest
puts "The message was signed correctly"
else
puts "ERROR: The message or signature is not correct!"
end
I'm trying to create a SAS url for my windows media services file using custom ruby code (ported from official php library for azure). The code looks like that:
def create_signature(path = '/', resource = 'b', permissions = 'r', start = '', expiry = '', identifier = '')
# If resource is a container, remove the last part (which is the filename)
path = path.split('/').reverse.drop(1).reverse.join('/') if resource == 'c'
canonicalizedResource = "/mediasvc78m7lfh2gnn2x/#{path}"
stringToSign = []
stringToSign << permissions
stringToSign << start
stringToSign << expiry
stringToSign << canonicalizedResource
stringToSign << identifier
stringToSign = stringToSign.join("\n")
signature = OpenSSL::HMAC.digest('sha256', wms_api_key, stringToSign.encode(Encoding::UTF_8))
signature = Base64.encode64(signature)
return signature
end
def createSignedQueryString(path = '/', query_string = '', resource = 'b', permissions = 'r', start = '', expiry = '', identifier = '')
base = 'https://mediasvc78m7lfh2gnn2x.blob.core.windows.net'
uri = Addressable::URI.new
# Parts
parts = {}
parts[:st] = URI.unescape(start) unless start == ''
parts[:se] = URI.unescape(expiry)
parts[:sr] = URI.unescape(resource)
parts[:sp] = URI.unescape(permissions)
parts[:si] = URI.unescape(identifier) unless identifier == ''
parts[:sig] = URI.unescape( create_signature(path, resource, permissions, start, expiry) )
uri.query_values = parts
return "#{base}/#{path}?#{uri.query}"
end
When running:
puts createSignedQueryString(
'asset-12514a3b-565f-4150-9543-e3c2b4531428/video.mp4',
nil,
'b',
'r',
(Time.now - 5*60).utc.iso8601,
(Time.now + 30*60).utc.iso8601
)
it gives me the following url: https://mediasvc78m7lfh2gnn2x.blob.core.windows.net/asset-12514a3b-565f-4150-9543-e3c2b4531428/video.mp4?se=2014-02-20T12%3A59%3A19Z&sig=RDc9nVMuf1dy%2BPrnzCkA8pZfgry2ZwrF08u9itf4v%2FA%3D%0A&sp=r&sr=b&st=2014-02-20T12%3A24%3A19Z
When i try to point my browser to it, i'm getting:
<Error>
<Code>AuthenticationFailed</Code>
<Message>
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:ee7fd18f-cd1f-4179-8a58-c8b746d0549c Time:2014-02-20T12:29:27.0468171Z
</Message>
<AuthenticationErrorDetail>
Signature did not match. String to sign used was r 2014-02-20T12:24:19Z 2014-02-20T12:59:19Z /mediasvc78m7lfh2gnn2x/asset-12514a3b-565f-4150-9543-e3c2b4531428/video.mp4
</AuthenticationErrorDetail>
</Error>
Have you any idea what can be causing that error (or how to debug it?) and how to deal with that? Thanks in advance.
Assuming you're using the storage key directly from the portal and using that in your wms_api_key variable (or in other words your wms_api_key is a Base64 encoded string, I believe you would need to convert it first as byte array for calculating signature. You would need to do something like:
signature = OpenSSL::HMAC.digest('sha256', Base64.strict_decode64(wms_api_key), stringToSign.encode(Encoding::UTF_8))
This is based on the source code for Azure SDK for Ruby on Github.
UPDATE
One more issue I discovered. If you notice your SAS URL, you would notice %0A at the end of sig query string parameter which is essentially a new line character. Not sure why this is coming but I think it is inserted automatically when you do the following:
signature = Base64.encode64(signature)
However if I use strict_encode64 instead of encode64 method, this is not inserted and everything works great. So try the following code:
signature = Base64.strict_encode64(signature)
I just tried it and it worked great for me.
I'm writing a program in Ruby that downloads a file from an RSS feed to my local hard drive. Previously, I'd written this application in Perl and figured a great way to learn Ruby would be to recreate this program using Ruby code.
In the Perl program (which works), I was able to download the original file directly from the server it was hosted on (keeping the original file name) and it worked great. In the Ruby program (which isn't working), I have to sort of "stream" the data from the file I want into a new file that I've created on my hard drive. Unfortunately, this isn't working and the "streamed" data is always coming back empty. My assumption is that there is some sort of redirect that Perl can handle to retrieve the file directly that Ruby cannot.
I'm going to post both programs (they're relatively small) and hope that this helps solve my issue. If you have questions, please let me know. As a side note, I pointed this program at a more static URL (a jpeg) and it downloaded the file just fine. This is why I'm theorizing that some sort of redirect is causing issues.
The Ruby Code (That Doesn't Work)
require 'net/http';
require 'open-uri';
require 'rexml/document';
require 'sqlite3';
# Create new SQLite3 database connection
db_connection = SQLite3::Database.new('fiend.db');
# Make sure I can reference records in the query result by column name instead of index number
db_connection.results_as_hash = true;
# Grab all TV shows from the shows table
query = '
SELECT
id,
name,
current_season,
last_episode
FROM
shows
ORDER BY
name
';
# Run through each record in the result set
db_connection.execute(query) { |show|
# Pad the current season number with a zero for later user in a search query
season = '%02d' % show['current_season'].to_s;
# Calculate the next episode number and pad with a zero
next_episode = '%02d' % (Integer(show['last_episode']) + 1).to_s;
# Store the name of the show
name = show['name'];
# Generate the URL of the RSS feed that will hold the list of torrents
feed_url = URI.encode("http://btjunkie.org/rss.xml?query=#{name} S#{season}E#{next_episode}&o=52");
# Generate a simple string the denotes the show, season and episode number being retrieved
episode_id = "#{name} S#{season}E#{next_episode}";
puts "Loading feed for #{name}..";
# Store the response from the download of the feed
feed_download_response = Net::HTTP.get_response(URI.parse(feed_url));
# Store the contents of the response (in this case, XML data)
xml_data = feed_download_response.body;
puts "Feed Loaded. Parsing items.."
# Create a new REXML Document and pass in the XML from the Net::HTTP response
doc = REXML::Document.new(xml_data);
# Loop through each in the feed
doc.root.each_element('//item') { |item|
# Find and store the URL of the torrent we wish to download
torrent_url = item.elements['link'].text + '/download.torrent';
puts "Downloading #{episode_id} from #{torrent_url}";
## This is where crap stops working
# Open Connection to the host
Net::HTTP.start(URI.parse(torrent_url).host, 80) { |http|
# Create a torrent file to dump the data into
File.open("#{episode_id}.torrent", 'wb') { |torrent_file|
# Try to grab the torrent data
data = http.get(torrent_url[19..torrent_url.size], "User-Agent" => "Mozilla/4.0").body;
# Write the data to the torrent file (the data is always coming back blank)
torrent_file.write(data);
# Close the torrent file
torrent_file.close();
}
}
break;
}
}
The Perl Code (That Does Work)
use strict;
use XML::Parser;
use LWP::UserAgent;
use HTTP::Status;
use DBI;
my $dbh = DBI->connect("dbi:SQLite:dbname=fiend.db", "", "", { RaiseError => 1, AutoCommit => 1 });
my $userAgent = new LWP::UserAgent; # Create new user agent
$userAgent->agent("Mozilla/4.0"); # Spoof our user agent as Mozilla
$userAgent->timeout(20); # Set timeout limit for request
my $currentTag = ""; # Stores what tag is currently being parsed
my $torrentUrl = ""; # Stores the data found in any node
my $isDownloaded = 0; # 1 or zero that states whether or not we've downloaded a particular episode
my $shows = $dbh->selectall_arrayref("SELECT id, name, current_season, last_episode FROM shows ORDER BY name");
my $id = 0;
my $name = "";
my $season = 0;
my $last_episode = 0;
foreach my $show (#$shows) {
$isDownloaded = 0;
($id, $name, $season, $last_episode) = (#$show);
$season = sprintf("%02d", $season); # Append a zero to the season (e.g. 6 becomes 06)
$last_episode = sprintf("%02d", ($last_episode + 1)); # Append a zero to the last episode (e.g. 6 becomes 06) and increment it by one
print("Checking $name S" . $season . "E" . "$last_episode \n");
my $request = new HTTP::Request(GET => "http://btjunkie.org/rss.xml?query=$name S" . $season . "E" . $last_episode . "&o=52"); # Retrieve the torrent feed
my $rssFeed = $userAgent->request($request); # Store the feed in a variable for later access
if($rssFeed->is_success) { # We retrieved the feed
my $parser = new XML::Parser(); # Make a new instance of XML::Parser
$parser->setHandlers # Set the functions that will be called when the parser encounters different kinds of data within the XML file.
(
Start => \&startHandler, # Handles start tags (e.g. )
End => \&endHandler, # Handles end tags (e.g.
Char => \&DataHandler # Handles data inside of start and end tags
);
$parser->parsestring($rssFeed->content); # Parse the feed
}
}
#
# Called every time XML::Parser encounters a start tag
# #param: $parseInstance {object} | Instance of the XML::Parser. Passed automatically when feed is parsed.
# #param: $element {string} | The name of the XML element being parsed (e.g. "title"). Passed automatically when feed is parsed.
# #attributes {array} | An array of all of the attributes of $element
# #returns: void
#
sub startHandler {
my($parseInstance, $element, %attributes) = #_;
$currentTag = $element;
}
#
# Called every time XML::Parser encounters anything that is not a start or end tag (i.e, all the data in between tags)
# #param: $parseInstance {object} | Instance of the XML::Parser. Passed automatically when feed is parsed.
# #param: $element {string} | The name of the XML element being parsed (e.g. "title"). Passed automatically when feed is parsed.
# #attributes {array} | An array of all of the attributes of $element
# #returns: void
#
sub DataHandler {
my($parseInstance, $element, %attributes) = #_;
if($currentTag eq "link" && $element ne "\n") {
$torrentUrl = $element;
}
}
#
# Called every time XML::Parser encounters an end tag
# #param: $parseInstance {object} | Instance of the XML::Parser. Passed automatically when feed is parsed.
# #param: $element {string} | The name of the XML element being parsed (e.g. "title"). Passed automatically when feed is parsed.
# #attributes {array} | An array of all of the attributes of $element
# #returns: void
#
sub endHandler {
my($parseInstance, $element, %attributes) = #_;
if($element eq "item" && $isDownloaded == 0) { # We just finished parsing an element so let's attempt to download a torrent
print("DOWNLOADING: $torrentUrl" . "/download.torrent \n");
system("echo.|lwp-download " . $torrentUrl . "/download.torrent"); # We echo the "return " key into the command to force it to skip any file-overwite prompts
if(unlink("download.torrent.html")) { # We tried to download a 'locked' torrent
$isDownloaded = 0; # Forces program to download next torrent on list from current show
}
else {
$isDownloaded = 1;
$dbh->do("UPDATE shows SET last_episode = '$last_episode' WHERE id = '$id'"); # Update DB with new show information
}
}
}
Yes, the URLs you are retrieving appear to be returning a 302 (redirect). Net::HTTP requires/allows you to handle the redirect yourself. You typically use a recursive techique like AboutRuby mentioned (although this http://www.ruby-forum.com/topic/142745 suggests you should not only look at the 'Location' field but also for META REFRESH in the response).
open-uri will handle redirects for you if you're not interested in the low-level interaction:
require 'open-uri'
File.open("#{episode_id}.torrent", 'wb') {|torrent_file| torrent_file.write open(torrent_url).read}
get_response will return a class from the HTTPResponse hierarchy. It's usually HTTPSuccess, but if there's a redirect, it will be HTTPRedirection. A simple recursive method can solve this, that follows redirects. How to handle this correctly is in the docs under the heading "Following Redirection."