Postman AWS S3 snapshot request Elasticsearch - elasticsearch

Hello I need some help with sending a PUT request to my ElasticSearch on AWS to create a snapshot in a S3 bucket, with POSTMAN.
I have created a S3 bucket called cb-search-es-backup.
I've created a role, and a policy for S3 (see:this post of mine for the steps I've taken).
REQUEST URL https://myelasticsearchendpoint.eu-west-1.es.amazonaws.com/
REQUEST METHOD: PUT
BODY : RAW / json
{
"type": "s3",
"settings": {
"bucket": "cb-search-es-backup", // my bucketname
"region": "eu-west-1", // region
"role_arn": "arn:aws:iam::12345676890:role/Role_ES_TO_S3" // my role arn
}
}
I've also tried the authorization type: 'AWS Signature', with access and secret key filled in.

It looks like you are not passing AWS credentials with this request.
There is a detailed guide how to make a Postman request with AWS authentication here: Use Postman to Call an API.
Your Postman window might look like this:
To do the same from python please check out Sample python client section of this documentation page, note that AWS4Auth object is created and it's passed as auth parameter to requests.put():
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
# Register repository
path = '_snapshot/my-snapshot-repo' # the Elasticsearch API endpoint
url = host + path
payload = {
...
}
headers = {"Content-Type": "application/json"}
r = requests.put(url, auth=awsauth, json=payload, headers=headers)

Related

AWS API Gateway not passing Endpoint request header even after transformation

I'm trying to pass an Authorization header through API Gateway into a Lambda function. I can see the key in the gateway logs. But, even after I transform the input with the standard script (see below), the Authorization head still doesn't make it to the Lambda function.
Any suggestions on what I'm missing?
API Log excerpt
Endpoint request headers:
{
X-Amz-Date=20220419T143450Z,
x-amzn-apigateway-api-id=?????????,
Accept=application/x-www-form-urlencoded,
User-Agent=AmazonAPIGateway_hhompg4,
Host=lambda.us-east-1.amazonaws.com,
X-Amz-Content-Sha256=??????????????????????????????????????????????????,
X-Amzn-Trace-Id=Root=1-????????-???????????????????,
x-amzn-lambda-integration-tag=abcd-4e32-1234-???????????????, Authorization=*********************************************************************************************************************************************************************************************************************************************************************************************************************************************70cc,
X-Amz-Source-Arn=arn:aws:execute-api:us-east-1:-----------------:asfd/test/POST/,
X-Amz-Security-Token=---------------------------------------// [TRUNCATED]
Method Execution / - POST - Integration Request Transformation script:
{
"method": "$context.httpMethod",
"body" : $input.json('$'),
"headers": {
#foreach($param in $input.params().header.keySet())
"$param": "$util.escapeJavaScript($input.params().header.get($param))"
#if($foreach.hasNext),#end
#end
}
}
event keys arriving to lambda function:
2022-04-19T14:29:34.457Z INFO Object.keys(event) [
'resource',
'path',
'httpMethod',
'headers',
'multiValueHeaders',
'queryStringParameters',
'multiValueQueryStringParameters',
'pathParameters',
'stageVariables',
'requestContext',
'body',
'isBase64Encoded'
]
Object.keys(event.headers)
[
'accept',
'accept-encoding',
'accept-language',
'cache-control',
'content-type',
'Host',
'origin',
'referer',
'sec-ch-ua',
'sec-ch-ua-mobile',
'sec-ch-ua-platform',
'sec-fetch-dest',
'sec-fetch-mode',
'sec-fetch-site',
'sec-fetch-user',
'upgrade-insecure-requests',
'User-Agent',
'X-Amzn-Trace-Id',
'X-Forwarded-For',
'X-Forwarded-Port',
'X-Forwarded-Proto'
]
I was able to see my request header (Authorization) in request headers but the same was not visible in the endpoint request headers. Found that you have to enable the 'Use HTTP Proxy integration' option while setting up the integration point.

How to exchange JWT token for Credentials in Cognito Identity Pool in .NET Core 3.1 WebApi

Broad Overview: I am trying to create a .Net Core 3.1 WebApi backend that is authenticated against Amazon Cognito. I want to use the Amazon-hosted sign-in page(s) provided by Cognito. I want to leverage Cognito Identity Pool to provide temporary scoped credentials for users after they have logged in. I cannot figure out how to exchange the Cognito token to create the Credentials to call AWS services.
Technology Overview
.NET Core 3.1 WebApi
Amazon Cognito User Pool for initial authentication
Amazon Identity Pool for defining permissions (Roles) for logged in users
Deployed on AWS via API Gateway + Lambda using the AWS Serverless framework (basically CloudFormation)
Currently both of the following work:
Add [Authorize] attribute to a controller endpoint and access the URL in a browser. This re-directs me to the Cognito-hosted login page and, upon successful login, returns me back to the controller/endpoint and I am authorized.
Create a separate Client application and login to AWS Cognito. Pass the JWT token in the Authorization HTTP header when calling APIs from the client and the Authorization succeeds and API access is granted.
In both cases, the access to the API is permitted however the AmazonServiceClient instances that are created in the WebApi are granted the permissions associated with the Lambda function (which is the proper behavior).
Problem
I need to create AmazonServiceClients whose credentials match the Role defined by the Cognito Identity Pool.
To do this, I need to exchange token provided by logging into Cognito User Pool for temporary credentials in the Identity Pool.
Virtually ALL examples and documentation I can find on this process define how to manually login to Cognito using the API (not the hosted web UI), and then using the API response to create a CognitoUser and then get credentials from the Identity Pool using that user.
The closest (though super brief) documentation I can find to do what I need is from AWS here: https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/cognito-creds-provider.html
// Authenticate user through Facebook
string facebookToken = GetFacebookAuthToken();
// Add Facebook login to credentials. This clears the current AWS credentials
// and retrieves new AWS credentials using the authenticated role.
credentials.AddLogin("graph.facebook.com", facebookAccessToken);
While that example uses Facebook, conceptually it should be the same for any provider (Facebook, Google, Twitter, OpenId, etc.).
My Current Attempt
I have registered CognitoAWSCredentials as a Scoped service as it is user-specific and therefore should only exist as long as the API request session exists.
RegionEndpoint region = Configuration.GetAWSOptions().Region;
services.AddScoped(_ => new CognitoAWSCredentials(Settings.CognitoIdentityPoolId, region));
I have created an event handler that gets triggered when the OpenIdConnect event 'OnTokenValidated' is fired. This happens after I login to the Cognito hosted web UI and am redirected back to my API.
In this handler I can call:
CognitoAWSCredentials creds = services.BuildServiceProvider().GetRequiredService<CognitoAWSCredentials>();
creds.AddLogin( ... ??? ...);
(note: since I'm setting all this up in the Startup.ConfigureServices(IServiceCollection services) method, I am building an IServiceProvider instance each time authentication succeeds... which may be inefficient but I haven't figured out another way to access a scoped service inside the ConfigureServices method)
All this preamble to say that I cannot find a set of values for the AddLogin call which allow this test call to succeed:
ImmutableCredentials immCreds = creds.GetCredentials();
Relevant Data Structures
In the event handler where I can call AddLogin, I have access to: Microsoft.AspNetCore.Authentication.OpenIdConnect.TokenValidatedContext which in particular contains:
Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectMessage with:
access_token
id_token
refresh_token
System.IdentityModel.Tokens.Jwt.JwtSecurityToken with:
{
{
"alg": "RS256",
"kid": "**************************"
}. {
"at_hash": "**************************",
"sub": "**************************",
"email_verified": true,
"iss": "https://cognito-idp.ca-central-1.amazonaws.com/**************************",
"cognito:username": "**************************",
"nonce": "**************************",
"aud": "**************************",
"event_id": "**************************",
"token_use": "id",
"auth_time": 1595260191,
"exp": 1595263791,
"iat": 1595260191,
"email": "**************************"
}
}
I have tried using the iss value as the providerName in AddLogin, and either the access_token or id_token but neither work.
Does anyone know what I need to use for AddLogin in order for Cognito to create Identity Pool credentials for me based upon a JWT token from a Cognito User Pool login?
unless I missed it, I haven't seen documentation that states this, but even though all the Issuer fields on the various data structures include the 'https://', you need to strip it before using the Issuer as the providerName on the AddLogin call. ugh.
CognitoAWSCredentials creds = services.BuildServiceProvider().GetRequiredService<CognitoAWSCredentials>();
string shortIssuer = tokenValidatedContext.SecurityToken.Issuer;
if (shortIssuer.StartsWith("https://", StringComparison.InvariantCultureIgnoreCase)) shortIssuer = shortIssuer.Substring("https://".Length);
if (shortIssuer.StartsWith("http://", StringComparison.InvariantCultureIgnoreCase)) shortIssuer = shortIssuer.Substring("http://".Length);
creds.AddLogin(shortIssuer, tokenValidatedContext.TokenEndpointResponse.IdToken);
now, the above code has a problem as the services.BuildServiceProvider(). part means the credentials object I modify isn't global (only local to the service provider I built here I think), but that's a different issue - just noting that in case anyone is copying this code.
services...<other authentication setup>...
.AddOpenIdConnect(OpenIdConnectDefaults.AuthenticationScheme, options =>
{
options.ClientId = Settings.CognitoClientId;
options.MetadataAddress = CognitoMetadataAddress;
options.ResponseType = OpenIdConnectResponseType.Code;
options.SaveTokens = true;
options.UsePkce = true;
options.TokenValidationParameters = new TokenValidationParameters()
{
ValidateIssuer = true,
ValidIssuers = new string[] { Settings.CognitoAuthority },
ValidateAudience = true,
ValidAudiences = new string[] { Settings.CognitoClientId }
};
options.Events = new OpenIdConnectEvents() {
OnTokenValidated = tokenValidatedContext => {
CognitoAWSCredentials creds = services.BuildServiceProvider().GetRequiredService<CognitoAWSCredentials>();
string shortIssuer = tokenValidatedContext.SecurityToken.Issuer;
if (shortIssuer.StartsWith("https://", StringComparison.InvariantCultureIgnoreCase)) shortIssuer = shortIssuer.Substring("https://".Length);
if (shortIssuer.StartsWith("http://", StringComparison.InvariantCultureIgnoreCase)) shortIssuer = shortIssuer.Substring("http://".Length);
creds.AddLogin(shortIssuer, tokenValidatedContext.TokenEndpointResponse.IdToken);
return Task.CompletedTask;
}
};
})
(some code removed to focus on specifically the OpenId Connect event and the CognitoAWSCredentials init)

AWS REST Request

Below is my request body. When I hit the request using postman it throws forbidden with 403 error code but when I remove the "paymentOkUrl": "https://ei-prelive.dasitha.com:8289/general/api-gateway/v1/upad/pb/termUrl" from request body it executes successfully. I found it blocks any thing starts from "https://". I am using AWS ECS (with EC2) to host my springboot app.
I am new to AWS can anyone tell me what should I change in the aws configurations to solve this problem.
Request:
{
"paymentProvider": "ELAVON_REALEX",
"systemId": "30",
"orderId": "26518203_3afb594b-f25e-40de-bca2-650a1a234b53",
"amount": "14100",
"currencyCode": "GBP",
"cardDetails": {
"cardType": "MASTERCARD",
"cardNumber": "5269275301730688",
"expiryDate": "0424",
"cardHolderName": "Perera H T P M"
},
"paymentOkUrl": "https://ei-prelive.dasitha.com:8289/general/api-gateway/v1/upad/pb/termUrl"
}
Response:
{403 Forbidden}

Terraform API Gateway for use with Lambda Proxy Integration

I've spent the day fighting with API Gateway and AWS Serverless Express to no avail. My goal is to deploy an API Gateway, via Terraform (v0.12), that proxies all requests to an AWS Serverless Express based lambda. The connection between API Gateway and Lambda seems to exist, but tenuous, as any invocation (from API Gateway console or Postman) respond with 502 Bad Gateway, apparently due to timeout (so states the lambda CloudWatch logs). It does not appear that the lambda code actually runs, only that it spins up ineffectually.
The API Gateway and Lambda should support path parameters and query strings:
GET /some/path/:id get by id
GET /some/path?query=param get
collection
POST /some/path create resource
PATCH /some/path/:id update resource
DELETE /some/path/:id remove resource
After several false starts, I've tried to make the API Gateway Terraform module as flexible as possible:
resource "aws_api_gateway_rest_api" "rest_api" {
name = "${var.application_name} API"
description = var.description
}
resource "aws_api_gateway_resource" "proxy" {
rest_api_id = aws_api_gateway_rest_api.rest_api.id
parent_id = aws_api_gateway_rest_api.rest_api.root_resource_id # aws_api_gateway_resource.version.id
path_part = "{proxy+}"
}
resource "aws_api_gateway_method" "method" {
rest_api_id = aws_api_gateway_rest_api.rest_api.id
resource_id = aws_api_gateway_resource.proxy.id
http_method = "ANY"
authorization = "NONE"
request_parameters = {
"method.request.path.proxy" = true
}
}
resource "aws_api_gateway_integration" "integration" {
rest_api_id = aws_api_gateway_rest_api.rest_api.id
resource_id = aws_api_gateway_resource.proxy.id
http_method = aws_api_gateway_method.method.http_method
integration_http_method = "POST"
type = "AWS_PROXY"
uri = "arn:aws:apigateway:${local.region}:lambda:path/2015-03-31/functions/${var.lambda_arn}/invocations"
}
resource "aws_api_gateway_deployment" "apig_deployment" {
depends_on = [
"aws_api_gateway_resource.proxy",
"aws_api_gateway_method.method",
"aws_api_gateway_integration.integration"
]
rest_api_id = aws_api_gateway_rest_api.rest_api.id
stage_name = var.api_stage_name
lifecycle {
create_before_destroy = true
}
}
resource "aws_lambda_permission" "apig_to_lambda" {
statement_id = "AllowExecutionFromAPIGateway"
action = "lambda:InvokeFunction"
function_name = var.function_name
principal = "apigateway.amazonaws.com"
source_arn = "arn:aws:execute-api:${local.region}:${local.account_id}:${aws_api_gateway_rest_api.rest_api.id}/*/*/*" # TODO: lock this down
}
You are getting 502 error that indicates that the response received by api gateway is not proper.
Having said that your application seems to return json response. Can you try below/confirm the settings?
Add the correct binary mime types in your lambda configuration. See here for more details.
Allow the same mime types or wildcard on api gateway side as below.
resource "aws_api_gateway_rest_api" "rest_api" {
name = "${var.application_name} API"
description = var.description
// Wildcard mimes, accept any
binary_media_types = ["*/*"]
}

Couldn't make new request verification for Slack API

I'm trying the new request verification process for Slack API on AWS Lambda but I can't produce a valid signature from a request.
The example showed in https://api.slack.com/docs/verifying-requests-from-slack is for a slash command but I'm using for an event subscription, especially, a subscription to a bot event (app_mention). Does the new process support event subscriptions as well?
If so, am I missing something?
Mapping template for Integration request in API Gateway. I can't get a raw request as the slack documentation says but did my best like this:
{
"body" : $input.body,
"headers": {
#foreach($param in $input.params().header.keySet())
"$param": "$util.escapeJavaScript($input.params().header.get($param))" #if($foreach.hasNext),#end
#end
}
}
My function for verification:
def is_valid_request(headers, body):
logger.info(f"DECODED_SECRET: {DECODED_SECRET}")
logger.info(f"DECRYPTED_SECRET: {DECRYPTED_SECRET}")
timestamp = headers.get(REQ_KEYS['timestamp'])
logger.info(f"timestamp: {timestamp}")
encoded_body = urlencode(body)
logger.info(f"encoded_body: {encoded_body}")
base_str = f"{SLACK_API_VER}:{timestamp}:{encoded_body}"
logger.info(f"base_str: {base_str}")
base_b = bytes(base_str, 'utf-8')
dgst_str = hmac.new(DECRYPTED_SECRET, base_b, digestmod=sha256).hexdigest()
sig_str = f"{SLACK_API_VER}={dgst_str}"
logger.info(f"signature: {sig_str}")
req_sig = headers.get(REQ_KEYS['sig'])
logger.info(f"req_sig: {req_sig}")
logger.info(f"comparing: {hmac.compare_digest(sig_str, req_sig)}")
return hmac.compare_digest(sig_str, req_sig)
Lambda Log in CloudWatch. I can't show the values for security reasons but it seems like each variable/constant has a reasonable value:
DECODED_SECRET: ...
DECRYPTED_SECRET: ...
timestamp: 1532011621
encoded_body: ...
base_str: v0:1532011621:token= ... &team_id= ... &api_app_id= ...
signature: v0=3 ...
req_sig: v0=1 ...
comparing: False
signature should match with req_sig but it doesn't. I guess there is something wrong with base_str = f"{SLACK_API_VER}:{timestamp}:{encoded_body}". I mean, the concatination or urlencoding of the request body, but I'm not sure. Thank you in advance!

Resources