I am trying to trigger a Lambda function - "TriggerLambda", once a file named "File.csv" is put in a folder named "Files" that is present in an s3 bucket called "Processed".
bucket_notification = s3.BucketNotification("Processed/Files")
response = bucket_notification.put(
NotificationConfiguration={'LambdaFunctionConfigurations': [
{
'LambdaFunctionArn': Lambda_arn,
'Events': [
's3:ObjectCreated:*'
],
},
]})
This code is giving an error -
"errorMessage": "Parameter validation failed:\nInvalid bucket name \"Processed/Files\": Bucket name must match the regex \"^[a-zA-Z0-9.\\-_]{1,255}$\"",
Related
The cli documentation (https://docs.aws.amazon.com/cli/latest/reference/quicksight/update-data-set-permissions.html) mentions
The ARN of an Amazon QuickSight user, group, or namespace associated with an analysis, dashboard, template, or theme. (This is common.)
for Principal but using an ARN of a principal is returning the error
An error occurred (InvalidParameterValueException) when calling the UpdateDataSetPermissions operation: Malformed principal arn:aws:quicksight:us-east-1: 11122233344:namespace/10
The command I used is
aws quicksight update-data-set-permissions --cli-input-json file://datapermissions.json
The Json file contains:
{
"AwsAccountId": "11122233344",
"DataSetId": "dummy-3d53-4e31-b6cb-c53d445ea75b",
"GrantPermissions": [
{
"Principal": "arn:aws:quicksight:us-east-1:11122233344:namespace/10",
"Actions": [
"quicksight:DescribeDataSet",
"quicksight:DescribeDataSetPermissions",
"quicksight:PassDataSet",
"quicksight:DescribeIngestion",
"quicksight:ListIngestions"
]
}
]
}
I was able to perform the task to copy data from the source bucket to a destination bucket using lambda function, however, I got an error while executing the lambda function in Step functions. Below are the steps I followed from the scratch.
Region chosen is ap-south-1
Created 2 buckets. Source bucket: start.bucket & Destination bucket: final.bucket
Created a Lambda function with the following information:
Author from scratch
Function name: CopyCopy
Runtime: Python 3.8
Had created a lambda IAM role: LambdaCopy and gave the necessary policies(S3 full access and Step functions full access) and attached it to the function.
Added a trigger and chose:
S3
Bucket: start.bucket
Event type: All object create events
I found a python code in GeeksforGeeks and applied in the code section.
import json
import boto3
s3_client=boto3.client('s3')
# lambda function to copy file from 1 s3 to another s3
def lambda_handler(event, context):
#specify source bucket
source_bucket_name=event['Records'][0]['s3']['bucket']['name']
#get object that has been uploaded
file_name=event['Records'][0]['s3']['object']['key']
#specify destination bucket
destination_bucket_name='final.bucket'
#specify from where file needs to be copied
copy_object={'Bucket':source_bucket_name,'Key':file_name}
#write copy statement
s3_client.copy_object(CopySource=copy_object,Bucket=destination_bucket_name,Key=file_name)
return {
'statusCode': 3000,
'body': json.dumps('File has been Successfully Copied')
}
- I deployed the code and it worked. Uploaded a csv file in start.bucket and it was copied to final.bucket.
Then, I created a State machine in Step functions with the following information:
Design your workflow visually
Type: Standard
Dragged the AWS Lambda between the Start and End state.
Changed its name to LambdaCopy
Integration type: Optimized
Under API Parameters, Function name(I chose the Lambda function that I had created): CopyCopy:$LATEST
Next State: End
Next and then again Next
State machine name: StepLambdaCopy
IAM Role: Create a new role (Later gave it S3 full access, Lambdafullaccess and Step function fullaccess too).
It showed error when I tried to execute it.
I know I am missing out on something. I would really appreciate the help.
Step functions now allows you to utilize the S3 Copy SDK directly completely bypassing the need for Lambda and boto3. Take a look here for more information.
So in your case you would need a simple task that looks like this:
{
"Comment": "A description of my state machine",
"StartAt": "CopyObject",
"States": {
"CopyObject": {
"Type": "Task",
"End": true,
"Parameters": {
"ServerSideEncryption": "AES256",
"Bucket.$": "$.destination_bucket",
"CopySource.$": "$.source_path",
"Key.$": "$.key"
},
"Resource": "arn:aws:states:::aws-sdk:s3:copyObject"
}
}
}
Then your input state will need to feed in the parameters you would normally use to copy a file with the copy command. Source Path, Destination Bucket, and Object Key exactly the same as the boto3 command.
Note: Your state machine IAM role will need direct S3 permissions and will need to be in the same region as the buckets.
It's always confusing what exactly you have to pass as parameters. Here is a template I use to copy the output of an Athena query. You can adapt it to your needs:
"athena_score": {
"Type": "Task",
"Resource": "arn:aws:states:::athena:startQueryExecution.sync",
"Parameters": {
"QueryExecutionContext": {
"Catalog": "${AthenaCatalog}",
"Database": "${AthenaDatabase}"
},
"QueryString": "SELECT ...",
"WorkGroup": "${AthenaWorkGroup}",
"ResultConfiguration": {
"OutputLocation": "s3://${BucketName}/${OutputPath}"
}
},
"TimeoutSeconds": 300,
"ResultPath": "$.responseBody",
"Next": "copy_csv"
},
"copy_csv": {
"Type": "Task",
"Resource": "arn:aws:states:::aws-sdk:s3:copyObject",
"Parameters": {
"Bucket": "${BucketName}",
"CopySource.$": "States.Format('/${BucketName}/${OutputPath}/{}.csv', $.responseBody.QueryExecution.QueryExecutionId)",
"Key": "${OutputPath}/latest.csv"
},
"ResultPath": "$.responseBody.CopyObject",
"Ent": "true"
}
How does one validate files uploaded straight to S3 using a Pre-Signed URL?
Normally when uploading through Laravel, you can just validate the request using rules, such as:
[
'image' => 'required|file|mimes:jpeg,png,pdf|max:2500',
]
However when using a Pre-Signed URL, the file goes straight to the S3 storage and Laravel only receives a string for the path or URL to the file/image.
Is there a way to determine rules when creating the S3 Pre-Signed URL for them to only accept certain files? Or can I get the Laravel app to retrieve the file from S3 storage afterwards and validate it using the same Validation rules?
In terms of your web app, there are no additional parameters to limit file type and such (as of now). You could implement additional client-side validation (which could help but most likely won't solve the problem).
However, you could use S3 policy statements to limit file types and other things:
Allow the s3:PutObject action only for objects that have the extension of the file type that you want
Explicitly deny the s3:PutObject action for objects that don't have the extension of the file type that you want Note: You need this
explicit deny statement to apply the file-type requirement to users
with full access to your Amazon S3 resources. The following example
bucket policy allows the s3:PutObject action only for objects with
.jpg, .png, or .gif file extensions.
Important: For the first Principal value, list the Amazon Resource
Names (ARNs) of the users that you want to grant upload permissions
to. For the Resource and NotResource values, be sure to replace
bucket-name with the name of your bucket.
{
"Version": "2012-10-17",
"Id": "Policy1464968545158",
"Statement": [
{
"Sid": "Stmt1464968483619",
"Effect": "Allow",
"Principal": {
"AWS": "IAM-USER-ARN"
},
"Action": "s3:PutObject",
"Resource": [
"arn:aws:s3:::bucket-name/*.jpg",
"arn:aws:s3:::bucket-name/*.png",
"arn:aws:s3:::bucket-name/*.gif"
]
},
{
"Sid": "Stmt1464968483619",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"NotResource": [
"arn:aws:s3:::bucket-name/*.jpg",
"arn:aws:s3:::bucket-name/*.png",
"arn:aws:s3:::bucket-name/*.gif"
]
}
]
}
Another thing is to use CORS to specify the allowed origin.
I am trying to use a wild card folder path that is being supplied by getmetadata and foreach. The actual file name within these folders ends with _Problem_1.csv however i get the following error. can anyone advise me on where i am going wrong?
{ "errorCode": "2200", "message": "ErrorCode=UserErrorSourceBlobNotExist,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The
required Blob is missing. Folder path: client-uploads/[{\"name\":\"A001\",\"type\":\"Folder\"},{\"name\":\"A002\",\"type\":\"Folder\"},{\"name\":\"A004\",\"type\":\"Folder\"},{\"name\":\"A006\",\"type\":\"Folder\"},{\"name\":\"A623\",\"type\":\"Folder\"}]/.,Source=Microsoft.DataTransfer.ClientLibrary,'",
"failureType": "UserError", "target": "Copy data1", "details": [] }
You can try having your Copy activity inside the forEach activity and having for each items dynamic expression as below, which will get the list of all Folder names (This will also include file names if any exists in the folder you are pointing in getmetadata activity).
ForEach Activity Dynamic expression:
Items : #activity('getFolderNames').output.childItems
Here are child Items from getMetaData:
{
"childItems": [
{
"name": "A001",
"type": "Folder"
},
{
"name": "A002",
"type": "Folder"
}
],
"effectiveIntegrationRuntime": "DefaultIntegrationRuntime (West US)",
"executionDuration": 0,
"durationInQueue": {
"integrationRuntimeQueue": 0
},
"billingReference": {
"activityType": "PipelineActivity",
"billableDuration": {
"Managed": 0.016666666666666666
}
}
Than you have to use the "item().name" in the wild card file path expression field of copy activity, to get the name of folder per iteration of forEach activity.
In my sample, I have tried below concat expression to point to the correct folder path name for each iteration.
Wildcard Folder path: #{Concat('input/MultipleFolders/', item().name)}
This will return:
For Iteration 1: input/MultipleFolders/A001
For Iteration 2: input/MultipleFolders/A002
Hope this helps..
I am following the suggested LUIS syntax when importing a file, but throws an error "A valid json file is required". I am not sure what it does not like about the file. I even tried a truncated file, and it does not like it. Here is complete content of my file:
[
{
"canonicalForm": "4214441",
"list": [
"SHASTA COUNTY RISK MANAGEMENT REDDING"
]
},
{
"canonicalForm": "4476278",
"list": [
"SEDGWICK UC 14533 SAN DIEGO"
]
}
]
The contents of your file looks fine, but from your screenshot, it looks like the file extension used is .txt. Change your file extension to .json, eg. insuranceco.json.