Delete Object from S3 after Upload with SDK - go

if I upload an object from dashboard, I can delete it with aws-sdk-go but if I upload with aws-sdk-go, I cannot delete object from AWS S3 and i don't have error.
This is my bucket policy:
{
"Version": "2008-10-17",
"Id": "PolicyForCloudFrontPrivateContent",
"Statement": [
{
"Sid": "1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity ..."
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::storage.test.com/*"
},
{
"Sid": "2",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::361908204985:user/caio"
},
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:DeleteObject",
"s3:DeleteObjectVersion"
],
"Resource": "arn:aws:s3:::storage.test.com/*"
}
]
}
I upload the file with this code:
_, err := s.client.PutObject(&s3.PutObjectInput{
Body: file,
Bucket: s.bucket,
Key: "test/foo/a.png",
ACL: aws.String("private"),
})
and I delete the file with this code:
_, err := s.client.DeleteObject(&s3.DeleteObjectInput{
Bucket: s.bucket,
Key: aws.String("test/foo/a.png"),
})
Why delete action fail?

Here is my code for your reference. I am using aws-sdk-go-v2.
Make sure your IAM user has sufficient S3 permissions.
import(
"context"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
func configS3() *s3.Client {
creds := credentials.NewStaticCredentialsProvider(os.Getenv("S3_ACCESS_KEY_ID"), os.Getenv("S3_SECRET_ACCESS_KEY"), "")
cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithCredentialsProvider(creds), config.WithRegion(os.Getenv("S3_REGION")))
if err != nil {
log.Fatal(err)
}
return s3.NewFromConfig(cfg)
}
func DeleteImageFromS3(echoCtx echo.Context) error {
awsClient := configS3()
input := &s3.DeleteObjectInput{
Bucket: aws.String("mybucket"),
Key: aws.String("pic.jpg"),
}
_, err := awsClient.DeleteObject(context.TODO(), input)
if err != nil {
fmt.Println("Got an error deleting item:")
fmt.Println(err)
return
}
return echoCtx.JSON(http.StatusOK, "Object Deleted Successfully")
}
Reference: https://aws.github.io/aws-sdk-go-v2/docs/code-examples/s3/deleteobject/

the same issue, after some short research about this case, I think s3 DeleteObject function it's not working properly when we want to use delete object s3.
I try to use s3manager to do it and it works, this function can make a delete marker on an s3 object (versioning enabled), and I think it is the best choice so far.

Related

SignatureDoesNotMatch on S3 PresignPutObject

I've got a golang backend in which I want to generate a signed s3 URL so that a React frontend can upload to. When using the generated URL, I'm always getting a 403 SignatureDoesNotMatch.
Here's how I generate the URL on the backend:
client := s3.NewFromConfig(aws.Config{
Region: cfg.Region,
Credentials: credentials.NewStaticCredentialsProvider(cfg.AccessKeyId, cfg.SecretAccessKey, ""),
})
presignClient := s3.NewPresignClient(client, func(opts *s3.PresignOptions) {
opts.Expires = time.Duration(60 * int(time.Second))
})
headOut, err := client.HeadBucket(ctx, &s3.HeadBucketInput{
Bucket: aws.String("bucket-name-example"),
}) // THIS RETURNS NO ERROR - AWS SECRETS ARE GOOD
req, err := presignClient.PresignPutObject(ctx, &s3.PutObjectInput{
Bucket: aws.String("bucket-name-example"),
Key: aws.String("randomFileName"),
})
return req.URL // Using this URL in the Frontend
And on the UI I do:
return await fetch(signedURLFromBackend, {
method: "PUT",
headers: {
"Content-Type": "multipart/form-data", // removing this doesn't fix it
},
body: file,
})
On AWS, I'm using an IAM user with all s3 policies right now (testing). The S3 bucket has
ACLs disabled
Following bucket policy:
{
"Version": "2012-10-17",
"Id": "Policy1675697346173",
"Statement": [
{
"Sid": "Stmt1675697339589",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::userled-assets-library/*"
}
]
}
Following CORS settings:
[
{
"AllowedHeaders": ["*"],
"AllowedMethods": ["PUT","HEAD","GET"],
"AllowedOrigins": ["*"],
"ExposeHeaders": []
}
]

InvalidCiphertextException when decrypt AWL Lambda environment variable using AWS go sdk

I'm stuck on this issue for days and have tried all the methods I can find online. I am trying to decrypt (using AWS go sdk) the AWS lambda environment variable encrypted with KMS. I encrypted the env variables in transit and at rest using the same KMS key.
Screenshot of environment variables encryption
I had also added the policy to lambda function to allow it to have permission to decrypt the variables.
The attached policy looks like the below:
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": "kms:*",
"Resource": "arn:aws:kms:us-west-2:602487775829:key/f91a9f1a-9b53-4544-xxxxxxxx",
"Condition": {
"StringEquals": {
"kms:EncryptionContext:LambdaFunctionName": "eLead"
}
}
}
}
And here is my Go code snippet for decrypting the variables(KeyId is not necessary for decrypting, I added it just to ensure it uses the same KMS key to encrypt and decrypt):
import (
"os"
"context"
"encoding/base64"
"github.com/aws/aws-sdk-go-v2/service/kms"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go-v2/config"
)
var API_key = os.Getenv("API_key")
var API_Secret = os.Getenv("API_Secret")
func Handler(ctx context.Context) error {
cfg, err := config.LoadDefaultConfig(context.TODO(),config.WithRegion("us-west-2"))
API_key_blob, err := base64.StdEncoding.DecodeString(API_key)
API_Secret_blob, err := base64.StdEncoding.DecodeString(API_Secret)
keyID := "arn:aws:kms:us-west-2:602487775829:key/f91a9f1a-9b53-4544-a05b-xxxxxx"
API_key_result, err := kmsClient.Decrypt(context.TODO(), &kms.DecryptInput{
CiphertextBlob: API_key_blob,
KeyId:aws.String(keyID),
})
API_Secret_result, err := kmsClient.Decrypt(context.TODO(),&kms.DecryptInput{
CiphertextBlob: API_Secret_blob,
KeyId:aws.String(keyID),
})
if err != nil {
fmt.Println("Got error decrypting data: ", err)
return err
}
func main() {
lambda.Start(Handler)
}
The error message is:
Got error decrypting data: operation error KMS: Decrypt, https response error StatusCode: 400, RequestID: 629a64b2-ce53-46ad-88fb-1f6ac684919e, InvalidCiphertextException:
I don't understand what else can cause this error.
Any suggestion would help! Thanks!

AWS Elasticsearch returning 403 forbidden error using ES client

I'm trying to create an index via an indexing job written in Go. I have all the access to ES cluster on AWS and using my access key and secret key.
I can easily create indices using Kibana but when I try to use Go client, it does not work and returns a 403 forbidden error.
AWS Elasticsearch Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "arn:aws:es:<region>:111111111111:domain/prod-elasticsearch/*"
}
]
}
indexing.go
package main
import (
"flag"
"fmt"
"log"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/olivere/elastic/v7"
"github.com/spf13/viper"
aws "github.com/olivere/elastic/v7/aws/v4"
)
func main() {
var (
accessKey = viper.GetString("aws.access_key")
secretKey = viper.GetString("aws.secret_key")
url = viper.GetString("elasticsearch.host")
sniff = flag.Bool("sniff", false, "Enable or disable sniffing")
region = flag.String("region", "ap-southeast-1", "AWS Region name")
)
if url == "" {
log.Fatal("please specify a URL with -url")
}
if accessKey == "" {
log.Fatal("missing -access-key or AWS_ACCESS_KEY environment variable")
}
if secretKey == "" {
log.Fatal("missing -secret-key or AWS_SECRET_KEY environment variable")
}
if *region == "" {
log.Fatal("please specify an AWS region with -region")
}
creds := credentials.NewStaticCredentials(accessKey, secretKey, "")
_, err := creds.Get()
if err != nil {
log.Fatal("Wrong credentials: ", err)
}
signingClient := aws.NewV4SigningClient(creds, *region)
// Create an Elasticsearch client
client, err := elastic.NewClient(
elastic.SetURL(url),
elastic.SetSniff(*sniff),
elastic.SetHealthcheck(*sniff),
elastic.SetHttpClient(signingClient),
)
if err != nil {
log.Fatal(err)
}
// This part gives 403 forbidden error
indices, err := client.IndexNames()
if err != nil {
log.Fatal(err)
}
// Just a status message
fmt.Println("Connection succeeded")
}
config.toml
[elasticsearch]
host = "https://vpc-prod-elasticsearch-111111.ap-southeast-1.es.amazonaws.com"
[aws]
access_key = <ACCESS_KEY>
secret_key = <SECRET_KEY>
The Go code looks good. The 403 error forbidden was coming up due to some issues in the AWS secret key.
To debug such an issue:
Check if the VPC subnet has been removed in which AWS Elasticsearch cluster is operational. If the VPC subnet has been removed, then it will always throw a 403 error.
If subnets configuration is fine, then it's most probably an issue with AWS secrets. You can create a new AWS application user for the AWS Elasticsearch and retry. This should work fine.
I solved the above issue following the above 2 points.

How to apply kubernetes workload

I'm writing a Go program that will run in a kubernetes cluster.
I want the program to apply a kubernetes workload using a yaml (yaml in json format)
import (
"encoding/json"
"fmt"
corev1 "k8s.io/api/core/v1"
)
var (
workload = `{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"name": "sleep",
},
"spec": {
"containers": [
{
"name": "sleep2",
"image": "tutum/curl",
"command": [
"/bin/sleep",
"infinity"
],
"imagePullPolicy": "Always",
"env": [
{
"name": "ENV_VAR",
"value": "i/love/sleeping"
}
]
}
]
}
}`
)
func ApplyWorkload(){
pod := corev1.Pod{}
if err := json.Unmarshal(workload, &pod); err != nil {
fmt.Errorf("%v", err)
}
// apply pod here
}
How can I apply the workload in Go?
What can I do if my workload is a deployment, should I run a switch-case for each possible workload?
You need kubernetes client-go to create pod or deployment.
import "k8s.io/client-go/kubernetes"
clientset, err := kubernetes.NewForConfig(cfg) // cfg is the *config
// .... ... .. create deployment
result, err := clientset.AppsV1().Deployments(namespaceName).Create(deployment) // deployment object
// ... .... ... create pod
result, err := clientset.CoreV1().Pods(namespaceName).Create(Pod) // Pod object

REST API with gojsonschema: first path segment in URL cannot contain colon

I'm having issues making github.com/xeipuuv/gojsonschema work for my REST API that I'm currently building.
The procedure would look like this
User sends request to /api/books/create (in this case I'm sending a PUT request)
User inputs body parameters name and content
The server converts these body parameters into readable JSON
The server tries to validate the JSON using a json schema
The server performs the request
or that is how it should work.
I get this error when trying to validate the JSON and I have no clue how to fix it.
http: panic serving [::1]:58611: parse {"name":"1","content":"2"}: first path segment in URL cannot contain colon
type CreateParams struct {
Name string
Content string
}
func Create(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
data := &CreateParams{
Name: r.Form.Get("name"),
Content: r.Form.Get("Content"),
}
jsonData, err := json.Marshal(data)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(jsonData))
schema := `{
"required": [
"Name",
"Content"
],
"properties": {
"Name": {
"$id": "#/properties/Name",
"type": "string",
"title": "The Name Schema",
"default": "",
"examples": [
"1"
],
"minLength": 3,
"pattern": "^(.*)$"
},
"Content": {
"$id": "#/properties/Content",
"type": "string",
"title": "The Content Schema",
"default": "",
"examples": [
"2"
],
"pattern": "^(.*)$"
}
}
}`
schemaLoader := gojsonschema.NewStringLoader(schema)
documentLoader := gojsonschema.NewReferenceLoader(string(jsonData))
result, err := gojsonschema.Validate(schemaLoader, documentLoader)
if err != nil {
panic(err.Error())
}
if result.Valid() {
fmt.Printf("The document is valid\n")
} else {
fmt.Printf("The document is not valid. see errors :\n")
for _, desc := range result.Errors() {
fmt.Printf("- %s\n", desc)
}
}
}
My first thought was that it breaks because r.ParseForm() outputs things in a weird way, but I'm not sure anymore.
Note that I would like to have a "universal" method as I'm dealing with all kinds of requests: GET, POST, PUT, etc. But if you have a better solution I could work with that.
Any help is appreciated!

Resources