if I upload an object from dashboard, I can delete it with aws-sdk-go but if I upload with aws-sdk-go, I cannot delete object from AWS S3 and i don't have error.
This is my bucket policy:
{
"Version": "2008-10-17",
"Id": "PolicyForCloudFrontPrivateContent",
"Statement": [
{
"Sid": "1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity ..."
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::storage.test.com/*"
},
{
"Sid": "2",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::361908204985:user/caio"
},
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:DeleteObject",
"s3:DeleteObjectVersion"
],
"Resource": "arn:aws:s3:::storage.test.com/*"
}
]
}
I upload the file with this code:
_, err := s.client.PutObject(&s3.PutObjectInput{
Body: file,
Bucket: s.bucket,
Key: "test/foo/a.png",
ACL: aws.String("private"),
})
and I delete the file with this code:
_, err := s.client.DeleteObject(&s3.DeleteObjectInput{
Bucket: s.bucket,
Key: aws.String("test/foo/a.png"),
})
Why delete action fail?
Here is my code for your reference. I am using aws-sdk-go-v2.
Make sure your IAM user has sufficient S3 permissions.
import(
"context"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
func configS3() *s3.Client {
creds := credentials.NewStaticCredentialsProvider(os.Getenv("S3_ACCESS_KEY_ID"), os.Getenv("S3_SECRET_ACCESS_KEY"), "")
cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithCredentialsProvider(creds), config.WithRegion(os.Getenv("S3_REGION")))
if err != nil {
log.Fatal(err)
}
return s3.NewFromConfig(cfg)
}
func DeleteImageFromS3(echoCtx echo.Context) error {
awsClient := configS3()
input := &s3.DeleteObjectInput{
Bucket: aws.String("mybucket"),
Key: aws.String("pic.jpg"),
}
_, err := awsClient.DeleteObject(context.TODO(), input)
if err != nil {
fmt.Println("Got an error deleting item:")
fmt.Println(err)
return
}
return echoCtx.JSON(http.StatusOK, "Object Deleted Successfully")
}
Reference: https://aws.github.io/aws-sdk-go-v2/docs/code-examples/s3/deleteobject/
the same issue, after some short research about this case, I think s3 DeleteObject function it's not working properly when we want to use delete object s3.
I try to use s3manager to do it and it works, this function can make a delete marker on an s3 object (versioning enabled), and I think it is the best choice so far.
Related
I've got a golang backend in which I want to generate a signed s3 URL so that a React frontend can upload to. When using the generated URL, I'm always getting a 403 SignatureDoesNotMatch.
Here's how I generate the URL on the backend:
client := s3.NewFromConfig(aws.Config{
Region: cfg.Region,
Credentials: credentials.NewStaticCredentialsProvider(cfg.AccessKeyId, cfg.SecretAccessKey, ""),
})
presignClient := s3.NewPresignClient(client, func(opts *s3.PresignOptions) {
opts.Expires = time.Duration(60 * int(time.Second))
})
headOut, err := client.HeadBucket(ctx, &s3.HeadBucketInput{
Bucket: aws.String("bucket-name-example"),
}) // THIS RETURNS NO ERROR - AWS SECRETS ARE GOOD
req, err := presignClient.PresignPutObject(ctx, &s3.PutObjectInput{
Bucket: aws.String("bucket-name-example"),
Key: aws.String("randomFileName"),
})
return req.URL // Using this URL in the Frontend
And on the UI I do:
return await fetch(signedURLFromBackend, {
method: "PUT",
headers: {
"Content-Type": "multipart/form-data", // removing this doesn't fix it
},
body: file,
})
On AWS, I'm using an IAM user with all s3 policies right now (testing). The S3 bucket has
ACLs disabled
Following bucket policy:
{
"Version": "2012-10-17",
"Id": "Policy1675697346173",
"Statement": [
{
"Sid": "Stmt1675697339589",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::userled-assets-library/*"
}
]
}
Following CORS settings:
[
{
"AllowedHeaders": ["*"],
"AllowedMethods": ["PUT","HEAD","GET"],
"AllowedOrigins": ["*"],
"ExposeHeaders": []
}
]
I'm stuck on this issue for days and have tried all the methods I can find online. I am trying to decrypt (using AWS go sdk) the AWS lambda environment variable encrypted with KMS. I encrypted the env variables in transit and at rest using the same KMS key.
Screenshot of environment variables encryption
I had also added the policy to lambda function to allow it to have permission to decrypt the variables.
The attached policy looks like the below:
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": "kms:*",
"Resource": "arn:aws:kms:us-west-2:602487775829:key/f91a9f1a-9b53-4544-xxxxxxxx",
"Condition": {
"StringEquals": {
"kms:EncryptionContext:LambdaFunctionName": "eLead"
}
}
}
}
And here is my Go code snippet for decrypting the variables(KeyId is not necessary for decrypting, I added it just to ensure it uses the same KMS key to encrypt and decrypt):
import (
"os"
"context"
"encoding/base64"
"github.com/aws/aws-sdk-go-v2/service/kms"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go-v2/config"
)
var API_key = os.Getenv("API_key")
var API_Secret = os.Getenv("API_Secret")
func Handler(ctx context.Context) error {
cfg, err := config.LoadDefaultConfig(context.TODO(),config.WithRegion("us-west-2"))
API_key_blob, err := base64.StdEncoding.DecodeString(API_key)
API_Secret_blob, err := base64.StdEncoding.DecodeString(API_Secret)
keyID := "arn:aws:kms:us-west-2:602487775829:key/f91a9f1a-9b53-4544-a05b-xxxxxx"
API_key_result, err := kmsClient.Decrypt(context.TODO(), &kms.DecryptInput{
CiphertextBlob: API_key_blob,
KeyId:aws.String(keyID),
})
API_Secret_result, err := kmsClient.Decrypt(context.TODO(),&kms.DecryptInput{
CiphertextBlob: API_Secret_blob,
KeyId:aws.String(keyID),
})
if err != nil {
fmt.Println("Got error decrypting data: ", err)
return err
}
func main() {
lambda.Start(Handler)
}
The error message is:
Got error decrypting data: operation error KMS: Decrypt, https response error StatusCode: 400, RequestID: 629a64b2-ce53-46ad-88fb-1f6ac684919e, InvalidCiphertextException:
I don't understand what else can cause this error.
Any suggestion would help! Thanks!
I'm trying to create an index via an indexing job written in Go. I have all the access to ES cluster on AWS and using my access key and secret key.
I can easily create indices using Kibana but when I try to use Go client, it does not work and returns a 403 forbidden error.
AWS Elasticsearch Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "arn:aws:es:<region>:111111111111:domain/prod-elasticsearch/*"
}
]
}
indexing.go
package main
import (
"flag"
"fmt"
"log"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/olivere/elastic/v7"
"github.com/spf13/viper"
aws "github.com/olivere/elastic/v7/aws/v4"
)
func main() {
var (
accessKey = viper.GetString("aws.access_key")
secretKey = viper.GetString("aws.secret_key")
url = viper.GetString("elasticsearch.host")
sniff = flag.Bool("sniff", false, "Enable or disable sniffing")
region = flag.String("region", "ap-southeast-1", "AWS Region name")
)
if url == "" {
log.Fatal("please specify a URL with -url")
}
if accessKey == "" {
log.Fatal("missing -access-key or AWS_ACCESS_KEY environment variable")
}
if secretKey == "" {
log.Fatal("missing -secret-key or AWS_SECRET_KEY environment variable")
}
if *region == "" {
log.Fatal("please specify an AWS region with -region")
}
creds := credentials.NewStaticCredentials(accessKey, secretKey, "")
_, err := creds.Get()
if err != nil {
log.Fatal("Wrong credentials: ", err)
}
signingClient := aws.NewV4SigningClient(creds, *region)
// Create an Elasticsearch client
client, err := elastic.NewClient(
elastic.SetURL(url),
elastic.SetSniff(*sniff),
elastic.SetHealthcheck(*sniff),
elastic.SetHttpClient(signingClient),
)
if err != nil {
log.Fatal(err)
}
// This part gives 403 forbidden error
indices, err := client.IndexNames()
if err != nil {
log.Fatal(err)
}
// Just a status message
fmt.Println("Connection succeeded")
}
config.toml
[elasticsearch]
host = "https://vpc-prod-elasticsearch-111111.ap-southeast-1.es.amazonaws.com"
[aws]
access_key = <ACCESS_KEY>
secret_key = <SECRET_KEY>
The Go code looks good. The 403 error forbidden was coming up due to some issues in the AWS secret key.
To debug such an issue:
Check if the VPC subnet has been removed in which AWS Elasticsearch cluster is operational. If the VPC subnet has been removed, then it will always throw a 403 error.
If subnets configuration is fine, then it's most probably an issue with AWS secrets. You can create a new AWS application user for the AWS Elasticsearch and retry. This should work fine.
I solved the above issue following the above 2 points.
I'm writing a Go program that will run in a kubernetes cluster.
I want the program to apply a kubernetes workload using a yaml (yaml in json format)
import (
"encoding/json"
"fmt"
corev1 "k8s.io/api/core/v1"
)
var (
workload = `{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"name": "sleep",
},
"spec": {
"containers": [
{
"name": "sleep2",
"image": "tutum/curl",
"command": [
"/bin/sleep",
"infinity"
],
"imagePullPolicy": "Always",
"env": [
{
"name": "ENV_VAR",
"value": "i/love/sleeping"
}
]
}
]
}
}`
)
func ApplyWorkload(){
pod := corev1.Pod{}
if err := json.Unmarshal(workload, &pod); err != nil {
fmt.Errorf("%v", err)
}
// apply pod here
}
How can I apply the workload in Go?
What can I do if my workload is a deployment, should I run a switch-case for each possible workload?
You need kubernetes client-go to create pod or deployment.
import "k8s.io/client-go/kubernetes"
clientset, err := kubernetes.NewForConfig(cfg) // cfg is the *config
// .... ... .. create deployment
result, err := clientset.AppsV1().Deployments(namespaceName).Create(deployment) // deployment object
// ... .... ... create pod
result, err := clientset.CoreV1().Pods(namespaceName).Create(Pod) // Pod object
I'm having issues making github.com/xeipuuv/gojsonschema work for my REST API that I'm currently building.
The procedure would look like this
User sends request to /api/books/create (in this case I'm sending a PUT request)
User inputs body parameters name and content
The server converts these body parameters into readable JSON
The server tries to validate the JSON using a json schema
The server performs the request
or that is how it should work.
I get this error when trying to validate the JSON and I have no clue how to fix it.
http: panic serving [::1]:58611: parse {"name":"1","content":"2"}: first path segment in URL cannot contain colon
type CreateParams struct {
Name string
Content string
}
func Create(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
data := &CreateParams{
Name: r.Form.Get("name"),
Content: r.Form.Get("Content"),
}
jsonData, err := json.Marshal(data)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(jsonData))
schema := `{
"required": [
"Name",
"Content"
],
"properties": {
"Name": {
"$id": "#/properties/Name",
"type": "string",
"title": "The Name Schema",
"default": "",
"examples": [
"1"
],
"minLength": 3,
"pattern": "^(.*)$"
},
"Content": {
"$id": "#/properties/Content",
"type": "string",
"title": "The Content Schema",
"default": "",
"examples": [
"2"
],
"pattern": "^(.*)$"
}
}
}`
schemaLoader := gojsonschema.NewStringLoader(schema)
documentLoader := gojsonschema.NewReferenceLoader(string(jsonData))
result, err := gojsonschema.Validate(schemaLoader, documentLoader)
if err != nil {
panic(err.Error())
}
if result.Valid() {
fmt.Printf("The document is valid\n")
} else {
fmt.Printf("The document is not valid. see errors :\n")
for _, desc := range result.Errors() {
fmt.Printf("- %s\n", desc)
}
}
}
My first thought was that it breaks because r.ParseForm() outputs things in a weird way, but I'm not sure anymore.
Note that I would like to have a "universal" method as I'm dealing with all kinds of requests: GET, POST, PUT, etc. But if you have a better solution I could work with that.
Any help is appreciated!