We have a Github repository that stores image files.
https://github.com/rollthecloudinc/ipe-objects/tree/dev/media
We would like to serve those image files via golang. The golang api runs on aws api gateway as a lambda function. The function in its current state which goes to a blank screen is below.
func GetMediaFile(req *events.APIGatewayProxyRequest, ac *ActionContext) (events.APIGatewayProxyResponse, error) {
res := events.APIGatewayProxyResponse{StatusCode: 500}
pathPieces := strings.Split(req.Path, "/")
siteName := pathPieces[1]
file, _ := url.QueryUnescape(pathPieces[3]) // pathPieces[2]
log.Printf("requested media site: " + siteName)
log.Printf("requested media file: " + file)
// buf := aws.NewWriteAtBuffer([]byte{})
// downloader := s3manager.NewDownloader(ac.Session)
/*_, err := downloader.Download(buf, &s3.GetObjectInput{
Bucket: aws.String(ac.BucketName),
Key: aws.String("media/" + file),
})
if err != nil {
return res, err
}*/
ext := strings.Split(pathPieces[len(pathPieces)-1], ".")
contentType := mime.TypeByExtension(ext[len(ext)-1])
if ext[len(ext)-1] == "md" {
contentType = "text/markdown"
}
suffix := ""
if os.Getenv("GITHUB_BRANCH") == "master" {
suffix = "-prod"
}
var q struct {
Repository struct {
Object struct {
ObjectFragment struct {
Text string
IsBinary bool
ByteSize int
} `graphql:"... on Blob"`
} `graphql:"object(expression: $exp)"`
} `graphql:"repository(owner: $owner, name: $name)"`
}
qVars := map[string]interface{}{
"exp": githubv4.String(os.Getenv("GITHUB_BRANCH") + ":media/" + file),
"owner": githubv4.String("rollthecloudinc"),
"name": githubv4.String(siteName + suffix),
}
err := ac.GithubV4Client.Query(context.Background(), &q, qVars)
if err != nil {
log.Print("Github latest file failure.")
log.Panic(err)
}
// log.Printf(q.Repository.Object.ObjectFragment.Text)
// json.Unmarshal([]byte(q.Repository.Object.ObjectFragment.Text), &obj)
// log.Printf("END GithubFileUploadAdaptor::LOAD %s", id)
log.Print("content")
log.Print(q.Repository.Object.ObjectFragment.Text)
res.StatusCode = 200
res.Headers = map[string]string{
"Content-Type": contentType,
}
res.Body = q.Repository.Object.ObjectFragment.Text //base64.StdEncoding.EncodeToString([]byte(q.Repository.Object.ObjectFragment.Text))
res.IsBase64Encoded = true
return res, nil
}
The full api file can viewed below but excludes the changes above for migration to Github. This api has been running fine using s3. However, we are now trying to migrate to Github for object storage instead. Have successfully implemented write but are having difficulties described above for reading the file using our lambda.
https://github.com/rollthecloudinc/verti-go/blob/master/api/media/main.go
Help requested to figure out how to serve image files from our Github repo using the golang lambda on aws which can be accessed here as a blank screen.
https://81j44yaaab.execute-api.us-east-1.amazonaws.com/ipe/media/Screen%20Shot%202022-02-02%20at%202.00.29%20PM.png
However, this repo is also a pages site which serves the image just fine.
https://rollthecloudinc.github.io/ipe-objects/media/Screen%20Shot%202022-02-02%20at%202.00.29%20PM.png
Thanks
Further debugging the Text property appears to be empty inside the log.
The IsBinary property value being false lead use to the discovery of a typo. The name input for the graph QL invocation was missing -objects. Once the typo was corrected IsBinary started showing up true. However, the Text property value is still empty.
Having managed to find some similar issues but for uploading many have suggested that graph QL isn't the right tool for uploading binary data to begin with. Therefore, rather than chase tail we have decided to try the Github REST v3 api. Specifically, the go-github package for golang instead.
https://github.com/google/go-github
Perhaps using the REST api instead will lead to successful results.
An additional step was necessary to fetch the blob contents of the object queried via the graph QL api. Once this was achieved the media file was served with success. This required using the go-github blob api to fetch the blob base64 contents from github.
https://81j44yaaab.execute-api.us-east-1.amazonaws.com/ipe/media/Screen%20Shot%202022-02-02%20at%202.00.29%20PM.png
GetMediaFile lambda
func GetMediaFile(req *events.APIGatewayProxyRequest, ac *ActionContext) (events.APIGatewayProxyResponse, error) {
res := events.APIGatewayProxyResponse{StatusCode: 500}
pathPieces := strings.Split(req.Path, "/")
siteName := pathPieces[1]
file, _ := url.QueryUnescape(pathPieces[3]) // pathPieces[2]
log.Print("requested media site: " + siteName)
log.Print("requested media file: " + file)
// buf := aws.NewWriteAtBuffer([]byte{})
// downloader := s3manager.NewDownloader(ac.Session)
/*_, err := downloader.Download(buf, &s3.GetObjectInput{
Bucket: aws.String(ac.BucketName),
Key: aws.String("media/" + file),
})
if err != nil {
return res, err
}*/
ext := strings.Split(pathPieces[len(pathPieces)-1], ".")
contentType := mime.TypeByExtension(ext[len(ext)-1])
if ext[len(ext)-1] == "md" {
contentType = "text/markdown"
}
suffix := ""
if os.Getenv("GITHUB_BRANCH") == "master" {
suffix = "-prod"
}
owner := "rollthecloudinc"
repo := siteName + "-objects" + suffix
var q struct {
Repository struct {
Object struct {
ObjectFragment struct {
Oid githubv4.GitObjectID
} `graphql:"... on Blob"`
} `graphql:"object(expression: $exp)"`
} `graphql:"repository(owner: $owner, name: $name)"`
}
qVars := map[string]interface{}{
"exp": githubv4.String(os.Getenv("GITHUB_BRANCH") + ":media/" + file),
"owner": githubv4.String(owner),
"name": githubv4.String(repo),
}
err := ac.GithubV4Client.Query(context.Background(), &q, qVars)
if err != nil {
log.Print("Github latest file failure.")
log.Panic(err)
}
oid := q.Repository.Object.ObjectFragment.Oid
log.Print("Github file object id " + oid)
blob, _, err := ac.GithubRestClient.Git.GetBlob(context.Background(), owner, repo, string(oid))
if err != nil {
log.Print("Github get blob failure.")
log.Panic(err)
}
res.StatusCode = 200
res.Headers = map[string]string{
"Content-Type": contentType,
}
res.Body = blob.GetContent()
res.IsBase64Encoded = true
return res, nil
}
Full Source: https://github.com/rollthecloudinc/verti-go/blob/master/api/media/main.go
Related
Since a few days, I'm trying to download file from Huawei, and more precisely on their cloud storage. The issue is, I haven't been able to connect to it...
I found a SDK from huawei : https://github.com/huaweicloud/huaweicloud-sdk-go-v3
But I'm a little bit lost on all the protocol I can use to connect to it, and I haven't been able to make one working, each time. And to be honest, documentations isn't really helping me...
I also found this : https://github.com/huaweicloud/huaweicloud-sdk-go-obs
There is an example of downloading a file. Here, I can't even connect to Huawei... In the AppGalery, project settings, I downloaded the configuration file and tried the endpoint, but without success...
Here is what I tried with obs (I know/guess it should be agc, but I haven't found a package for it), but not working due to the host...
/**
* This sample demonstrates how to download an object
* from OBS in different ways using the OBS SDK for Go.
*/
package huawei
import (
"fmt"
"io"
"github.com/huaweicloud/huaweicloud-sdk-go-obs/obs"
)
type DownloadSample struct {
bucketName string
objectKey string
location string
obsClient *obs.ObsClient
}
func newDownloadSample(ak, sk, endpoint, bucketName, objectKey, location string) *DownloadSample {
obsClient, err := obs.New(ak, sk, endpoint)
if err != nil {
panic(err)
}
return &DownloadSample{obsClient: obsClient, bucketName: bucketName, objectKey: objectKey, location: location}
}
func (sample DownloadSample) GetObject() {
input := &obs.GetObjectInput{}
input.Bucket = sample.bucketName
input.Key = sample.objectKey
fmt.Printf("%+v\n", input)
output, err := sample.obsClient.GetObject(input)
if err != nil {
panic(err)
}
defer func() {
errMsg := output.Body.Close()
if errMsg != nil {
panic(errMsg)
}
}()
fmt.Println("Object content:")
body, err := io.ReadAll(output.Body)
if err != nil {
panic(err)
}
fmt.Println(string(body))
fmt.Println()
}
func RunDownloadSample() {
const (
endpoint = "theEndPointInConfigJSONFile"
ak = "prettySureOfThis"
sk = "prettySureOfThis"
bucketName = "prettySureOfThis"
objectKey = "test.txt" // a txt in the bucket to try to download it
location = ""
)
sample := newDownloadSample(ak, sk, endpoint, bucketName, objectKey, location)
fmt.Println("Download object to string")
sample.GetObject()
}
Thank you for your help
I'm switching from the legacy streaming API to the storage write API following this example in golang:
https://github.com/alexflint/bigquery-storage-api-example
In the old code I used bigquery's null types to indicate a field can be null:
type Person struct {
Name bigquery.NullString `bigquery:"name"`
Age bigquery.NullInt64 `bigquery:"age"`
}
var persons = []Person{
{
Name: ToBigqueryNullableString(""), // this will be null in bigquery
Age: ToBigqueryNullableInt64("20"),
},
{
Name: ToBigqueryNullableString("David"),
Age: ToBigqueryNullableInt64("60"),
},
}
func main() {
ctx := context.Background()
bigqueryClient, _ := bigquery.NewClient(ctx, "project-id")
inserter := bigqueryClient.Dataset("dataset-id").Table("table-id").Inserter()
err := inserter.Put(ctx, persons)
if err != nil {
log.Fatal(err)
}
}
func ToBigqueryNullableString(x string) bigquery.NullString {
if x == "" {
return bigquery.NullString{Valid: false}
}
return bigquery.NullString{StringVal: x, Valid: true}
}
func ToBigqueryNullableInt64(x string) bigquery.NullInt64 {
if x == "" {
return bigquery.NullInt64{Valid: false}
}
if s, err := strconv.ParseInt(x, 10, 64); err == nil {
return bigquery.NullInt64{Int64: s, Valid: true}
}
return bigquery.NullInt64{Valid: false}
}
After switching to the new API:
var persons = []*personpb.Row{
{
Name: "",
Age: 20,
},
{
Name: "David",
Age: 60,
},
}
func main() {
ctx := context.Background()
client, _ := storage.NewBigQueryWriteClient(ctx)
defer client.Close()
stream, err := client.AppendRows(ctx)
if err != nil {
log.Fatal("AppendRows: ", err)
}
var row personpb.Row
descriptor, err := adapt.NormalizeDescriptor(row.ProtoReflect().Descriptor())
if err != nil {
log.Fatal("NormalizeDescriptor: ", err)
}
var opts proto.MarshalOptions
var data [][]byte
for _, row := range persons {
buf, err := opts.Marshal(row)
if err != nil {
log.Fatal("protobuf.Marshal: ", err)
}
data = append(data, buf)
}
err = stream.Send(&storagepb.AppendRowsRequest{
WriteStream: fmt.Sprintf("projects/%s/datasets/%s/tables/%s/streams/_default", "project-id", "dataset-id", "table-id"),
Rows: &storagepb.AppendRowsRequest_ProtoRows{
ProtoRows: &storagepb.AppendRowsRequest_ProtoData{
WriterSchema: &storagepb.ProtoSchema{
ProtoDescriptor: descriptor,
},
Rows: &storagepb.ProtoRows{
SerializedRows: data,
},
},
},
})
if err != nil {
log.Fatal("AppendRows.Send: ", err)
}
_, err = stream.Recv()
if err != nil {
log.Fatal("AppendRows.Recv: ", err)
}
}
With the new API I need to define the types in a .proto file, so I need to use something else to define nullable fields, I tried with optional fields:
syntax = "proto3";
package person;
option go_package = "/personpb";
message Row {
optional string name = 1;
int64 age = 2;
}
but it gives me error when trying to stream (not in the compile time):
BqMessage.proto: person_Row.Name: The [proto3_optional=true] option may only be set on proto3fields, not person_Row.Name
Another option I tried is to use oneof, and write the proto file like this
syntax = "proto3";
import "google/protobuf/struct.proto";
package person;
option go_package = "/personpb";
message Row {
NullableString name = 1;
int64 age = 2;
}
message NullableString {
oneof kind {
google.protobuf.NullValue null = 1;
string data = 2;
}
}
Then use it like this:
var persons = []*personpb.Row{
{
Name: &personpb.NullableString{Kind: &personpb.NullableString_Null{
Null: structpb.NullValue_NULL_VALUE,
}},
Age: 20,
},
{
Name: &personpb.NullableString{Kind: &personpb.NullableString_Data{
Data: "David",
}},
Age: 60,
},
}
...
But this gives me the following error:
Invalid proto schema: BqMessage.proto: person_Row.person_NullableString.null: FieldDescriptorProto.oneof_index 0 is out of range for type "person_NullableString".
I guess because the api doesn't know how to handle oneof type, I need to tell it somehow about this.
How can I use something like bigquery.Nullable types when using the new storage API? Any help will be appreciated
Take a look at this sample for an end to end example using a proto2 syntax file in go.
proto3 is still a bit of a special beast when working with the Storage API, for a couple reasons:
The current behavior of the Storage API is to operate using proto2 semantics.
Currently, the Storage API doesn't understand wrapper types, which was the original way in which proto3 was meant to communicate optional presence (e.g. NULL in BigQuery fields). Because of this, it tends to treat wrapper fields as a submessage with a value field (in BigQuery, a STRUCT with a single leaf field).
Later in its evolution, proto3 reintroduced the optional keyword as a way of marking presence, but in the internal representation this meant adding another presence marker (the source of the proto3_optional warning you were observing in the backend error).
It looks like you've using bits of the newer veneer, particularly adapt.NormalizeDescriptor(). I suspect if you're using this, you may be using an older version of the module, as the normalization code was updated in this PR and released as part of bigquery/v1.33.0.
There's work to improve the experiences with the storage API and make the overall experience smoother, but there's still work to be done.
Using Go and AWS-SDK
I'm attempting to query route53 CNAME and A records as listed in the AWS Console under Route53 -> Hosted Zones. I'm able to query using the following code, but it requires the (cryptic) HostedZoneId I have to know ahead of time.
Is there a different function, or a HostedZoneId lookup based on the Domain Name such as XXX.XXX.com ?
AWSLogin(instance)
svc := route53.New(instance.AWSSession)
listParams := &route53.ListResourceRecordSetsInput{
HostedZoneId: aws.String("Z2798GPJN9CUFJ"), // Required
// StartRecordType: aws.String("CNAME"),
}
respList, err := svc.ListResourceRecordSets(listParams)
if err != nil {
fmt.Println(err.Error())
return
}
// Pretty-print the response data.
fmt.Println("All records:")
fmt.Println(respList)
edit: oh, additionally, the StartRecordType with the value "CNAME" throws a validation error, so I'm not sure what I should be using there.
You first have to do a lookup to get the HostedZoneID. Here is the func I wrote for it. :
func GetHostedZoneIdByNameLookup(awsSession string, HostedZoneName string) (HostedZoneID string, err error) {
svc := route53.New(awsSession)
listParams := &route53.ListHostedZonesByNameInput{
DNSName: aws.String(HostedZoneName), // Required
}
req, resp := svc.ListHostedZonesByNameRequest(listParams)
err = req.Send()
if err != nil {
return "", err
}
HostedZoneID = *resp.HostedZones[0].Id
// remove the /hostedzone/ path if it's there
if strings.HasPrefix(HostedZoneID, "/hostedzone/") {
HostedZoneID = strings.TrimPrefix(HostedZoneID, "/hostedzone/")
}
return HostedZoneID, nil
}
I am attempting to create a REST API in Go. I have it partially working in that it will return 4 separate json objects like such:
[{"Name":"QA1","Server":"BOT1","Description":"Tools","Apps":""},
{"Name":"QA1","Server":"","Description":"","Apps":"Duo"},
{"Name":"QA1","Server":"","Description":"","Apps":"Git"},
{"Name":"QA1","Server":"","Description":"","Apps":"php"}]
What I want is a single returned object like:
[{"Name":"QA1","Server":"BOT1","Description":"Tools","Apps": "Duo|Git|php"}]
I obviously have the way that I am either making my queries or the structs (or both or something else) not quite correct. I want to make sure I understand how to do this right because I would like to expand on it for other queries and such down the road. I have included the "full" go code below.
To be clear, I'm not simply looking for the solution (though I would of course appreciate that to compare with), but where I've gone wrong in my thinking and what the correct approach would be.
package main
import (
"database/sql"
"encoding/json"
"fmt"
_ "github.com/go-sql-driver/mysql"
"io/ioutil"
"log"
"net/http"
)
// There can be zero or more apps on a volume
type Apps struct {
Name string
}
// Volumes have a name, description, are on a server and have multiple services/apps
type Volume struct {
Name string
Server string
Description string
Services Apps
}
//Handle all requests
func Handler(response http.ResponseWriter, request *http.Request) {
response.Header().Set("Content-type", "text/html")
webpage, err := ioutil.ReadFile("index.html")
if err != nil {
http.Error(response, fmt.Sprintf("home.html file error %v", err), 500)
}
fmt.Fprint(response, string(webpage))
}
// DB Connection
const (
DB_HOST = "mydbhost"
DB_NAME = "mydb"
DB_USER = "mydbuser"
DB_PASS = "mydbpass"
)
// Respond to URLs of the form /api
func APIHandler(response http.ResponseWriter, request *http.Request) {
//Connect to database
dsn := DB_USER + ":" + DB_PASS + "#" + DB_HOST + "/" + DB_NAME + "?charset=utf8"
db, err := sql.Open("mysql", dsn)
if err != nil {
fmt.Println(err.Error())
}
defer db.Close()
// Open doesn't open a connection. Validate DSN data:
err = db.Ping()
if err != nil {
fmt.Println(err.Error())
}
//set mime type to JSON
response.Header().Set("Content-type", "application/json")
result := []*Volume{}
switch request.Method {
case "GET":
srvrnm := request.URL.Query().Get("srvrnm")
appnm := request.URL.Query().Get("appnm")
srvrs, err := db.Prepare("select VOLUMES.name as volnm, SERVERS.name as srvrnm, VOLUMES.description as descr From VOLUMES LEFT JOIN SERVERS ON VOLUMES.server_id = SERVERS.id where SERVERS.name = ?")
if err != nil {
fmt.Print(err)
}
srvcs, err := db.Prepare("select VOLUMES.name as volnm, SUPPRTSVCS.name as app_name From VOLUMES as VOLUMES JOIN HOSTSVCS ON VOLUMES.id = HOSTSVCS.volume_id JOIN SUPPRTSVCS ON SUPPRTSVCS.id = HOSTSVCS.supportsvcs_id where VOLUMES.name = ?")
if err != nil {
fmt.Print(err)
}
// Run the SQL Query to Get Volum & Description From Hostname
srvrrows, err := srvrs.Query(srvrnm)
if err != nil {
fmt.Print(err)
}
for srvrrows.Next() {
var volnm string
var srvrnm string
var descr string
// Scan the First Query
err = srvrrows.Scan(&volnm, &srvrnm, &descr)
if err != nil {
fmt.Println("Error scanning: " + err.Error())
return
}
// Append Slice with results from the scan
result = append(result, &Volume{Name: volnm, Server: srvrnm, Description: descr})
}
// Run the SQL Query for Services/Apps
srvcrows, err := srvcs.Query(appnm)
if err != nil {
fmt.Print(err)
}
for srvcrows.Next() {
var volnm string
var appnm string
// Scan the Second Query
err = srvcrows.Scan(&volnm, &appnm)
if err != nil {
fmt.Println("Error scanning: " + err.Error())
return
}
// Append Slice with results from the scan
result = append(result, &Volume{Name: volnm, Apps: appnm})
}
default:
}
json, err := json.Marshal(result)
if err != nil {
fmt.Println(err)
return
}
fmt.Fprintf(response, string(json))
db.Close()
}
func main() {
port := "1236"
var err string
mux := http.NewServeMux()
mux.Handle("/api", http.HandlerFunc(APIHandler))
mux.Handle("/", http.HandlerFunc(Handler))
// Start listing on a given port with these routes on this server.
log.Print("Listening on port " + port + " ... ")
errs := http.ListenAndServe(":"+port, mux)
if errs != nil {
log.Fatal("ListenAndServe error: ", err)
}
}
From the sounds of it, you want to your result to look like:
[
{"Name":"QA1","Server":"BOT1","Description":"Tools","Apps": ["Duo","Git","php"]
]
Hence you want your Volumes struct to look like:
type Volume struct {
Name string
Server string
Description string
Services []Apps
}
If you want the Apps to actually output Duo|Git|php then you could create a custom type instead of []Apps with a JSON Marshaler implementation. This could simply return json.Marshal(strings.join(names,"|"))
Rather than run two separate queries, it would be more efficient to run a single query that selects the product of volumes & apps together. It is important that this query is sorted by volume so all volume rows are contiguous. Example query output would be:
Name | Server | Desc | App
---- | ------ | ----- | ---
Vol1 | Srv1 | Desc1 | App1
Vol1 | Srv1 | Desc1 | App2
Vol2 | Srv2 | Desc2 | App3
You would then loop over this and detect if you are looking at a new volume. If so, create a new entry in the result. If not, add the App to the list of apps. For example:
var (
volnm string
srvrnm string
descr string
appnm string
v *Volume
result []*Volume
)
for srvrrows.Next() {
if err = srvcrows.Scan(&volnm, &srvrnm, &descr, &appnm);err!=nil {
// Handle error
}
// Add App to current volume if same, otherwise start a new volume
if v!=nil && v.Name == volnm {
v.Services = append(v.Services,Apps{appnm})
} else {
v = &Volume{
Name: volnm,
Server: svrnm,
Description: descr,
Services: []Apps{appnm}}
result = append(result,v)
}
}
// Finished, return result etc...
When taking this approach, you need an appropriate parent record discriminator. I'd just used v.Name == volnm for illustration purposes but this should really be checking the primary key. You can make this an unexported (lowercase) field in the struct if you do not wish to export it through the API.
I have the following working code to delete an object from Amazon s3
params := &s3.DeleteObjectInput{
Bucket: aws.String("Bucketname"),
Key : aws.String("ObjectKey"),
}
s3Conn.DeleteObjects(params)
But what i want to do is to delete all files under a folder using wildcard **. I know amazon s3 doesn't treat "x/y/file.jpg" as a folder y inside x but what i want to achieve is by mentioning "x/y*" delete all the subsequent objects having the same prefix. Tried amazon multi object delete
params := &s3.DeleteObjectsInput{
Bucket: aws.String("BucketName"),
Delete: &s3.Delete{
Objects: []*s3.ObjectIdentifier {
{
Key : aws.String("x/y/.*"),
},
},
},
}
result , err := s3Conn.DeleteObjects(params)
I know in php it can be done easily by s3->delete_all_objects as per this answer. Is the same action possible in GOlang.
Unfortunately the goamz package doesn't have a method similar to the PHP library's delete_all_objects.
However, the source code for the PHP delete_all_objects is available here (toggle source view): http://docs.aws.amazon.com/AWSSDKforPHP/latest/#m=AmazonS3/delete_all_objects
Here are the important lines of code:
public function delete_all_objects($bucket, $pcre = self::PCRE_ALL)
{
// Collect all matches
$list = $this->get_object_list($bucket, array('pcre' => $pcre));
// As long as we have at least one match...
if (count($list) > 0)
{
$objects = array();
foreach ($list as $object)
{
$objects[] = array('key' => $object);
}
$batch = new CFBatchRequest();
$batch->use_credentials($this->credentials);
foreach (array_chunk($objects, 1000) as $object_set)
{
$this->batch($batch)->delete_objects($bucket, array(
'objects' => $object_set
));
}
$responses = $this->batch($batch)->send();
As you can see, the PHP code will actually make an HTTP request on the bucket to first get all files matching PCRE_ALL, which is defined elsewhere as const PCRE_ALL = '/.*/i';.
You can only delete 1000 files at once, so delete_all_objects then creates a batch function to delete 1000 files at a time.
You have to create the same functionality in your go program as the goamz package doesn't support this yet. Luckily it should only be a few lines of code, and you have a guide from the PHP library.
It might be worth submitting a pull request for the goamz package once you're done!
Using the mc tool you can do:
mc rm -r --force https://BucketName.s3.amazonaws.com/x/y
it will delete all the objects with the prefix "x/y"
You can achieve the same with Go using minio-go like this:
package main
import (
"log"
"github.com/minio/minio-go"
)
func main() {
config := minio.Config{
AccessKeyID: "YOUR-ACCESS-KEY-HERE",
SecretAccessKey: "YOUR-PASSWORD-HERE",
Endpoint: "https://s3.amazonaws.com",
}
// find Your S3 endpoint here http://docs.aws.amazon.com/general/latest/gr/rande.html
s3Client, err := minio.New(config)
if err != nil {
log.Fatalln(err)
}
isRecursive := true
for object := range s3Client.ListObjects("BucketName", "x/y", isRecursive) {
if object.Err != nil {
log.Fatalln(object.Err)
}
err := s3Client.RemoveObject("BucketName", object.Key)
if err != nil {
log.Fatalln(err)
continue
}
log.Println("Removed : " + object.Key)
}
}
Since this question was asked, the AWS GoLang lib for S3 has received some new methods in S3 Manager to handle this task (in response to #Itachi's pr).
See Github record: https://github.com/aws/aws-sdk-go/issues/448#issuecomment-309078450
Here is their example in v1: https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/go/s3/DeleteObjects/DeleteObjects.go#L36
To get "wildcard matching" on paths inside the bucket, add the Prefix param to the example's ListObjectsInput call, as shown here:
iter := s3manager.NewDeleteListIterator(svc, &s3.ListObjectsInput{
Bucket: bucket,
Prefix: aws.String("somePathString"),
})
A bit late in the game, but since I was having the same problem, I created a small pkg that you can copy to your code base and import as needed.
func ListKeysInPrefix(s s3iface.S3API, bucket, prefix string) ([]string, error) {
res, err := s.Client.ListObjectsV2(&s3.ListObjectsV2Input{
Bucket: aws.String(bucket),
Prefix: aws.String(prefix),
})
if err != nil {
return []string{}, err
}
var keys []string
for _, key := range res.Contents {
keys = append(keys, *key.Key)
}
return keys, nil
}
func createDeleteObjectsInput(keys []string) *s3.Delete {
rm := []*s3.ObjectIdentifier{}
for _, key := range keys {
rm = append(rm, &s3.ObjectIdentifier{Key: aws.String(key)})
}
return &s3.Delete{Objects: rm, Quiet: aws.Bool(false)}
}
func DeletePrefix(s s3iface.S3API, bucket, prefix string) error {
keys, err := s.ListKeysInPrefix(bucket, prefix)
if err != nil {
panic(err)
}
_, err = s.Client.DeleteObjects(&s3.DeleteObjectsInput{
Bucket: aws.String(bucket),
Delete: s.createDeleteObjectsInput(keys),
})
if err != nil {
return err
}
return nil
}
So, in the case you have a bucket called "somebucket" with the following structure: s3://somebucket/foo/some-prefixed-folder/bar/test.txt and wanted to delete from some-prefixed-folder onwards, usage would be:
func main() {
// create your s3 client here
// client := ....
err := DeletePrefix(client, "somebucket", "some-prefixed-folder")
if err != nil {
panic(err)
}
}
This implementation only allows to delete a maximum of 1000 entries from the given prefix due ListObjectsV2 implementation - but it is paginated, so it's a matter of adding the functionality to keep refreshing results until results are < 1000.