Download a file from Huawei cloud using official SDK - go

Since a few days, I'm trying to download file from Huawei, and more precisely on their cloud storage. The issue is, I haven't been able to connect to it...
I found a SDK from huawei : https://github.com/huaweicloud/huaweicloud-sdk-go-v3
But I'm a little bit lost on all the protocol I can use to connect to it, and I haven't been able to make one working, each time. And to be honest, documentations isn't really helping me...
I also found this : https://github.com/huaweicloud/huaweicloud-sdk-go-obs
There is an example of downloading a file. Here, I can't even connect to Huawei... In the AppGalery, project settings, I downloaded the configuration file and tried the endpoint, but without success...
Here is what I tried with obs (I know/guess it should be agc, but I haven't found a package for it), but not working due to the host...
/**
* This sample demonstrates how to download an object
* from OBS in different ways using the OBS SDK for Go.
*/
package huawei
import (
"fmt"
"io"
"github.com/huaweicloud/huaweicloud-sdk-go-obs/obs"
)
type DownloadSample struct {
bucketName string
objectKey string
location string
obsClient *obs.ObsClient
}
func newDownloadSample(ak, sk, endpoint, bucketName, objectKey, location string) *DownloadSample {
obsClient, err := obs.New(ak, sk, endpoint)
if err != nil {
panic(err)
}
return &DownloadSample{obsClient: obsClient, bucketName: bucketName, objectKey: objectKey, location: location}
}
func (sample DownloadSample) GetObject() {
input := &obs.GetObjectInput{}
input.Bucket = sample.bucketName
input.Key = sample.objectKey
fmt.Printf("%+v\n", input)
output, err := sample.obsClient.GetObject(input)
if err != nil {
panic(err)
}
defer func() {
errMsg := output.Body.Close()
if errMsg != nil {
panic(errMsg)
}
}()
fmt.Println("Object content:")
body, err := io.ReadAll(output.Body)
if err != nil {
panic(err)
}
fmt.Println(string(body))
fmt.Println()
}
func RunDownloadSample() {
const (
endpoint = "theEndPointInConfigJSONFile"
ak = "prettySureOfThis"
sk = "prettySureOfThis"
bucketName = "prettySureOfThis"
objectKey = "test.txt" // a txt in the bucket to try to download it
location = ""
)
sample := newDownloadSample(ak, sk, endpoint, bucketName, objectKey, location)
fmt.Println("Download object to string")
sample.GetObject()
}
Thank you for your help

Related

Creating and seeding a torrent in golang

I want to use the golang library https://github.com/anacrolix/torrent to create to torrent and get a magnet and seed the torrent. Below you can find the code I wrote. Yet, if I use the magnet the code generates I can not download anything not even the metainfo.
Am I missing something here?
package main
import (
"log"
"time"
"github.com/anacrolix/torrent"
"github.com/anacrolix/torrent/bencode"
"github.com/anacrolix/torrent/metainfo"
)
var builtinAnnounceList = [][]string{
{"http://p4p.arenabg.com:1337/announce"},
{"udp://tracker.opentrackr.org:1337/announce"},
{"udp://tracker.openbittorrent.com:6969/announce"},
}
func main() {
tmpComment:="Cool torrent description"
tmpCreatedBy:="coolboys"
tmpInfoName:="CoolInfoName"
mi := metainfo.MetaInfo{
AnnounceList: builtinAnnounceList,
}
mi.SetDefaults()
mi.Comment = tmpComment
mi.CreatedBy = tmpCreatedBy
//}
//mi.UrlList = args.Url//???????????
info := metainfo.Info{
PieceLength: 256 * 1024,
}
err := info.BuildFromFilePath("./TorrentFiles")//args.Root)
if err != nil {
log.Fatal(err)
}
info.Name =tmpInfoName
mi.InfoBytes, err = bencode.Marshal(info)
if err != nil {
log.Fatal(err)
}
tmpMagnet:=mi.Magnet(nil,nil)
log.Println("****",tmpMagnet)
//
cfg := torrent.NewDefaultClientConfig()
cfg.Seed = true
mainclient, ncerr := torrent.NewClient(cfg)
if ncerr != nil {
log.Println("new torrent client:", ncerr)
return
}
defer mainclient.Close()
t, _ := mainclient.AddMagnet(tmpMagnet.String())
for {
log.Println("******", t.Seeding())
time.Sleep(8 * time.Second)
}
}
I think you might need to take a closer look at this example. Among other things, I don't see any invocation of t.DownloadAll() to do the actual download or mainclient.WaitAll() to tell you when the downloads are complete.
I corrected the code I wrote and the correct code can be found here:
enter link description here
In the original code I should not have used addmagnet as it assumes that I don't have the info available, which is why it would fail to seed.

Serve Image File From Github Graph QL API Using Golang

We have a Github repository that stores image files.
https://github.com/rollthecloudinc/ipe-objects/tree/dev/media
We would like to serve those image files via golang. The golang api runs on aws api gateway as a lambda function. The function in its current state which goes to a blank screen is below.
func GetMediaFile(req *events.APIGatewayProxyRequest, ac *ActionContext) (events.APIGatewayProxyResponse, error) {
res := events.APIGatewayProxyResponse{StatusCode: 500}
pathPieces := strings.Split(req.Path, "/")
siteName := pathPieces[1]
file, _ := url.QueryUnescape(pathPieces[3]) // pathPieces[2]
log.Printf("requested media site: " + siteName)
log.Printf("requested media file: " + file)
// buf := aws.NewWriteAtBuffer([]byte{})
// downloader := s3manager.NewDownloader(ac.Session)
/*_, err := downloader.Download(buf, &s3.GetObjectInput{
Bucket: aws.String(ac.BucketName),
Key: aws.String("media/" + file),
})
if err != nil {
return res, err
}*/
ext := strings.Split(pathPieces[len(pathPieces)-1], ".")
contentType := mime.TypeByExtension(ext[len(ext)-1])
if ext[len(ext)-1] == "md" {
contentType = "text/markdown"
}
suffix := ""
if os.Getenv("GITHUB_BRANCH") == "master" {
suffix = "-prod"
}
var q struct {
Repository struct {
Object struct {
ObjectFragment struct {
Text string
IsBinary bool
ByteSize int
} `graphql:"... on Blob"`
} `graphql:"object(expression: $exp)"`
} `graphql:"repository(owner: $owner, name: $name)"`
}
qVars := map[string]interface{}{
"exp": githubv4.String(os.Getenv("GITHUB_BRANCH") + ":media/" + file),
"owner": githubv4.String("rollthecloudinc"),
"name": githubv4.String(siteName + suffix),
}
err := ac.GithubV4Client.Query(context.Background(), &q, qVars)
if err != nil {
log.Print("Github latest file failure.")
log.Panic(err)
}
// log.Printf(q.Repository.Object.ObjectFragment.Text)
// json.Unmarshal([]byte(q.Repository.Object.ObjectFragment.Text), &obj)
// log.Printf("END GithubFileUploadAdaptor::LOAD %s", id)
log.Print("content")
log.Print(q.Repository.Object.ObjectFragment.Text)
res.StatusCode = 200
res.Headers = map[string]string{
"Content-Type": contentType,
}
res.Body = q.Repository.Object.ObjectFragment.Text //base64.StdEncoding.EncodeToString([]byte(q.Repository.Object.ObjectFragment.Text))
res.IsBase64Encoded = true
return res, nil
}
The full api file can viewed below but excludes the changes above for migration to Github. This api has been running fine using s3. However, we are now trying to migrate to Github for object storage instead. Have successfully implemented write but are having difficulties described above for reading the file using our lambda.
https://github.com/rollthecloudinc/verti-go/blob/master/api/media/main.go
Help requested to figure out how to serve image files from our Github repo using the golang lambda on aws which can be accessed here as a blank screen.
https://81j44yaaab.execute-api.us-east-1.amazonaws.com/ipe/media/Screen%20Shot%202022-02-02%20at%202.00.29%20PM.png
However, this repo is also a pages site which serves the image just fine.
https://rollthecloudinc.github.io/ipe-objects/media/Screen%20Shot%202022-02-02%20at%202.00.29%20PM.png
Thanks
Further debugging the Text property appears to be empty inside the log.
The IsBinary property value being false lead use to the discovery of a typo. The name input for the graph QL invocation was missing -objects. Once the typo was corrected IsBinary started showing up true. However, the Text property value is still empty.
Having managed to find some similar issues but for uploading many have suggested that graph QL isn't the right tool for uploading binary data to begin with. Therefore, rather than chase tail we have decided to try the Github REST v3 api. Specifically, the go-github package for golang instead.
https://github.com/google/go-github
Perhaps using the REST api instead will lead to successful results.
An additional step was necessary to fetch the blob contents of the object queried via the graph QL api. Once this was achieved the media file was served with success. This required using the go-github blob api to fetch the blob base64 contents from github.
https://81j44yaaab.execute-api.us-east-1.amazonaws.com/ipe/media/Screen%20Shot%202022-02-02%20at%202.00.29%20PM.png
GetMediaFile lambda
func GetMediaFile(req *events.APIGatewayProxyRequest, ac *ActionContext) (events.APIGatewayProxyResponse, error) {
res := events.APIGatewayProxyResponse{StatusCode: 500}
pathPieces := strings.Split(req.Path, "/")
siteName := pathPieces[1]
file, _ := url.QueryUnescape(pathPieces[3]) // pathPieces[2]
log.Print("requested media site: " + siteName)
log.Print("requested media file: " + file)
// buf := aws.NewWriteAtBuffer([]byte{})
// downloader := s3manager.NewDownloader(ac.Session)
/*_, err := downloader.Download(buf, &s3.GetObjectInput{
Bucket: aws.String(ac.BucketName),
Key: aws.String("media/" + file),
})
if err != nil {
return res, err
}*/
ext := strings.Split(pathPieces[len(pathPieces)-1], ".")
contentType := mime.TypeByExtension(ext[len(ext)-1])
if ext[len(ext)-1] == "md" {
contentType = "text/markdown"
}
suffix := ""
if os.Getenv("GITHUB_BRANCH") == "master" {
suffix = "-prod"
}
owner := "rollthecloudinc"
repo := siteName + "-objects" + suffix
var q struct {
Repository struct {
Object struct {
ObjectFragment struct {
Oid githubv4.GitObjectID
} `graphql:"... on Blob"`
} `graphql:"object(expression: $exp)"`
} `graphql:"repository(owner: $owner, name: $name)"`
}
qVars := map[string]interface{}{
"exp": githubv4.String(os.Getenv("GITHUB_BRANCH") + ":media/" + file),
"owner": githubv4.String(owner),
"name": githubv4.String(repo),
}
err := ac.GithubV4Client.Query(context.Background(), &q, qVars)
if err != nil {
log.Print("Github latest file failure.")
log.Panic(err)
}
oid := q.Repository.Object.ObjectFragment.Oid
log.Print("Github file object id " + oid)
blob, _, err := ac.GithubRestClient.Git.GetBlob(context.Background(), owner, repo, string(oid))
if err != nil {
log.Print("Github get blob failure.")
log.Panic(err)
}
res.StatusCode = 200
res.Headers = map[string]string{
"Content-Type": contentType,
}
res.Body = blob.GetContent()
res.IsBase64Encoded = true
return res, nil
}
Full Source: https://github.com/rollthecloudinc/verti-go/blob/master/api/media/main.go

How do I set the S3 Request Payer option when using the aws-sdk-v2?

I am trying to use the aws-sdk-go-v2 to retrieve some data from an S3 bucket. In order to do so I need to be able to set the Request Payer option, however, since I am new to using the SDK, I have no idea how to do so.
I've tried setting this as an env variable AWS_REQUEST_PAYER=requester, but scanning the source code for this golang SDK quickly, I couldn't find that it would be picked up by the SDK as an option.
Using the SDK as directed also fails with an Unauthorized response:
import (
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
type awsArchive struct {
bucket string
config config.Config
client *s3.Client
}
func (s *awsArchive) download(uri string, dest string) error {
downloader := manager.NewDownloader(s.client)
paginator := s3.NewListObjectsV2Paginator(s.client, &s3.ListObjectsV2Input{
Bucket: &s.bucket,
Prefix: &uri,
})
for paginator.HasMorePages() {
page, err := paginator.NextPage(context.TODO())
if err != nil {
return err
}
for _, obj := range page.Contents {
if err := downloadToFile(downloader, dest, s.bucket, aws.ToString(obj.Key)); err != nil {
return err
}
}
}
return nil
}
func downloadToFile(downloader *manager.Downloader, targetDirectory, bucket, key string) error {
// Create the directories in the path
file := filepath.Join(targetDirectory, key)
if err := os.MkdirAll(filepath.Dir(file), 0775); err != nil {
return err
}
// Set up the local file
fd, err := os.Create(file)
if err != nil {
return err
}
defer fd.Close()
// Download the file using the AWS SDK for Go
fmt.Printf("Downloading s3://%s/%s to %s...\n", bucket, key, file)
_, err = downloader.Download(context.TODO(), fd, &s3.GetObjectInput{Bucket: &bucket, Key: &key})
return err
}
Error: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: ..., HostID: ..., api error AccessDenied: Access Denied
Would anyone be able to provide me with an example of using the Golang SDK to get S3 files from a Requestor Pays enabled bucket please, i.e. the equivalent of:
aws s3 sync --request-payer requester source_bucket destination_folder
It seems you can use field named 'RequestPayer' in the GetObjectInput struct. Found it from pkg document.
From the link:
type GetObjectInput struct {
// The bucket name containing the object. When using this action with an access
...
Bucket *string
// Key of the object to get.
// This member is required.
Key *string
...
...
// Confirms that the requester knows that they will be charged for the request.
// Bucket owners need not specify this parameter in their requests. For information
// about downloading objects from requester pays buckets, see Downloading Objects
// in Requestor Pays Buckets
// (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html)
// in the Amazon S3 Developer Guide.
RequestPayer types.RequestPayer
You can refer to 'RequestPayer' definition here.

Route53: query domain records by Domain Name

Using Go and AWS-SDK
I'm attempting to query route53 CNAME and A records as listed in the AWS Console under Route53 -> Hosted Zones. I'm able to query using the following code, but it requires the (cryptic) HostedZoneId I have to know ahead of time.
Is there a different function, or a HostedZoneId lookup based on the Domain Name such as XXX.XXX.com ?
AWSLogin(instance)
svc := route53.New(instance.AWSSession)
listParams := &route53.ListResourceRecordSetsInput{
HostedZoneId: aws.String("Z2798GPJN9CUFJ"), // Required
// StartRecordType: aws.String("CNAME"),
}
respList, err := svc.ListResourceRecordSets(listParams)
if err != nil {
fmt.Println(err.Error())
return
}
// Pretty-print the response data.
fmt.Println("All records:")
fmt.Println(respList)
edit: oh, additionally, the StartRecordType with the value "CNAME" throws a validation error, so I'm not sure what I should be using there.
You first have to do a lookup to get the HostedZoneID. Here is the func I wrote for it. :
func GetHostedZoneIdByNameLookup(awsSession string, HostedZoneName string) (HostedZoneID string, err error) {
svc := route53.New(awsSession)
listParams := &route53.ListHostedZonesByNameInput{
DNSName: aws.String(HostedZoneName), // Required
}
req, resp := svc.ListHostedZonesByNameRequest(listParams)
err = req.Send()
if err != nil {
return "", err
}
HostedZoneID = *resp.HostedZones[0].Id
// remove the /hostedzone/ path if it's there
if strings.HasPrefix(HostedZoneID, "/hostedzone/") {
HostedZoneID = strings.TrimPrefix(HostedZoneID, "/hostedzone/")
}
return HostedZoneID, nil
}

how to use redis geo radius command go lang

I'm developing with Go lang and very new to it. I would like to use the redis GEORADIUS command and get back my results including the distance.
I have now used two packages radix.v2 and redigo to try and get results but had no joy. I have no issues using simple commands such as SET and GET.
Any help will be greatly appreciated.
Edit: added code :-)
package main
import (
"fmt"
"github.com/mediocregopher/radix.v2/redis"
//"reflect"
"encoding/json"
)
var (
client *redis.Client
)
type Facility struct {
Id string
Dist string
}
func main() {
client, err := redis.Dial("tcp", "192.168.99.100:6379")
if err != nil {
// handle err
}
surname, err := client.Cmd("GET", "surname").Str()
if err != nil {
// handle err
}
fmt.Println(surname)
reply, _ := client.Cmd("GEORADIUS", "venues", 50, 0, 25, "mi", "WITHDIST", "ASC").Array()
if err != nil {
fmt.Println(err)
}
//fmt.Println(reflect.TypeOf(reply))
//fmt.Println(reply)
var facilities []Facility
for _, results := range reply {
facility, _ := results.Array()
id, _ := facility[0].Str()
dist, _ := facility[1].Str()
facilities = append(facilities, Facility{
Id: id,
Dist: dist,
})
}
//fmt.Println(facilities)
resp, _ := json.Marshal(facilities)
fmt.Fprintf(w, string(resp))
}
Cheers
Stephen
You can look at the code in the redis package.
the file commands.go has all the signatures
a few signatures are below
GeoAdd(key string, geoLocation ...*GeoLocation) *IntCmd
GeoPos(key string, members ...string) *GeoPosCmd
GeoRadius(key string, longitude, latitude float64, query *GeoRadiusQuery) GeoRadiusByMember(key, member string, query *GeoRadiusQuery)
GeoDist(key string, member1, member2, unit string) *FloatCmd

Resources