I'm using the go-elasticsearch and I need to use the operations in the Aliases API:
GET _alias/my-alias-name to get the information on an alias
POST _aliases to add multiple index names to an alias in an atomic operation
I can't find the representations of these APIs in the go-elasticsearch client, is there any alternative way to make a request to these endpoints with the library, or I just need to use a plain HTTP client only for these methods?
Finally found a way to do it, the Aliases API is inside the Indices API:
GET _alias/my-alias-name: we can get the alias information using the GetAlias() method, building a request like this:
res, err := s.client.Indices.GetAlias(
s.client.Indices.GetAlias.WithName("my-alias-name"),
s.client.Indices.GetAlias.WithContext(context.Background()),
)
if err != nil {
log.Fatal(err)
}
// ...parse response
POST _aliases to add or delete indices to aliases in an single operation we can use UpdateAliases():
type UpdateAliasRequest struct {
Actions []map[string]*UpdateAliasAction `json:"actions"`
}
// UpdateAliasAction represents an action in the Elasticsearch Aliases API.
type UpdateAliasAction struct {
Index string `json:"index"`
Alias string `json:"alias"`
}
updateActions := make([]map[string]*UpdateAliasAction, 0)
removeAction := make(map[string]*UpdateAliasAction)
removeAction["remove"] = &UpdateAliasAction{
Index: "old-index-00",
Alias: "my-alias-name",
}
updateActions = append(updateActions, removeAction)
addAction := make(map[string]*UpdateAliasAction)
addAction["add"] = &UpdateAliasAction{
Index: "new-index-00",
Alias: "my-alias-name",
}
updateActions = append(updateActions, addAction)
jsonBody, err := json.Marshal(&UpdateAliasRequest{
Actions: updateActions,
})
if err != nil {
log.Fatal(err)
}
// make API request
res, err := s.client.Indices.UpdateAliases(
bytes.NewBuffer(jsonBody),
s.client.Indices.UpdateAliases.WithContext(context.Background()),
)
if err != nil {
log.Fatal(err)
}
// ...parse response
Related
Using the neo4j Go bolt driver, I am able to get nodes, but not relationships from the graph db.
The Run() API in neo4j.transaction return type is Result which can give nodes, but not to the relationships?
If I try the query in the neo4j browser, it shows me the properties of the relationships, but if I send the same query programmatically, I don’t get anything. Am I missing something?
MATCH (:a {name: ‘foo’})-[r:bar]->() RETURN properties(r)
The above query works
{
"X":"20",
"Y":"40"
}
But the same query sent via the driver returns no error, but has nothing in it.
There might be something wrong with your mapping code.
It should look like this:
func readCoordinates(driver neo4j.Driver) ([]Coordinates, error) {
session := driver.NewSession(neo4j.SessionConfig{})
defer session.Close()
result, err := session.ReadTransaction(executeReadCoordinates)
if err != nil {
return nil, err
}
return result.([]Coordinates), nil
}
func executeReadCoordinates(tx neo4j.Transaction) (interface{}, error) {
records, err := tx.Run("MATCH (:A {name: 'foo'})-[r:BAR]->() RETURN properties(r)", map[string]interface{}{})
if err != nil {
return nil, err
}
var results []Coordinates
for records.Next() {
record := records.Record()
if props, found := record.Get("properties(r)"); !found {
return nil, fmt.Errorf("expected properties not found")
} else {
properties := props.(map[string]interface{})
coordinates := Coordinates{
X: properties["x"].(int64),
Y: properties["y"].(int64),
}
results = append(results, coordinates)
}
}
return results, nil
}
I changed the case of the node label (convention: PascalCase), relationship type (convention: SCREAMING_SNAKE_CASE) and properties (convention: snake_case).
The code assumes those properties are of type int64 and fetches a list.
If you want a single pair of coordinates, then remove the for loop and use records.Single() instead.
Im using postman to post data and in the body Im putting some simple json
Request Body
{
"order":"1",
"Name":"ts1"
}
I need to transfer the data to json and I try like following,
and I wasnt able to get json, any idea what is missing
router.POST("/user", func(c *gin.Context) {
var f interface{}
//value, _ := c.Request.GetBody()
//fmt.Print(value)
err2 := c.ShouldBindJSON(&f)
if err2 == nil {
err = client.Set("id", f, 0).Err()
if err != nil {
panic(err)
}
}
The f is not a json and Im getting an error, any idea how to make it work?
The error is:
redis: can't marshal map[string]interface {} (implement encoding.BinaryMarshaler)
I use https://github.com/go-redis/redis#quickstart
If I remove the the body and use hard-coded code like this I was able to set the data, it works
json, err := json.Marshal(Orders{
order: "1",
Name: "tst",
})
client.Set("id", json, 0).Err()
If you only want to pass the request body JSON to Redis as a value, then you do not need to bind the JSON to a value. Read the raw JSON from the request body directly and just pass it through:
jsonData, err := ioutil.ReadAll(c.Request.Body)
if err != nil {
// Handle error
}
err = client.Set("id", jsonData, 0).Err()
let's do it with an example. Assume your request body has a user email like this:
{ email: "test#test.com" }
and now you want to get this email on the back-end. first, define a struct like the following:
type EmailRequestBody struct {
Email string
}
Now you can easily bind the email value in your request body to the struct you defined: first, define a variable for your struct and then bind the value:
func ExampleFunction(c *gin.Context) {
var requestBody EmailRequestBody
if err := c.BindJSON(&requestBody); err != nil {
// DO SOMETHING WITH THE ERROR
}
fmt.Println(requestBody.Email)
}
you can easily access the email value and print it out or do whatever you need :
fmt.Println(requestBody.Email)
Or you can use GetRawData() function as:
jsonData, err := c.GetRawData()
if err != nil{
//Handle Error
}
err = client.Set("id", jsonData, 0).Err()
If you want to get json body like other frameworks like express(Nodejs), you can do the following
bodyAsByteArray, _ := ioutil.ReadAll(c.Request.Body)
jsonBody := string(bodyAsByteArray)
Using Go and AWS-SDK
I'm attempting to query route53 CNAME and A records as listed in the AWS Console under Route53 -> Hosted Zones. I'm able to query using the following code, but it requires the (cryptic) HostedZoneId I have to know ahead of time.
Is there a different function, or a HostedZoneId lookup based on the Domain Name such as XXX.XXX.com ?
AWSLogin(instance)
svc := route53.New(instance.AWSSession)
listParams := &route53.ListResourceRecordSetsInput{
HostedZoneId: aws.String("Z2798GPJN9CUFJ"), // Required
// StartRecordType: aws.String("CNAME"),
}
respList, err := svc.ListResourceRecordSets(listParams)
if err != nil {
fmt.Println(err.Error())
return
}
// Pretty-print the response data.
fmt.Println("All records:")
fmt.Println(respList)
edit: oh, additionally, the StartRecordType with the value "CNAME" throws a validation error, so I'm not sure what I should be using there.
You first have to do a lookup to get the HostedZoneID. Here is the func I wrote for it. :
func GetHostedZoneIdByNameLookup(awsSession string, HostedZoneName string) (HostedZoneID string, err error) {
svc := route53.New(awsSession)
listParams := &route53.ListHostedZonesByNameInput{
DNSName: aws.String(HostedZoneName), // Required
}
req, resp := svc.ListHostedZonesByNameRequest(listParams)
err = req.Send()
if err != nil {
return "", err
}
HostedZoneID = *resp.HostedZones[0].Id
// remove the /hostedzone/ path if it's there
if strings.HasPrefix(HostedZoneID, "/hostedzone/") {
HostedZoneID = strings.TrimPrefix(HostedZoneID, "/hostedzone/")
}
return HostedZoneID, nil
}
I am trying to test some golang code and I have a method that calls several other methods from its body. All these methods perform some kind of operations using an elastic search client. I wanted to know whether it will be a good practice if I used a test server for testing this method that will write different responses depending upon the request method and path it received from the request that is made when the methods inside the body execute and make calls to the elasticsearch client that sends the requests to my test server?
Update:
I am testing an elasticsearch middleware. It implements a reindex service like this
type reindexService interface {
reindex(ctx context.Context, index string, mappings, settings map[string]interface{}, includes, excludes, types []string) error
mappingsOf(ctx context.Context, index string) (map[string]interface{}, error)
settingsOf(ctx context.Context, index string) (map[string]interface{}, error)
aliasesOf(ctx context.Context, index string) ([]string, error)
createIndex(ctx context.Context, name string, body map[string]interface{}) error
deleteIndex(ctx context.Context, name string) error
setAlias(ctx context.Context, index string, aliases ...string) error
getIndicesByAlias(ctx context.Context, alias string) ([]string, error)
}
I can easily test all the methods using this pattern. Creating a simple elastic search client using a httptest server url and making requests to that server
var createIndexTests = []struct {
setup *ServerSetup
index string
err string
}{
{
&ServerSetup{
Method: "PUT",
Path: "/test",
Body: `null`,
Response: `{"acknowledged": true, "shards_acknowledged": true, "index": "test"}`,
},
"test",
"",
},
// More test cases here
}
func TestCreateIndex(t *testing.T) {
for _, tt := range createIndexTests {
t.Run("Should successfully create index with a valid setup", func(t *testing.T) {
ctx := context.Background()
ts := buildTestServer(t, tt.setup)
defer ts.Close()
es, _ := newTestClient(ts.URL)
err := es.createIndex(ctx, tt.index, nil)
if !compareErrs(tt.err, err) {
t.Fatalf("Index creation should have failed with error: %v got: %v instead\n", tt.err, err)
}
})
}
}
But in case of reindex method this approach poses a problem since reindex makes calls to all the other methods inside its body. reindex looks something like this:
func (es *elasticsearch) reindex(ctx context.Context, indexName string, mappings, settings map[string]interface{}, includes, excludes, types []string) error {
var err error
// Some preflight checks
// If mappings are not passed, we fetch the mappings of the old index.
if mappings == nil {
mappings, err = es.mappingsOf(ctx, indexName)
// handle err
}
// If settings are not passed, we fetch the settings of the old index.
if settings == nil {
settings, err = es.settingsOf(ctx, indexName)
// handle err
}
// Setup the destination index prior to running the _reindex action.
body := make(map[string]interface{})
body["mappings"] = mappings
body["settings"] = settings
newIndexName, err := reindexedName(indexName)
// handle err
err = es.createIndex(ctx, newIndexName, body)
// handle err
// Some additional operations
// Reindex action.
_, err = es.client.Reindex().
Body(reindexBody).
Do(ctx)
// handle err
// Fetch all the aliases of old index
aliases, err := es.aliasesOf(ctx, indexName)
// handle err
aliases = append(aliases, indexName)
// Delete old index
err = es.deleteIndex(ctx, indexName)
// handle err
// Set aliases of old index to the new index.
err = es.setAlias(ctx, newIndexName, aliases...)
// handle err
return nil
}
For testing the reindex method I have tried mocking and DI but that turns out to be hard since the methods are defined on a struct instead of passing an interface as an argument to them. (So now I want to keep the implementation same since it would require making changes to all the plugin implementations and I want to avoid that)
I wanted to know whether I can use a modified version of my build server funtion (the one I am using is given below) to return responses for different methods for the reindex service which will write the appropriate responses based on the HTTP method and the request path that is used by that method?
type ServerSetup struct {
Method, Path, Body, Response string
HTTPStatus int
}
// This function is a modified version of: https://github.com/github/vulcanizer/blob/master/es_test.go
func buildTestServer(t *testing.T, setup *ServerSetup) *httptest.Server {
handlerFunc := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requestBytes, _ := ioutil.ReadAll(r.Body)
requestBody := string(requestBytes)
matched := false
if r.Method == setup.Method && r.URL.EscapedPath() == setup.Path && requestBody == setup.Body {
matched = true
if setup.HTTPStatus == 0 {
w.WriteHeader(http.StatusOK)
} else {
w.WriteHeader(setup.HTTPStatus)
}
_, err := w.Write([]byte(setup.Response))
if err != nil {
t.Fatalf("Unable to write test server response: %v", err)
}
}
// TODO: remove before pushing
/*if !reflect.DeepEqual(r.URL.EscapedPath(), setup.Path) {
t.Fatalf("wanted: %s got: %s\n", setup.Path, r.URL.EscapedPath())
}*/
if !matched {
t.Fatalf("No requests matched setup. Got method %s, Path %s, body %s\n", r.Method, r.URL.EscapedPath(), requestBody)
}
})
return httptest.NewServer(handlerFunc)
}
Something like this function but it takes a map of request methods and past mapped to appropriate responses and writes them to the writer?
I have the following working code to delete an object from Amazon s3
params := &s3.DeleteObjectInput{
Bucket: aws.String("Bucketname"),
Key : aws.String("ObjectKey"),
}
s3Conn.DeleteObjects(params)
But what i want to do is to delete all files under a folder using wildcard **. I know amazon s3 doesn't treat "x/y/file.jpg" as a folder y inside x but what i want to achieve is by mentioning "x/y*" delete all the subsequent objects having the same prefix. Tried amazon multi object delete
params := &s3.DeleteObjectsInput{
Bucket: aws.String("BucketName"),
Delete: &s3.Delete{
Objects: []*s3.ObjectIdentifier {
{
Key : aws.String("x/y/.*"),
},
},
},
}
result , err := s3Conn.DeleteObjects(params)
I know in php it can be done easily by s3->delete_all_objects as per this answer. Is the same action possible in GOlang.
Unfortunately the goamz package doesn't have a method similar to the PHP library's delete_all_objects.
However, the source code for the PHP delete_all_objects is available here (toggle source view): http://docs.aws.amazon.com/AWSSDKforPHP/latest/#m=AmazonS3/delete_all_objects
Here are the important lines of code:
public function delete_all_objects($bucket, $pcre = self::PCRE_ALL)
{
// Collect all matches
$list = $this->get_object_list($bucket, array('pcre' => $pcre));
// As long as we have at least one match...
if (count($list) > 0)
{
$objects = array();
foreach ($list as $object)
{
$objects[] = array('key' => $object);
}
$batch = new CFBatchRequest();
$batch->use_credentials($this->credentials);
foreach (array_chunk($objects, 1000) as $object_set)
{
$this->batch($batch)->delete_objects($bucket, array(
'objects' => $object_set
));
}
$responses = $this->batch($batch)->send();
As you can see, the PHP code will actually make an HTTP request on the bucket to first get all files matching PCRE_ALL, which is defined elsewhere as const PCRE_ALL = '/.*/i';.
You can only delete 1000 files at once, so delete_all_objects then creates a batch function to delete 1000 files at a time.
You have to create the same functionality in your go program as the goamz package doesn't support this yet. Luckily it should only be a few lines of code, and you have a guide from the PHP library.
It might be worth submitting a pull request for the goamz package once you're done!
Using the mc tool you can do:
mc rm -r --force https://BucketName.s3.amazonaws.com/x/y
it will delete all the objects with the prefix "x/y"
You can achieve the same with Go using minio-go like this:
package main
import (
"log"
"github.com/minio/minio-go"
)
func main() {
config := minio.Config{
AccessKeyID: "YOUR-ACCESS-KEY-HERE",
SecretAccessKey: "YOUR-PASSWORD-HERE",
Endpoint: "https://s3.amazonaws.com",
}
// find Your S3 endpoint here http://docs.aws.amazon.com/general/latest/gr/rande.html
s3Client, err := minio.New(config)
if err != nil {
log.Fatalln(err)
}
isRecursive := true
for object := range s3Client.ListObjects("BucketName", "x/y", isRecursive) {
if object.Err != nil {
log.Fatalln(object.Err)
}
err := s3Client.RemoveObject("BucketName", object.Key)
if err != nil {
log.Fatalln(err)
continue
}
log.Println("Removed : " + object.Key)
}
}
Since this question was asked, the AWS GoLang lib for S3 has received some new methods in S3 Manager to handle this task (in response to #Itachi's pr).
See Github record: https://github.com/aws/aws-sdk-go/issues/448#issuecomment-309078450
Here is their example in v1: https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/go/s3/DeleteObjects/DeleteObjects.go#L36
To get "wildcard matching" on paths inside the bucket, add the Prefix param to the example's ListObjectsInput call, as shown here:
iter := s3manager.NewDeleteListIterator(svc, &s3.ListObjectsInput{
Bucket: bucket,
Prefix: aws.String("somePathString"),
})
A bit late in the game, but since I was having the same problem, I created a small pkg that you can copy to your code base and import as needed.
func ListKeysInPrefix(s s3iface.S3API, bucket, prefix string) ([]string, error) {
res, err := s.Client.ListObjectsV2(&s3.ListObjectsV2Input{
Bucket: aws.String(bucket),
Prefix: aws.String(prefix),
})
if err != nil {
return []string{}, err
}
var keys []string
for _, key := range res.Contents {
keys = append(keys, *key.Key)
}
return keys, nil
}
func createDeleteObjectsInput(keys []string) *s3.Delete {
rm := []*s3.ObjectIdentifier{}
for _, key := range keys {
rm = append(rm, &s3.ObjectIdentifier{Key: aws.String(key)})
}
return &s3.Delete{Objects: rm, Quiet: aws.Bool(false)}
}
func DeletePrefix(s s3iface.S3API, bucket, prefix string) error {
keys, err := s.ListKeysInPrefix(bucket, prefix)
if err != nil {
panic(err)
}
_, err = s.Client.DeleteObjects(&s3.DeleteObjectsInput{
Bucket: aws.String(bucket),
Delete: s.createDeleteObjectsInput(keys),
})
if err != nil {
return err
}
return nil
}
So, in the case you have a bucket called "somebucket" with the following structure: s3://somebucket/foo/some-prefixed-folder/bar/test.txt and wanted to delete from some-prefixed-folder onwards, usage would be:
func main() {
// create your s3 client here
// client := ....
err := DeletePrefix(client, "somebucket", "some-prefixed-folder")
if err != nil {
panic(err)
}
}
This implementation only allows to delete a maximum of 1000 entries from the given prefix due ListObjectsV2 implementation - but it is paginated, so it's a matter of adding the functionality to keep refreshing results until results are < 1000.