Scylla gocqlx how to implement pagination similar to a cursor - go

I'm using Scylla to save parties created by a users. The method below returns a list of parties created by a user. I currently return all parties without allowing pagination, but I'm trying to implement Pagination for the method below but I still don't quite understand how pagination is handled with Scylla.
My guess would be that a cursor can be passed to a query. Based on this example it looks like the PageState can be used to pass something similar to a cursor.
I would appreciate a short explanation what PageState is and if I should use it to accomplish token based pagination. It would also be great if an example could be provided that shows how a new PageState can be returned to the client and used to fetch a new page on a second request.
func (pq *partyQuery) GetByUser(ctx context.Context, uId string) ([]datastruct.Party, error) {
var result []datastruct.Party
stmt, names := qb.
Select(TABLE_NAME).
Where(qb.Eq("user_id")).
ToCql()
err := pq.sess.
Query(stmt, names).
BindMap((qb.M{"user_id": uId})).
PageSize(10).
Iter().
Select(&result)
if err != nil {
log.Println(err)
return []datastruct.Party{}, errors.New("no parties found")
}
return result, nil
}
Thanks in advance and I appreciate your time.
Edit
For anybody interested, this is how I transformed my function to allow paging:
func (pq *partyQuery) GetByUser(ctx context.Context, uId string, page []byte) (result []datastruct.Party, nextPage []byte, err error) {
stmt, names := qb.
Select(TABLE_NAME).
Where(qb.Eq("user_id")).
ToCql()
q := pq.sess.
Query(stmt, names).
BindMap((qb.M{"user_id": uId}))
defer q.Release()
q.PageState(page)
q.PageSize(10)
iter := q.Iter()
err = iter.Select(&result)
if err != nil {
log.Println(err)
return []datastruct.Party{}, nil, errors.New("no parties found")
}
return result, iter.PageState(), nil
}

Hi gocqlx author here.
Please take a look at this example https://github.com/scylladb/gocqlx/blob/25d81de30ebcdfa02d3d849b518fc57b839e4399/example_test.go#L482
getUserVideos := func(userID int, page []byte) (userVideos []Video, nextPage []byte, err error) {
q := videoTable.SelectQuery(session).Bind(userID)
defer q.Release()
q.PageState(page)
q.PageSize(itemsPerPage)
iter := q.Iter()
return userVideos, iter.PageState(), iter.Select(&userVideos)
}
You need to send page state to caller.

Related

How to design RestAPI for too many tables in Golang

I think if i keep using the method below, i'll have to write too much code.
I declared structures for all the tables. and i used the go validate package for validation.
[types.go]
type TableA struct {
Field1 string `json:"field1" validate:"required, max=10"`
Field2 int `json:"field2" validate:"number"`
}
type TableB struct {
...
}
And i initialized the router for each method and connected the handlers.
[tableA.go]
router.Get("/table-a", r.Get_tableA_Handler),
router.Post("/table-a", r.Post_tableA_Handler),
router.Patch("/table-a", r.Patch_tableA_Handler),
router.Delete("/table-a", r.Delete_tableA_Handler)
...
Each handler parses the json in the request body, validates the data and call the db function.
[tableA_router.go]
func (rt *tableARouter) Post_tableA_Handler(w http.ResponseWriter, r *http.Request) error {
//Json to Struct
req := new(types.tableA)
if err := httputils.DecodeJsonBody(r, req); err != nil {
return err
}
// Validation
if err := validCheck(req); err != nil {
return err
}
// DB function
err := rt.insert_tableA_DB(r.Context(), req)
if err != nil {
return err
}
return rt.rd.JSON(w, http.StatusCreated, "Create Success")
}
...
func validCheck(data interface{}) error {
validate := validator.New()
err := validate.Struct(data)
return err
}
This is a DB function called from the handler function above (using Gorm)
[tableA_db.go]
func (rt *tableARouter) insert_tableA_DB(ctx context.Context, data *types.TableA) error {
// DB Connect
db, err := db.Open(rt.dbcfg)
if err != nil {
return err
}
defer db.Close()
tx := db.Begin()
defer tx.Rollback()
// == INSERT ==
query := `INSERT INTO table_a
(field1, field2, ...)
VALUES (?, ?, ...)`
result := tx.WithContext(ctx).Exec(query,
data.Field1, data.Field2, ...)
//Result
if result.Error != nil {
...
}
There are too many tables now... If there are 100 tables i have to write 100 handlers and 100 DB functions.
Is there any way to use something like /tables/{tableName}?
Please give me any advice.... Thank you.
You can use an ORM package, like GORM to make easier your work.
Or you can make an universal handler and with the reflect package, analyze your defined structs and make every SQL query dinamically. But it's not the best solution if any of your struct has inner slices, other embedded structs, or if you need to use joined tables you also have to deal with it manually. I have servers where we have more than 200 endpoints with more than 3-400 methods with 200+ SQL tables and the whole server was written by hand. But I can say, it's very rare when a handler and the DB func can be reused without modifying.
Maybe you can wrap the error handling, rollback/commit, json parse and response parts in a func then use it to call the DB methods.

How to serialize `LastEvaluatedKey` from DynamoDB's Golang SDK?

When working with DynamoDB in Golang, if a call to query has more results, it will set LastEvaluatedKey on the QueryOutput, which you can then pass in to your next call to query as ExclusiveStartKey to pick up where you left off.
This works great when the values stay in Golang. However, I am writing a paginated API endpoint, so I would like to serialize this key so I can hand it back to the client as a pagination token. Something like this, where something is the magic package that does what I want:
type GetDomainObjectsResponse struct {
Items []MyDomainObject `json:"items"`
NextToken string `json:"next_token"`
}
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
// ... parse query params, set up dynamoIn ...
dynamoIn.ExclusiveStartKey = something.Decode(params.NextToken)
dynamoOut, _ := db.Query(dynamoIn)
response := GetDomainObjectsResponse{}
dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)
response.NextToken := something.Encode(dynamoOut.LastEvaluatedKey)
// ... marshal and write the response ...
}
(please forgive any typos in the above, it's a toy version of the code I whipped up quickly to isolate the issue)
Because I'll need to support several endpoints with different search patterns, I would love a way to generate pagination tokens that doesn't depend on the specific search key.
The trouble is, I haven't found a clean and generic way to serialize the LastEvaluatedKey. You can marshal it directly to JSON (and then e.g. base64 encode it to get a token), but doing so is not reversible. LastEvaluatedKey is a map[string]types.AttributeValue, and types.AttributeValue is an interface, so while the json encoder can read it, it can't write it.
For example, the following code panics with panic: json: cannot unmarshal object into Go value of type types.AttributeValue.
lastEvaluatedKey := map[string]types.AttributeValue{
"year": &types.AttributeValueMemberN{Value: "1993"},
"title": &types.AttributeValueMemberS{Value: "Benny & Joon"},
}
bytes, err := json.Marshal(lastEvaluatedKey)
if err != nil {
panic(err)
}
decoded := map[string]types.AttributeValue{}
err = json.Unmarshal(bytes, &decoded)
if err != nil {
panic(err)
}
What I would love would be a way to use the DynamoDB-flavored JSON directly, like what you get when you run aws dynamodb query on the CLI. Unfortunately the golang SDK doesn't support this.
I suppose I could write my own serializer / deserializer for the AttributeValue types, but that's more effort than this project deserves.
Has anyone found a generic way to do this?
OK, I figured something out.
type GetDomainObjectsResponse struct {
Items []MyDomainObject `json:"items"`
NextToken string `json:"next_token"`
}
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
// ... parse query params, set up dynamoIn ...
eskMap := map[string]string{}
json.Unmarshal(params.NextToken, &eskMap)
esk, _ = dynamodbattribute.MarshalMap(eskMap)
dynamoIn.ExclusiveStartKey = esk
dynamoOut, _ := db.Query(dynamoIn)
response := GetDomainObjectsResponse{}
dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)
lek := map[string]string{}
dynamodbattribute.UnmarshalMap(dynamoOut.LastEvaluatedKey, &lek)
response.NextToken := json.Marshal(lek)
// ... marshal and write the response ...
}
(again this is my real solution hastily transferred back to the toy problem, so please forgive any typos)
As #buraksurdar pointed out, attributevalue.Unmarshal takes an inteface{}. Turns out in addition to a concrete type, you can pass in a map[string]string, and it just works.
I believe this will NOT work if the AttributeValue is not flat, so this isn't a general solution [citation needed]. But my understanding is the LastEvaluatedKey returned from a call to Query will always be flat, so it works for this usecase.
Inspired by Dan, here is a solution to serialize and deserialize to/from base64
package dynamodb_helpers
import (
"encoding/base64"
"encoding/json"
"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)
func Serialize(input map[string]types.AttributeValue) (*string, error) {
var inputMap map[string]interface{}
err := attributevalue.UnmarshalMap(input, &inputMap)
if err != nil {
return nil, err
}
bytesJSON, err := json.Marshal(inputMap)
if err != nil {
return nil, err
}
output := base64.StdEncoding.EncodeToString(bytesJSON)
return &output, nil
}
func Deserialize(input string) (map[string]types.AttributeValue, error) {
bytesJSON, err := base64.StdEncoding.DecodeString(input)
if err != nil {
return nil, err
}
outputJSON := map[string]interface{}{}
err = json.Unmarshal(bytesJSON, &outputJSON)
if err != nil {
return nil, err
}
return attributevalue.MarshalMap(outputJSON)
}

Mapping Gorm queries to lists of structs

Go here, using gorm to help me with database stuff.
I have the following function which is working for me:
func (d DbPersister) FetchOrderById(orderId string) (Order,error) {
order := &Order{}
if err := d.GormDB.Table("orders").
Select(`orders`.`order_id`,
`orders`.`quantity`,
`addresses`.`line_1`,
`addresses`.`state`,
`addresses`.`zip`).
Joins("join addresses on addresses.address_id = orders._address_id").
Where("orders.order_id = ?", orderId).
First(order).Error; err != nil {
return Order{}, err
}
return *order,nil
}
So, you give it an orderId, and it fetches that from the DB and maps it to an Order instance.
I now want to look up all a particular user's orders:
func (d DbPersister) FetchAllOrdersByCustomerId(userId string) ([]Order,error) {
orders := []&Order{}
if err := d.GormDB.Table("orders").
Select(`orders`.`order_id`,
`orders`.`quantity`,
`orders`.`status`,
`addresses`.`line_1`,
`addresses`.`state`,
`addresses`.`zip`).
Joins("join addresses on addresses.address_id = orders.shipping_address_id").
Joins("join users on users.user_id = orders.user_id").
Where("users.user_id = ?", userId).
First(orders).Error; err != nil {
return []Order{}, err
}
return orders,nil
}
However, I'm getting compiler errors. I don't believe First(orders) is the correct function to be calling here. Basically do a join between orders and users and give me all of a particular user's orders, which should be a list of Order instances. Can anyone spot where I'm going awry?
Firstly orders := []&Order{} should be orders := make([]Order,0)
Then use Find() instead of First() to get multiple values.
For structs, slices, maps, and other composite types, if no data is contained you can return nil
So your code should be
func (d DbPersister) FetchAllOrdersByCustomerId(userId string) ([]Order,error) {
orders := make([]Order,0)
if err := d.GormDB.Table("orders").
Select(`orders`.`order_id`,
`orders`.`quantity`,
`orders`.`status`,
`addresses`.`line_1`,
`addresses`.`state`,
`addresses`.`zip`).
Joins("join addresses on addresses.address_id = orders.shipping_address_id").
Joins("join users on users.user_id = orders.user_id").
Where("users.user_id = ?", userId).
Find(&orders).Error; err != nil {
return nil, err
}
return orders,nil
}

How do I use olivere's elastic client.Update() service to change array fields, generic string fields and possibly nested struct fields?

https://github.com/olivere/elastic
Version 5.x
The wiki documentation isn't really clear on how client.Update() works. It's needed to completely change a field and to modify arrays. i.e. in the example in the wiki documentation, how would one go about appending and removing tags to a tweet or changing a tweet's content? Also if a tweet was represented in go as a struct and I added a nested struct called "echo" which contains a foo of type int, content of type string and another type string array, how would one go about changing any of these fields using client.Update() if it's even possible?
In my personal example I have this function:
func UpdateEntryContent(eclient *elastic.Client, entryID string, newContent []rune) error{
ctx:=context.Background()
exists, err := eclient.IndexExists(ENTRY_INDEX).Do(ctx)
if err != nil {return err}
if !exists {return errors.New("Index does not exist")}
_, err = eclient.Update().Index(ENTRY_INDEX).Type(ENTRY_TYPE).Id(entryID).
Script("ctx._source.Content = newCont").
ScriptParams(map[string]interface{}{"newCont": newContent}).
Do(ctx)
if err != nil {return err}
return nil
}
But I get this following error when I try to compile:
cannot use "ctx._source.Content = newCont" (type string) as type *elastic.Script in argument to eclient.Update().Index(ENTRY_INDEX).Type(ENTRY_TYPE).Id(entryID).Script
eclient.Update().Index(ENTRY_INDEX).Type(ENTRY_TYPE).Id(entryID).Script("ctx._source.Content = newCont").ScriptParams undefined (type *elastic.UpdateService has no field or method ScriptParams)
The Script method accepts a *elastic.Script, not a string. The ScriptParams method is also found on *elastic.Script as Params instead of being on *elastic.UpdateService.
func UpdateEntryContent(eclient *elastic.Client, entryID string, newContent []rune) error{
ctx:=context.Background()
exists, err := eclient.IndexExists(ENTRY_INDEX).Do(ctx)
if err != nil {return err}
if !exists {return errors.New("Index does not exist")}
script := elastic.NewScript("ctx._source.Content = newCont").Params(map[string]interface{}{"newCont": newContent})
_, err = eclient.Update().Index(ENTRY_INDEX).Type(ENTRY_TYPE).Id(entryID).
Script(script).
Do(ctx)
if err != nil {return err}
return nil
}
You can see more information about the package with GoDoc or by looking through the source code.
The following code should resolve the issue
_, err = eclient.Update().Index(INDEX).
Type(TYPE).
Id(ID).
Doc(map[string]interface{}{field: message}).
Do(ctx)
Credit where it's due Gavin's answer put me on the right track. This is for another .Index but the full function that acts as a generic single field update is as follows:
func UpdateUser(eclient *elastic.Client, userID string, field string, newContent interface{})error {
//CHANGES A SINGLE FIELD OF ES USER DOCUMENT(requires an elastic client pointer,
// the user DocID, the feild you wish to modify as a string,
// and what you want to change that field to as any type necessary)
//RETURN AN error IF SUCESSFUL error = nil
ctx := context.Background()
exists, err := eclient.IndexExists(USER_INDEX).Do(ctx)
if err != nil {return err}
if !exists {return errors.New("Index does not exist")}
_, err = eclient.Update().
Index(USER_INDEX).
Type(USER_TYPE).
Id(userID).
Doc(map[string]interface{}{field: newContent}).
Do(ctx)
return nil
}
You can change the .Index, .Type, and .Id and it works with all fields and types as far as I can tell

Simplifying Go Code

I have two functions as shown below which look similar, but using different functions to query the db. Since Go doesn't encourage overloaading methods, is the redundancy acceptable? Or should I refactor them into one function? All comments are welcomed.
var getCustomers = func() ([]customer, error) {
return nil, nil
}
var getCustomerById = func(int64) (*customer, error) {
return nil, nil
}
func listCustomer(w http.ResponseWriter, r *http.Request) *appError {
cus, err := getCustomers()
if err != nil {
return &appError{err, "No customer found", 404}
}
res, err := json.Marshal(cus)
if err != nil {
return &appError{err, "Can't display record", 500}
}
fmt.Fprint(w, string(res))
return nil
}
func viewCustomer(w http.ResponseWriter, r *http.Request, id int64) *appError {
cus, err := getCustomerByID(id)
if err != nil {
return &appError{err, "No customer found", 404}
}
res, err := json.Marshal(cus)
if err != nil {
return &appError{err, "Can't display record", 500}
}
fmt.Fprint(w, string(res))
return nil
}
Suggestion to use -1 to list all customer, but I'm not sure if this is the best it can be:
func viewCustomer(w http.ResponseWriter, r *http.Request, id int64) *appError {
var c *customer
var clist []customer
var err error
if id < 0 {
clist, err = getCustomers()
res, err := json.Marshal(clist)
if err != nil {
return &appError{err, "Can't display record", 500}
}
fmt.Fprint(w, string(res))
return nil
} else {
c, err = getCustomerById(id)
res, err := json.Marshal(c)
if err != nil {
return &appError{err, "Can't display record", 500}
}
fmt.Fprint(w, string(res))
return nil
}
}
Since Go doesn't encourage overloaading methods, is the redundancy acceptable? Or should I refactor them into one function?
The answer to your question depends heavily on the real code and your task. If listing one customer is very different from listing several customers (that is, you need different information and have different presentation logic), I'd say that duplication here is not that bad, since the difference may grow larger in the future, so a DRY solution could turn into an if and switch mess quickly. (I've had a similar experience on a non-Go project and since then I think that DRY is good but you should not be fanatic about it.)
On the other hand, if you're making, say a JSON API, you could make it more DRY. Define your getCustomers function like this:
func customers(ids ...int64) ([]customer, error) { /* ... */ }
This way you can:
call it like customers() and get all customers;
call customers(42) and get a customer whose ID is 42;
call customers(someIDs...) and get multiple customers by their IDs.
All can be done in one handler in a straightforward way.
All comments are welcomed.
Two nitpicks on your code:
Getter methods in Go are idiomatically named foo and not getFoo, so I used customers in my example. You would most probably call it as DB.Customers(ids...), so it looks a lot like a getter method.
What's up with var foo = func() notation? Unless there is a reason for that (like using these functions as closures), I'd suggest sticking with the func foo() notation, since it's more idiomatic and generally easier to grep. EDIT: as OP pointed out in the comments, another case for var foo = func() notation is of course functions that can be faked in testing, as shown in Andrew Gerrand's Testing Techniques talk.
I hope that helps.

Resources