Golang implementing pagination on map[string]interface{} data - go

I have a json file (nested json) that I am unmarshalling its content into a map[string]interface. Now I have to implement pagination as the data is large. The client side will send as a query parameter the desired page, how can I slice the data I have?
This is a snippet of the data I am dealing with:
"packages":{
"pkg1": {
"meta": {
"description": "description1",
"name": "pkg1.1"
},
"name": "pkg1.1"
},
"pkg2": {
"meta": {
"description": "description2",
"name": "pkg2.2"
},
"name": "pkg2.2"
},
}
So what I did is that I recursively iterated through the data and created an array of a custom type containing the data I need (name, description) for each entry so that I can use it for pagination. Here is the code I used:
type Object struct {
name string
description string
}
func iterate(aMap map[string]interface{}, result *[]Object){
for key, val := range aMap {
switch val.(type) {
case map[string]interface{}:
if(key == "meta"){
switch reflect.TypeOf(val).Kind() {
case reflect.Map:
s := reflect.ValueOf(val)
var tmpData Object
if(s.MapIndex(reflect.ValueOf("name")).IsValid()){
tmpData.name = s.MapIndex(reflect.ValueOf("name")).Interface().(string)
}
if(s.MapIndex(reflect.ValueOf("description")).IsValid()){
tmpData.description = s.MapIndex(reflect.ValueOf("description")).Interface().(string)
}
*result = append(*result, tmpData)
}
}
iterate(val.(map[string]interface{}), result)
default: //DO NOTHING!!
}
}
}

If you're doing pagination, somewhere the data must be represented as a list instead of an object? I assume at some place in your JSON, you have a list of items, otherwise pagination doesn't make sense.
It shouldn't be very hard, something simple like this should work:
const (
itemsPerPage = 10
)
var data []map[string]interface{}
// pages start at 1, can't be 0 or less.
func GetDataPage(page int) []map[string]interface{} {
start := (page - 1) * itemsPerPage
stop := start + itemsPerPage
if start > len(data) {
return nil
}
if stop > len(data) {
stop = len(data)
}
return data[start:stop]
}

You are unmarshalling your json into a map which has no order by itself. In order to be able to paginate your results you need to order them in some way.
One way of doing it is to sort your data and then store it into an array. But in order to paginate you need to have ordered data and that is not possible with a map.

Related

Transform a golang map into another structure of map

I need help for transforming this input map into the output map. I try with switch/case and for but I didn't succeed it. Thanks a lot !
Input :
Values{
"toto_voiture_brand": Ad{
"CITROEN": "CITROEN",
},
"toto_voiture_model": Ad{
"CITROEN_toto": "C3",
},
"toto_moto_brand": Ad{
"KAWASAKI": "KAWASAKI",
},
"toto_moto_model": Ad{
"KAWASAKI_tata": "Ninja 1000 SX",
},
"toto_camion_brand": Ad{
"RENAULT": "RENAULT",
"PEUGEOT": "PEUGEOT",
},
"toto_camion_model": Ad{
"RENAULT_toto": "J5",
"PEUGEOT_tata": "255",
},
},
}
Output
Values{
"toto_voiture_model": {
"Citroen": {
{Value: "C3"},
},
},
"toto_moto_model": {
"Kawasaki": {
{Value: "Ninja 1000 SX"},
},
},
"toto_camion_model": {
"RENAULT": {
{Value: "J5"},
},
"PEUGEOT": {
{Value: "255"},
},
},
}
I've tried with switch case and loop for and map. But I don't have the result attendee, I didn't found how to match every map, key and value. Thanks a lot
I should have managed what you need with the following code:
package main
import (
"encoding/json"
"fmt"
"strings"
)
type Output struct {
Value string `json:"Value"`
}
func main() {
// declare output
output := make(map[string]map[string]Output, 0)
// input
input := make(map[string]map[string]string, 0)
input["toto_voiture_brand"] = map[string]string{
"CITROEN": "CITROEN",
}
input["toto_voiture_model"] = map[string]string{
"CITROEN_toto": "C3",
}
input["toto_moto_model"] = map[string]string{
"KAWASAKI_tata": "Ninja 1000 SX",
}
input["toto_camion_model"] = map[string]string{
"RENAULT_toto": "J5",
"PEUGEOT_tata": "255",
}
// transformation
for k, v := range input {
if strings.HasSuffix(k, "_model") {
tempMap := make(map[string]Output, len(v))
for kk, vv := range v {
key := strings.Split(kk, "_")[0]
tempMap[key] = Output{
Value: vv,
}
}
output[k] = tempMap
}
}
data, _ := json.MarshalIndent(&output, "", "\t")
fmt.Println(string(data))
}
I put some comments within the code just to separate sections. The first two parts are only supposed to define your input and output variables.
The section starting with // transformation is a good candidate to become a function but I preferred to leave it within the main function for demo purposes. Let me recap what's happens in the loop:
You range over the entries of your input variable
If the key has the suffix _model, you take it into consideration
You define a locally-scoped map (called tempMap) of type map[string]Output with the right number of elements that we're gonna add
You range over the v variable (that's why we're dealing with nested maps)
For each item, you're gonna add an entry to the tempMap
At the end of the nested loop, you add an entry to the parent map (output)
The last part is only for printing a beautiful JSON that can be easily read and checked.
Note that this code is simplified just to show off how to achieve your goal, adjust it before putting it into production.
Let me know if this helps, thanks!

How to make pagination from inline buttons Telegram in Golang

I am using the library https://github.com/go-telegram-bot-api/telegram-bot-api
But I do not understand how I can implement the output of buttons and arrows. A lot of data with pagination comes from my api, but I don't understand how to make pages in telegrams. If someone gives an example, I will be very grateful!
This is perfectly implemented in python: https://pypi.org/project/python-telegram-bot-pagination/
But i need for Golang :(
As I checked there were no plugins or wrappers for telegram-bot-api package, so you're gonna have to handle this manually.
Suppose we have this data:
var data = []string{"DummyData1", "DummyData2", "DummyData3", "DummyData4", "DummyData5", "DummyData6", "DummyData7", "DummyData8", "DummyData9", "DummyData10"}
If we're going to show 2 items on each page for 10 items, we would have 5 pages:
var count = 2
var maxPages = len(data) / count // = 5
First we should have a function that calculates data slice and gives us the inlineKeyboardMarkup:
func DummyDataTextMarkup(currentPage, count int) (text string, markup tgbotapi.InlineKeyboardMarkup) {
text = strings.Join(data[currentPage*count:currentPage*count+count], "\n")
var rows []tgbotapi.InlineKeyboardButton
if currentPage > 0 {
rows = append(rows, tgbotapi.NewInlineKeyboardButtonData("Previous", fmt.Sprintf("pager:prev:%d:%d", currentPage, count)))
}
if currentPage < maxPages-1 {
rows = append(rows, tgbotapi.NewInlineKeyboardButtonData("Next", fmt.Sprintf("pager:next:%d:%d", currentPage, count)))
}
markup = tgbotapi.NewInlineKeyboardMarkup(rows)
return
}
Then there is going to be a function to send/edit the calculated data and keyboard by passing chatId, currentPage, count and messageId:
func SendDummyData(chatId int64, currentPage, count int, messageId *int) {
text, keyboard := DummyDataTextMarkup(currentPage, count)
var cfg tgbotapi.Chattable
if messageId == nil {
msg := tgbotapi.NewMessage(chatId, text)
msg.ReplyMarkup = keyboard
cfg = msg
} else {
msg := tgbotapi.NewEditMessageText(chatId, *messageId, text)
msg.ReplyMarkup = &keyboard
cfg = msg
}
bot.Send(cfg)
}
(Note: If we pass a messageId to this function it's going to edit our message with new data)
The next step is to have a CallbackQueryHandler to handle a user's click on inline buttons:
func CallbackQueryHandler(query *tgbotapi.CallbackQuery) {
split := strings.Split(query.Data, ":")
if split[0] == "pager" {
HandleNavigationCallbackQuery(query.Message.MessageID, split[1:]...)
return
}
}
func HandleNavigationCallbackQuery(messageId int, data ...string) {
pagerType := data[0]
currentPage, _ := strconv.Atoi(data[1])
itemsPerPage, _ := strconv.Atoi(data[2])
if pagerType == "next" {
nextPage := currentPage + 1
if nextPage < maxPages {
SendDummyData(chatId, nextPage, itemsPerPage, &messageId)
}
}
if pagerType == "prev" {
previousPage := currentPage - 1
if previousPage >= 0 {
SendDummyData(chatId, previousPage, itemsPerPage, &messageId)
}
}
}
(Note: The first function CallbackQueryHandler is a global callback handler that calls our desired callbackhandler by splitting its query with : and getting the handler's name, here it is pager as we have defined and its handler is called HandleNavigationCallbackQuery).
The last step would be to call CallbackQueryHandler in your update loop as well as sending the initial data to your desired chat:
var chatId = int64(0) // <--- Place Chat Id Here
SendDummyData(chatId, 0, 2, nil) // Send initial data
for update := range updates {
if update.CallbackQuery != nil {
CallbackQueryHandler(update.CallbackQuery)
continue
}
}
You can check the full example on my GitHub's gist here

How to build an array of structs in Go and gRPC?

I am trying to add an array of strings into the content part of a struct with the Id as the array index. I have the code working for one element, but get various errors when I try and add the loop. Any ideas are welcome. I am using Go.
func buildRequest(s []string) []*storepb.LongStoreRequest {
// ss:= []storepb.LongStoreRequest
// int32 i =0 stringv := s[0]
// for i := 0; i < len(s); i++ {
// println(i, apps[i])
ss := []*storepb.LongStoreRequest{
&storepb.LongStoreRequest {
Msg: &storepb.StoreMessage{
Content: stringv,
Account: "trevor3",
Parent: "parentrec",
Id: 0,
},
},
} // }
return ss
}
If I understand your description correctly, you want to build an array of LongStoreRequests, where each element corresponding to an item in the string array, with Id giving the array index. If that's really what you need, something like this should work:
ss := []*storepb.LongStoreRequest{}
for i,str:=range s {
ss=append(ss,&storepb.LongStoreRequest {
Msg: &storepb.StoreMessage{
Content: str,
Account: "trevor3",
Parent: "parentrec",
Id: i,
}})
}

Remove Element From Struct But Only For This One Function

So I have an Struct that holds data that has a AddedByUser which links to my User Struct.
What I want to be able to do it remove the UserLevel from the AddedByUser
Now I want to be able to do it from this function only, so using the json:"-" is not an option. That would remove it from all json output. I only want to remove it form this one function.
I should also say that these are Gorm models and when I have been trying to remove the 10 option (UserLevels) it only removes the outer data set not the UserLevel from all of the data.
{
"ID": 1,
"CreatedAt": "2019-01-08T16:33:09.514711Z",
"UpdatedAt": "2019-01-08T16:33:09.514711Z",
"DeletedAt": null,
"UUID": "00000000-0000-0000-0000-000000000000",
"Title": "title000",
"Information": "info999",
"EventDate": "2006-01-02T15:04:05Z",
"AddedByUser": {
"ID": 2,
"CreatedAt": "2019-01-08T15:27:52.435397Z",
"UpdatedAt": "2019-01-08T15:27:52.435397Z",
"DeletedAt": null,
"UUID": "b019df80-a7e4-4397-814a-795e7e84b4ca",
"Firstname": "Me",
"Surname": "admin",
"Password": "....",
"Email": "admin#email.co.uk",
"UserLevel": {
"ID": 0,
"CreatedAt": "0001-01-01T00:00:00Z",
"UpdatedAt": "0001-01-01T00:00:00Z",
"DeletedAt": null,
"LevelTitle": "",
"UserLevel": null
},
So this is what I have tried,
data := []models.MyData{}
data = append(data[0:2])
I have about 14 results, with out the append it loads all the results but with this is only loads two results. The idea was to remove either UpdateAt or Title. As I am not sure if the gorm model information is all 0 or if the slice sees them as 0,1,2,3,4 etc.
I have also tried to range over the slice of models, while I can access each of the sections, I can not seem to find a simple method to remove data by name from a struct? Maps seem to have that but not structs which I am not sure why?
Thanks.
UPDATE
This is the model I am using:
//Model
type MyData struct {
gorm.Model
UUID uuid.UUID
Title string
Information string
EventDate time.Time
AddedByUser Users `gorm:"ForeignKey:added_by_user_fk"`
AddedByUserFK uint
}
//Users Model
type Users struct {
gorm.Model
UUID uuid.UUID
Firstname string
Surname string
Password string
Email string
UserLevel UserLevels `gorm:"ForeignKey:user_level_fk" json:",omitempty"`
UserLevelFK uint
}
As mentioned in the comments, you cannot remove fields from a struct value, because that would yield a value of a different type.
However, you can set fields to their zero value. Combined with the omitempty JSON tag, you can exclude fields from the JSON encoding. To make this work properly, you have to change the UserLevel field to a pointer type (otherwise you end up with empty objects in the JSON document).
Types shortened for brevity:
package main
import (
"encoding/json"
"fmt"
)
type MyData struct {
Title string
AddedByUser Users
}
type Users struct {
ID int
UserLevel *UserLevels `json:",omitempty"` // pointer type with omitempty
}
type UserLevels struct {
LevelTitle string
}
func main() {
var x MyData
x.Title = "foo"
x.AddedByUser.ID = 2
x.AddedByUser.UserLevel = &UserLevels{}
f(x)
b, _ := json.MarshalIndent(x, "", " ")
fmt.Println("main:\n" + string(b))
}
func f(x MyData) {
// "unset" UserLevel. Since we are receiving a copy of MyData, this is
// invisible to the caller.
x.AddedByUser.UserLevel = nil
b, _ := json.MarshalIndent(x, "", " ")
fmt.Println("f:\n" + string(b))
}
// Output:
// f:
// {
// "Title": "foo",
// "AddedByUser": {
// "ID": 2
// }
// }
// main:
// {
// "Title": "foo",
// "AddedByUser": {
// "ID": 2,
// "UserLevel": {
// "LevelTitle": ""
// }
// }
// }
Try it on the playground: https://play.golang.org/p/trUgnYamVOA
Alternatively, you can define new types that exclude the AddedByUser field. However, since this field isn't at the top level, this is a lot of work, and it's easy to forget to update those types when new fields are added to the original types.
If the field were at the top level, the compiler would do most of the work for you, because types that only differ in their field tags can be directly converted to one another:
type MyData struct {
ID int
Title string
}
func main() {
var x MyData
x.ID = 1
x.Title = "foo"
f(x)
}
func f(x MyData) {
type data struct { // same as MyData, except the field tags
ID int
Title string `json:"-"`
}
b, _ := json.MarshalIndent(data(x), "", " ")
fmt.Println("main:\n" + string(b))
}

Couchbase sort options

I am using Go (1.10.3) with the Gin Framework. I am connecting to a Couchbase database (5.1.1) but I am having some issues with sorting the data.
So this is the code I am working with right now,
//Build Search Sort
func searchSort(c *gin.Context) (cbft.SearchSortField, *cbft.SearchSortScore, string) {
mapData := getQuery(c)
sortData := mapData["sort"]
var sortType string
var sortScore *cbft.SearchSortScore
var sortOps *cbft.SearchSortField
if sortData == "amount:asc" {
fmt.Println( sortData )
sortType = "asc"
sortScore = cbft.NewSearchSortScore().Descending(false)
sortOps = cbft.NewSearchSortField("AmountBase").Descending(false)
}
if sortData == "amount:desc" {
fmt.Println( sortData )
sortType = "desc"
sortScore = cbft.NewSearchSortScore().Descending(true)
sortOps = cbft.NewSearchSortField("AmountBase").Descending(true)
}
return sortOps, sortScore
}
Which I load into the query like so,
sortOpsOne, sortOpsTwo := searchSort(c)
ftSearch := gocb.NewSearchQuery("Search", searchOps).Sort(sortOpsOne).Sort(sortOpsTwo).Limit(size).Skip(from)
To explain, the above loads a get string request, e.g. sort=amount:asc to sort the JSON reply by the AmountBase field I have.
However this does not seem to be working at all, I started out with just the sortOps field but I added the sortScore field as the sort field was not doing anything.
Adding the descending(false) / descending(true) to the end of the sort field does not seem to do anything either.
The search is being run on a FTS index based from the data I have loaded into my bucket.
I am not sure what I am doing wrong?
It seems not to be working with descending options the ascending query seems to be working. :)

Resources