I have a widget.json file which is loaded into a document in couchbase:
{
"type": "widget",
"name": "clicker",
"description": "clicks!"
}
I also have a couchbase design document, couchbase.ddoc, for a bucket. It is registered with the name "views":
{
"_id": "_design/accesscheck",
"language": "javascript",
"views": {
"all_widgets": {
"map": "function(doc, meta) { if(doc.type == 'widget') { emit(meta.id, doc.name); } }"
}
}
}
and some golang code, using the couchbase golang API, to fetch it:
opts := map[string]interface{}{
"stale": false
}
data, _ := bucket.View("views", "all_widgets", opts)
and at this point i still have A Few Questions:
what is the best way to determine what i can do with the "data" variable? I suspect it's a list of integer-indexed rows, each of which contains a key/value map where the value can be of different types. I see plenty of trivial examples for map[string]interface{}, but this seems to have an additional level of indirection. Easily understandable in C, IMHO, but the interface{}{} is puzzling to me.
probably an extension of the above answer, but how can i effectively search using the design document? I would rather have the Couchbase server doing the sifting.
some of the Get() examples pass in a struct type i.e.
type User struct {
Name string `json:"name"`
Id string `json:"id"`
}
err = bucket.Get("1", &user)
if err != nil {
log.Fatalf("Failed to get data from the cluster (%s)\n", err)
}
fmt.Printf("Got back a user with a name of (%s) and id (%s)\n", user.Name, user.Id)
Is it possible to do something like this with a View()?
does somebody know of a good way to flexibly hook the View mechanism into a net/http REST handler? Just hoping..
I didn't find these questions covered in the golang client API examples or documentation. I probably missed something. If someone has links please let me know.
Thanks for any help!
Customize View Result In Go
If you are using github.com/couchbaselabs/go-couchbase, you can use bucket.ViewCustom to solve your problem. It accepts a item which view result parsed to.
The bucket.View is just calling bucket.ViewCustom with predefined struct and returns it.
An executed view result is a json object like below;
{
"total_rows": 123,
"rows": [
{
"id": "id of document",
"key": {key you emitted},
"value": {value you emitted},
"doc": {the document}
},
...
]
}
As we are know this structure, we can set struct type manually to bucket.ViewCustom.
In your case, you can write custom view struct for "all_widgets" like this;
type AllWidgetsView struct {
Total int `json:"total_rows"`
Rows []struct {
ID string `json:"id"` // document id
Key string `json:"string"` // key you emitted, the 'meta.id'
Value string `json:"value"` // value you emitted, the 'doc.name'
} `json:"rows"`
}
And use bucket.ViewCustom to retrieve the value.
var result AllWidgetsView
opts := map[string]interface{}{}
viewErr := bucket.ViewCustom("views", "all_widgets", opts, &result)
if viewErr != nil {
// handle error
}
If this pattern of code appears frequently, you can refactor it.
Effective searching
For the question 2, it's not related to golang but Couchbase itself.
View has some feature that bucket has not. View results are sorted by key so you can specify startkey option to where the result start. You can use this attribute and make view for searching. See Querying views for more information.
But you need more detailed search like full-text search, you should use ElasticSearch plugin or N1QL to do it.
(note the N1QL is preview and not officially released yet)
Related
I'm building a graphql interface using golang. I'm using gqlgen package to implement it.
Here I need to pass all field names in a query to get it in response, But the problem is my data is huge, it is having more than 30 fields it would be difficult to pass all fields in a query.
This is my query
{Model{id, name, email, mobile,...............}}
Like this I need to pass all fields name.
Instead Im looking for a result which will return all fields without passing any fields. I mean if not passing any field names it should return all.
For example
{Model{}}
First, you really should list out all the fields in your query. That is the nature of graphql. It is verbose, but most client libraries get the fields from your data structure anyway, so it's not that bad.
So I recommend listing out all fields manually!
Using Scalars (must be on v0.11.3 or below, see https://github.com/99designs/gqlgen/issues/1293)
But if you insist, if there is a will, there is way. You can use GraphQL's scalar types and make your own. See this doc for how to make them with gqlgen: https://gqlgen.com/reference/scalars/
In your schema, you can make a JSON scalar:
scalar JSON
type Query {
random: JSON!
}
Make a model for this
// in your own models.go
// You can really play with this to make it better, easier to use
type JSONScalar json.RawMessage
// UnmarshalGQL implements the graphql.Unmarshaler interface
func (y *JSONScalar) UnmarshalGQL(v interface{}) error {
data, ok := v.(string)
if !ok {
return fmt.Errorf("Scalar must be a string")
}
*y = []byte(data)
return nil
}
// MarshalGQL implements the graphql.Marshaler interface
func (y JSONScalar) MarshalGQL(w io.Writer) {
_, _ = w.Write(y)
}
Then link the scalar to your custom type in the gql.yml
models:
JSON:
model: github.com/your-project/path/graph/model.JSONScalar
When you run the generate (use gqlgen v0.11.3 or below, gqlgen version), your resolvers will now use the custom type you made. And it's easy to use:
func (r *queryResolver) random(ctx context.Context) (model.JSONScalar, error) {
// something is the data structure you want to return as json
something := struct {
Value string
}{
Value: "Hello World",
}
d, _ := json.Marshal(something)
return model1.JSONScalar(d), nil
}
The resulting query of
// Query
{
random
}
// Response
{
"random" : {
"Value": "Hello World!"
}
}
I am trying to set up a many to many relationship in my demo API, between a Job, and a list of skills []Skill.
Job Struct
type Job struct {
ID string `sql:"type:uuid;primary_key;"`
Title string `json:"title,omitempty"`
Skills []*skill.Skill `json:"skills,omitempty"gorm:"many2many:job_skill;"`
CreatedAt time.Time
UpdatedAt time.Time
}
Skill Struct
type Skill struct {
gorm.Model
Name string `json:"name,omitempty"`
}
I then use gorm.DB.AutoMigrate() to generate the join table automatically.
When I send a POST request to my API, the data is created correctly the first time around, and the join table populates as you'd expect.
Example POST data
{
"title": "Senior Python Engineer",
"skills": [{"name": "javascript"}, {"name": "python"}]
}
But then when I send a PATCH request to add a new skill, it duplicates the skill in the skills table and then creates a new record in the join table for the skills that already exist.
Example PATCH data
{
"title": "Lead Engineer",
"skills": [{"name": "javascript"}, {"name": "python"}, {"name": "management"}]
}
Doing a get request for the data will show the following:
{
"title": "Lead Engineer",
"skills": [{"name": "javascript"}, {"name": "python"}, {"name": "javascript"}, {"name": "python"}, {"name": "management"}]
}
I have also tried setting gorm:"unique" on the skill struck Name but when adding a new Skill it fails as it says the other two already exist, which is good but then won't add the new one.
I am assuming I can only send back new values? Not the entire list?
Some of my Go code for clarity
func GetJobs(w http.ResponseWriter, r *http.Request) {
j := &[]Job{}
o := database.DB.Preload("Skills").Find(&j)
render.JSON(w, r, o)
}
func UpdateJob(w http.ResponseWriter, r *http.Request) {
j := &Job{}
err := json.NewDecoder(r.Body).Decode(j)
if err != nil {
return
}
o := database.DB.Model(&j).Where("id = ?", j.ID).Update(&j)
render.JSON(w, r, o)
}
These are two things with gorm associations that I feel aren't adequately conveyed in its documentation which continue to confuse developers.
1) It uses IDs to identify the associated entities
When you Update that list of skills, if they have no IDs in them to gorm they are just new entities to be added. This leads to your duplicated values. Even if you have a unique field, if that field isn't the entity's primary key, then gorm will try to create a new record and result in a constraint violation instead.
There are a few ways of dealing with this:
Making sure the API user must supply an ID for the related entities
Pulling out of the DB the entity IDs via some other surrogate key that the user does provide, and populating those in your to-save entity. In your case that could be name since it's unique.
Making that surrogate key your primary key (making name the gorm:"primaryKey" of your Skill struct).
2) When Updating, it won't delete existing associations that don't appear in the association slice
When you call Save/Update gorm doesn't delete entities in the far side of collection associations. This is a safety feature to avoid accidentally deleting data on a simple Save/Update. You have to be explicit about wanting that behaviour.
To deal with that you can use Association mode to replace the collection as part of your update: db.Model(&job).Association('Skills').Replace(&job.Skills).
To elaborate on Eziquel's answer and show what worked for me.
I updated my Skill struct to use the Name as the primary key
type Skill struct {
Name string `json:"name,omitempty" gorm:"primary_key"`
}
I then read some documentation saing to just use gorm.DB.Session(&gorm.Session{FullSaveAssociations: true}).Updates(&j) Not sure what it does, but without it, the associations were duplicating or not updating.
func UpdateJob(w http.ResponseWriter, r *http.Request) {
j := &Job{}
err := json.NewDecoder(r.Body).Decode(j)
if err != nil {
return
}
database.DB.Session(&gorm.Session{FullSaveAssociations: true}).Updates(&j)
database.DB.Model(&j).Association("Skills").Replace(&j.Skills)
render.JSON(w, r, &j)
}
I dont know what database.DB.Session(&gorm.Session{FullSaveAssociations: true}).Updates(&j) does but I saw Jihnzu say to use it in the docs, no further explanation. Using that in combination with the .Association("Skills").Replace(&j.Skills) ensure there are no duplicates and that the association updates.
If there was more than one you would do:
database.DB.Session(&gorm.Session{FullSaveAssociations: true}).Updates(&j)
database.DB.Model(&j).Association("Skills").Replace(&j.Skills)
database.DB.Model(&j).Association("Locations").Replace(&j.Locations)
type Users struct {
ID int64
Email string
Permissions string
}
type UserPermissions struct {
ID int64
Description json.RawMessage
}
The user json should be like this:
{
"status": 200,
"data": {
"id": 1,
"email": "hello#hello.com",
"permisions": [{
"id":"1",
"description":"Create permission"
},
{
"id":"3",
"description":"Edit permission"
}]
}
}
I have the following string in the Permissions of my User:
';1;3;5;7;' every number is the id related to the UserPermissions struct/table
How can I match the string with the user permission table using gorm?.
I'm using mysql
An answer to your question (if you are using PostgreSQL as your DB).
If you want to query for the Permissions that a given User record has, given its ID
perms := []Permission{}
err := db.Joins("JOIN users u ON id::string IN STRING_TO_ARRAY(TRIM(';' FROM u.permissions), ';')").
Where("u.id = ?", userID).
Find(&perms).
Error
if err != nil {
log.Fatal(err) // or something like that
}
As you can see, we are doing a funky join with functions that clean and split the permission field into individual string IDs. This is clunky, brittle and super slow and convoluted, and stems from the faulty relational design that you started with.
EDIT: Nevermind, for MySQL the equivalent solution is very much more complicated, so I suggest you instead use the advice below.
A better way
If you have control over the database design, the better way to do this would be
by not storing an array of IDs as a string, never a good thing in database design, but instead using a foreign key on the many side of the HasMany relationship*.
This would look like this as go structs:
type User struct {
ID int64 `json:"id"`
Email string `json:"email"`
Permissions []UserPermission `json:"permissions"`
}
type UserPermissions struct {
ID int64 `json:"id"`
UserID int64 `json:"-"`
Description json.RawMessage `json:"description"`
}
// now you can use gorm to query the permissions that are related to a user
user := User{}
err := db.Preload("Permissions").First(&user, userID).Error
if err != nil {
log.Fatal(err)
}
// now you can access user.Permissions[i].Description for example.
// or marshal to json
out, err := json.Marshal(user)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(out))
*: I am assuming that the relationship between User and UserPermission is One-to-many. This may very well not be the case, and a Many-to-many relationship actually makes sense (One user can have many permissions, one permission can belong to many users). If this is the case, the concepts are the same and you should be able to modify this solution following the Gorm guide on many to many relationships.
What do I have: two structs for some kind of API
type BaseUser struct {
ID int64 `json:"user_id"`
Name string `json:"user_name"`
Email string `json:"user_email"`
}
and
type UserWithAddress struct {
BaseUser
Postal string `json:"user_postal"`
City string `json:"user_city"`
Street string `json:"user_street"`
}
What do I want to do: convert json keys from snake_case to camelCase.
Lets say, this is a request body
{
"user_id": 123,
"user_name": "test",
"user_email": "test#mail.com",
"user_postal": "12312",
"user_city": "city",
"user_street": "street"
}
So as a result, after some kind of transformation, I'd like to have this output
{
"userId": 123,
"userName": "test",
"userEmail": "test#mail.com",
"userPostal": "12312",
"userCity": "city",
"userStreet": "street"
}
How do I handle this at the moment: I made another two structs with camelCase json tag
type BaseUserCamelCase struct {
ID int64 `json:"userId"`
Name string `json:"userName"`
Email string `json:"userEmail"`
}
and
type UserWithAddressCamelCase struct {
BaseUserCamelCase
Postal string `json:"userPostal"`
City string `json:"userCity"`
Street string `json:"userStreet"`
}
My transformation looks like
var userWithAddressCamelCase UserWithAddressCamelCase
userWithAddressCamelCase.BaseUserCamelCase = BaseUserCamelCase(userWithAddress.BaseUser)
//I can't cast whole userWithAddressCamelCase object to another type because of different field names - BaseUser and BaseUserCamelCase
userWithAddressCamelCase.Name = userWithAddress.Name
userWithAddressCamelCase.Email = userWithAddress.Email
userWithAddressCamelCase.Postal = userWithAddress.Postal
//and so on
and I don't like it, because if BaseUser or UserWithAddress will grow up, I have to add appropriate field to %CamelCase structs.
My question: is there another more efficient way to handle keys transformation?
Is there another more efficient way to handle keys transformation?
No.
(Well, based on your definition of "efficient". You could use reflection, but I will not recommend this. Your code is perfectly fine. If any struct grows you add a few lines of simple code. There is nothing wrong with simple code which is not going to produce errors and is fast during execution. Just because it doesn't look fancy it doesn't mean that there is anything to "improve" here.)
If the need to maintain the field list is what concerns you the most, then I'd suggest making type aliases for your user types and implementing the json.Marshaler interface for those aliases, where you'd implement custom JSON encoding. You can even introduce an alternative set of tags and use those there.
Something along these lines:
type BaseUser struct {
ID int64 `json:"user_id" jsonCC:"userId"`
Name string `json:"user_name" jsonCC:"userName"`
Email string `json:"user_email" jsonCC:"userEmail"`
}
type BaseUserCamelCase BaseUser
func (bucc BaseUserCamelCase) MarshalJSON() ([]byte, error) {
buccVal := reflect.ValueOf(bucc)
kvpairs := []string{}
for i := 0; i < buccVal.NumField(); i++ {
k := buccVal.Type().Field(i).Tag.Get("jsonCC")
v := buccVal.Field(i).Interface() //TODO: proper JSON encoding of things
kvpairs = append(kvpairs, fmt.Sprintf("\"%s\":%#v", k, v))
}
return []byte(fmt.Sprintf("{%s}", strings.Join(kvpairs, ","))), nil
}
Then you can choose marshaling style:
user := BaseUser{
ID: 123,
Name: "Johnny D03",
Email: "j#example.com",
}
json.Marshal(user)
// {"user_id":123,"user_name":"Johnny D03","user_email":"j#example.com"}
json.Marshal(BaseUserCamelCase(user))
// {"userId":123,"userName":"Johnny D03","userEmail":"j#example.com"}
Assume a JSON object with the general format
"accounts": [
{
"id": "<ACCOUNT>",
"tags": []
}
]
}
I can create a struct with corresponding json tags to decode it like so
type AccountProperties struct {
ID AccountID `json:"id"`
MT4AccountID int `json:"mt4AccountID,omitempty"`
Tags []string `json:"tags"`
}
type Accounts struct {
Accounts []AccountProperties `json:"accounts"`
}
But the last struct with just one element seems incorrect to me. Is there a way I could simply say type Accounts []AccountProperties `json:"accounts"` instead of creating an entire new struct just to decode this object?
You need somewhere to store the json string accounts. Using a:
var m map[string][]AccountProperties
suffices, though of course you then need to know to use the string literal accounts to access the (single) map entry thus created:
type AccountProperties struct {
ID string `json:"id"`
MT4AccountID int `json:"mt4AccountID,omitempty"`
Tags []string `json:"tags"`
}
func main() {
var m map[string][]AccountProperties
err := json.Unmarshal([]byte(data), &m)
fmt.Println(err, m["accounts"])
}
See complete Go Playground example (I had to change the type of ID to string and fix the missing { in the json).
As Dave C points out in comments, this is no shorter than just using an anonymous struct type:
var a struct{ Accounts []AccountProperties }
in terms of the Unmarshall call (and when done this way it's more convenient to use). Should you want to use an anonymous struct like this in a json.Marshall call, you'll need to tag its single element to get a lowercase encoding: without a tag it will be called "Accounts" rather than "accounts".
(I don't claim the map method to be better, just an alternative.)