I am trying to implement very simple shard using GORM and DBResolver
Here is the code for it
func (mysqlDB *MySQLDB) InitDB() {
dsn_master := "root:rootroot#tcp(127.0.0.1:3306)/workspace_master?charset=utf8mb4&parseTime=True&loc=Local"
dsn_shard1 := "root:rootroot#tcp(127.0.0.1:3306)/workspace_shard1?charset=utf8mb4&parseTime=True&loc=Local"
db, err := gorm.Open(mysql.Open(dsn_master), &gorm.Config{
Logger: logger.Default.LogMode(logger.Info)})
if err != nil {
log.Println("Connection Failed to Open")
} else {
log.Println("Connection Established")
}
db.Use(dbresolver.Register(dbresolver.Config{
Sources: []gorm.Dialector{mysql.Open(dsn_shard1)}},
&models.WorkspaceGroup{}, "shard1"))
db.AutoMigrate(&models.Workspace{}, &models.WorkspaceMember{})
//db.AutoMigrate(&models.Workspace{}, &models.WorkspaceMember{}, &models.WorkspaceGroup{}, &models.GroupMember{})
db.Clauses(dbresolver.Use("shard1")).AutoMigrate(&models.WorkspaceGroup{}, &models.GroupMember{})
mysqlDB.Database = db
}
I am creating two databases workspace_master and workspace_shard1
Issue is that automigration is not working as expected. Shard Tables are not getting created in respective database. I have tried the commented code as well (automigrate having all the tables and setting db resolver earlier)
Expected Result:
Workspace and WorkspaceMember will get created in workspace_master database
WorkspaceGroup and GroupMember will get created in workspace_shard1 database
Current Result:
All tables are created in workspace_master database
However if I create WorkspaceGroup and GroupMember manually in workspace_shard1, any subsequent queries for create, select, delete etc is going correctly to workspace_shard1. So DBResolver seems to be working as expected.
Only issue is db.AutoMigrate is not working as expected. Can anyone suggest how it can be achieved?
Related
I'm trying to implement mariadb integration to my todo app, as I'm learning go. I decided to use gorm. The error I'm getting is
2022/10/16 21:47:49 C:/Users/xxx/go/src/go-todo-app/server/main.go:44 Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'RETURNING `id`,`id`' at line 1
[0.000ms] [rows:0] INSERT INTO `todos` (`created_at`,`updated_at`,`deleted_at`,`title`,`done`,`body`) VALUES ('2022-10-16 21:47:49.1','2022-10-16 21:47:49.1',NULL,'Testing',0,'Finish Tutorial') RETURNING `id`,`id`
For my http server im using gofiber v2.
app.Post("/api/todos", func(c *fiber.Ctx) error {
todo := &Todo{}
if err := c.BodyParser(todo); err != nil {
return err
}
newTodo := &Todo{
Title: todo.Title,
Body: todo.Body,
Done: 0,
}
db.Create(&newTodo) // fails here
var todos []Todo
db.Find(&todos)
return c.JSON(todos)
})
and my Todo struct looks like this:
type Todo struct {
gorm.Model
ID int `json:"id"`
Title string `json:"title"`
Done int `json:"done"`
Body string `json:"body"`
}
Explicitly disabling the returning in your older MariaDB version will prevent the syntax error:
gorm.Open(mysql.New(mysql.Config{Conn: conn, DisableWithReturning: true}))
The gorm mysql driver shouldn't be enabling WithReturning without a version check. This is a bug that should be reported to hem.
Problem resolved.
The issue was that I had MariaDb installed with Xampp for windows (PHP 7.4). In that version, RETURNING statement is not available for MariaDb. I followed this guide to install MySQL 5.7 instead of MariaDb. How can I change MariaDB to MySQL in XAMPP?
I'm using the Operator SDK to build a custom Kubernetes operator. I have created a custom resource definition and a controller using the respective Operator SDK commands:
operator-sdk add api --api-version example.com/v1alpha1 --kind=Example
operator-sdk add controller --api-version example.com/v1alpha1 --kind=Example
Within the main reconciliation loop (for the example above, the auto-generated ReconcileExample.Reconcile method) I have some custom business logic that requires me to query the Kubernetes API for other objects of the same kind that have a certain field value. It's occurred to me that I might be able to use the default API client (that is provided by the controller) with a custom field selector:
func (r *ReconcileExample) Reconcile(request reconcile.Request) (reconcile.Result, error) {
ctx := context.TODO()
listOptions := client.ListOptions{
FieldSelector: fields.SelectorFromSet(fields.Set{"spec.someField": "someValue"}),
Namespace: request.Namespace,
}
otherExamples := v1alpha1.ExampleList{}
if err := r.client.List(ctx, &listOptions, &otherExamples); err != nil {
return reconcile.Result{}, err
}
// do stuff...
return reconcile.Result{}, nil
}
When I run the operator and create a new Example resource, the operator fails with the following error message:
{"level":"info","ts":1563388786.825384,"logger":"controller_example","msg":"Reconciling Example","Request.Namespace":"default","Request.Name":"example-test"}
{"level":"error","ts":1563388786.8255732,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"example-controller","request":"default/example-test","error":"Index with name field:spec.someField does not exist","stacktrace":"..."}
The most important part being
Index with name field:spec.someField does not exist
I've already searched the Operator SDK's documentation on the default API client and learned a bit about the inner workings of the client, but no detailed explanation on this error or how to fix it.
What does this error message mean, and how can I create this missing index to efficiently list objects by this field value?
The default API client that is provided by the controller is a split client -- it serves Get and List requests from a locally-held cache and forwards other methods like Create and Update directly to the Kubernetes API server. This is also explained in the respective documentation:
The SDK will generate code to create a Manager, which holds a Cache and a Client to be used in CRUD operations and communicate with the API server. By default a Controller's Reconciler will be populated with the Manager's Client which is a split-client. [...] A split client reads (Get and List) from the Cache and writes (Create, Update, Delete) to the API server. Reading from the Cache significantly reduces request load on the API server; as long as the Cache is updated by the API server, read operations are eventually consistent.
To query values from the cache using a custom field selector, the cache needs to have a search index for this field. This indexer can be defined right after the cache has been set up.
To register a custom indexer, add the following code into the bootstrapping logic of the operator (in the auto-generated code, this is done directly in main). This needs to be done after the controller manager has been instantiated (manager.New) and also after the custom API types have been added to the runtime.Scheme:
package main
import (
k8sruntime "k8s.io/apimachinery/pkg/runtime"
"example.com/example-operator/pkg/apis/example/v1alpha1"
// ...
)
function main() {
// ...
cache := mgr.GetCache()
indexFunc := func(obj k8sruntime.Object) []string {
return []string{obj.(*v1alpha1.Example).Spec.SomeField}
}
if err := cache.IndexField(&v1alpha1.Example{}, "spec.someField", indexFunc); err != nil {
panic(err)
}
// ...
}
When a respective indexer function is defined, field selectors on spec.someField will work from the local cache as expected.
I am using GORM ORM for Postgres access in a Go application.
I have got 4 functions, Create, Update, Delete, and Read in a database repository.
In each of these functions, I open a database connection, perform a CRUD operation and then close the connection just after the operation is performed as you will see here and here and in the code snippet below, using GORM
func (e *Example) Create(m *model.Example) (*model.Example, error) {
// open a database session
dbSession, err := e.OpenDB() //gorm.Open("postgres", connStr)
if err != nil {
log.Log(err)
return nil, err
}
// close database connection after operation is completed
defer dbSession.Close()
// create item
db := dbSession.Create(m)
if db.Error != nil {
return nil, db.Error
}
return m, nil
}
Is that the correct practice or should one database connection be shared in the whole application and let the ORM handle managing connections? as stated here?
You should reuse a DB connection as much as you can. Also gorm has a built-in connection pool, so, you don't need to manage the db handle. Simply share it amongst all goroutines and they can share the handle safely, allocating new connections as needed.
I'm working on a simple application written in golang, using tiedot as NoSQL database engine.
I need to store some users in the database.
type User struct {
Login string
PasswordHash string
Salt string
}
Of course two users cannot have the same login, and - as this engine does not provide any transaction mechanism - I'm wondering how to ensure that there's no duplicated login in the database when writing.
I first thought that I could just search for user by login before inserting, but as the database will be
used concurently, it is not reliable.
Maybe I could wait for a random time and if there is another user with the same login in the collection, delete it, but that does not sound reliable either.
Is this even possible, or should I switch to a database engine that support transactions ?
Below is my solution. It is not Tiedot specific, but It uses CQRS and can be applied to various DBs.
You can also have other benefits using it, such as caching and bulk write (in case DB supports it) to prevent asking DB on every request.
package main
import (
"sync"
"log"
"errors"
)
type User struct {
Login string
PasswordHash string
Salt string
}
type MutexedUser struct {
sync.RWMutex
Map map[string]User
}
var u = &MutexedUser{}
func main() {
var user User
u.Sync()
// Get new user here
//...
if err := u.Insert(user); err != nil {
// Ask to provide new login
//...
log.Println(err)
}
}
func (u *MutexedUser) Insert(user User) (err error) {
u.Lock()
if _, ok := u.Map[user.Login]; !ok {
u.Map[user.Login] = user
// Add user to DB
//...
u.Unlock()
return err
}
u.Unlock()
return errors.New("duplicated login")
}
func (u *MutexedUser) Read(login string) User {
u.RLock()
value := u.Map[login]
u.RUnlock()
return value
}
func (u *MutexedUser) Sync() (err error) {
var users []User
u.Lock()
defer u.Unlock()
// Read users from DB
//...
u.Map = make(map[string]User)
for _, user := range users {
u.Map[user.Login] = user
}
return err
}
I first thought that I could just search for user by login before inserting, but as the database will be used concurently, it is not reliable.
Right, it creates a race condition. The only way to resolve this is:
Lock the table
Search for the login
Insert if the login is not found
Unlock the table
Table-locks are not a scalable solution, because it creates an expensive bottleneck in your application. It's why non-transactional storage engines like MySQL's MyISAM are being phased out. It's why MongoDB has to use clusters to scale up.
It can work if you have a small dataset size and a light amount of concurrency, so perhaps it's adequate for login creation on a lightly-used website. New logins probably aren't created so frequently that they need to scale up so much.
But users logging in, or password changes, or other changes to account attributes, do happen more frequently.
The solution for this is to make this operation atomic, to avoid race conditions. For example, attempt the insert and have the database engine verify uniqueness and reject the insert if it violates that constraint.
Unfortunately, I don't see any documentation in tiedot that shows that it supports a unique constraint or a uniqueness enforcement on indexes.
Tiedot is 98% written by a single developer, in a period of about 2 years (May 2013 - April 2015). Very little activity since then (see https://www.openhub.net/p/tiedot). I would consider tiedot to be an experimental project, unlikely to expand in feature set.
I'm trying to insert a simple record as in GoDoc. But this returns,
rpc error: code = 7 desc = "User can't access project: tidy-groove"
When I searched for grpc codes, it says..
PermissionDenied Code = 7
// Unauthenticated indicates the request does not have valid
// authentication credentials for the operation.
I've enabled Big table in my console and created a cluster and a service account and recieved the json. What I'm doing wrong here?
package main
import (
"fmt"
"golang.org/x/net/context"
"golang.org/x/oauth2/google"
"google.golang.org/cloud"
"google.golang.org/cloud/bigtable"
"io/ioutil"
)
func main() {
fmt.Println("Start!")
put()
}
func getClient() *bigtable.Client {
jsonKey, err := ioutil.ReadFile("TestProject-7854ea9op741.json")
if err != nil {
fmt.Println(err.Error())
}
config, err := google.JWTConfigFromJSON(
jsonKey,
bigtable.Scope,
) // or bigtable.AdminScope, etc.
if err != nil {
fmt.Println(err.Error())
}
ctx := context.Background()
client, err := bigtable.NewClient(ctx, "tidy-groove", "asia-east1-b", "test1-bigtable", cloud.WithTokenSource(config.TokenSource(ctx)))
if err != nil {
fmt.Println(err.Error())
}
return client
}
func put() {
ctx := context.Background()
client := getClient()
tbl := client.Open("table1")
mut := bigtable.NewMutation()
mut.Set("links", "maps.google.com", bigtable.Now(), []byte("1"))
mut.Set("links", "golang.org", bigtable.Now(), []byte("1"))
err := tbl.Apply(ctx, "com.google.cloud", mut)
if err != nil {
fmt.Println(err.Error())
}
}
I've solved the problem. It's nothing wrong with the code, but config json itself. So anyone who out there want to authenticate and came here by google search... This code is correct and working perfectly. What I've done wrong is follows.
First I made a service account and got the json. But google warned me that im not an owner of project hence it wont be added to accept list but anyway it let me download the json.
Then I deleted that key from console and requested project owner to create a key for me.
There he has created another key with the same name I given.. And since he's the owner no error/warning msgs displayed and successfully json file was downloaded.
When I tried with that... my question begun. That's when i posted this question.
After that with no solutions. I asked owner to delete that key and create another key but with a different name..
Then it worked! It seems if you try to create a key with non-owner account and then again create with same name ( after deleting original of course ) has no effect. Hope this helps everyone out there :)
Take a look at: helloworld.go or search.go which uses GOOGLE_APPLICATION_CREDENTIALS environment variable.
For most environments, you no longer even need to set GOOGLE_APPLICATION_CREDENTIALS. Google Cloud Platform, Managed VMs or Google App Engine all have the right thing set for you. Your desktop environment will also be correct if you've used gcloud init or it's predecessor gcloud auth login followed by gcloud config set project <projectID>.