Golang Redis error on heroku - heroku

I am having a hard time on this one because I cannot reproduce it locally. Intention originally was to test if a Redis connection is accessible from a go method if so continue else inform.
In my code:
if os.Getenv("REDISTOGO_URL") != "" {
host, password = parseRedistogoUrl()
} else {
host = "tcp:localhost:6379"
password = ""
}
db := 0
redis = godis.New(host, db, password)
// Try to set a "dummy" value:
err = redis.Set("INIT", time.Now().UTC())
With a Redis server available locally this executes without an error, but fails when on heroku.
What makes the problem more interesting is that if the last line (with err) is commented out then the next redis.Set in the program succeeds, which indicates that the server is accessible later on, but not "just after" the instanciation. Any ideas on how I should resolve this? Is godis.New(...) running asynchronously?
Update: This is the parsing function currently used from https://gist.github.com/TheDudeWithTheThing/6468746:
func ParseRedistogoUrl() (string, string) {
redisUrl := os.Getenv("REDISTOGO_URL")
redisInfo, _ := url.Parse(redisUrl)
server := redisInfo.Host
password := ""
if redisInfo.User != nil {
password, _ = redisInfo.User.Password()
}
return server, password
}
Rewriting the same method with redigo and https://github.com/soveran/redisurl:
// Gets redis instance from configuration parameters or environment's REDISTOGO_URL.
func redis_instance() (redis_con redis.Conn, err error) {
if os.Getenv("REDISTOGO_URL") != "" {
redis_con, err = redisurl.ConnectToURL(os.Getenv("REDISTOGO_URL"))
} else {
redis_con, err = redis.Dial("tcp", ":6379")
}
if err != nil {
return
}
// Try to set a "dummy" value, panic if redis is not accessible.
err = redis_con.Send("Set", "INIT", time.Now().UTC())
return
}
and from program's main:
redis, err := redis_instance()
redis.Send("Set", "SOMETHING", "ELSE")
if err != nil {
log.Critical("Cannot access Redis server")
log.Critical(fmt.Sprintf("%s", err))
os.Exit(2)
}
On logs I get the following output:
2014-09-16T18:15:10.084908+00:00 app[web.1]: 2014/09/16 18:15:10 CRITICAL003 Cannot access Redis server
2014-09-16T18:15:10.084974+00:00 app[web.1]: 2014/09/16 18:15:10 CRITICAL004 Connection error 10942
About redis access:
~ $ echo $REDISTOGO_URL
redis://redistogo:47███████████#hoki.redistogo.com:10███/
~ $ ping hoki.redistogo.com
bash: ping: command not found
~ $ nc hoki.redistogo.com 10███
flushdb
-NOAUTH Authentication required.
AUTH 47███████████
+OK
get XXX
$-1
set XXX 2
+OK
get XXX
$1
2

Related

GORM and PostgreSQL: search_path is used only when doing the first request

I'm currently working on an api gateway that forwards requests to an grpc client. The routes use a middleware. Within that middleware the Validate function from the dialed grpc client is used. The function is in the last code snippet at the end.
The gateway is based on a gin router, which, as far as I understand handles each request in a separat go routine. I can send as many requests simultaneously to either to endpoint /protected/1 or to endpoint /protected/2 and they get handled correctly.
The Validate function is having problems handling request sent simultaneously to /protected/1 and /protected/2. Sending two request to both endpoints simultaneously results in that either both, none or only one of the requests is handled correctly. The error message that I receive is actually from the gorm function trying to query the user in the Validate function (ERROR: relation "users" does not exist (SQLSTATE 42P01)). I use a gorm database object that is connected to a postgres database. This behaviour suggests that some how a resource is shared. Using run -race does not give any insights.
So my questions are, why does this setup not work to send requests simultaneously to both endpoints?
I tried to make a working minimal example, but somehow I could nor reproduce the error in a minimal setting, hence I would like to share my code snippets here.
Important snippets of the the Api Gateway implementation
func main() {
c, err := config.LoadConfig()
...
r := gin.Default()
auth.RegisterRoutes(r, &c)
r.Run(":" + c.Port)
}
whereas the routes are defined as
func RegisterRoutes(r *gin.Engine, c *config.Config){
svc := &ServiceClient{
Client: InitServiceClient(c),
}
a := InitAuthMiddleware(svc)
// routes
protected := r.Group("/protected")
protected.Use(a.AuthRequired)
protected.GET("/1", svc.DoStuff)
protected.GET("/2", svc.DoStuff)
}
// dummy handler
func (svc *ServiceClient) DoStuff(ctx *gin.Context) {
handler.DoStuff(ctx, svc.Client)
}
The grpc client is dialed with the following function
type ServiceClient struct {
Client pb.AuthServiceClient
}
func InitServiceClient(c *config.Config) pb.AuthServiceClient {
// using WithInsecure() because no SSL running
cc, err := grpc.Dial(c.AuthSvcUrl, grpc.WithInsecure())
if err != nil {
fmt.Println("Could not connect:", err)
}
return pb.NewAuthServiceClient(cc)
}
and the middleware is implemented in the following way:
type AuthMiddlewareConfig struct {
svc *ServiceClient
}
func InitAuthMiddleware(svc *ServiceClient) AuthMiddlewareConfig {
return AuthMiddlewareConfig{svc}
}
func (c *AuthMiddlewareConfig) AuthRequired(ctx *gin.Context) {
access_token, _ := ctx.Cookie("access_token")
res, err := c.svc.Client.Validate(context.Background(), &pb.ValidateRequest{
Token: access_token,
TokenType: "ACCESS_TOKEN",
Url: ctx.Request.URL.String(),
})
if err != nil || res.Status != http.StatusOK {
ctx.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"status": http.StatusUnauthorized, "error": res.Error})
return
}
ctx.Set("userId", res.UserId)
ctx.Next()
}
Important snippets of the Auth Service:
func serve(c *conf.Configuration) {
/*
Dial returns:
type Repository struct {
DB *gorm.DB
}
*/
h := storage.Dial(&c.DB)
serviceUri := fmt.Sprintf(":%s", c.API.Port)
lis, err := net.Listen("tcp", serviceUri)
if err != nil {
logrus.Fatal("Failed to listen on: ", err)
}
s := api.Server{
R: h,
//
}
grpcServer := grpc.NewServer()
pb.RegisterAuthServiceServer(grpcServer, &s)
if err := grpcServer.Serve(lis); err != nil {
logrus.Fatalln("Failed to serve:", err)
}
}
and the validate function is implemented by
type Server struct {
R storage.Repository
//
// #https://github.com/grpc/grpc-go/issues/3794:
pb.UnimplementedAuthServiceServer
}
func (s *Server) Validate(ctx context.Context, req *pb.ValidateRequest) (*pb.ValidateResponse, error) {
// token validation stuff works
var user models.User
// causes error sometimes
if result := s.R.DB.Where(&models.User{Id: claims.Id}).First(&user); result.Error != nil {
return &pb.ValidateResponse{
Status: http.StatusNotFound,
Error: "User not found",
}, nil
}
// send response
}
EDIT
For completeness here the function that connects to the postgres client.
func openDb(c *conf.PostgresConfiguration, gormConfig *gorm.Config, database string) *gorm.DB {
connectionString := fmt.Sprintf("postgres://%s:%s#%s:%s/%s", c.User, c.Password, c.Host, c.Port, database)
db, err := gorm.Open(postgres.Open(connectionString), gormConfig)
if err != nil {
logrus.Fatal(fmt.Sprintf("Unable to connect to database '%s' given url: ", database), err)
}
logrus.Info(fmt.Sprintf("Connected to database %s", database))
return db
}
func Dial(c *conf.PostgresConfiguration) Repository {
gormDefaultConfig := &gorm.Config{}
gormAppConfig := &gorm.Config{}
// connect to default database
logrus.Info(fmt.Sprintf("Use default database %s for initialization", c.DefaultDatabase))
defaultDb := openDb(c, gormDefaultConfig, c.DefaultDatabase)
// check if database exists and create it if necessary
var dbexists bool
dbSQL := fmt.Sprintf("SELECT EXISTS (SELECT FROM pg_database WHERE datname = '%s') AS dbexists;", c.AppDatabase)
defaultDb.Raw(dbSQL).Row().Scan(&dbexists)
if !dbexists {
logrus.Info(fmt.Sprintf("Created database %s", c.AppDatabase))
db := defaultDb.Exec(fmt.Sprintf("CREATE DATABASE %s;", c.AppDatabase))
if db.Error != nil {
logrus.Fatal("Unable to create app database: ", db.Error)
}
}
// connect to app databse
appDb := openDb(c, gormAppConfig, c.AppDatabase)
// check if schema exists and create it if necessary
var schemaexists bool
schemaSQL := fmt.Sprintf("SELECT EXISTS(SELECT FROM pg_namespace WHERE nspname = '%s') AS schemaexisits;", c.AppDatabaseSchema)
appDb.Raw(schemaSQL).Row().Scan(&schemaexists)
// create app specfic database, if not already existing
if !schemaexists {
// create service specific schema
db := appDb.Exec(fmt.Sprintf("CREATE SCHEMA %s;", c.AppDatabaseSchema))
if db.Error != nil {
logrus.Fatal(fmt.Sprintf("Unable to create database schema %s", c.AppDatabaseSchema), db.Error)
}
logrus.Info(fmt.Sprintf("Created database schema %s", c.AppDatabaseSchema))
}
db := appDb.Exec(fmt.Sprintf(`set search_path='%s';`, c.AppDatabaseSchema))
logrus.Info(fmt.Sprintf("Use existing database schema %s", c.AppDatabaseSchema))
if db.Error != nil {
logrus.Fatal(fmt.Sprintf("Unable to set search_path for database to schema %s", c.AppDatabaseSchema), db.Error)
}
// migrate table
appDb.AutoMigrate(&models.User{})
return Repository{appDb}
}
Edit2
Changing the connection string in the openDb function to connectionString := fmt.Sprintf("postgres://%s:%s#%s:%s/%s?search_path=%s", c.User, c.Password, c.Host, c.Port, database, c.AppDatabaseSchema) fixes the issue (see how the search_path is part of the url).
Thank you!

Can I connect to Memgraph using Go?

I'd like to connect from Go to the running instance of the Memgraph database. I'm using Docker and I've installed the Memgraph Platform. What exactly do I need to do?
The procedure for connecting fro Go to Memgraph is rather simple. For this you need to use Bolt protocol. Here are the needed steps:
First, create a new directory for your app, /MyApp, and position yourself in it. Next, create a program.go file with the following code:
package main
import (
"fmt"
"github.com/neo4j/neo4j-go-driver/v4/neo4j"
)
func main() {
dbUri := "bolt://localhost:7687"
driver, err := neo4j.NewDriver(dbUri, neo4j.BasicAuth("username", "password", ""))
if err != nil {
panic(err)
}
// Handle driver lifetime based on your application lifetime requirements driver's lifetime is usually
// bound by the application lifetime, which usually implies one driver instance per application
defer driver.Close()
item, err := insertItem(driver)
if err != nil {
panic(err)
}
fmt.Printf("%v\n", item.Message)
}
func insertItem(driver neo4j.Driver) (*Item, error) {
// Sessions are short-lived, cheap to create and NOT thread safe. Typically create one or more sessions
// per request in your web application. Make sure to call Close on the session when done.
// For multi-database support, set sessionConfig.DatabaseName to requested database
// Session config will default to write mode, if only reads are to be used configure session for
// read mode.
session := driver.NewSession(neo4j.SessionConfig{})
defer session.Close()
result, err := session.WriteTransaction(createItemFn)
if err != nil {
return nil, err
}
return result.(*Item), nil
}
func createItemFn(tx neo4j.Transaction) (interface{}, error) {
records, err := tx.Run(
"CREATE (a:Greeting) SET a.message = $message RETURN 'Node ' + id(a) + ': ' + a.message",
map[string]interface{}{"message": "Hello, World!"})
// In face of driver native errors, make sure to return them directly.
// Depending on the error, the driver may try to execute the function again.
if err != nil {
return nil, err
}
record, err := records.Single()
if err != nil {
return nil, err
}
// You can also retrieve values by name, with e.g. `id, found := record.Get("n.id")`
return &Item{
Message: record.Values[0].(string),
}, nil
}
type Item struct {
Message string
}
Now, create a go.mod file using the go mod init example.com/hello command.
I've mentioned the Bolt driver earlier. You need to add it with go get github.com/neo4j/neo4j-go-driver/v4#v4.3.1. You can run your program with go run .\program.go.
The complete documentation is located at Memgraph site.

Memcached Ping() doesn't return an error on an invalid server

I use memcache for caching and the client I use is https://github.com/bradfitz/gomemcache. When I tried initiate new client with dummy/invalid server address and then pinging to it, it return no error.
package main
import (
"fmt"
m "github.com/bradfitz/gomemcache"
)
func main() {
o := m.New("dummy_adress")
fmt.Println(o.Ping()) // return no error
}
I think it suppose to return error as the server is invalid. What do I miss?
It looks like the New() call ignores the return value for SetServers:
func New(server ...string) *Client {
ss := new(ServerList)
ss.SetServers(server...)
return NewFromSelector(ss)
}
The SetServers() function will only set the server list to valid servers (in
your case: no servers) and the Ping() funtion will only ping servers that are
set, and since there are no servers set it doesn't really do anything.
This is arguably a feature; if you have 4 servers and one is down then that's
not really an issue. Even with just 1 server memcache is generally optional.
You can duplicate the New() logic with an error check:
ss := new(memcache.ServerList)
err := ss.SetServers("example.localhost:11211")
if err != nil {
panic(err)
}
c := memcache.NewFromSelector(ss)
err = c.Ping()
if err != nil {
panic(err)
}
Which gives:
panic: dial tcp 127.0.0.1:11211: connect: connection refused

how to query redis Db using golang if its up and running for taking connections?

My redis db has huge number of keys, so when I start my code, at that time if I do info persistence (on redis CLI) it gives me following, Loading: 1 means that redis is not ready to take the connections,while loading :0 means redis up and running and ready to take connections.
loading:1
rdb_changes_since_last_save:1024
rdb_bgsave_in_progress:0
rdb_last_save_time:1530558451
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
My code to establish the connection with redis is as follows:
var DefaultPool = NewPool("redis", 6379)
/*NewPool - create a new redis pool accessible at the given address */
func NewPool(host string, port int) *redis.Pool {
var address string
if os.Getenv("DOCKER") != "" {
address = fmt.Sprintf("%v:6379", host)
} else {
address = fmt.Sprintf("127.0.0.1:%v", port)
}
return &redis.Pool{
MaxIdle: 80,
MaxActive: 1000, // max number of connections
Dial: func() (redis.Conn, error) {
c, err := redis.Dial("tcp", address)
if err != nil {
panic(err.Error())
}
return c, err
},
}
}
How I can constantly check if loading value is 0.
Get the info:
info, err := redis.String(c.Do("INFO", "persistence"))
if err != nil {
// handle error
}
Check the loading field:
re := regexp.MustCompile("^loading:0$") // this can be done once and stored as a package level variable.
if re.MatchString(info) {
// loading is zero!
}

How to create a socket using WSASocket function in go programming language?

Does anyone know how to create a SOCKET as returned by WSASocket() function in go programming language?
Using a normal syscall.Socket type syscall.Bind results in:
WSAENOTSOCK - Error 10038 - An operation was attempted on something that is not a socket. The specified socket parameter refers to a file, not a socket.
Thanks
We don't use such low level API, we use net.Dial. ex.
func main() {
var (
host = "127.0.0.1"
port = "9998"
remote = host + ":" + port
msg string = "test"
)
con, error := net.Dial("tcp4", remote)
if error != nil {
fmt.Printf("Host not found: %s\n", error)
os.Exit(1)
} else {
defer con.Close()
}
in, error := con.Write([]byte(msg))
if error != nil {
fmt.Printf("Error sending data: %s, in: %d\n", error, in)
os.Exit(2)
}
fmt.Println("Connection OK")
}
Or, you could trace the code $GOROOT/src/pkg/net/dial.go

Resources