Connecting to Redigo and manipulating data inside a function is easy like butter, but the problem comes when you have to re-use its connection, obviously for performance/practicality reasons.
Doing it inside a function like this works:
func main() {
client, err := redis.Dial("tcp", ":6379")
if err != nil {
panic(err)
}
defer client.Close()
client.Do("GET", "test:1")
}
But bringing it outside doesn't:
var Client = redis.Dial("tcp", ":6379")
defer Client.Close()
func main() {
Client.Do("GET", "test:1")
}
With the following error(s) returned:
./main.go:1: multiple-value redis.Dial() in single-value context
./main.go:2: non-declaration statement outside function body
I've tried putting the connection as a const(ant), putting defer inside the main function to my dismay not working too.
This is an even bigger concern as I have many other functions that have to communicate to Redis, but recreating the connection to Redis everytime seems silly.
The Redigo API just shows how to create a Dial instance but doesn't go further by explaining how to re-use it.
You may've been lost in my talk, but I wanted to put a bit of context here, so my clear and concise question is: How do you go about re-using (not recreating everytime) a Redigo connection?
The best way turned out to be using Pools, which are briefly documented here: Redigo Pools.
A global variable won't eventually reuse a connection, so I ended up with something like this (using Pools as noted before):
func newPool() *redis.Pool {
return &redis.Pool{
MaxIdle: 80,
MaxActive: 12000, // max number of connections
Dial: func() (redis.Conn, error) {
c, err := redis.Dial("tcp", ":6379")
if err != nil {
panic(err.Error())
}
return c, err
},
}
}
var pool = newPool()
func main() {
c := pool.Get()
defer c.Close()
test,_:=c.Do("HGETALL", "test:1")
fmt.Println(test)
}
If for example you want to reuse a pool inside another function you do it like this:
func test() {
c := pool.Get()
defer c.Close()
test2,_:=c.Do("HGETALL", "test:2")
fmt.Println(test2)
}
The redis.Dial() method returns client error. To fix it, you should replace:
var Client = redis.Dial("tcp", ":6379")
with:
var Client, _ = redis.Dial("tcp", ":6379")
Related
I'm doing an online course on Golang. The following piece of code is presented in the course material as an example of misuse of sync.Once:
var (
once sync.Once
db *sql.DB
)
func DbOnce() (*sql.DB, error) {
var err error
once.Do(func() {
fmt.Println("Am called")
db, err = sql.Open("mysql", "root:test#tcp(127.0.0.1:3306)/test")
if err != nil {
return
}
err = db.Ping()
})
if err != nil {
return nil, err
}
return db, nil
}
Supposedly, the above is a faulty implementation of an SQL connection manager. We, the students, are to find the error ourselves, which I struggle with. The code runs fine even in parallel. This is how I used it:
func main() {
wg := sync.WaitGroup{}
wg.Add(10)
for i := 0; i < 10; i++ {
go (func() {
db, err := DbOnce()
if err != nil {
panic(err)
}
var v int
r := db.QueryRow("SELECT 1")
err = r.Scan(&v)
fmt.Println(v, err)
wg.Done()
})()
}
wg.Wait()
}
I understand that homework questions are discouraged here, so I'm not asking for a complete solution, just a hint would be fine. Is the error related to concurrency (i.e. I need to run it in a specific concurrent context)? Is it usage of sql.Open specifically?
Initialization of the db variable is OK. The problem is with the returned error.
If you call DbOnce() for the first time and opening a DB connection fails, that error will be returned properly. But what about subsequent calls? The db initialization code will not be run again, so nil db may be returned, and since the initialization code is not run, the default value of the err variable is returned, which will be nil. To sum it up, the initialization error is lost and will not be reported anymore.
One solution is to stop the app if connection fails (at the first call). Another option is to store the initialization error too in a package level variable along with db, and return that from DbOnce() (and not use a local variable for that). The former has the advantage that you don't have to handle errors returned from DbOnce(), as it doesn't even have to return an error (if there's an error, your app will terminate).
The latter could look like this:
var (
once sync.Once
db *sql.DB
dbErr error
)
func DbOnce() (*sql.DB, error) {
once.Do(func() {
fmt.Println("Am called")
db, dbErr = sql.Open("mysql", "root:test#tcp(127.0.0.1:3306)/test")
if dbErr != nil {
return
}
dbErr = db.Ping()
})
return db, dbErr
}
How can I test func main? Like this:
func main(){
Engine := GetEngine() // returns gin router with handlers atttached
Engine.Run(":8080")
}
It has only 2 lines but I'd like to have them covered.
TestMain' is reserved for test preparation, does that mean testing main was not planned by language creators?
I can move the contents to another function mainReal but it seems to be some over engineering?
How to test gin has started well? Can I launch main in separate goroutine, check reply and stop it?
Thanks.
P.S. Possible duplicate is not precise duplicate because it is dedicated not to testing of func main() itself, but rather ideas to move in outside and so contains different issue and approach.
Solution.
You may test function main() from package main the same way, just do not name it TestMain. I launch it as a separate goroutine, than try to connect to it and perform any request.
I decided to connect to auxilary handler which should respond with a simple json {"status": "ok"}.
In my case:
func TestMainExecution(t *testing.T) {
go main()
resp, err := http.Get("http://127.0.0.1:8080/checkHealth")
if err != nil {
t.Fatalf("Cannot make get: %v\n", err)
}
bodySb, err := ioutil.ReadAll(resp.Body)
if err != nil {
t.Fatalf("Error reading body: %v\n", err)
}
body := string(bodySb)
fmt.Printf("Body: %v\n", body)
var decodedResponse interface{}
err = json.Unmarshal(bodySb, &decodedResponse)
if err != nil {
t.Fatalf("Cannot decode response <%p> from server. Err: %v", bodySb, err)
}
assert.Equal(t, map[string]interface{}{"status": "ok"}, decodedResponse,
"Should return status:ok")
}
I have a number of routes in my routes.go file and they all call my redis database. I'm wondering how I can avoid calling the dial and AUTH calls in every route.
I've tried setting variables outside the functions like this:
var (
c, err = redis.Dial("tcp", ADDRESS)
_, err = c.Do("AUTH", "testing")
)
but then the compiler doesn't like err being used twice.
First, only use var for declaring variables. You can't run code outside of functions, so there's no use in trying to create connections inside a var statement. Use init() if you need something run at startup.
The redis connections can't be used with concurrent requests. If you want to share a redis connection across multiple routes, you need to have a safe method for concurrent use. In the case of github.com/garyburd/redigo/redis you want to use a Pool. You can do the AUTH call inside the Dial function, returning a ready connection each time.
var redisPool *redis.Pool
func init() {
redisPool = &redis.Pool{
MaxIdle: 3,
IdleTimeout: 240 * time.Second,
Dial: func() (redis.Conn, error) {
c, err := redis.Dial("tcp", server)
if err != nil {
return nil, err
}
if _, err := c.Do("AUTH", password); err != nil {
c.Close()
return nil, err
}
return c, err
},
}
}
Then each time you need a connection, you get one from the pool, and return it when you're done.
conn := redisPool.Get()
// conn.Close() just returns the connection to the pool
defer conn.Close()
if err := conn.Err(); err != nil {
// conn.Err() will have connection or Dial related errors
return nil, err
}
What I would do is instantiate a connection pool in main.go and pass the reference to the pool to your routes. This way you are setting up your redis client once, and your routes can leverage it.
I created a decorator around redigo that makes creating a Redis Client with a Connection pool very straightforward. Plus it is type-safe.
You can check it out here: https://github.com/shomali11/xredis
I am playing wit golang and orientdb to test them. i have written a tiny web app which uppon a request fetches a single document from local orientdb instance and returns it. when i bench this app with apache bench, when concurrency is above 1 it get following error:
2015/04/08 19:24:07 http: panic serving [::1]:57346: Get http://localhost:2480/d
ocument/t1/9:1441: EOF
when i bench Orientdb itself, it runs perfectley OK with any cuncurrency factor.
also when i change the url to fetch from this document to anything (other program whritten in golang, some internet site etc) the app runs OK.
here is the code:
func main() {
fmt.Println("starting ....")
var aa interface{}
router := gin.New()
router.GET("/", func(c *gin.Context) {
ans := getdoc("http://localhost:2480/document/t1/9:1441")
json.Unmarshal(ans, &aa)
c.JSON(http.StatusOK, aa)
})
router.Run(":3000")
}
func getdoc(addr string) []byte {
client := new(http.Client)
req, err := http.NewRequest("GET", addr, nil)
req.SetBasicAuth("admin","admin")
resp, err := client.Do(req)
if err != nil {
fmt.Println("oops", resp, err)
panic(err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
panic(err)
}
return body
}
thanks in advance
The keepalive connections are getting closed on you for some reason. You might be overwhelming the server, or going past the max number of connections the database can handle.
Also, the current http.Transport connection pool doesn't work well with synthetic benchmarks that make connections as fast as possible, and can quickly exhaust available file descriptors or ports (issue/6785).
To test this, I would set Request.Close = true to prevent the Transport from using the keepalive pool. If that works, one way to handle this with keepalive, is to specifically check for an io.EOF and retry that request, possibly with some backoff delay.
I have been playing around with golang and redis. I just stood up a simple http server and wanted to increment requests on redis. I am blowing up the connections (I think). I found that with redigo you can use connection pooling, but not sure how to implment that in go when I am serving the requests (where do you instantiate / call the pool from).
error: can't assign requested address.
Any suggestions would be appreciated....I am sure I am incorrectly making the connections, but just not sure how to change.
EDIT: Modified per pauljz's suggestions -- Works great now
var pool redis.Pool
func qryJson(rw http.ResponseWriter, req *http.Request){
incrementRedis()
}
func incrementRedis () {
t := time.Now().Format("2006-01-02 15:04:05")
conn := pool.Get()
defer conn.Close()
if _, err := conn.Do("HINCRBY", "messages", t, 1); err != nil {
log.Fatal(err)
}
}
func main() {
runtime.GOMAXPROCS(runtime.NumCPU())
pool = redis.Pool{
MaxIdle: 50,
MaxActive: 500, // max number of connections
Dial: func() (redis.Conn, error) {
c, err := redis.Dial("tcp", ":6379")
if err != nil {
panic(err.Error())
}
return c, err
},
}
http.HandleFunc("/testqry", qryJson)
log.Fatal(http.ListenAndServe(":8082", nil))
}
The redigo docs have a good starter example for connection pooling: http://godoc.org/github.com/garyburd/redigo/redis#Pool
In your case you would have a var pool redis.Pool in your package (i.e. not inside of a function).
In main(), before your ListenAndServe call, you would call pool = redis.Pool{ ... } from the redigo example to initialize the pool.
In incrementRedis() you would then do something like:
func incrementRedis () {
conn := pool.Get()
defer conn.Close()
if _, err := conn.Do("HINCRBY", "messages", t, 1); err != nil {
log.Fatal(err)
}
}
In your code, you create a connection to redis for each http request. Use a global variable to store connected redis connection and reuse it.