Golang database/sql hangs with SetMaxOpenConns - go

I have pasted a minimally reproducible test code. In short, with SetMaxOpenConns set to 10, the program hangs forever after 10. I found this relevant thread from way back when, but it seems resolved and tested: https://github.com/golang/go/issues/6593
Note that by commenting out the SetMaxOpenConns the code runs normally.
What am I doing wrong? or should I open a new issue?
1 package main
2
3 import (
4 "database/sql"
5 "log"
6 "time"
7 _ "github.com/lib/pq"
8 )
9
10 func main(){
11 // Establish db connection
12 db, err := sql.Open("postgres", "host=0.0.0.0 port=5432 user=postgres password=password dbname=test sslmode=disable")
13 if err != nil {
14 log.Fatal(err)
15 }
16
17 db.SetMaxOpenConns(10) // commenting this line will resolve the problem
18 db.SetMaxIdleConns(10)
19 db.SetConnMaxLifetime(10 * time.Second)
20
21 // Query more than max open; note that hangs forever
22 for i:=0; i<12; i++ {
23 rows, err := Query(db)
24 if err != nil {
25 log.Fatal(err)
26 }
27 log.Println(i)
28 log.Println(rows)
29 }
30 }
31
32 func Query(db *sql.DB) (*sql.Rows, error){
33 stmt, err := db.Prepare("SELECT * FROM test;")
34 if err != nil {
35 log.Fatal(err)
36 }
37
38 defer stmt.Close()
39
40 rows, err := stmt.Query()
41 if err != nil {
42 log.Fatal(err)
43 }
44
45 return rows, nil
46 }

You need to either fully iterate through the result set with rows.Next and/or call rows.Close(); as per the docs:
Close closes the Rows, preventing further enumeration. If Next is called and returns false and there are no further result sets, the Rows are closed automatically and it will suffice to check the result of Err. Close is idempotent and does not affect the result of Err.
Something like:
for i:=0; i<12; i++ {
rows, err := Query(db)
if err != nil {
log.Fatal(err)
}
log.Println(i)
log.Println(rows)
if err = rows.Close(); err != nil {
panic(err)
}
}
For this to be useful you need to iterate through the rows (see the example in the docs).
The connection to the database will remain in use until the result set is closed (at which point it is returned to the pool). Because you are doing this in a loop you will end up with 10 active result sets and when you call Query() again the sql package will wait for a connection to become available (which will never happen).
Note that because your query has no parameters (and you are only using the stmt once) calling Prepare has no benefit; the following is simpler and will have the same result:
func Query(db *sql.DB) (*sql.Rows, error) {
return db.Query("SELECT * FROM test;")
}

Related

How does gRPC handle requests from the same client concurrently?

I was reading gRPC tutorial (https://grpc.io/docs/languages/go/basics/#bidirectional-streaming-rpc) and found the following code:
1 func (s *routeGuideServer) RouteChat(stream pb.RouteGuide_RouteChatServer) error {
2 for {
3 in, err := stream.Recv()
4 if err == io.EOF {
5 return nil
6 }
7 if err != nil {
8 return err
9 }
10 key := serialize(in.Location)
11 ... // look for notes to be sent to client
12 for _, note := range s.routeNotes[key] {
13 if err := stream.Send(note); err != nil {
14 return err
15 }
16 }
17 }
18 }
My current understanding from reading the code is that, when a request A from client is received in line 3, server proceeds to line 4 - 16 to handle that request, and this is a blocking process. If the client sends request B immediately after A is sent, the server cannot receive and handle it until request A is done processing.
So if processing A takes a very long time and B is a request to stop processing A, then the code will need to first finish processing A before starting to stop processing it, which is weird. I thought the code in this case should be something like:
1 func (s *routeGuideServer) RouteChat(stream pb.RouteGuide_RouteChatServer) error {
2 for {
3 in, err := stream.Recv()
4 if err == io.EOF {
5 return nil
6 }
7 if err != nil {
8 return err
9 }
10 if GetTypeofRequest(in) == A {
11 go HandleRequestA(stream)
12 } else if GetTypeofRequest(in) == B {
13 go HandleRequestB(stream)
14 }
15 }
16 }
Then in HandleRequestA and HandleRequestB, stream sends different responses.
I was under the impression that gRPC should recv and handle requests concurrently and doesn't require such goroutines. Please correct me if I'm wrong.

executable exits early when using io.WriteString

I'm using the io package to work with an executable defined in my PATH.
The executable is called "Stockfish" (Chess Engine) and obviously usable via command line tools.
In order to let the engine search for the best move, you use "go depth n" - the higher the depth - the longer it takes to search.
Using my command line tool it searches for about 5 seconds using a depth of 20, and it looks like this:
go depth 20
info string NNUE evaluation using nn-3475407dc199.nnue enabled
info depth 1 seldepth 1 multipv 1 score cp -161 nodes 26 nps 3714 tbhits 0 time 7 pv e7e6
info depth 2 seldepth 2 multipv 1 score cp -161 nodes 51 nps 6375 tbhits 0 time 8 pv e7e6 f1d3
info depth 3 seldepth 3 multipv 1 score cp -161 nodes 79 nps 7900 tbhits 0 time 10 pv e7e6 f1d3 g8f6
info depth 4 seldepth 4 multipv 1 score cp -161 nodes 113 nps 9416 tbhits 0 time 12 pv e7e6 f1d3 g8f6 b1c3
[...]
bestmove e7e6 ponder h2h4
Now, using io.WriteString it finishes after milliseconds without any (visible) calculation:
(That's also the output of the code below)
Stockfish 14 by the Stockfish developers (see AUTHORS file)
info string NNUE evaluation using nn-3475407dc199.nnue enabled
bestmove b6b5
Here's the code I use:
func useStockfish(commands []string) string {
cmd := exec.Command("stockfish")
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
for _, cmd := range commands {
writeString(cmd, stdin)
}
err = stdin.Close()
if err != nil {
log.Fatal(err)
}
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatal(err)
}
return string(out)
}
func writeString(cmd string, stdin io.WriteCloser) {
_, err := io.WriteString(stdin, cmd)
if err != nil {
log.Fatal(err)
}
And this is an example of how I use it. The first command is setting the position, the second one is calculation the next best move with a depth of 20. The result is showed above.
func FetchComputerMove(game *internal.Game) {
useStockfish([]string{"position exmaplepos\n", "go depth 20"})
}
To leverage engines like stockfish - you need to start the process and keep it running.
You are executing it, passing 2 commands via a Stdin pipe, then closing the pipe. Closing the pipe indicates to the program that you are no longer interested in what the engine has to say.
To run it - and keep it running - you need something like:
func startEngine(enginePath string) (stdin io.WriteCloser, stdout io.ReadCloser, err error) {
cmd := exec.Command(enginePath )
stdin, err = cmd.StdinPipe()
if err != nil {
return
}
stdout, err = cmd.StdoutPipe()
if err != nil {
return
}
err = cmd.Start() // start command - but don't wait for it to complete
return
}
The returned pipes allow you to send commands & see the output live:
stdin, stdout, err := startEngine("/usr/local/bin/stockfish")
sendCmd := func(cmd string) error {
_, err := stdin.Write([]byte(cmd + "\n"))
return err
}
sendCmd("position examplepos")
sendCmd("go depth 20")
then to crudely read the asynchronous response:
b := make([]byte, 10240)
for {
n, err := stdout.Read(b)
if err != nil {
log.Fatalf("read error: %v", err)
}
log.Println(string(b[:n]))
}
once a line like bestmove d2d4 ponder g8f6 appears, you know the current analysis command has completed.
You can then either close the engine (by closing the stdin pipe) if that's all you need, or keep it open for further command submissions.

Testing echo handlers with path variables

I’m testing one of my http handlers.
A curl like this works: curl localhost:8080/v1/parts/2bb834c9-8e17-4e2c-80b9-a20b80732899.
But the corresponding test fails as the handler is not able to extract the id from the URL.
Please have a look at this test file. I don’t understand why my curl command works but the newRequest on line 17 fails.
1 package part
2
3 import (
4 "net/http"
5 "net/http/httptest"
6 "testing"
7
8 "github.com/labstack/echo/v4"
9 "github.com/stretchr/testify/assert"
10 )
11
12 var handler *Handler
13
14 func TestGetPart(t *testing.T) {
15 e := echo.New()
16
17 req := httptest.NewRequest("GET", "/v1/parts/2bb834c9-8e17-4e2c-80b9-a20b80732899", nil)
18 rec := httptest.NewRecorder()
19 c := e.NewContext(req, rec)
20
21 if assert.NoError(t, handler.handleGetPart(c)) {
22 assert.Equal(t, http.StatusOK, rec.Code)
23 }
24 }
go test -v ./internal/handler/part
=== RUN TestGetPart
2020/09/08 08:51:10 UUID PASSED:
time="2020-09-08T08:51:10+02:00" level=error msg="Could not decode string to uuid"
TestGetPart: handler_test.go:21:
Error Trace: handler_test.go:21
Error: Received unexpected error:
invalid UUID length: 0
Test: TestGetPart
--- FAIL: TestGetPart (0.00s)
The handler
ID := ctx.Param("id")
log.Println("UUID PASSED: ", ID)
uuid, err := uuid.Parse(ID)
if err != nil {
logrus.Error("Could not decode string to uuid")
return err
}
// Fetch data.json from S3 bucket
filename := helper.PartFileName(uuid)
content, err := a.GetObject(filename)
if err != nil {
logrus.Error(err)
return ctx.JSON(http.StatusNotFound, "file not found")
}
return ctx.String(http.StatusOK, content)
}
Any my route:
func Register(router *echo.Echo, handler *Handler) {
router.GET("/v1/parts/:id", handler.handleGetPart)
}
You should set a path in this way:
e := echo.New()
req := httptest.NewRequest("GET", "/", nil)
rec := httptest.NewRecorder()
c := e.NewContext(req, rec)
c.SetPath("/:id")
c.SetParamNames("id")
c.SetParamValues("2bb834c9-8e17-4e2c-80b9-a20b80732899")
// rest of the test goes below

How do I stop a NewScanner Scan() after a timeout in background if there is no new output in tailed process [duplicate]

I'd like to run a command and print each line from its stdout (as it becomes available). Further, if the command doesn't complete within N seconds, I would like to terminate it as well.
There are some examples of implementing timeouts in golang (notably Terminating a Process Started with os/exec in Golang). I have a time.After() clause in select that I expect to hit after 2 seconds, at which point RunTraceroute should return - but this doesn't happen.
My code is below (and on go playground: http://play.golang.org/p/D4AcoaweMt)
package main
import (
"bufio"
"fmt"
"log"
"os"
"os/exec"
"time"
)
func RunTraceroute(host string) {
errch := make(chan error, 1)
cmd := exec.Command("/usr/bin/traceroute", host)
stdout, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
go func() {
errch <- cmd.Wait()
}()
select {
case <-time.After(time.Second * 2):
log.Println("Timeout hit..")
return
case err := <-errch:
if err != nil {
log.Println("traceroute failed:", err)
}
default:
for _, char := range "|/-\\" {
fmt.Printf("\r%s...%c", "Running traceroute", char)
time.Sleep(100 * time.Millisecond)
}
scanner := bufio.NewScanner(stdout)
fmt.Println("")
for scanner.Scan() {
line := scanner.Text()
log.Println(line)
}
}
}
func main() {
RunTraceroute(os.Args[1])
}
Output (snipped out first few lines of local ip addresses and gateway):
$ go run pipe.go xinhua.com
Running traceroute...\
2016/04/01 11:16:43 traceroute to xinhua.com (58.64.200.76), 30 hops max, 60 byte packets
.......
.......
.....More deleted lines.....
2016/04/01 11:16:49 12 * * *
2016/04/01 11:16:49 13 4.68.63.214 (4.68.63.214) 3.208 ms 3.176 ms 3.241 ms
2016/04/01 11:16:49 14 * * *
2016/04/01 11:16:49 15 if-ae-9-2.tcore1.TV2-Tokyo.as6453.net (180.87.180.18) 160.314 ms 158.837 ms 161.438 ms
2016/04/01 11:16:49 16 if-ae-5-7.tcore1.HK2-Hong-Kong.as6453.net (180.87.112.189) 157.497 ms if-ae-3-2.tcore1.HK2-Hong-Kong.as6453.net (180.87.112.5) 161.397 ms if-ae-5-7.tcore1.HK2-Hong-Kong.as6453.net (180.87.112.189) 159.351 ms
2016/04/01 11:16:49 17 if-ge-10-0-0-1128.core1.undefined.as6453.net (180.87.160.73) 156.412 ms 156.522 ms if-ge-14-0-0-1126.core1.undefined.as6453.net (180.87.112.30) 156.605 ms
2016/04/01 11:16:49 18 * * *
2016/04/01 11:16:49 19 * * *
2016/04/01 11:16:49 20 * * *
2016/04/01 11:16:49 21 113.10.229.113 (113.10.229.113) 165.578 ms 165.818 ms 163.451 ms
2016/04/01 11:16:49 22 113.10.229.74 (113.10.229.74) 163.564 ms ae5.10g-idc.wpc.nwtgigalink.com (113.10.229.66) 162.384 ms 113.10.229.74 (113.10.229.74) 167.026 ms
2016/04/01 11:16:49 23 113.10.230.162 (113.10.230.162) 162.988 ms 162.777 ms 163.807 ms
2016/04/01 11:16:49 24 58.64.160.164 (58.64.160.164) 161.902 ms 162.396 ms 164.986 ms
2016/04/01 11:16:54 25 * * *
2016/04/01 11:16:54 26 58.64.200.76 (58.64.200.76) 162.178 ms !X 162.509 ms !X 162.356 ms !X
I think you want to put the body of your default: case into a goroutine; I suspect it's preventing your case <-time.After(time.Second * 2): label from being hit.
Also, keep in mind that time.After does not guarantee that it will be hit exactly after that duration, it only says that any time after that duration it will send the signal on the channel, which could be a while after the designated duration. See the docs on the underlying time.NewTimer
I modified your example: http://play.golang.org/p/TggNQ1d57Y
package main
import (
"bufio"
"fmt"
"log"
"os/exec"
"time"
)
func RunTraceroute(host string) {
errch := make(chan error, 1)
cmd := exec.Command("/usr/bin/traceroute", host)
stdout, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
go func() {
errch <- cmd.Wait()
}()
go func() {
for _, char := range "|/-\\" {
fmt.Printf("\r%s...%c", "Running traceroute", char)
time.Sleep(100 * time.Millisecond)
}
scanner := bufio.NewScanner(stdout)
fmt.Println("")
for scanner.Scan() {
line := scanner.Text()
log.Println(line)
}
}()
select {
case <-time.After(time.Second * 1):
log.Println("Timeout hit..")
return
case err := <-errch:
if err != nil {
log.Println("traceroute failed:", err)
}
}
}
func main() {
RunTraceroute("8.8.8.8")
}
Which works for me

Send ISO8583 Message Golang

I'm trying to build a ISO8583 Client using a Golang,
when using java, i don't have any problem when creating client.
But when trying creating a client using golang (i'm just starting learning golang btw), i can't send a message to the server. Can someone help me, why i can't send a message?
I've tried to send a SIGN IN Message, The client and server already connected, but the message i send not readable by the server.
This My Code
package main
import (
"bufio"
"fmt"
"net"
"os"
"time"
"github.com/ideazxy/iso8583"
)
type ISOSignIn struct {
Bit3 *iso8583.Numeric `field:"3" length:"6" encode:"bcd"`
Bit7 *iso8583.Numeric `field:"7" length:"10" encode:"ascii`
Bit11 *iso8583.Numeric `field:"11" length:"6" encode:"rbcd`
Bit32 *iso8583.Llnumeric `field:"32" length:"11" encode:"ascii`
Bit70 *iso8583.Numeric `field:"70" length:"3" encode:"ascii`
}
func main() {
testIso()
}
func testIso() {
data := ISOSignIn{
Bit3: iso8583.NewNumeric("001111"),
Bit7: iso8583.NewNumeric("0913110004"),
Bit11: iso8583.NewNumeric("000001"),
Bit32: iso8583.NewLlnumeric("9999"), // Client ID
Bit70: iso8583.NewNumeric("301"), // Login Code
}
msg := iso8583.NewMessage("0800", data)
msg.MtiEncode = iso8583.BCD
b, err := msg.Bytes()
if err != nil {
fmt.Println(err.Error())
}
fmt.Printf("% x\n", b)
tcpClientNew(b)
}
func tcpClientNew(b []byte) {
tcpAddr, err := net.ResolveTCPAddr("tcp", "192.168.100.5:12346")
if err != nil {
println("ResolveTCPAddr failed:", err.Error())
os.Exit(1)
}
conn, err := net.DialTCP("tcp", nil, tcpAddr)
if err != nil {
println("Dial failed:", err.Error())
os.Exit(1)
}
timeoutDuration := 30 * time.Second
_, err = conn.Write(b)
if err != nil {
println("Write to server failed:", err.Error())
os.Exit(1)
}
conn.SetReadDeadline(time.Now().Add(timeoutDuration))
bufReader := bufio.NewReader(conn)
resp, _ := bufReader.ReadByte()
fmt.Print("Message from server: " + string(resp))
conn.Close()
}
Server Already Connected
<log realm="Server-A.server.session/192.168.100.1:32218" at="Mon Jan 07 09:37:15.747 WIB 2019">
<session-start/>
</log>
<log realm="channel/192.168.100.1:32218" at="Mon Jan 07 09:37:19.034 WIB 2019" lifespan="3287ms">
<receive>
<peer-disconnect/>
</receive>
</log>
<log realm="Server-A.server.session/192.168.100.1:32218" at="Mon Jan 07 09:37:19.035 WIB 2019">
<session-end/>
</log>
Output from Client Terminal :
GOROOT=/Users/ivanaribanilia/Applications/go
GOPATH=/Users/ivanaribanilia/Project/Workspace-Github/Project-Go/pclient
/Users/ivanaribanilia/Applications/go/bin/go build -i -o /Users/ivanaribanilia/Project/Workspace-Github/Project-Go/pclient/build/pclient /Users/ivanaribanilia/Project/Workspace-Github/Project-Go/pclient/src/github.com/ivanj4u/pclient/main.go
/Users/ivanaribanilia/Project/Workspace-Github/Project-Go/pclient/build/pclient
08 00 22 20 00 01 00 00 00 00 00 11 11 30 39 31 33 31 31 30 30 30 34 30 30 30 30 30 31 30 34 31 31 31 34
Message from server:
Process finished with exit code 0
I expect a response from the server, so i can develop other message like INQUIRY or PAYMENT.
Thank you
ReadByte reads and returns a single byte. If no byte is available,
returns an error.
seems that what you read from server is only one byte, which is a white-space char.
The server and client should make a protocol when to close the connection. Thus, if server don't close the conn actively, client should read all bytes from server and close the connection. Like this:
recvBuf := make([]byte, 1024)
n, err := bufReader.Read(recvBuf)
for err == nil {
println("Recv data from server:", string(recvBuf[:n]))
n, err = bufReader.Read(recvBuf)
}
if err != io.EOF {
println("recv from server failed, err:", err)
}
conn.Close()
Or if the protocol defines the client should close the connection when received a certain byte, client can use ReadBytes() and close the connection actively.
func (b *Reader) ReadBytes(delim byte) ([]byte, error)
ReadBytes reads until the first occurrence of delim in the input,
returning a slice containing the data up to and including the
delimiter. If ReadBytes encounters an error before finding a
delimiter, it returns the data read before the error and the error
itself (often io.EOF). ReadBytes returns err != nil if and only if the
returned data does not end in delim.

Resources