I currently have an issue with Golang 1.7.1
My purpose is to create a pool with TLS connections from lib "crypto/tls"
My purpose is to forward stdin entry to the network via TLS. I am reading the stdin with reader := bufio.NewReader(os.Stdin) and loop at it. before the loop I tried to create the pool.
I followed the TCP pool sample from : https://github.com/fatih/pool
Sample from my code :
factory := func() (tls.Conn, error) {
log.Println("************** factory start :) :() **************")
conn, err := tls.Dial("tcp", "discover.logs.ovh.com:12202" , &tls.Config{
InsecureSkipVerify: true, })
if err != nil {
panic(err)
}
log.Println("************** factory stop :) **************")
return *conn, err
}
p, err := NewChannelPool(3, 10, factory)
if err != nil {
panic(err)
}
reader := bufio.NewReader(os.Stdin)
log.Println("NewReader done")
for {
line, overflow, err := reader.ReadLine()
if err == io.EOF {
/*err = p.Close()
if err != nil {
log.Fatal(err)
}*/
os.Exit(0)
}
......
......
......
conn, err := p.Get()
if err != nil {
log.Fatal(err)
}
bytes, err := conn.Write(databytes)
if err != nil {
log.Fatal(err)
}
conn.Close()
}
The main difference between tls / tcp lib (apart from the secure part) is the Dial return *tls.Conn
and tcp tcp.Conn
So I thought I was doing well by returning *conn, I return the object pointed by the pointer.
I always end up open by this error when I start the second connection:
journald-gateway_1 | 2016/12/13 16:19:20 socat[15] E write(6, 0x55fc5a73db80, 1006): Broken pipe
Any Idea?
Thank you for reading me.
Related
I'm trying to make a music app that sends file through tcp protocol using go and microservice architecture. Now I'm creating a player service that should:
Get user token and get claims from it
Check is user exists using claims and user_service microservice
Get song from redis
Check is song exists using music_service
Read file by chunks and send it to client using tcp
Redis data looks like this:
{
"user_id": [{
"song_id": "<song_id>"
}]
}
But I faced with a small problem. My music files stored in a flac format and when I receive it on the client, my player doesn't play it. I don't really know what can be the problem. So here's my code:
SERVER
service_setup.go
//this function is called in main function
func setService() {
ln, err := net.Listen("tcp", config.TCPAddress)
if err != nil {
panic("couldn't start tcp server")
}
defer ln.Close()
for {
conn, err := ln.Accept()
if err != nil {
logger.ErrorLog(fmt.Sprintf("Error: couldn't accept connection. Details: %v", err))
return
}
service.DownloadSong(conn)
}
}
downloader_service.go
func DownloadSong(conn net.Conn) {
token, err := bufio.NewReader(conn).ReadString('\n')
if err != nil {
logger.ErrorLog(fmt.Sprintf("Error: couldn't get token. Details: %v", token))
conn.Close()
return
}
claims, err := jwt_funcs.DecodeJwt(token)
if err != nil {
conn.Close()
return
}
songs, err := redis_repo.Get(claims.Id)
if err != nil {
conn.Close()
return
}
for _, song := range songs {
download(song, conn)
}
}
func download(song models.SongsModel, conn net.Conn) {
filePath, err := filepath.Abs(fmt.Sprintf("./songs/%s.flac", song.SongId))
if err != nil {
logger.ErrorLog(fmt.Sprintf("Errror: couldn't create filepath. Details: %v", err))
conn.Close()
return
}
file, err := os.Open(filePath)
defer file.Close()
if err != nil {
logger.ErrorLog(fmt.Sprintf("Errror: couldn't open file. Details: %v", err))
conn.Close()
return
}
read(file, conn)
}
func read(file *os.File, conn net.Conn) {
reader := bufio.NewReader(file)
buf := make([]byte, 15)
defer conn.Close()
for {
_, err := reader.Read(buf)
if err != nil && err == io.EOF {
logger.InfoLog(fmt.Sprintf("Details: %v", err))
fmt.Println()
return
}
conn.Write(buf)
}
}
CLIENT
main.go
func main() {
conn, _ := net.Dial("tcp", "127.0.0.1:6060")
var glMessage []byte
text := "eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjYzYzlhNmE1OWI3ZmQyNTQ2ZjA4ZWEyYSIsInVzZXJuYW1lIjoiMTIiLCJleHAiOjE2NzQyMTE5ODl9.aarSDhrFF1df3i2pIRyjNxTfSHKObqLU3kHJiPreredIhLNCzs7z7jMgRHQIcLaIvCOECN7bX0OaSvKdW7VKsQ\n"
fmt.Fprint(conn, text)
reader := bufio.NewReader(conn)
b := make([]byte, 15)
c := 0
for i, _ := reader.Read(b); int(i) != 0; i, _ = reader.Read(b) {
c += i
glMessage = append(glMessage, b...)
}
os.WriteFile("./test.flac", glMessage, 0644)
}
If you know what can be the problem, please tell me. I'd really appreciate it!
It looks like you're trying to send the music file over the network in 15 byte chunks, which is likely not enough to play the song on the client side.
You can try increasing the chunk size, for example, to 8192 bytes. To do this, replace buf := make([]byte, 15) with buf := make([]byte, 8192).
Also, it's better to write the received data directly to the file rather than storing it in memory. You can do this by creating a file and using os.Create to write the received data to it:
file, err := os.Create("./test.flac")
if err != nil {
fmt.Println("Error: couldn't create file")
return
}
defer file.Close()
for {
i, err := reader.Read(buf)
if err != nil && err == io.EOF {
break
}
file.Write(buf[:i])
}
I believe that this can solve the issue.
Implementing an ssh proxy in Go, errors out with bad packet length, these are the errors with ssh in debug mode:
debug1: SSH2_MSG_KEXINIT sent
Bad packet length 1231976033.
ssh_dispatch_run_fatal: Connection to ::1 port 8080: message authentication code incorrect
Code:
func handleSSH(conn net.Conn, r *bufio.Reader, protocol string) {
target, err := url.Parse("ssh://localhost:3333")
if err != nil {
fmt.Println("Error parsing target", err)
conn.Close()
return
}
targetConn, err := net.Dial("tcp", target.Host)
if err != nil {
fmt.Println("error dialing SSH target:", err)
conn.Close()
return
}
defer targetConn.Close()
var wg sync.WaitGroup
wg.Add(2)
go func() {
_, err := io.Copy(targetConn, conn)
if err != nil {
fmt.Println("error copying data to target:", err)
}
wg.Done()
}()
go func() {
_, err := io.Copy(conn, targetConn)
if err != nil {
fmt.Println("error copying data from target:", err)
}
wg.Done()
}()
wg.Wait()
conn.Close()
}
// EDIT
func connection(conn net.Conn) {
r := bufio.NewReader(conn)
protocol, err := r.ReadString('\n')
if err != nil {
fmt.Println("Error reading first line", err)
conn.Close()
return
}
if protocol[0:3] == "SSH" {
handleSSH(conn, r, protocol)
}
}
func main() {
ln, err := net.Listen("tcp", ":8080")
if err != nil {
panic(err)
}
defer ln.Close()
for {
conn, err := ln.Accept()
if err != nil {
panic(err)
}
go connection(conn)
}
}
EDIT: added code for relevant information on how the connection is initiated and reproduce the error.
My best guess is the ssh negotiation process is being interrupted, and things goes out of sync.
The code is reading the first line from the client and checks the kind of protocol in order to call the appropriate handler:
protocol, err := r.ReadString('\n')
...
if protocol[0:3] == "SSH" {
handleSSH(conn, r, protocol)
}
}
But the code fails to forward the already read bytes to the connected server. These bytes are in protocol and are given to handleSSH. But it fails to send these bytes to the connected server once the connection is established. Instead it only copies new data between client and server.
This means the server does not get the first line from the client. It therefore likely complains about a protocol error with something like Invalid SSH identification string. which gets forwarded to the client and misinterpreted as valid data from an SSH connection.
I have a Go GRPC server-side streaming function:
func (server *Server) GetClients(req *iam.GetClientsRequest, client iam.IAM_GetClientsServer) error {
ctx := client.(interface{ Context() context.Context }).Context()
userID, err := getUserIDStream(client)
if err != nil {
return err
}
clients, err := server.db.QueryByUserID(ctx, userID)
if err != nil {
return grpc.Errorf(codes.Internal, apiutils.ServerError)
}
for _, value := range clients {
converted, err := server.fromInternalClient(value)
if err != nil {
return err
}
if err := client.Send(converted); err != nil {
return err
}
}
return nil
}
and I'm testing it like this:
It("GetClients - Send fails - Error", func() {
handler := createHandler(db)
lis := bufconn.Listen(bufSize)
server := grpc.NewServer()
iam.RegisterIAMServer(server, NewServer(handler))
go func() {
if err := server.Serve(lis); err != nil {
log.Fatalf("Server exited with error: %v", err)
}
}()
defer lis.Close()
defer server.GracefulStop()
conn, err := grpc.DialContext(context.Background(), "bufnet",
grpc.WithContextDialer(createBufDialier(lis)), grpc.WithInsecure())
Expect(err).ShouldNot(HaveOccurred())
defer conn.Close()
client := iam.NewIAMClient(conn)
cclient, _ := client.GetClients(addAccessToken(context.Background()), new(iam.GetClientsRequest))
resp, err := cclient.Recv()
Expect(resp).Should(BeNil())
Expect(err).Should(HaveOccurred())
Expect(err.Error()).Should(Equal(message))
})
My issue is that I'm not sure how to induce a failure on Send so I can test the response. Since I'm using an actual test server and client, I can't just mock out the object and I'd prefer not to go that route anyway. Is there a way I can do this?
Originally, I was trying to force Send to fail by setting bufSize to an artificially low value. However, this wasn't producing an error so I decided to try modifying the maxSendMessageSize on the server:
opts := []grpc.ServerOption{}
if sendFails {
opts = append(opts, grpc.MaxSendMsgSize(10))
}
lis := bufconn.Listen(bufSize)
server := grpc.NewServer(opts...)
And this worked in producing the error.
I have a weird issue with this code:
func PrepareFileUpload(filePath, url string) (*http.Request, error) {
pr, pw := io.Pipe()
mpw := multipart.NewWriter(pw)
go func() {
defer pw.Close()
part, err := mpw.CreateFormFile("file", filepath.Base(filePath))
if err != nil {
return
}
file, err := os.Open(filePath)
if err != nil {
return
}
defer file.Close()
if _, err = io.Copy(part, file); err != nil {
return
}
err = mpw.Close()
if err != nil {
return
}
}()
req, err := http.NewRequest("POST", url, pr)
req.Header.Set("Content-Type", mpw.FormDataContentType())
return req, err
}
which I use like this:
filePath := "foo.bar"
s := []byte("Test file")
ioutil.WriteFile(filePath, s, 0644)
values := url.Values{}
values.Set("folderid", "123456")
values.Set("filename", filepath.Base(filePath))
values.Set("nopartial", "1")
u := url.URL{
Scheme: "https",
Host: "eapi.pcloud.com",
Path: "/uploadfile",
RawQuery: values.Encode(),
}
req, err := PrepareFileUpload(filePath, u.String())
if err != nil {
log.Fatal(err)
}
req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", "ACCESS_TOKEN"))
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
retData, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(retData))
For some reason, when used with the pCloud API, this hangs when running http.DefaultClient.Do(req). I have tried creating my own test server in Go, and there are no issues there, so I'm thinking it's some issues with the communication with the Go client and the pCloud server, but I can't figure out what it is (I've tried forcing HTTP/1.1, but no dice).
When uploading files without io.Pipe and with bytes.Buffer instead, everything is OK, but that doesn't work with large files (OOM).
The only warning I get when enabling verbose HTTP debugging is:
2022/04/21 10:43:29 http2: Transport failed to get client conn for eapi.pcloud.com:443: http2: no cached connection was available
This doesn't happen when I force HTTP/1.1, but the connection still hangs, so I'm not sure how relevant this error is.
Does anyone have any idea what could be the cause and how to fix it? Any help is appreciated.
I'm taking a dive into the networking side of Go, and I'd thought I'd start with a TCP Client and Server.
I am able to get the client to connect to the server and send a simple message ("Hello") successfully. However, I can not get the server to send back a response (or the get the client to read the response).
Here is the code.
Server
Address := "localhost:9999"
Addr, err := net.ResolveTCPAddr("tcp", Address)
if err != nil {
log.Fatal(err)
}
listener, err := net.ListenTCP("tcp", Addr)
if err != nil {
log.Fatal(err)
}
defer listener.Close()
//server loop
for {
conn, err := listener.Accept()
if err != nil {
continue
}
go handle(conn)
}
func handle(c net.Conn) {
totalBytes, message := connRead(c)
fmt.Println(c.RemoteAddr())
fmt.Println(string(message[:totalBytes]))
c.Write([]byte("Hi"))
fmt.Println("Replied")
c.Close()
}
func connRead(c net.Conn) (int, []byte) {
buffer := make([]byte, 4096)
totalBytes := 0
for {
n, err := c.Read(buffer)
totalBytes += n
if err != nil {
if err != io.EOF {
log.Printf("Read error: %s", err)
}
break
}
}
return totalBytes, buffer
}
Client
tcpAddr, err := net.ResolveTCPAddr("tcp", "localhost:9999")
if err != nil {
log.Fatal(err)
}
conn, err := net.DialTCP("tcp", nil, tcpAddr)
if err != nil {
log.Fatal(err)
}
defer conn.Close()
_, err = conn.Write([]byte("Hello"))
if err != nil {
log.Fatal(err)
}
tBytes, resp := connRead(conn)
fmt.Println(tBytes)
fmt.Println(string(resp[:tBytes]))
func connRead(c net.Conn) (int, []byte) {
buffer := make([]byte, 4096)
totalBytes := 0
for {
fmt.Println("Stuck?")
n, err := c.Read(buffer)
fmt.Println("Stuck.")
totalBytes += n
fmt.Println(totalBytes)
if err != nil {
if err != io.EOF {
log.Printf("Read error: %s", err)
}
break
}
}
return totalBytes, buffer
}
From what I can tell it's not a problem with the server. When I run the client, everything stops right after fmt.Println("Stuck?"). This leads me to belive that it's messing up in the n, err := c.Read(buffer) statement somehow. The server doesn't even print out the messeage length (5) and message ("Hello") untill after I Ctrl-C the client. If I comment out the read and printings in the client, then things run smoothly.
I've tried googling for answers, but nothing has come up.
What am I doing wrong? Am I using conn.Read() wrong in the client?
EDIT:
I actually do have access to Linux, so here are the SIGQUIT dumps for the pertinent functions.
Server
http://pastebin.com/itevngCq
Client
http://pastebin.com/XLiKqkvs
for {
n, err := c.Read(buffer)
totalBytes += n
if err != nil {
if err != io.EOF {
log.Printf("Read error: %s", err)
}
break
}
}
It is because you are reading from connection till EOF error occurs
conn.Write([]byte("Hello"))
The above statement won't reach EOF at all until you actually closes the connection
On pressing ctrl+c client side the connection will be closed, So EOF occurs at server side, That is the reason why it is exiting server side for loop and printing these
127.0.0.1:****
Hello
Replied
If you want to make this work you should not read the connection till EOF
There are many other alternatives for this
Chose a delimiter and read at the server until the delimiter occurs and respond back after that. Check out this link
Send number of bytes to read from client side before sending the actual message, First read number of bytes to read from the server side and then read those many bytes from the connection