I need to write multicast listener on Go. I faces the problem of twicing packets when I read it. It seems that I need to set IP_MULTICAST_LOOP to false. It is not easy to do in Go.
I found this post. It seems that it should work. But I still get copies of the same host. How should it be done?
ipAddr, err := net.ResolveUDPAddr("udp", groupAddress)
if err != nil {...}
iface, err := net.InterfaceByName("en0")
if err != nil {...}
conn, err := net.ListenPacket("udp4", groupAddress)
if err != nil {...}
pc := ipv4.NewPacketConn(conn)
if err := pc.JoinGroup(iface, ipAddr); err != nil {...}
if err := pc.SetMulticastLoopback(false); err != nil {...}
if loop, err := pc.MulticastLoopback(); err == nil {...}
buf := make([]byte, 1024)
for {
n, _, addr, err := pc.ReadFrom(buf)
if err != nil {...}
fmt.Printf("recv from %v: [%s] \n", addr, buf[:n])
}
The simplest way is to use the ListenMulticastUDP wrapper in the net package, as actually already explained in the other SO answer you point to, How to set IP_MULTICAST_LOOP on multicast UDPConn in Golang.
If you follow the implementation of ListenMulticastUDP(), you will see that at a certain point it calls setIPv4MulticastLoopback(fd, false).
If you need something more advanced, the documentation of ListenMulticastUDP() suggests to look at https://godoc.org/golang.org/x/net/ipv4 and https://godoc.org/golang.org/x/net/ipv6, which document extensively how to do multicast in Go.
Here is some minimal code (tested on MacOS, but platform-independent) that shows how to use ListenMulticastUDP():
func main() {
// MDNS (https://en.wikipedia.org/wiki/Multicast_DNS)
groupAddress := "224.0.0.251:5353"
ifaceName := "en0"
if err := run(groupAddress, ifaceName); err != nil {
fmt.Fprintln(os.Stderr, "error:", err)
os.Exit(1)
}
}
func run(groupAddr string, ifaceName string) error {
iface, err := net.InterfaceByName(ifaceName)
if err != nil {
return err
}
gaddr, err := net.ResolveUDPAddr("udp", groupAddr)
if err != nil {
return err
}
conn, err := net.ListenMulticastUDP("udp", iface, gaddr)
if err != nil {
return err
}
buf := make([]byte, 1024)
for {
n, addr, err := conn.ReadFromUDP(buf)
if err != nil {
return err
}
fmt.Printf("recv %4d bytes from %v\n", n, addr)
}
}
Related
I'm trying to create a TCP server that will timeout if the client does not respond within the span of every second.
I tried:
func main() {
listener, err := net.Listen("tcp", "localhost:8000")
if err != nil {
log.Fatal(err)
}
for {
conn, err := listener.Accept()
conn.SetDeadline(time.Now().Add(timeout))
if err != nil {
log.Print(err)
}
go handleConn(conn)
}
}
where the timeout is a single second but the disconnects immediately, not even waiting for a reply.
What you want can be achieved by setting socket options on your listener. Tweak the values as per your needs
Note that this is its own KeepAlive and does not depend on incoming/outgoing data by application
func enableTCPKeepAlive(listener *net.TCPListener) error {
rawConn, err := listener.SyscallConn()
if err != nil {
return err
}
cfg := config.TLSServerConfig()
rawConn.Control(
func(fdPtr uintptr) {
// got socket file descriptor. Setting parameters.
fd := int(fdPtr)
//Idle time before sending probe.
err = syscall.SetsockoptInt(fd, syscall.IPPROTO_TCP, syscall.TCP_KEEPIDLE, cfg.TCPKAIdleTime)
if err != nil {
return err
}
//Number of probes.
err := syscall.SetsockoptInt(fd, syscall.IPPROTO_TCP, syscall.TCP_KEEPCNT, cfg.TCPKANumProbes)
if err != nil {
return err
}
//Wait time after an unsuccessful probe.
err = syscall.SetsockoptInt(fd, syscall.IPPROTO_TCP, syscall.TCP_KEEPINTVL, cfg.TCPKAInterval)
if err != nil {
return err
}
// go syscall doesn't have the constant 0x12 (18) for TCP_USER_TIMEOUT.
// 0x12 value referenced from linux kernel source code header:
// include/uapi/linux/tcp.h
err = syscall.SetsockoptInt(fd, syscall.IPPROTO_TCP, 0x12, cfg.TCPKAUserTimeout)
if err != nil {
return err
}
})
return nil
}
There are more options available than the ones I have mentioned above.
Call this function on your listener before the for loop.
func main() {
listener, err := net.Listen("tcp", "localhost:8000")
if err != nil {
log.Fatal(err)
}
err = enableTCPKeepAlive(listener)
if err != nil {
log.Fatal(err)
}
for {
conn, err := listener.Accept()
conn.SetDeadline(time.Now().Add(timeout))
if err != nil {
log.Print(err)
}
go handleConn(conn)
}
}
The problem is almost always in code that is not posted here. The function obviously works like a charme:
package main
import (
"crypto/rand"
"log"
"net"
"time"
)
func main() {
listener, err := net.Listen("tcp", "localhost:8000")
if err != nil {
log.Fatal(err)
}
go func() {
for {
conn, err := listener.Accept()
if err != nil {
log.Print(err)
return
}
go func(c net.Conn) {
defer c.Close()
conn.SetDeadline(time.Now().Add(2 * time.Second))
if err != nil {
log.Print(err)
return
}
buf := make([]byte, 1<<19) // 512 KB
for {
_, err := conn.Read(buf)
if err != nil {
log.Print(err)
break
}
}
}(conn)
}
}()
payload := make([]byte, 1<<20)
_, err = rand.Read(payload) // generate a random payload
if err != nil {
log.Print(err)
}
conn, err := net.Dial("tcp", listener.Addr().String())
if err != nil {
log.Fatal(err)
}
log.Println("Connected to server.")
time.Sleep(5 * time.Second)
_, err = conn.Write(payload)
if err != nil {
log.Print(err)
}
listener.Close()
}
I'm trying with tls.Conn i.e, conn.Read from golang crypto/tls to read a large size of data. However when all the data is read out, then the program will never stop. Why? My simplified code is as follow:
// server
func main() {
log.SetFlags(log.Lshortfile)
cer, err := tls.LoadX509KeyPair("server.pem", "server.key")
if err != nil {
log.Println(err)
return
}
config := &tls.Config{Certificates: []tls.Certificate{cer}}
ln, err := tls.Listen("tcp", ":2000", config)
if err != nil {
log.Println(err)
return
}
defer ln.Close()
for {
conn, err := ln.Accept()
if err != nil {
log.Println(err)
continue
}
go handleConnection(conn)
}
}
func handleConnection(conn net.Conn) {
defer conn.Close()
buf := make([]byte, 0)
tmp := make([]byte, 1000)
totalSize := 0
for {
n, err := conn.Read(tmp) //after read all the data, server stucked here
totalSize += n
if err != nil {
if err != io.EOF {
log.Printf("prover: conn: read: %s", err)
}
break
}
buf = append(buf, tmp[:n]...)
}
n, err := conn.Write([]byte("finished"))
if err != nil {
log.Println(n, err)
return
}
}
The logic of the client:
// client
func main() {
log.SetFlags(log.Lshortfile)
conf := &tls.Config{
InsecureSkipVerify: true,
}
conn, err := tls.Dial("tcp", "127.0.0.1:2000", conf)
if err != nil {
log.Println(err)
return
}
defer conn.Close()
writeData := make([]byte, 4096)
n, err := conn.Write(writeData)
if err != nil {
log.Println(n, err)
return
}
fmt.Println("finish writing")
buf := make([]byte, 4096)
n, err = conn.Read(buf)
if err != nil {
log.Println(n, err)
}
fmt.Print("finish reading")
}
How can I read a large size of data in the server and after the data is completely read out, then the server will send a response to the client. I tried with net.Conn (TCPConn), this logic works. WHy?
I have a TCP server server :
l, err := net.Listen("tcp", "localhost:"+strconv.Itoa(tcpPort))
The server listens to incoming client requests as is :
for {
c, err := l.Accept()
if err != nil {
log.Fatal(err)
}
b := make([]byte, 1024)
c.Read(b)
fmt.Println(string(b)) // "Hello"
}
I have a client :
conn, err := net.Dial("tcp", address)
Now I I write once with conn.Write([]byte("Hello"))
The server catches Hello. But if I have these :
_, err := conn.Write([]byte("Hello"))
if err != nil {
log.Fatal(err)
}
_, err = conn.Write([]byte("World"))
if err != nil {
log.Fatal(err)
}
Then I will get Hello, but not World.
How can I write multiple time on the same connection ?
Full function below
func main() {
l, err := net.Listen("tcp", "localhost:1234")
if err != nil {
log.Fatal(err)
}
defer l.Close()
go func() {
for {
c, err := l.Accept()
if err != nil {
log.Fatal(err)
}
b := make([]byte, 1024)
c.Read(b)
fmt.Println(string(b))
}
}()
conn, err := net.Dial("tcp", "localhost:1234")
if err != nil {
log.Fatal(err)
}
_, err = conn.Write([]byte("hello"))
if err != nil {
log.Fatal(err)
}
_, err = conn.Write([]byte("world"))
if err != nil {
log.Fatal(err)
}
}
Your problem is in the server code, the one that receives data. It only reads once from the tcp stream. If you want it to read "world", replace the single read operation by a loop:
go func() {
for {
c, err := l.Accept()
if err != nil {
log.Fatal(err)
}
for {
b := make([]byte, 1024)
c.Read(b)
fmt.Println(string(b))
}
}
}()
Your problem is not with conn.Write but with reading from connection. Now you read just once from every new opened connection accepted by l.Accept(). Solution is to read repeatedly.
Your code is also limited to handle just one connection. And do not forget to check error from c.Read(b) to know when stop listen on this connection.
go func() {
for {
c, err := l.Accept()
if err != nil {
log.Fatal(err)
}
go func(conn net.Conn) {
for {
b := make([]byte, 1024)
_, err := conn.Read(b)
if err != nil {
if err != io.EOF {
fmt.Println("read error:", err)
}
break
}
fmt.Println(string(b))
}
fmt.Println("Stopping handle connection")
}(c)
}
}()
Im doing some testing for receiving multicast UDP stream and save it to file using the below functions:
func Record2(udp string, filePath string) {
c1, err := net.ListenPacket("udp4", udp)
if err != nil {
return
}
defer c1.Close()
p1 := ipv4.NewPacketConn(c1)
en0, err := net.InterfaceByName("enp3s0")
addr, _ := net.ResolveUDPAddr("udp4", udp)
if err := p1.JoinGroup(en0, addr); err != nil {
}
Bee.DelFile(filePath)
fo, err := os.Create(filePath)
defer fo.Close()
if err != nil {
panic(err)
}
io.Copy(fo, p1)
}
func Record(udp string, filePath string) {
addr, err := net.ResolveUDPAddr("udp4", udp)
if err != nil {
return
}
Bee.DelFile(filePath)
fo, err := os.Create(filePath)
defer fo.Close()
if err != nil {
panic(err)
}
l, err := net.ListenMulticastUDP("udp4", nil, addr)
defer l.Close()
l.SetReadBuffer(maxDatagramSize)
io.Copy(fo, l)
}
go Record2("224.1.2.2:1234", "file1")
go Record2("224.1.2.1:1234", "file2")
go Record("224.1.2.2:1234", "file3")
Record("224.1.2.1:1234", "file4")
the problem is that udp stream have different content while all files saved got the same content of the first address
this is not occuring when using UDP addresses with different ports
im not sure if this is related to SO_REUSEADDR
how to fix that?
I'm trying to download a remote file over ssh
The following approach works fine on shell
ssh hostname "tar cz /opt/local/folder" > folder.tar.gz
However the same approach on golang giving some difference in output artifact size. For example the same folders with pure shell produce artifact gz file 179B and same with go script 178B.
I assume that something has been missed from io.Reader or session got closed earlier. Kindly ask you guys to help.
Here is the example of my script:
func executeCmd(cmd, hostname string, config *ssh.ClientConfig, path string) error {
conn, _ := ssh.Dial("tcp", hostname+":22", config)
session, err := conn.NewSession()
if err != nil {
panic("Failed to create session: " + err.Error())
}
r, _ := session.StdoutPipe()
scanner := bufio.NewScanner(r)
go func() {
defer session.Close()
name := fmt.Sprintf("%s/backup_folder_%v.tar.gz", path, time.Now().Unix())
file, err := os.OpenFile(name, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0644)
if err != nil {
panic(err)
}
defer file.Close()
for scanner.Scan() {
fmt.Println(scanner.Bytes())
if err := scanner.Err(); err != nil {
fmt.Println(err)
}
if _, err = file.Write(scanner.Bytes()); err != nil {
log.Fatal(err)
}
}
}()
if err := session.Run(cmd); err != nil {
fmt.Println(err.Error())
panic("Failed to run: " + err.Error())
}
return nil
}
Thanks!
bufio.Scanner is for newline delimited text. According to the documentation, the scanner will remove the newline characters, stripping any 10s out of your binary file.
You don't need a goroutine to do the copy, because you can use session.Start to start the process asynchronously.
You probably don't need to use bufio either. You should be using io.Copy to copy the file, which has an internal buffer already on top of any buffering already done in the ssh client itself. If an additional buffer is needed for performance, wrap the session output in a bufio.Reader
Finally, you return an error value, so use it rather than panic'ing on regular error conditions.
conn, err := ssh.Dial("tcp", hostname+":22", config)
if err != nil {
return err
}
session, err := conn.NewSession()
if err != nil {
return err
}
defer session.Close()
r, err := session.StdoutPipe()
if err != nil {
return err
}
name := fmt.Sprintf("%s/backup_folder_%v.tar.gz", path, time.Now().Unix())
file, err := os.OpenFile(name, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0644)
if err != nil {
return err
}
defer file.Close()
if err := session.Start(cmd); err != nil {
return err
}
n, err := io.Copy(file, r)
if err != nil {
return err
}
if err := session.Wait(); err != nil {
return err
}
return nil
You can try doing something like this:
r, _ := session.StdoutPipe()
reader := bufio.NewReader(r)
go func() {
defer session.Close()
// open file etc
// 10 is the number of bytes you'd like to copy in one write operation
p := make([]byte, 10)
for {
n, err := reader.Read(p)
if err == io.EOF {
break
}
if err != nil {
log.Fatal("err", err)
}
if _, err = file.Write(p[:n]); err != nil {
log.Fatal(err)
}
}
}()
Make sure your goroutines are synchronized properly so output is completeky written to the file.