I want to develop a UDP server based on Linux. There are some IP set on that host machine (such as 1.1.1.1,1.1.1.2, 2001::1:1:1:1), and I want server listen on all IP as follows (9090 as sample)
udp6 0 0 :::9090 :::*
The server code as follows
package main
import (
"fmt"
"net"
)
func main() {
udpAddr, err := net.ResolveUDPAddr("udp", ":9090")
conn, err := net.ListenUDP("udp", udpAddr)
if err != nil {
fmt.Println(err)
return
}
var data [1024]byte
n, addr, err := conn.ReadFromUDP(data[:])
if err != nil {
fmt.Println(err)
}
fmt.Println(n)
fmt.Println(addr)
// this is not my wanted result. it will print [::]:9090
fmt.Println(conn.LocalAddr())
}
When the client dial this server (dst_string is 1.1.1.1:9090);
Actual result:
the server will print conn.LocalAddr() with
[::]:9090
excepted result:
the server should print
1.1.1.1:9090
How to achieve that?
BTW: I know if UDP server only listen 1.1.1.1:9090 can make that. But server has many IP, I want the server listen all IP and LocalAddr() can print 1.1.1.1:9090
Let's cite this post from a PowerDNS developer:
There are two ways to listen on all addresses, one of which is to enumerate all interfaces, grab all their IP addresses, and bind to all of them. Lots of work, and non-portable work too. We really did not want to do that. You also need to monitor new addresses arriving.
Secondly, just bind to 0.0.0.0 and ::! This works very well for TCP and other connection-oriented protocols, but can fail silently for UDP and other connectionless protocols. How come? When a packet comes in on 0.0.0.0, we don’t know which IP address it was sent to. And this is a problem when replying to such a packet – what would the correct source address be? Because we are connectionless (and therefore stateless), the kernel doesn’t know what to do.
So it picks the most appropriate address, and that may be the wrong one. There are some heuristics that make some kernels do the right thing more reliably, but there are no guarantees.
When receiving packets on datagram sockets, we usually use recvfrom(2), but this does not provide the missing bit of data: which IP address the packet was actually sent to. There is no recvfromto(). Enter the very powerful recvmsg(2). Recvmsg() allows for the getting of a boatload of parameters per datagram, as requested via setsockopt().
One of the parameters we can request is the original destination IP address of the packet.
<…>
IPv4
<..> For Linux, use the setsockopt() called IP_PKTINFO, which will get you a parameter over recvmsg() called IP_PKTINFO, which carries a struct in_pktinfo, which has a 4 byte IP address hiding in its ipi_addr field.
It appears, that the closest to recvmsg() thing there exists in the net package is net.IPConn.ReadMsgIP, and its documentation states that
The packages golang.org/x/net/ipv4 and golang.org/x/net/ipv6 can be used to manipulate IP-level socket options in oob.
Hence, looks like a way forward.
I'd also make the following points explicit (they are not obvious from the text above):
It seems, the net package of the Go's stdlib does not have a standard and easy-to-use way to have what you want.
It appears that the approach to getting the destination address of a datagram when receiving them on a wildcard address is not really standardized, and hence implementations vary between different kernels.
While it looks that net.IPConn.ReadMsgIP wraps recvmsg(2), I'd first verify that in the source code of the Go standard library. Pay attention to the fact that the stdlib contains code for all platforms it supports, so make sure you understand what build constraints are.
https://godoc.org/golang.org/x/net/ may help. And so do the syscall package and https://godoc.org/golang.org/x/sys — if the stock one falls short.
Thanks response of kostix very much.
And according to IP_PKTINFO prompt, I found the following code can resolve my ipv4 issue directly
https://gist.github.com/omribahumi/5da8517497042152691e
But for ipv6, the result is still NOT excepted
package main
import (
"bytes"
"encoding/binary"
"fmt"
"net"
"syscall"
)
func main() {
serverAddr, _ := net.ResolveUDPAddr("udp", ":9999")
sConn, _ := net.ListenUDP("udp", serverAddr)
file, _ := sConn.File()
syscall.SetsockoptInt(int(file.Fd()), syscall.IPPROTO_IPV6, syscall.IP_PKTINFO, 1)
data := make([]byte, 1024)
oob := make([]byte, 2048)
sConn.ReadMsgUDP(data, oob)
oob_buffer := bytes.NewBuffer(oob)
msg := syscall.Cmsghdr{}
binary.Read(oob_buffer, binary.LittleEndian, &msg)
if msg.Level == syscall.IPPROTO_IPV6 && msg.Type == syscall.IP_PKTINFO {
packet_info := syscall.Inet6Pktinfo{}
binary.Read(oob_buffer, binary.LittleEndian, &packet_info)
fmt.Println(packet_info)
// the ipv6 address is not my wanted result
fmt.Println(packet_info.Addr)
}
}
The result as follows
root#test-VirtualBox:/home/test/mygo/src/udp# go run s2.go
{[64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0}
[64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
the tcpdump sniffer as follows
22:20:47.009712 IP6 ::1.43305 > ::1.1025: UDP, length 6
I resolve this ipv6 issue by Setting the source IP for a UDP socket via set ipv6 options
err = syscall.SetsockoptInt(int(fd), syscall.IPPROTO_IPV6, syscall.IPV6_RECVPKTINFO, 1)
thanks everyone
Related
SSRF vul should forbidden local ip request.
How did go net.LookupIP work with Base 8 (octal )?
#go code
net.LookupIP("0177.0.0.01") // NOT_OK 0177.0.0.01
net.LookupIP("0266.075.0310.07") // OK 182.61.200.7
#shell
ping 0177.0.0.01 //OK 127.0.0.1
ping 0266.075.0310.07 // OK 182.61.200.7
input http:address to chrome is the same as ping ,but go not stablize
debug-snapshot-img net.LookupIP("0177.0.0.01") Has Not Decode Base8
debug-snapshot-img ping 0177.0.0.01 in shell Has Decode Base8
debug-snapshot-img net.LookupIP("0266.075.0310.07") Has Decode Base8
The problem
I think you cannot for a simple reason: that A.B.C.D format of IPv4 addresses is called "dot-decimal" (or "dotted-decimal") for a reason: the numbers representing the octets of an address are in the decimal format, and no "special modifiers"—such as 0x, 0, 0o etc—to make it "not decimal" are allowed.
I'm inclined to think that the fact ping is OK to parse the address with some octets in octal is some weird artefact of its implementation; for instance, it also parses hexadecimal:
$ ping 0xAB.0.0.0x01
PING 0xAB.0.0.0x01 (171.0.0.1) 56(84) bytes of data.
^C
Also note that there's no such thing as "the standard ping tool": any OS can have any implementation of it; for instance, ping available on Windows definitely shares no source code with ping available on a typical GNU/Linux-based OS, and that one is probably different from ping implementations available on systems tracing their heritage to *BSD.
To further illustrate my point, consider host:
$ host -t PTR 0177.0.0.01
Host 0177.0.0.01 not found: 3(NXDOMAIN)
This tool, as you can see, failed to cope with the leading 0s.
So, I think it's sensible to assume that the fact net.LookupIP (and net.ParseIP for that matter) definitely cannot be blamed for not supporting "octals" when they parse the dotted-decimal representation of an IPv4 address.
Possible solutions
Possible solutions depend on what really you're after.
If you have to accept IP addresses as strings in such a "dotted-not-so-decimal" format, I'd propose to write a simplistic custom parser which would "preprocess" the strings before calling net.LookupIP on them.
Such a preprocessor would split the string and try to parse each octet, then assemble the address back.
Go has strconv.ParseInt in its standard library, which accepts the "base" argument telling it which base the digits forming the string representation to parse are, and if you pass 0 as the base, the function uses the usual heuristics to guess the base; in particular, the leading 0 signalizes the base 8—just as you'd want it to.
So, we could roll something like this:
package main
import (
"errors"
"fmt"
"net"
"strconv"
"strings"
)
func normalizeIPv4Addr(s string) (string, error) {
parts := strings.SplitN(s, ".", 4)
if len(parts) != 4 || strings.IndexByte(parts[3], '.') != -1 {
return "", errors.New("invalid IPv4 address: invalid number of octets")
}
var dst [4]byte
for i, p := range parts {
n, err := strconv.ParseInt(p, 0, 8)
if err != nil {
return "", fmt.Errorf("invalid IPv4 address: invalid octet %d: %s", i+1, err)
}
if n < 0 || 255 < n {
return "", fmt.Errorf("invalid IPv4 address: invalid octet %d: out of range", i+1)
}
dst[i] = byte(n)
}
return net.IP(dst[:]).String(), nil
}
func main() {
fmt.Println(normalizeIPv4Addr("0177.0.0.01"))
}
Playground.
How can I get the start and end for v4 and v6 ip addresses of a CIDR? I don't care about addresses which are between.
I have checked the net library and parseCIDR does not return this information. Is there an idiomatic way to calculate the ip range of a CIDR which will work for v6 and v4 addresses alike?
For example, given the CIDR 2001:db8:a0b:12f0::1/32 I would expect
2001:0db8:0000:0000:0000:0000:0000:0000 and 2001:0db8:ffff:ffff:ffff:ffff:ffff:ffff returned as the start and end addresses respectively.
Using the IPAddress Go library, this code will do it for either IPv4 or IPv6. Disclaimer: I am the project manager.
import (
"fmt"
"github.com/seancfoley/ipaddress-go/ipaddr"
)
func main() {
ipRange("2001:db8:a0b:12f0::1/32")
ipRange("1.2.3.1/16")
}
func ipRange(cidr string) {
block := ipaddr.NewIPAddressString(cidr).GetAddress().ToPrefixBlock()
addr := block.WithoutPrefixLen()
fmt.Printf("addr %s block %s lower %s upper %s\n",
cidr, block, addr.GetLower(), addr.GetUpper())
}
Output:
addr 2001:db8:a0b:12f0::1/32 block 2001:db8::/32 lower 2001:db8:: upper 2001:db8:ffff:ffff:ffff:ffff:ffff:ffff
addr 1.2.3.1/16 block 1.2.0.0/16 lower 1.2.0.0 upper 1.2.255.255
There are many things I don't understand about ipv6 and networking in general, which is why I need some further clarification on some the answers already posted to other questions. I'll list my questions, what I grasped from other answers, and what I'm still confused about.
Say I have a VPS with a /56 ipv6 subnet (256 * residential /64 subnets) allotted to it. How can I programmatically find the range (prefix?) of the ip's I "own".
How to get IPv4 and IPv6 address of local machine?. This is the answer I saw for this question: and what I think I understand is that I get the DNS hostname for the machine, then look up that same hostname to find the range. I'm wondering two things: How do I do this in Go, and
How do I transfer this range ^ into a slice (array) of ipv6 addresses. For this specific use case: the ideal solution would be to only get one ipv6 address per \64 subnet, resulting in 256 seperate ips
DNS is not very helpful in determining the local IP addresses, because a DNS entry is not required to make an IP address work, nor is it required to point to (only) the machine that you happen to run your program on.
Instead, inspect the network interfaces and their configuration:
package main
import (
"fmt"
"log"
"net"
"os"
"text/tabwriter"
)
func main() {
tw := tabwriter.NewWriter(os.Stdout, 0, 0, 1, ' ', 0)
ifaces, err := net.Interfaces()
if err != nil {
log.Fatal(err)
}
for _, iface := range ifaces {
addrs, err := iface.Addrs()
if err != nil {
log.Fatal(err)
}
for _, addr := range addrs {
addr, ok := addr.(*net.IPNet)
if !ok {
// Not an IP interface
continue
}
if addr.IP.To4() != nil {
// Skip IPv4 addresses
continue
}
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\n",
iface.Name, addr.String(), addr.IP, addr.Mask)
}
}
tw.Flush()
}
For my local machine the output is:
lo ::1/128 ::1 ffffffffffffffffffffffffffffffff
enp2s0 fe80::52e5:49ff:fe3b:107a/64 fe80::52e5:49ff:fe3b:107a ffffffffffffffff0000000000000000
docker0 fe80::42:afff:fedb:7389/64 fe80::42:afff:fedb:7389 ffffffffffffffff0000000000000000
tun0 fe80::f22c:2d3b:a5a0:1b61/64 fe80::f22c:2d3b:a5a0:1b61 ffffffffffffffff0000000000000000
vethd176f0c fe80::1cc1:65ff:fe39:feff/64 fe80::1cc1:65ff:fe39:feff ffffffffffffffff0000000000000000
Note that these addresses are not necessarily reachable from the Internet. This all depends on how the routing of the hoster works. In any kind of cloud setup, you are almost always better off querying the providers APIs.
To list all /64 subnets in a particular /56 subnet, you have to leave the 56 upper bits of the subnet address as they are and permute the following 64-56 = 8 bits (which happens to be the eigth byte):
package main
import (
"fmt"
"net"
)
func main() {
_, subnet, _ := net.ParseCIDR("2001:db8::/56")
fmt.Println(subnet)
subnet.Mask = net.CIDRMask(64, 128) // change mask to /64
for i := 0; i <= 0xff; i++ {
subnet.IP[7] = byte(i) // permute the 8th byte
fmt.Println("\t", subnet)
}
// Output:
// 2001:db8::/56
// 2001:db8::/64
// 2001:db8:0:1::/64
// 2001:db8:0:2::/64
// 2001:db8:0:3::/64
// 2001:db8:0:4::/64
// 2001:db8:0:5::/64
// 2001:db8:0:6::/64
// 2001:db8:0:7::/64
// 2001:db8:0:8::/64
// [...]
}
Regarding the second part of your question, "How do I transfer this range ^ into a slice (array) of ipv6 addresses"
The IPAddress Go library can do this with polymorphic code that works with both IPv4 and IPv6 addresses and all prefix lengths. Repository here. Disclaimer: I am the project manager.
addrStr := "2001:db8::/56"
addr := ipaddr.NewIPAddressString(addrStr).GetAddress()
addrAdjusted := addr.SetPrefixLen(64) // adjust prefix
iterator := addrAdjusted.PrefixIterator()
var blocks []*ipaddr.IPAddress
for iterator.HasNext() {
blocks = append(blocks, iterator.Next())
}
// print the details
fmt.Println("first and last blocks are",
addrAdjusted.GetLower().ToPrefixBlock(), "and",
addrAdjusted.GetUpper().ToPrefixBlock())
fmt.Print("list: ")
for i, addr := range blocks {
if i < 3 || len(blocks)-i <= 3 {
if i > 0 {
fmt.Print(", ")
}
fmt.Print(addr)
} else if i == 3 {
fmt.Print(", ...")
}
}
Output:
first and last blocks are 2001:db8::/64 and 2001:db8:0:ff::/64
list: 2001:db8::/64, 2001:db8:0:1::/64, 2001:db8:0:2::/64, ..., 2001:db8:0:fd::/64, 2001:db8:0:fe::/64, 2001:db8:0:ff::/64
I have a device, which continues to send data over a serial port.
Now I want to read this and process it.
The data send this delimiter "!" and
as soon as this delimiter appears I want to pause reading to processing the data thats already been received.
How can I do that? Is there any documentation or examples that I can read or follow.
For reading data from a serial port you can find a few packages on Github, e.g. tarm/serial.
You can use this package to read data from your serial port. In order to read until a specific delimiter is reached, you can use something like:
config := &serial.Config{Name: "/dev/ttyUSB", Baud: 9600}
s, err := serial.OpenPort(config)
if err != nil {
// stops execution
log.Fatal(err)
}
// golang reader interface
r := bufio.NewReader(s)
// reads until delimiter is reached
data, err := r.ReadBytes('\x21')
if err != nil {
// stops execution
log.Fatal(err)
}
// or use fmt.Printf() with the right verb
// https://golang.org/pkg/fmt/#hdr-Printing
fmt.Println(data)
See also: Reading from serial port with while-loop
bufio's reader unfortunately did not work for me - it kept crashing after a while. This was a no-go since I needed a stable solution for a low-performance system.
My solution was to implement this suggestion with a small tweak. As noted, if you don't use bufio, the buffer gets overwritten every time you call
n, err := s.Read(buf0)
To fix this, append the bytes from buf0 to a second buffer, buf1:
if n > 0 {
buf1 = append(buf1, buf0[:n]...)
}
Then parse the bytes stored in buf1. If you find a subset you're looking for, process it further.
make sure to clear the buffers in a suitable manner
make sure to limit the frequency the loop is running with (e.g. time.Sleep)
I'm trying to develop a simple job queue server with some worker that query it but I encountered a problem with my net/http server. I'm surely doing something bad but after ~3 minutes my server start displaying :
http: Accept error: accept tcp [::]:4200: accept4: too many open files; retrying in 1s
For information it receive 10 request per second in my test case.
Here's two files to reproduce this error :
// server.go
package main
import (
"net/http"
)
func main() {
http.HandleFunc("/get", func(rw http.ResponseWriter, r *http.Request) {
http.Error(rw, "Try again", http.StatusInternalServerError)
})
http.ListenAndServe(":4200", nil)
}
// worker.go
package main
import (
"net/http"
"time"
)
func main() {
for {
res, _ := http.Get("http://localhost:4200/get")
defer res.Body.Close()
if res.StatusCode == http.StatusInternalServerError {
time.Sleep(100 * time.Millisecond)
continue
}
return
}
}
I already done some search about this error and I found some interesting response but none of these fixed my issue.
The first response I saw was to correctly close the Body in the http.Get response, as you can see I did it.
The second response was to change the file descriptor ulimit of my system but as I will not control where my app will run I prefer to not use this solution (But for information it's set at 1024 on my system)
Can someone explain me why this problem happen and how I can fix it in my code ?
Thanks a lot for your time
EDIT :
EDIT 2 : In comment Martin says that I'm not closing the Body, I tried to close it (without defer so) and it fixed the issue. Thanks Martin ! I was thinking that continue will execute my defer, I was wrong.
I found a post explaining the root problem in a lot more detail.
Nathan Smith even explains how to control timeouts on the TCP level, if needed.
Below is a summary of everything I could find on this particular problem, as well as the best practices to avoid this problem in future.
Problem
When a response is received regardless of whether response-body is required or not, the connection is kept alive until the response-body stream is closed. So, as mentioned in this thread, always close the response-body. Even if you do not need to use/read the body content:
func Ping(url string) (bool) {
// simple GET request on given URL
res, err := http.Get(url)
if err != nil {
// if unable to GET given URL, then ping must fail
return false
}
// always close the response-body, even if content is not required
defer res.Body.Close()
// is the page status okay?
return res.StatusCode == http.StatusOK
}
Best Practice
As mentioned by Nathan Smith never use the http.DefaultClient in production systems, this includes calls like http.Get as it uses http.DefaultClient at its base.
Another reason to avoid http.DefaultClient is that it is a Singleton (package level variable), meaning that the garbage collector will not try to clean it up, which will leave idling subsequent streams/sockets alive.
Instead create your own instance of http.Client and remember to always specify a sane Timeout:
func Ping(url string) (bool) {
// create a new instance of http client struct, with a timeout of 2sec
client := http.Client{ Timeout: time.Second * 2 }
// simple GET request on given URL
res, err := client.Get(url)
if err != nil {
// if unable to GET given URL, then ping must fail
return false
}
// always close the response-body, even if content is not required
defer res.Body.Close()
// is the page status okay?
return res.StatusCode == http.StatusOK
}
Safety Net
The safety net is for that newbie on the team, who does not know the shortfalls of http.DefaultClient usage. Or even that very useful, but not so active, open-source library that is still riddled with http.DefaultClient calls.
Since http.DefaultClient is a Singleton we can easily change the Timeout setting, just to ensure that legacy code does not cause idle connections to remain open.
I find it best to set this on the package main file in the init function:
package main
import (
"net/http"
"time"
)
func init() {
/*
Safety net for 'too many open files' issue on legacy code.
Set a sane timeout duration for the http.DefaultClient, to ensure idle connections are terminated.
Reference: https://stackoverflow.com/questions/37454236/net-http-server-too-many-open-files-error
*/
http.DefaultClient.Timeout = time.Minute * 10
}
As Martin say in comment I don't really closed the Body after the Get request. I used defer res.Body.Close() but it's not executed since I'm staying in the for loop. So continue dont't trigger defer
Please note that in some cases the setting in /etc/sysctl.conf
net.ipv4.tcp_tw_recycle = 1
Could cause this error because TCP connections remain open.
A temporary solution, just increase the number of open files:
ulimit -Sn 10000