I'm working on detecting NTP using golang and the gopacket package. I'm using a pcap I downloaded from wireshark. I've got the following code for opening PCAPs and handling them :
func (d *DPI) readPCAP(pcapFile string) (*pcap.Handle, error) {
// Open file instead of device
handle, err := pcap.OpenOffline(pcapFile)
if err != nil {
return nil, err
}
return handle, nil
}
And this is the code I'm writing to perform the actual detection
func TestNTP(t *testing.T) {
dpi := newDPI()
handle, _ := dpi.readPCAP("data/pcap/NTP_sync.pcap")
var filter = "udp and port 123"
dpi.setFilter(handle,filter)
ntpPackets := 0
for packet := range dpi.getPacketChan(handle) {
fmt.Println("stuff: ",packet.ApplicationLayer().Payload())
if dpi.detectNTP(packet) == 1 {
ntpPackets++
} else {
fmt.Println(" Output : ", dpi.detectNTP(packet))
}
}
fmt.Println(" Total ntp packets ", ntpPackets)
}
The Payload content in the ApplicationLayer is coming up empty and I'm unable to figure out why this is happening.
Example screenshot when I print out the ApplicationLayer itself :
https://i.gyazo.com/6257f298a09e7403bbc0be5b8ac84ccc.png
Example screenshot when I print out the Payload :
https://i.gyazo.com/7f4abd449025f5d65160fdbecffa8181.png
Could use some help figuring out what I'm doing wrong.
Thanks!
Reading through the golang soure code, I came across this :
// NTP packets do not carry any data payload, so the empty byte slice is retured.
// In Go, a nil slice is functionally identical to an empty slice, so we
// return nil to avoid a heap allocation.
func (d *NTP) Payload() []byte {
return nil
}
So, apparently it's not supposed to carry a Payload. I've managed to perform the detection using layers.
Related
Go 1.12 on Linux 4.19.93 armv6l.
Hardware is a raspberypi zero w (BCM2835) running a yocto linux image.
I've got a gpio driven SRF04 proximity sensor driven by the srf04 linux driver.
It works great over sysfs and the busybox shell.
# cat /sys/bus/iio/devices/iio:device0/in_distance_raw
1646
I've used Go before with IIO devices that support triggers and buffered output at high sample rates on this hardware platform. However for this application the srf04 driver doesn't implement those IIO features. Drat. I don't really feel like adding buffer / trigger support to the driver myself (at this time) since I do not have a need for a 'high' sample rate. A handful of pings per second should suffice for my purpose. I figure I'll calculate mean & std. dev. for a rolling window of data points and 'divine' the signal out of the noise.
So with that - I'd be perfectly happy to Read the bytes from the published sysfs file with Go.
Which brings me to the point of this post.
When I open the file for reading, and try to Read() any number of bytes, I always get a generic -EIO error.
func (s *Srf04) Read() (int, error) {
samp := make([]byte, 16)
f, err := os.OpenFile(s.readPath, OS.O_RDONLY, os.ModeDevice)
if err != nil {
return 0, err
}
defer f.Close()
n, err := f.Read(samp)
if err != nil {
// This block is always executed.
// The error is never a timeout, and always 'input/output error' (-EIO aka -5)
log.Fatal(err)
}
...
}
This seems like strange behavior to me.
So I decided to mess with using io.ReadFull. This yielded unreliable results.
func (s *Srf04) Read() (int, error) {
samp := make([]byte, 16)
f, err := os.OpenFile(s.readPath, OS.O_RDONLY, os.ModeDevice)
if err != nil {
return 0, err
}
defer f.Close()
for {
n, err := io.ReadFull(readFile, samp)
log.Println("ReadFull ", n, " bytes.")
if err == io.EOF {
break
}
if err != nil {
log.Println(err)
}
}
...
}
I ended up adding it to a loop, as I found behavior changes from 'one-off' reads to multiple read calls subsequent to one another. I have it exiting if it gets an EOF, and repeatedly trying to read otherwise.
The results are straight-up crazy unreliable, seemingly returning random results. Sometimes I get the -5, other times I read between 2 - 5 bytes from the device. Sometimes I get bytes without an eof file before the EOF. The bytes appear to represent character data for numbers (each rune is a rune between [0-9]) -- which I'd expect.
Aside: I expect this is related to file polling and the go blocking IO implementation, but I have no way to really tell.
As a temporary workaround, I decided try using os.exec, and now I get results I'd expect to see.
func (s *Srf04)Read() (int, error) {
out, err := exec.Command("cat", s.readPath).Output()
if err != nil {
return 0, err
}
return strconv.Atoi(string(out))
}
But Yick. os.exec. Yuck.
I'd try to run that cat whatever encantation under strace and then peer at what read(2) calls cat actually manages to do (including the number of bytes actually read), and then I'd try to re-create that behaviour in Go.
My own sheer guess at the problem's cause is that the driver (or the sysfs layer) is not too well prepared to deal with certain access patterns.
For a start, consider that GNU cat is not a simple-minded byte shoveler but is rather a reasonably tricky piece of software, which, among other things, considers optimal I/O block sizes for both input and output devices (if available), calls fadvise(2) etc. It's not that any of that gets actually used when you run it on your sysfs-exported file, but it may influence how the full stack (starting with the sysfs layer) performs in the case of using cat and with your code, respectively.
Hence my advice: start with strace-ing the cat and then try to re-create its usage pattern in your Go code; then try to come up with a minimal subset of that, which works; then profoundly comment your code ;-)
I'm sure I've been looking at this too long tonight, and this code is probably terrible. That said, here's the snippet of what I came up with that works just as reliably as the busybox cat, but in Go.
The Srf04 struct carries a few things, the important bits are included below:
type Srf04 struct {
readBuf []byte `json:"-"`
readFile *os.File `json:"-"`
samples *ring.Ring `json:"-"`
}
func (s *Srf04) Read() (int, error) {
/** Reliable, but really really slow.
out, err := exec.Command("cat", s.readPath).Output()
if err != nil {
log.Fatal(err)
}
val, err := strconv.Atoi(string(out[:len(out) - 2]))
if err == nil {
s.samples.Value = val
s.samples = s.samples.Next()
}
*/
// Seek should tell us the new offset (0) and no err.
bytesRead := 0
_, err := s.readFile.Seek(0, 0)
// Loop until N > 0 AND err != EOF && err != timeout.
if err == nil {
n := 0
for {
n, err = s.readFile.Read(s.readBuf)
bytesRead += n
if os.IsTimeout(err) {
// bail out.
bytesRead = 0
break
}
if err == io.EOF {
// Success!
break
}
// Any other err means 'keep trying to read.'
}
}
if bytesRead > 0 {
val, err := strconv.Atoi(string(s.readBuf[:bytesRead-1]))
if err == nil {
fmt.Println(val)
s.samples.Value = val
s.samples = s.samples.Next()
}
return val, err
}
return 0, err
}
I'm trying to create a tool to connect to a network device by Telnet and send some commands (Expect-like with certain additional requirements) using go-telnet.
To the moment I managed to create a connection and send commands with something like this:
func main() {
var loginBuffer = [6]byte{'r', 'o', 'o', 't', '\r', '\n'}
var login = loginBuffer[:]
conn, err := telnet.DialTo("10.10.10.2:23")
if nil != err {
fmt.Println(err)
}
defer conn.Close()
conn.Write(login)
}
Using Wireshark I can see the device responding, however I cannot read any response data. Guess I'm using Read() in a wrong way, not sure.
Would appreciate a working example or an explanation of how to capture and process response data in this case.
Thanks everyone, who spared their time to answer my question. I managed to identify the problem:
Every time I created a read buffer it was too big (1024 bytes) so the program was waiting for it to fill up. Now I'm using a cycle reading to a 1 byte buffer.
It seems, I also needed some criterion for the function to stop reading and proceed with sending commands.
Here is the working piece of code:
// Thin function reads from Telnet session. "expect" is a string I use as signal to stop reading
func ReaderTelnet(conn *telnet.Conn, expect string) (out string) {
var buffer [1]byte
recvData := buffer[:]
var n int
var err error
for {
n, err = conn.Read(recvData)
fmt.Println("Bytes: ", n, "Data: ", recvData, string(recvData))
if n <= 0 || err != nil || strings.Contains(out, expect) {
break
} else {
out += string(recvData)
}
}
return out
}
//convert a command to bytes, and send to Telnet connection followed by '\r\n'
func SenderTelnet(conn *telnet.Conn, command string) {
var commandBuffer []byte
for _, char := range command {
commandBuffer = append(commandBuffer, byte(char))
}
var crlfBuffer [2]byte = [2]byte{'\r', '\n'}
crlf := crlfBuffer[:]
fmt.Println(commandBuffer)
conn.Write(commandBuffer)
conn.Write(crlf)
}
func main() {
conn, err := telnet.DialTo("10.10.10.2:23")
if nil != err {
fmt.Println(err)
}
fmt.Print(ReaderTelnet(conn, "Login"))
SenderTelnet(conn, "root")
fmt.Print(ReaderTelnet(conn, "Password"))
SenderTelnet(conn, "root")
fmt.Print(ReaderTelnet(conn, ">"))
}
Where is your read operation from connection ?
I think you need to call conn.read(buffer) to read from the connection and write it to buffer
https://godoc.org/github.com/reiver/go-telnet#Conn.Read
The examples are not helping much with hte package that you are using. May be the following example taken out of different telnet go package would be more helpful.
https://github.com/ziutek/telnet/blob/master/examples/unix-cisco/main.go
I'm performing a bunch of operations over SSH on a remote machine and I'm streaming its stdout and stderr and then consuming it to by a writer, which writes to the local stdout and stderr, along with byte buffers.
Just before the writer consumes it, I want to perform a series of string manipulations on it and then write to my screen and buffer. Up to this point, it all works fine and dandy.
My issue is now it's not a stream anymore, it hangs and then outputs the whole glob in one chunk. I want it to be real time, so I put channels in my go routines but with no improvement. Below are my functions, let me know if you can spot a reason why, or possibly a better way of achieving this.
// sending
func handleStdStream(filters []string, replaceFilters map[string]string, pipe io.Reader, readers chan io.Reader) {
if filters != nil {
// filters exist
// read first 8 bytes
res := readPipe(8, pipe)
// get each line from the resulting streamed output
for _, str := range strings.Split(res, "\n") {
if str != "" {
out := lineFilterAndReplace(str, filters, replaceFilters)
// instantiate an io.Reader obj from the given string
outReader := strings.NewReader(out)
readers <- outReader
}
}
} else {
// filters dont exist
if len(replaceFilters) > 0 {
res := readPipe(8, pipe)
for _, str := range strings.Split(res, "\n") {
if str != "" {
out := lineReplace(str, replaceFilters)
// instantiate an io.Reader obj from the given string
outReader := strings.NewReader(out)
readers <- outReader
}
}
} else {
readers <- pipe
}
}
}
// recieving
outReaders := make(chan io.Reader)
go handleStdStream(outFilters, replaceFilters, stdoutIn, outReaders)
go func() {
for {
pipe := <-outReaders
_, errStdout = io.Copy(outWriter, pipe)
}
// _, errStdout = io.Copy(outWriter, stdoutIn)
}()
I don't think you need channels or goroutines to accomplish this. The Writer and Reader interfaces are already streaming; you sip bytes from a Reader continuously until you hit EOF or an error and you hand off bytes to a Writer continuously until you're done or you get an error. On its own, processing a stream does not require any concurrency, so doing this sequentially in a single goroutine is quite appropriate.
You shouldn't ignore error returns. If a function or method returns an error value, you need to check it. In the case of I/O, you usually need to stop reading from a Reader when it returns an error and you usually need to stop writing to a Writer when it returns an error. In the case of a Reader you also have to check for the special "error" value io.EOF.
I think using Scanner from the bufio package is better than trying to do your own buffering/splitting. By default, Scanner splits input on newlines (Unix-style LF or DOS-style CRLF). It also gets rid of the need to check for io.EOF, provided you only interact with the Reader through the Scanner.
Consider the following version of handleStdStream:
func handleStdStream(filters []string, replaceFilters map[string]string, pipe io.Reader, w io.Writer) error {
scanner := bufio.NewScanner(pipe)
for scanner.Scan() {
str := scanner.Text()
if str == "" {
continue
}
out := ""
if len(filters) != 0 {
out = lineFilterAndReplace(str, filters, replaceFilters)
} else {
out = lineReplace(str, replaceFilters)
}
if _, err := w.Write([]byte(out)); err != nil {
return err
}
}
if err := scanner.Err(); err != nil {
return err
}
return nil
}
You would use it like this:
err := handleStdStream(filters, replaceFilters, pipe, outWriter)
if err != nil {
// do something, like printing the error to a log or stderr
}
I'm building some server/client application in Go (the language is new to me). I searched a lot and read a whole bunch of different examples but there is still one thing I can't find. Lets say I have a single server client up and running. The client will send some kind of a message to the server and vice versa. Encoding and decoding is done by the package gob.
This example is not my application, it is only a quick example:
package main
import (
"bytes"
"encoding/gob"
"fmt"
"log"
)
type Message struct {
Sender string
Receiver string
Command uint8
Value int64
}
func (message *Message) Set(sender string, receiver string, command uint8, value int64) *Message {
message.Sender = sender
message.Receiver = receiver
message.Command = command
message.Value = value
return message
}
func main() {
var network bytes.Buffer // Stand-in for a network connection
enc := gob.NewEncoder(&network) // Will write to network.
dec := gob.NewDecoder(&network) // Will read from network.
message := new(Message).Set("first", "second", 10, -1)
err := enc.Encode(*message) // send message
if err != nil {
log.Fatal("encode error:", err)
}
var m Message
err = dec.Decode(&m) // receice message
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Printf("%q %q %d %d\n", m.Sender, m.Receiver, m.Command, m.Value)
}
This works fine, but I want the server to block until a new message is received so I can put the receiving process inside a infinite for loop inside a goroutine.
Something like that:
for {
// The server blocks HERE until a message from the client is received
fmt.Println("Received message:")
// Decode the new message
var m Message
err = dec.Decode(&m) // receice message
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Printf("%q %q %d %d\n", m.Sender, m.Receiver, m.Command, m.Value)
}
The gob decoder blocks until it has read a full message or there's an error. The read loop in the question works as is.
working example on the playground
add a length header to the raw tcp stream.
that means, send a 4-bytes-length-header information to server before send the real load. and in server side read 4 bytes, allocate buffer, full read total message, and finally decode.
assume you have a tcp connection conn, in server side we could have:
func getInt(v []byte) int {
var r uint
r = 0
r |= uint(v[0]) << 24
r |= uint(v[1]) << 16
r |= uint(v[2]) << 8
r |= uint(v[3]) << 0
return int(r)
}
buf := make([]byte, 4)
_, err := io.ReadFull(conn, buf)
if err != nil {
return
}
length := getInt(buf)
buf = make([]byte, length)
_, err = io.ReadFull(conn, buf)
if err != nil {
return
}
//do gob decode from `buf` here
you may know client side refer the the server side source I think.
I want to write test for function which includes a call to fmt.Scanf(), but I am having problem in passing the required parameter to function.
Is there a better way to do this or I need to mock fmt.Scanf()
Function to be tested is given here:
https://github.com/apsdehal/Konsoole/blob/master/parser.go#L28
// Initializes the network interface by finding all the available devices
// displays them to user and finally selects one of them as per the user
func Init() *pcap.Pcap {
devices, err := pcap.Findalldevs()
if err != nil {
fmt.Fprintf(errWriter, "[-] Error, pcap failed to iniaitilize")
}
if len(devices) == 0 {
fmt.Fprintf(errWriter, "[-] No devices found, quitting!")
os.Exit(1)
}
fmt.Println("Select one of the devices:")
var i int = 1
for _, x := range devices {
fmt.Println(i, x.Name)
i++
}
var index int
fmt.Scanf("%d", &index)
handle, err := pcap.Openlive(devices[index-1].Name, 65535, true, 0)
if err != nil {
fmt.Fprintf(errWriter, "Konsoole: %s\n", err)
errWriter.Flush()
}
return handle
}
It's theoretically possible to change the behavior of Scanf by hotswapping the value of os.Stdin with some other os.File. I wouldn't particularly recommend it just for testing purposes, though.
A better option would just be to make your Init take in an io.Reader that you pass to Fscanf.
Overall, however, it would likely be better to separate your device initialization code from your input as much as possible. This probably means having a device list returning function and a device opening function. You only need to prompt for selection in live/main code.