http://play.golang.org/p/RQXB-hCq_M
type Header struct {
ByteField1 uint32 // 4 bytes
ByteField2 [32]uint8 // 32 bytes
ByteField3 [32]uint8 // 32 bytes
SkipField1 []SomethingElse
}
func main() {
var header Header
headerBytes := make([]byte, 68) // 4 + 32 + 32 == 68
headerBuf := bytes.NewBuffer(headerBytes)
err := binary.Read(headerBuf, binary.LittleEndian, &header)
if err != nil {
fmt.Println(err)
}
fmt.Println(header)
}
I don't want to read from the buffer into the header struct in chunks. I want to read into the bytefield in one step but skip non byte fields. If you run the program in the given link (http://play.golang.org/p/RQXB-hCq_M) you will find that binary.Read to throw an error: binary.Read: invalid type []main.SomethingElse
Is there a way that I can skip this field?
Update:
Based on dommage's answer, I decided to embed the fields inside the struct instead like this
http://play.golang.org/p/i0xfmnPx4A
You can cause a field to be skipped by prefixing it's name with _ (underscore).
But: binary.Read() requires all fields to have a known size. If SkipField1 is of variable or unknown length then you have to leave it out of your struct.
You could then use io.Reader.Read() to manually skip over the skip field portion of your input and then call binary.Read() again.
Related
I'm trying to do direct i/o on linux, so I need to create memory aligned buffers. I copied some code to do it, but I don't understand how it works:
package main
import (
"fmt"
"golang.org/x/sys/unix"
"unsafe"
"yottaStore/yottaStore-go/src/yfs/test/utils"
)
const (
AlignSize = 4096
BlockSize = 4096
)
// Looks like dark magic
func Alignment(block []byte, AlignSize int) int {
return int(uintptr(unsafe.Pointer(&block[0])) & uintptr(AlignSize-1))
}
func main() {
path := "/path/to/file.txt"
fd, err := unix.Open(path, unix.O_RDONLY|unix.O_DIRECT, 0666)
defer unix.Close(fd)
if err != nil {
panic(err)
}
file := make([]byte, 4096*2)
a := Alignment(file, AlignSize)
offset := 0
if a != 0 {
offset = AlignSize - a
}
file = file[offset : offset+BlockSize]
n, readErr := unix.Pread(fd, file, 0)
if readErr != nil {
panic(readErr)
}
fmt.Println(a, offset, offset+utils.BlockSize, len(file))
fmt.Println("Content is: ", string(file))
}
I understand that I'm generating a slice twice as big than what I need, and then extracting a memory aligned block from it, but the Alignment function doesn't make sense to me.
How does the Alignment function works?
If I try to fmt.Println the intermediate steps of that function I get different results, why? I guess because observing it changes its memory alignment (like in quantum physics :D)
Edit:
Example with fmt.println, where I don't need any more alignment:
package main
import (
"fmt"
"golang.org/x/sys/unix"
"unsafe"
)
func main() {
path := "/path/to/file.txt"
fd, err := unix.Open(path, unix.O_RDONLY|unix.O_DIRECT, 0666)
defer unix.Close(fd)
if err != nil {
panic(err)
}
file := make([]byte, 4096)
fmt.Println("Pointer: ", &file[0])
n, readErr := unix.Pread(fd, file, 0)
fmt.Println("Return is: ", n)
if readErr != nil {
panic(readErr)
}
fmt.Println("Content is: ", string(file))
}
Your AlignSize has a value of a power of 2. In binary representation it contains a 1 bit followed by full of zeros:
fmt.Printf("%b", AlignSize) // 1000000000000
A slice allocated by make() may have a memory address that is more or less random, consisting of ones and zeros following randomly in binary; or more precisely the starting address of its backing array.
Since you allocate twice the required size, that's a guarantee that the backing array will cover an address space that has an address in the middle somewhere that ends with as many zeros as the AlignSize's binary representation, and has BlockSize room in the array starting at this. We want to find this address.
This is what the Alignment() function does. It gets the starting address of the backing array with &block[0]. In Go there's no pointer arithmetic, so in order to do something like that, we have to convert the pointer to an integer (there is integer arithmetic of course). In order to do that, we have to convert the pointer to unsafe.Pointer: all pointers are convertible to this type, and unsafe.Pointer can be converted to uintptr (which is an unsigned integer large enough to store the uninterpreted bits of a pointer value), on which–being an integer–we can perform integer arithmetic.
We use bitwise AND with the value uintptr(AlignSize-1). Since AlignSize is a power of 2 (contains a single 1 bit followed by zeros), the number one less is a number whose binary representation is full of ones, as many as trailing zeros AlignSize has. See this example:
x := 0b1010101110101010101
fmt.Printf("AlignSize : %22b\n", AlignSize)
fmt.Printf("AlignSize-1 : %22b\n", AlignSize-1)
fmt.Printf("x : %22b\n", x)
fmt.Printf("result of & : %22b\n", x&(AlignSize-1))
Output:
AlignSize : 1000000000000
AlignSize-1 : 111111111111
x : 1010101110101010101
result of & : 110101010101
So the result of & is the offset which if you subtract from AlignSize, you get an address that has as many trailing zeros as AlignSize itself: the result is "aligned" to the multiple of AlignSize.
So we will use the part of the file slice starting at offset, and we only need BlockSize:
file = file[offset : offset+BlockSize]
Edit:
Looking at your modified code trying to print the steps: I get an output like:
Pointer: 0xc0000b6000
Unsafe pointer: 0xc0000b6000
Unsafe pointer, uintptr: 824634466304
Unpersand: 0
Cast to int: 0
Return is: 0
Content is:
Note nothing is changed here. Simply the fmt package prints pointer values using hexadecimal representation, prefixed by 0x. uintptr values are printed as integers, using decimal representation. Those values are equal:
fmt.Println(0xc0000b6000, 824634466304) // output: 824634466304 824634466304
Also note the rest is 0 because in my case 0xc0000b6000 is already a multiple of 4096, in binary it is 1100000000000000000100001110000000000000.
Edit #2:
When you use fmt.Println() to debug parts of the calculation, that may change escape analysis and may change the allocation of the slice (from stack to heap). This depends on the used Go version too. Do not rely on your slice being allocated at an address that is (already) aligned to AlignSize.
See related questions for more details:
Mix print and fmt.Println and stack growing
why struct arrays comparing has different result
Addresses of slices of empty structs
Assume I have the following structure
type Hdr struct{
Src uint16
Dst uint16
Priotity byte
Pktcnt byte
Opcode byte
Ver byte
}
I have two functions Marshal and Unmarshal that encode Hdr to and from a binary format of:
0 1
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Src |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Dst |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Prio | Cnt | Opcode| Ver |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
I'd like to use Go Fuzz to make random, valid Hdr instances, Marshal then to binary, Unmarshal the binary and make sure the output matches the original input.
The main issue I am having is that I cannot figure out how to tell Go Fuzz that fields like Priotity cannot be greater than 15 otherwise they will get truncated when they are marshalled (only 4 bits). How do I set this constraint?
Update
This is just a toy case. There are many times with protocols like the above where something like the opcode would trigger secondary more complex parsing/vetting. Fuzzing could still find very useful issues within a constraint (IE: if Prio 0x00 and Cnt 0x2F secondary parser will error because delimiter is \ ).
EDIT
I'm not sure Fuzzing is a good fit here. Fuzzing is designed to find unexpected inputs: multi-byte UTF8 inputs (valid and non-valid); negative values; huge values, long lengths etc. These will try to catch "edge" cases.
In your case here, you know the:
Unmarshal input payload must be 6 bytes (should error otherwise)
you know precisely your internal "edges"
so vanilla testing.T tests may be a better fit here.
Keep it simple.
If you don't want to "waste" a Fuzz input & you know the input constraints of your code, you can try something like this:
func coerce(h *Hdr) (skip bool) {
h.Priotity &= 0x0f // ensure priority is 0-15
h.OpCode %= 20 // ensure opcode is 0-19
return false // optionally skip this test
}
and in your test - the coerced value can be tested - or skipped (as #jch showed):
import "github.com/google/go-cmp/cmp"
f.Fuzz(func(t *testing.T, src, dst uint16, pri, count, op, ver byte) {
h := Hdr{src, dst, pri, count, op, ver}
if coerce(&h) {
t.Skip()
return
}
bs, err := Marshal(h) // check err
h2, err := Unmarhsal(bs) // check err
if !cmp.Equal(h, h2) {
t.Errorf("Marshal/Unmarshal validation failed for: %+v", h)
}
}
In order to skip uninteresting results, call t.Skip in your fuzzing function. Something like this:
f.Fuzz(func(t *testing.T, b []byte) {
a, err := Unmarshal(b)
if err != nil {
t.Skip()
return
}
c, err := Marshal(a)
if err != nil || !bytes.Equal(b, c) {
t.Errorf("Eek!")
}
})
I've been trying to make a syscall to FreeBSD's jail_set without success.
It uses an array of iovec when receiving the syscall, but returns me always a "Bad Address".
Here's the part of the code that creates the IOVEC:
func (p Params) buildIovec() ([]unix.Iovec, error) {
iovSize := len(p) * 2
iovec := make([]unix.Iovec, iovSize)
var itr int
for paramKey, paramValue := range p {
arrayParamKey := []byte(paramKey)
// Adds a nullbyte
// Jail parameters are passed as an array
// of name-value pairs in the array iov, containing niov
// elements. Parameter names are a null-terminated
// string, and values may be strings, integers,
// or other arbitrary data.
// Ref: https://www.freebsd.org/cgi/man.cgi?query=jail&sektion=3&manpath=FreeBSD+11.1-stable
arrayParamKey = append(arrayParamKey, 0)
size := len(arrayParamKey)
iovec[itr] = unix.Iovec{
Base: (*byte)(unsafe.Pointer(&arrayParamKey)),
Len: uint64(size),
}
itr++
rv := reflect.ValueOf(paramValue)
var arrayValueBytes []byte
switch rv.Kind() {
case reflect.String:
arrayValueBytes = []byte(rv.String())
arrayValueBytes = append(arrayValueBytes, 0)
if err != nil {
return nil, err
}
default:
return nil, errors.New("invalid value passed in for key: " + paramKey)
}
arrayValueSize := len(arrayValueBytes)
iovec[itr] = unix.Iovec{
Base: (*byte)(unsafe.Pointer(&arrayValueBytes)),
Len: uint64(arrayValueSize),
}
itr++
}
return iovec, nil
}
And then the syscall is called with:
_, _, e1 := unix.Syscall(uintptr(call), uintptr(unsafe.Pointer(&iov)), uintptr(len(iov)), flags)
Where call represents the jail_set int 507, and flags is the uintptr of 1 (represents the CREATE).
Also, Params is a map[string]interface{} where the key is the key of the jail config, and the value is the value of the config (this explains the loop in the buildIovec).
This is based in https://github.com/briandowns/jail but I've been changing some things to see if it works.
Thanks.
Your value is passing the address of the slice header for the slice of bytes, not the address of the start of the actual bytes.
[]byte(rv.String()) will create a small slice header.
unsafe.Pointer(C.CString(rv.String())) will probably get you closer to working code.
The documentation in the registry package for GetValue() says :
GetValue retrieves the type and data for the specified value associated with an open key k. It fills up buffer buf and returns the retrieved byte count n. If buf is too small to fit the stored value it returns ErrShortBuffer error along with the required buffer size n. If no buffer is provided, it returns true and actual buffer size n. If no buffer is provided, GetValue returns the value's type only. If the value does not exist, the error returned is ErrNotExist.
GetValue is a low level function. If value's type is known, use the appropriate Get*Value function instead."
In my case, I don't know the value type of the registry key. However, I only need to print the value as a string. GetValue() takes in the value name and a "buffer" but the buffer is of type []byte. It is not passed by reference so I can't just create var buf []byte, pass that in and read it. I can't pass it in with &buf (type *[]byte). I can't use byte.Buffer (also type mismatch). I feel like there is something really simple I'm missing.
Code:
var buf []byte //????
_, _, e := myKey.GetValue(valuename, buf)
if e != nil {
panic(e)
}
fmt.Printf("Value: %s\n", string(buf)) // Prints blank
I suppose the registry API you mention is the Windows registry. To use these kinds of APIs, you have to take your best guess on the size of output you expect from the call:
buf:=make([]byte,1024)
typ, n, e := myKey.GetValue(valuename, buf)
if e==ErrShortBuffer {
// Go back, try with a larger buffer size
buf=make([]byte,n)
typ, n, e = myKey.GetValue(valuename, buf)
}
I'm writing a simple program that takes in input from a form, populates an instance of a struct with the received data and the writes this received data to a file.
I'm a bit stuck at the moment with figuring out the best way to iterate over the populated struct and write its contents to the file.
The struct in question contains 3 different types of fields (ints, strings, []strings).
I can iterate over them but I am unable to get their actual type.
Inspecting my posted code below with print statements reveals that each of their types is coming back as structs rather than the aforementioned string, int etc.
The desired output format is be plain text.
For example:
field_1="value_1"
field_2=10
field_3=["a", "b", "c"]
Anyone have any ideas? Perhaps I'm going about this the wrong way entirely?
func (c *Config) writeConfigToFile(file *os.File) {
listVal := reflect.ValueOf(c)
element := listVal.Elem()
for i := 0; i < element.NumField(); i++ {
field := element.Field(i)
myType := reflect.TypeOf(field)
if myType.Kind() == reflect.Int {
file.Write(field.Bytes())
} else {
file.WriteString(field.String())
}
}
}
Instead of using the Bytes method on reflect.Value which does not work as you initially intended, you can use either the strconv package or the fmt to format you fields.
Here's an example using fmt:
var s string
switch fi.Kind() {
case reflect.String:
s = fmt.Sprintf("%q", fi.String())
case reflect.Int:
s = fmt.Sprintf("%d", fi.Int())
case reflect.Slice:
if fi.Type().Elem().Kind() != reflect.String {
continue
}
s = "["
for j := 0; j < fi.Len(); j++ {
s = fmt.Sprintf("%s%q, ", s, fi.Index(i).String())
}
s = strings.TrimRight(s, ", ") + "]"
default:
continue
}
sf := rv.Type().Field(i)
if _, err := fmt.Fprintf(file, "%s=%s\n", sf.Name, s); err!= nil {
panic(err)
}
Playground: https://play.golang.org/p/KQF3CicVzA
Why not use the built-in gob package to store your struct values?
I use it to store different structures, one per line, in files. During decoding, you can test the type conversion or provide a hint in a wrapper - whichever is faster for your given use case.
You'd treat each line as a buffer when Encoding and Decoding when reading back the line. You can even gzip/zlib/compress, encrypt/decrypt, etc the stream in real-time.
No point in re-inventing the wheel when you have a polished and armorall'd wheel already at your disposal.