Want to have an empty char/byte, which has zero size/length, in Go, such as byte("").
func main() {
var a byte = '' // not working
var a byte = 0 // not working
}
A more specific example is
func removeOuterParentheses(S string) string {
var stack []int
var res []byte
for i, b := range []byte(S) {
if b == '(' {
stack = append(stack, i)
} else {
if len(stack) == 1 {
res[stack[0]] = '' // set this byte to be empty
res[i] = '' // / set this byte to be empty
}
stack = stack[:len(stack)-1]
}
}
return string(res)
}
There is an equivalent question in Java
A byte is an alias to the uint8 type. Having an "empty byte" doesn't really make any sense, just as having an "empty number" doesn't make any sense (you can have the number 0, but what is an "empty" number?)
You can assign a value of zero (b := byte(0), or var b byte), which can be used to indicate that nothing is assigned yet ("zero value"). The byte value of 0 is is known as a "null byte". It normally never occurs in regular text, but often occurs in binary data (e.g. images, compressed files, etc.)
This is different from byte(""), which is a sequence of bytes. You can have a sequence of zero bytes. To give an analogy: I can have a wallet with no money in it, but I can't have a coin that is worth "empty".
If you really want to distinguish between "value of 0" and "never set" you can use either a pointer or a struct. An example with a pointer:
var b *byte
fmt.Println(b) // <nil>, since it's a pointer which has no address to point to.
one := byte(0)
b = &one // Set address to one.
fmt.Println(b, *b) // 0xc000014178 0 (first value will be different, as it's
// a memory address).
You'll need to be a little bit careful here, as *b will be a panic if you haven't assigned a value yet. Depending on how it's used it can either work quite well, or be very awkward to work with. An example where this is used in the standard library is the flag package.
Another possibility is to use a struct with separate fiels for the byte itself and a flag to record whether it's been set or not. In the database/sql library there are already the Null* types (e.g. NullInt64, which you can use as a starting point.
a single byte is a number. 0 would transform into a 8bit number. 00000000.
A byte slice/array can have a length of 0.
var a byte = 0
var b = [0]byte{}
Related
I have a struct with a int16 in it represent the offset of indexes in a slice. There are 2 states for this value: 1. it can be that the offset is invalid; 2. it's meaningful and will actually represent a positive, negative, or zero value offset. Now the question is that I will allocate an array of this struct, and the default value for this int16 should be the invalid state, but the default value is 0, and 0 can also represent a valid offset with value of 0.
Currently, I use -32768 as the value to represent the invalid state, and manually initialize the array after using make. However, I don't like this approach and it doesn't follow the Go way of making the zero value meaningful. Is there a better way to approach this? I don't really want to introduce extra variables as that will take more space in memory.
When accessing the offset value, you can remap the zero value to the invalid constant through addition and subtraction. This will require obscuring the type through a Get and Set, since accessing the internal value directly will no longer be valid.
package main
import "fmt"
type Offset struct {
// This value is not safe to use directly,
// since it has to be mapped to/from a real offset
mappedOffset int16
}
const InvalidOffset = -32768
func (o Offset) Get() int16 {
// Shift values so 0 becomes InvalidOffset
return o.mappedOffset + InvalidOffset
}
func (o *Offset) Set(i int16) {
// Shift values so InvalidOffset becomes 0
o.mappedOffset = i - InvalidOffset
}
func (o Offset) IsValid() bool {
// Check if it is the actual "Go Zero"
return o != Offset{}
}
func main() {
var offset Offset
fmt.Println("Apparent value:", offset.Get(), " Internal:", offset, " IsValid:", offset.IsValid())
offset.Set(123)
fmt.Println("Apparent value:", offset.Get(), " Internal:", offset, " IsValid:", offset.IsValid())
offset.Set(InvalidOffset)
fmt.Println("Apparent value:", offset.Get(), " Internal:", offset, " IsValid:", offset.IsValid())
}
Output:
Apparent value: -32768 Internal: {0} IsValid: false
Apparent value: 123 Internal: {-32645} IsValid: true
Apparent value: -32768 Internal: {0} IsValid: false
Found a solution. I can store it using an uint16 and offset it with math.minint16. This way the default value 0 is -32768 and is treated as the invalid state, all other numbers are valid.
I'm trying to do direct i/o on linux, so I need to create memory aligned buffers. I copied some code to do it, but I don't understand how it works:
package main
import (
"fmt"
"golang.org/x/sys/unix"
"unsafe"
"yottaStore/yottaStore-go/src/yfs/test/utils"
)
const (
AlignSize = 4096
BlockSize = 4096
)
// Looks like dark magic
func Alignment(block []byte, AlignSize int) int {
return int(uintptr(unsafe.Pointer(&block[0])) & uintptr(AlignSize-1))
}
func main() {
path := "/path/to/file.txt"
fd, err := unix.Open(path, unix.O_RDONLY|unix.O_DIRECT, 0666)
defer unix.Close(fd)
if err != nil {
panic(err)
}
file := make([]byte, 4096*2)
a := Alignment(file, AlignSize)
offset := 0
if a != 0 {
offset = AlignSize - a
}
file = file[offset : offset+BlockSize]
n, readErr := unix.Pread(fd, file, 0)
if readErr != nil {
panic(readErr)
}
fmt.Println(a, offset, offset+utils.BlockSize, len(file))
fmt.Println("Content is: ", string(file))
}
I understand that I'm generating a slice twice as big than what I need, and then extracting a memory aligned block from it, but the Alignment function doesn't make sense to me.
How does the Alignment function works?
If I try to fmt.Println the intermediate steps of that function I get different results, why? I guess because observing it changes its memory alignment (like in quantum physics :D)
Edit:
Example with fmt.println, where I don't need any more alignment:
package main
import (
"fmt"
"golang.org/x/sys/unix"
"unsafe"
)
func main() {
path := "/path/to/file.txt"
fd, err := unix.Open(path, unix.O_RDONLY|unix.O_DIRECT, 0666)
defer unix.Close(fd)
if err != nil {
panic(err)
}
file := make([]byte, 4096)
fmt.Println("Pointer: ", &file[0])
n, readErr := unix.Pread(fd, file, 0)
fmt.Println("Return is: ", n)
if readErr != nil {
panic(readErr)
}
fmt.Println("Content is: ", string(file))
}
Your AlignSize has a value of a power of 2. In binary representation it contains a 1 bit followed by full of zeros:
fmt.Printf("%b", AlignSize) // 1000000000000
A slice allocated by make() may have a memory address that is more or less random, consisting of ones and zeros following randomly in binary; or more precisely the starting address of its backing array.
Since you allocate twice the required size, that's a guarantee that the backing array will cover an address space that has an address in the middle somewhere that ends with as many zeros as the AlignSize's binary representation, and has BlockSize room in the array starting at this. We want to find this address.
This is what the Alignment() function does. It gets the starting address of the backing array with &block[0]. In Go there's no pointer arithmetic, so in order to do something like that, we have to convert the pointer to an integer (there is integer arithmetic of course). In order to do that, we have to convert the pointer to unsafe.Pointer: all pointers are convertible to this type, and unsafe.Pointer can be converted to uintptr (which is an unsigned integer large enough to store the uninterpreted bits of a pointer value), on which–being an integer–we can perform integer arithmetic.
We use bitwise AND with the value uintptr(AlignSize-1). Since AlignSize is a power of 2 (contains a single 1 bit followed by zeros), the number one less is a number whose binary representation is full of ones, as many as trailing zeros AlignSize has. See this example:
x := 0b1010101110101010101
fmt.Printf("AlignSize : %22b\n", AlignSize)
fmt.Printf("AlignSize-1 : %22b\n", AlignSize-1)
fmt.Printf("x : %22b\n", x)
fmt.Printf("result of & : %22b\n", x&(AlignSize-1))
Output:
AlignSize : 1000000000000
AlignSize-1 : 111111111111
x : 1010101110101010101
result of & : 110101010101
So the result of & is the offset which if you subtract from AlignSize, you get an address that has as many trailing zeros as AlignSize itself: the result is "aligned" to the multiple of AlignSize.
So we will use the part of the file slice starting at offset, and we only need BlockSize:
file = file[offset : offset+BlockSize]
Edit:
Looking at your modified code trying to print the steps: I get an output like:
Pointer: 0xc0000b6000
Unsafe pointer: 0xc0000b6000
Unsafe pointer, uintptr: 824634466304
Unpersand: 0
Cast to int: 0
Return is: 0
Content is:
Note nothing is changed here. Simply the fmt package prints pointer values using hexadecimal representation, prefixed by 0x. uintptr values are printed as integers, using decimal representation. Those values are equal:
fmt.Println(0xc0000b6000, 824634466304) // output: 824634466304 824634466304
Also note the rest is 0 because in my case 0xc0000b6000 is already a multiple of 4096, in binary it is 1100000000000000000100001110000000000000.
Edit #2:
When you use fmt.Println() to debug parts of the calculation, that may change escape analysis and may change the allocation of the slice (from stack to heap). This depends on the used Go version too. Do not rely on your slice being allocated at an address that is (already) aligned to AlignSize.
See related questions for more details:
Mix print and fmt.Println and stack growing
why struct arrays comparing has different result
Addresses of slices of empty structs
The structure is like:
type Auth_msg struct {
Msg_class [2]byte
Msg_content_pty [2]byte
I am fresh to use Reflect in Go and I encounter this:
panic: reflect: call of reflect.Value.Bytes on array Value
This occurs when I run val.Field(i).Bytes(), however the when I try to print it: fmt.PrintLn(val.Field(i)), it prints out the right arrays.
I just wonder, how I can retrieve the Msg_class in an array or a slice?
In Go, there is a distinction between an array and a slice. Value.Bytes() explicitly works only for a byte slice (link to docs).
note : I don't know why it doesn't handle byte arrays ; it probably was written that way, and it makes the implementation of reflect.Bytes() simpler. Anyway : slices are definitely the common use case in Go, and it is easy to convert an array to a slice :
You can create a slice pointing to the array using [:] :
v := reflect.ValueOf(msg.Msg_class)
fmt.Println("kind :", v.Kind()) // prints 'array'
// fmt.Printf("bytes : % x\n", v.Bytes()) // panics
v = reflect.ValueOf(msg.Msg_class[:])
fmt.Println("kind :", v.Kind()) // prints 'slice'
fmt.Printf("bytes : % x\n", v.Bytes()) // works
https://play.golang.org/p/sKcGaru4rOq
To turn an array into a slice using reflect, you can call .Slice() on a reflect.Value.
One constraint, mentioned in the doc, is that the array value must be addressable.
I haven't got all the details sorted out, but one way to make sure the reflect Value is addressable is to call reflect.ValueOf() on a pointer, and then call .Elem() on that pointer value :
var arr [2]byte
arr[0] = 'g'
arr[1] = 'o'
// take ValueOf a *pointer* to your array, and let reflect dereference it :
v := reflect.ValueOf(&arr).Elem()
// this sets the "canAddr" flag on this value
fmt.Println("arr value - CanAddr() :", v.CanAddr()) // prints 'true'
slice := v.Slice(0, v.Len())
fmt.Printf("arr bytes : % x\n", slice.Bytes()) // prints '67 6f'
// for a field inside a struct : take a pointer to the struct
var msg Auth_msg
msg.Msg_class[0] = 'a'
msg.Msg_class[1] = 'z'
v = reflect.ValueOf(&msg).Elem()
fmt.Println("msg value - CanAddr() :", v.CanAddr()) // prints 'true'
// now reflect accepts to call ".Slice()" on one of its fields :
field := v.FieldByName("Msg_class")
slice = field.Slice(0, field.Len())
fmt.Printf("msg.Msg_class bytes : % x\n", slice.Bytes()) // prints '61 7a'
https://play.golang.org/p/SqM7yxl2D96
I am doing the http://tour.golang.org/. Could anyone explain this function to me lines 1,3,5 and 7, especially what '*' and '&' do? By mentioning them in a function declaration, what are they supposed/expected to do? A toy example:
1: func intial1(var1 int, var2 int, func1.newfunc[]) *callproperfunction {
2:
3: addition:= make ([] add1, var1)
4: for i:=1;i<var2;i++ {
5: var2 [i] = *addtother (randomstring(lengthofcurrent))
6: }
7: return &callproperfunction {var1 int, var2 int, func1.newfunc[], jackpot}
8: }
It seems that they are pointers like what we have in C++. But I cannot connect those concepts to what we have here. In other words, what '*' an '&' do when I use them in function declaration in Go.
I know what reference and dereference mean. I cannot understand how we can use a pointer to a function in Go? For example lines 1 and 7, what do these two lines do? The function named intial1 is declared that returns a pointer? And in line 7, we call it with arguments using the return function.
This is possibly one of the most confusing things in Go. There are basically 3 cases you need to understand:
The & Operator
& goes in front of a variable when you want to get that variable's memory address.
The * Operator
* goes in front of a variable that holds a memory address and resolves it (it is therefore the counterpart to the & operator). It goes and gets the thing that the pointer was pointing at, e.g. *myString.
myString := "Hi"
fmt.Println(*&myString) // prints "Hi"
or more usefully, something like
myStructPointer = &myStruct
// ...
(*myStructPointer).someAttribute = "New Value"
* in front of a Type
When * is put in front of a type, e.g. *string, it becomes part of the type declaration, so you can say "this variable holds a pointer to a string". For example:
var str_pointer *string
So the confusing thing is that the * really gets used for 2 separate (albeit related) things. The star can be an operator or part of a type.
Your question doesn't match very well the example given but I'll try to be straightforward.
Let's suppose we have a variable named a which holds the integer 5 and another variable named p which is going to be a pointer.
This is where the * and & come into the game.
Printing variables with them can generate different output, so it all depends on the situation and how well you use.
The use of * and & can save you lines of code (that doesn't really matter in small projects) and make your code more beautiful/readable.
& returns the memory address of the following variable.
* returns the value of the following variable (which should hold the memory address of a variable, unless you want to get weird output and possibly problems because you're accessing your computer's RAM)
var a = 5
var p = &a // p holds variable a's memory address
fmt.Printf("Address of var a: %p\n", p)
fmt.Printf("Value of var a: %v\n", *p)
// Let's change a value (using the initial variable or the pointer)
*p = 3 // using pointer
a = 3 // using initial var
fmt.Printf("Address of var a: %p\n", p)
fmt.Printf("Value of var a: %v\n", *p)
All in all, when using * and & in remember that * is for setting the value of the variable you're pointing to and & is the address of the variable you're pointing to/want to point to.
Hope this answer helps.
Those are pointers like we have in C++.
The differences are:
Instead of -> to call a method on a pointer, you always use ., i.e. pointer.method().
There are no dangling pointers. It is perfectly valid to return a pointer to a local variable. Golang will ensure the lifetime of the object and garbage-collect it when it's no longer needed.
Pointers can be created with new() or by creating a object object{} and taking the address of it with &.
Golang does not allow pointer-arithmetic (arrays do not decay to pointers) and insecure casting. All downcasts will be checked using the runtime-type of the variable and either panic or return false as second return-value when the instance is of the wrong type, depending on whether you actually take the second return type or not.
This is by far the easiest way to understand all the three cases as explained in the #Everett answer
func zero(x int) {
x = 0
}
func main() {
x := 5
zero(x)
fmt.Println(x) // x is still 5
}
If you need a variable to be changed inside a function then pass the memory address as a parmeter and use the pointer of this memory address to change the variable permanently.
Observe the use of * in front of int in the example. Here it just represents the variable that is passed as a parameter is the address of type int.
func zero(xPtr *int) {
*xPtr = 0
}
func main() {
x := 5
zero(&x)
fmt.Println(x) // x is 0
}
Simple explanation.. its just like, you want to mutate the original value
func zero(num *int){ // add * to datatype
*num = 0 // can mutate the original number
}
i := 5
zero(&i) // passing variable with & will allows other function to mutate the current value of variable```
& Operator gets the memory address where as * Opeartor holds the memory address of particular variable.
I'm trying to manipulate processes on Windows using Go language,
and I'm starting off by reading other process' memory by using ReadProcessMemory.
However, for most of the addresses I get Error: Only part of a ReadProcessMemory or WriteProcessMemory request was completed. error. Maybe my list of arguments is wrong, but I can't find out why.
Can anyone point out what I am doing wrong here?
package main
import (
"fmt"
)
import (
windows "golang.org/x/sys/windows"
)
func main() {
handle, _ := windows.OpenProcess(0x0010, false, 6100) // 0x0010 PROCESS_VM_READ, PID 6100
procReadProcessMemory := windows.MustLoadDLL("kernel32.dll").MustFindProc("ReadProcessMemory")
var data uint = 0
var length uint = 0
for i := 0; i < 0xffffffff; i += 2 {
fmt.Printf("0x%x\n", i)
// BOOL ReadProcessMemory(HANDLE hProcess, LPCVOID lpBaseAddress, LPVOID lpBuffer, DWORD nSize, LPDWORD lpNumberOfBytesRead)
ret, _, e := procReadProcessMemory.Call(uintptr(handle), uintptr(i), uintptr(data), 2, uintptr(length)) // read 2 bytes
if (ret == 0) {
fmt.Println(" Error:", e)
} else {
fmt.Println(" Length:", length)
fmt.Println(" Data:", data)
}
}
windows.CloseHandle(handle)
}
uintptr(data) is incorrect: it takes the value from data (0 of type uint) and converts that to unitptr type — yielding the same value converted to another type — producing, on x86, a null pointer.
Note that Go is not C, and you can't really play dirty games with pointers in it, or, rather, you can, but only through using the unsafe built-in package and its Pointer type which is like void* (pointing somewhere in a data memory block) in C.
What you need is something like
import "unsafe"
var (
data [2]byte
length uint32
)
ret, _, e := procReadProcessMemory.Call(uintptr(handle), uintptr(i),
uintptr(unsafe.Pointer(&data[0])),
2, uintptr(unsafe.Pointer(&length))) // read 2 bytes
Observe what was done here:
A variable of type "array of two bytes" is declared;
The address of the first element of this array is taken;
That address is type-converted to the type unsafe.Pointer;
The obtained value is then type-converted to uintptr.
The last two steps are needed because Go features garbage collection:
In Go, when you take an address of a value in memory and store it in a variable, the GC knows about this "implicit" pointer and the value which address was taken won't be garbage-collected even if it becomes unreachable with that value holding its address being the only reference left.
Even if you make that address value lose the type information it maintains — through type-converting it to unsafe.Pointer, the new value is still considered by GC and behaves like "normal" values containing addresses — as explained above.
By type-converting such value to uintptr you make GC stop considering it as a pointer. Hence this type is there only for FFI/interop.
In other words, in
var data [2]byte
a := &data[0]
p := unsafe.Pointer(a)
i := uintptr(p)
there are only three references to the value in data: that variable itself, a and p, but not i.
You should consider these rules when dealing with calling outside code because you should never ever pass around unitptr-typed values: they're only for marshaling data to the called functions and unmarshaling it back, and have to be used "on the spot" — in the same scope as the values they are type-converted from/to.
Also observe that in Go, you can't just take the address of a variable of an integer type and supply that address to a function which expects a pointer to a memory block of an appropriate size. You have to deal with byte arrays and after the data has been written by the called function, you need to explicitly convert it to a value of the type you need. That's why there's no "type casts" in Go but only "type conversions": you can't reinterpret the data type of a value through type-conversion, with the uintptr(unsafe.Pointer) (and back) being a notable exception for the purpose of FFI/interop, and even in this case you basically convert a pointer to a pointer, just transfer it through the GC boundary.
To "serialize" and "deserialize" a value of an integer type you might use the encoding/binary standard package or hand-roll no-brainer simple functions which do bitwise shifts and or-s and so on ;-)
2015-10-05, updated as per the suggestion of James Henstridge.
Note that after the function returns, and ret signalizes there's no error
you have to check the value of the length variable.