I'm quite new to Go, so this might be obvious. The compiler does not allow the following code:
(http://play.golang.org/p/3sTLguUG3l)
package main
import "fmt"
type Card string
type Hand []Card
func NewHand(cards []Card) Hand {
hand := Hand(cards)
return hand
}
func main() {
value := []string{"a", "b", "c"}
firstHand := NewHand(value)
fmt.Println(firstHand)
}
The error is:
/tmp/sandbox089372356/main.go:15: cannot use value (type []string) as type []Card in argument to NewHand
From the specs, it looks like []string is not the same underlying type as []Card, so the type conversion cannot occur.
Is it, indeed, the case, or did I miss something?
If it is the case, why is it so? Assuming, in a non-pet-example program, I have as input a slice of string, is there any way to "cast" it into a slice of Card, or do I have to create a new structure and copy the data into it? (Which I'd like to avoid since the functions I'll need to call will modify the slice content).
There is no technical reason why conversion between slices whose elements have identical underlying types (such as []string and []Card) is forbidden. It was a specification decision to help avoid accidental conversions between unrelated types that by chance have the same structure.
The safe solution is to copy the slice. However, it is possible to convert directly (without copying) using the unsafe package:
value := []string{"a", "b", "c"}
// convert &value (type *[]string) to *[]Card via unsafe.Pointer, then deref
cards := *(*[]Card)(unsafe.Pointer(&value))
firstHand := NewHand(cards)
https://play.golang.org/p/tto57DERjYa
Obligatory warning from the package documentation:
unsafe.Pointer allows a program to defeat the type system and read and write arbitrary memory. It should be used with extreme care.
There was a discussion on the mailing list about conversions and underlying types in 2011, and a proposal to allow conversion between recursively equivalent types in 2016 which was declined "until there is a more compelling reason".
The underlying type of Card might be the same as the underlying type of string (which is itself: string), but the underlying type of []Card is not the same as the underlying type of []string (and therefore the same applies to Hand).
You cannot convert a slice of T1 to a slice of T2, it's not a matter of what underlying types they have, if T1 is not identical to T2, you just can't. Why? Because slices of different element types may have different memory layout (different size in memory). For example the elements of type []byte occupy 1 byte each. The elements of []int32 occupy 4 bytes each. Obviously you can't just convert one to the other even if all values are in the range 0..255.
But back to the roots: if you need a slice of Cards, why do you create a slice of strings in the first place? You created the type Card because it is not a string (or at least not just a string). If so and you require []Card, then create []Card in the first place and all your problems go away:
value := []Card{"a", "b", "c"}
firstHand := NewHand(value)
fmt.Println(firstHand)
Note that you are still able to initialize the slice of Card with untyped constant string literals because it can be used to initialize any type whose underlying type is string. If you want to involve typed string constants or non-constant expressions of type string, you need explicit conversion, like in the example below:
s := "ddd"
value := []Card{"a", "b", "c", Card(s)}
If you have a []string, you need to manually build a []Card from it. There is no "easier" way. You can create a helper toCards() function so you can use it everywhere you need it.
func toCards(s []string) []Card {
c := make([]Card, len(s))
for i, v := range s {
c[i] = Card(v)
}
return c
}
Some links for background and reasoning:
Go Language Specification: Conversions
why []string can not be converted to []interface{} in golang
Cannot convert []string to []interface {}
What about memory layout means that []T cannot be converted to []interface in Go?
From the specs, it looks like []string is not the same underlying type as []Card, so the type conversion cannot occur.
Exactly right. You have to convert it by looping and copying over each element, converting the type from string to Card on the way.
If it is the case, why is it so? Assuming, in a non-pet-example program, I have as input a slice of string, is there any way to "cast" it into a slice of Card, or do I have to create a new structure and copy the data into it? (Which I'd like to avoid since the functions I'll need to call will modify the slice content).
Because conversions are always explicit and the designers felt that when a conversion implicitly involves a copy it should be made explicit as well.
Related
The following code generates panic: binary.Write: invalid type main.test:
type (
config struct {
Key uint16
Val uint16
}
test struct {
Mode uint32
Data []config
}
)
func main() {
t := test{
Mode: 5,
Data: []config{
{1, 2},
{3, 4},
},
}
var bs bytes.Buffer
assert(binary.Write(&bs, binary.LittleEndian, t))
}
The key point is:
length of the config data structure is fixed, but the test structure contains a slice of config, whose number varies.
I need to interact with other program written in C, so cannot use things like GOB.
Is there anyway to binary-encode such data structure, apart from do it manually?
The problem is not writing out a slice, as slices are supported. Quoting from binary.Write():
Write writes the binary representation of data into w. Data must be a fixed-size value or a slice of fixed-size values, or a pointer to such data.
The problem is that the size of config is not fixed. It is not fixed because it contains a field of slice type, and the binary representation of the slice is not fixed (depends on its length).
So writing a slice value is supported, writing a value of composite type (e.g. struct) holding a slice is not supported for the above mentioned reason.
You may change the field to an array type (e.g. [2]config) but I assume this isn't sufficient to you.
You may write the fields using encoding/binary individually, in which case you can write a slice value.
For example:
var bs bytes.Buffer
fmt.Println(binary.Write(&bs, binary.LittleEndian, t.Mode))
fmt.Println(binary.Write(&bs, binary.LittleEndian, t.Data))
This will output (try it on the Go Playground):
<nil>
<nil>
There was a proposal to extend encoding/binary to support similar cases (see here), but was rejected. encoding/binary is for simple things.
If you need more flexibility, use encoding/gob (although Go specific), or use encoding/json (supported by all languages).
Consider the following code in Go
type A struct {
f int
}
type B struct {
f int `somepkg:"somevalue"`
}
func f() {
var b *B = (*B)(&A{1}) // <-- THIS
fmt.Printf("%#v\n", b)
}
Will the marked line result in a memory copy (which I would like to avoid as A has many fields attached to it) or will it be just a reinterpretation, similar to casting an int to an uint?
EDIT: I was concerned, whether the whole struct would have to be copied, similarly to converting a byte slice to a string. A pointer copy is therefore a no-op for me
It is called a conversion. The expression (&A{}) creates a pointer to an instance of type A, and (*B) converts that pointer to a *B. What's copied there is the pointer, not the struct. You can validate this using the following code:
a:=A{}
var b *B = (*B)(&a)
b.f=2
fmt.Printf("%#v\n", a)
Prints 2.
The crucial points to understand is that
First, unlike C, C++ and some other languages of their ilk, Go does not have type casting, it has type conversions.
In most, but not all, cases, type conversion changes the type but not the internal representation of a value.
Second, as to whether a type conversion "is a no-op", depends on how you define the fact of being a no-op.
If you are concerned with a memory copy being made, there are two cases:
Some type conversions are defined to drastically change the value's representation or to copy memory; for example:
Type-converting a value of type string to []rune would interpret the value as a UTF-8-encoded byte stream, decode each encoded Unicode code point and produce a freshly-allocated slice of decoded Unicode runes.
Type-converting a value of type string to []byte, and vice-versa, will clone the backing array underlying the value.
Other type-conversions are no-op in this sense but in order for them to be useful you'd need to either assign a type-converted value to some variable or to pass it as an argument to a function call or send to a channel etc — in other words, you have to store the result or otherwise make use of it.
All of such operations do copy the value, even though it does not "look" like this; consider:
package main
import (
"fmt"
)
type A struct {
X int
}
type B struct {
X int
}
func (b B) Whatever() {
fmt.Println(b.X)
}
func main() {
a := A{X: 42}
B(a).Whatever()
b := B(a)
b.Whatever()
}
Here, the first type conversion in main does not look like a memory copy, but the resulting value will serve as a receiver in the call to B.Whatever and will be physically copied there.
The second type conversion stores the result in a variable (and then copies it again when a method is called).
Reasonong about such things is easy in Go as there everything, always, is passed by value (and pointers are values, too).
It may worth adding that variables in Go does not store the type of the value they hold, so a type conversion cannot mutate the type of a variable "in place". Values do not have type information stored in them, either. This basically means that type conversions is what compiler is concerned with: it knows the types of all the participating values and variables and performs type checking.
I'm writing a package in go to send messages between services, using a specific type of transport.
I'd like the package to not understand the type of messages being sent.
My first thought is to serialize the message object into json, send that, deserialize on the receiving end, and pass the go object (as an interface{}) to the subscribing code.
The serialization isn't a problem, but I don't see how the generic package code can deserialize the message since it doesn't know the type. I thought of using reflect TypeOf(), and passing that value as part of the message. But I don't see how to accomplish this since Type is an interface and the implementing rtype is not exported.
If the receiving app gets an interface{}, it is going to have to check the type anyways, so maybe it should just do the deserialization. Or the receiver could provide a reflect Type so the package can deserialize?
Or it could give the receiver a map of field name to value, but I'd prefer the actual type.
Any suggestions?
Let me add an example:
I have a go channel for sending change notifications of different types of objects. Since go doesn't support tagged unions, I define the channel type as:
type UpdateInfo struct {
UpdateType UpdateType
OldObject interface{}
NewObject interface{}
}
The receiving end of the channel gets an UpdateInfo with OldObject and NewObject as the actual concrete object types that were sent.
I want to extend this to work between applications, where the transport will be via a message queue to support pub/sub, multiple consumers, etc.
TL;DR
Just use json.Unmarshal. You can wrap it lightly, using your transport, and call json.Unmarshal (or with a json.Decoder instance, use d.Decode) on your prebuilt JSON bytes and the v interface{} argument from your caller.
Somewhat longer, with an example
Consider how json.Unmarshal does its own magic. Its first argument is the JSON (data []byte), but its second argument is of type interface{}:
func Unmarshal(data []byte, v interface{}) error
As the documentation goes on to say, if v really is just an interface{}:
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
but if v has an underlying concrete type, such as type myData struct { ... }, it's much fancier. It only does the above if v's underlying type is interface{}.
Its actual implementation is particularly complex because it's optimized to do the de-JSON-ification and the assignment into the target object at the same time. In principle, though, it's mostly a big type-switch on the underlying (concrete) type of the interface value.
Meanwhile, what you are describing in your question is that you will first deserialize into generic JSON—which really means a variable of type interface{}—and then do your own assignment out of this pre-decoded JSON into another variable of type interface{}, where the type signature of your own decoder would be:
func xxxDecoder(/* maybe some args here, */ v interface{}) error {
var predecoded interface{}
// get some json bytes from somewhere into variable `data`
err := json.Unmarshal(data, &predecoded)
// now emulate json.Unmarshal by getting field names and assigning
... this is the hard part ...
}
and you would then call this code by writing:
type myData struct {
Field1 int `xxx:"Field1"`
Field2 string `xxx:"Field2"`
}
so that you know that JSON object key "Field1" should fill in your Field1 field with an integer, and JSON object key "Field2" should fill in your Field2 field with a string:
func whatever() {
var x myData
err := xxxDecode(..., &x)
if err != nil { ... handle error ... }
... use x.Field1 and x.Field2 ...
}
But this is silly. You can just write:
type myData struct {
Field1 int `json:"Field1"`
Field2 string `json:"Field2"`
}
(or even omit the tags since the field's names are the default json tags), and then do this:
func xxxDecode(..., v interface{}) error {
... get data bytes as before ...
return json.Unmarshal(data, v)
}
In other words, just let json.Unmarshal do all the work by providing json tags in the data structures in question. You still get—and transmit across your special transport—the JSON data bytes from json.Marshal and json.Unmarshal. You do the transmitting and receiving. json.Marshal and json.Unmarshal do all the hard work: you don't have to touch it!
It's still fun to see how Json.Unmarshal works
Jump down to around line 660 of encoding/json/decode.go, where you will find the thing that handles a JSON "object" ({ followed by either } or a string that represents a key), for instance:
func (d *decodeState) object(v reflect.Value) error {
There are some mechanics to handle corner cases (including the fact that v might not be settable and/or might be a pointer that should be followed), then it makes sure that v is either a map[T1]T2 or struct, and if it is a map, that it's suitable—that both T1 and T2 will work when decoding the "key":value items in the object.
If all goes well, it gets into the JSON key-and-value scanning loop starting at line 720 (for {, which will break or return as appropriate). On each trip through this loop, the code reads the JSON key first, leaving the : and value part for later.
If we're decoding into a struct, the decoder now uses the struct's fields—names and json:"..." tags—to find a reflect.Value that we'll use to store right into the field.1 This is subv, found by calling v.Field(i) for the right i, with some slightly complicated goo to handle embedded anonymous structs and pointer-following. The core of this is just subv = v.Field(i), though, where i is whichever field this key names, within the struct. So subv is now a reflect.Value that represents the actual struct instance's value, which we should set once we've decoded the value part of the JSON key-value pair.
If we're decoding into a map, we will decode the value into a temporary first, then store it into the map after decoding. It would be nice to share this with the struct-field storing, but we need a different reflect function to do the store into the map: v.SetMapIndex, where v is the reflect.Value of the map. That's why for a map, subv points to a temporary Elem.
We're now ready to convert the actual value to the target type, so we go back to the JSON bytes and consume the colon : character and read the JSON value. We get the value and store it into our storage location (subv). This is the code starting at line 809 (if destring {). The actual assigning is done through the decoder functions (d.literalStore at line 908, or d.value at line 412) and these actually decode the JSON value while doing the storing. Note that only d.literalStore really stores the value—d.value calls on d.array, d.object, or d.literalStore to do the work recursively if needed.
d.literalStore therefore contains many switch v.Kind()s: it parses a null or a true or false or an integer or a string or an array, then makes sure it can store the resulting value into v.Kind(), and chooses how to store that resulting value into v.Kind() based on the combination of what it just decoded, and the actual v.Kind(). So there's a bit of a combinatorial explosion here, but it gets the job done.
If all that worked, and we're decoding to a map, we may now need to massage the type of the temporary, find the real key, and store the converted value into the map. That's what lines 830 (if v.Kind() == reflect.Map {) through the final close brace at 867 are about.
1To find fields, we first look over at encoding/json/encode.go to find cachedTypeFields. It is a caching version of typeFields. This is where the json tags are found and put into a slice. The result is cached via cachedTypeFields in a map indexed by the reflect-type value of the struct type. So what we get is a slow lookup the first time we use a struct type, then a fast lookup afterwards, to get a slice of information about how to do the decoding. This slice-of-information maps from json-tag-or-field name to: field; type; whether it's a sub-field of an anonymous structure; and so on: everything we will need to know to decode it properly—or to encode it, on the encoding side. (I didn't really look closely at this code.)
You can encode/decode several message on the same buffer, whether that be a "gob" or "json" or some other encoding.
Assuming there's a limited set of concrete types that you want to support, you can always encode a type tag as the first thing, then encode the actual object. This way the decode can decode the type tag first, and depending on its value, decide how to decode the next item.
// encoder side
enc := json.NewEncoder(buffer) // or gob.NewEncoder(buffer)
enc.Encode("player")
enc.Encode(playerInstance)
// decoder side
dec := json.NewDecoder(buffer) // or gob.NewDecoder(buffer)
var tag string
dec.Decode(&tag)
switch tag {
case "player":
var playerInstance Player
dec.Decode(&player)
// do something with it
case "somethingelse":
// decode something else
}
Try dynobuffers instead of struct. It provides get and set by name ability for byte array. Also it is much more faster than json.
I'm interfacing with C code in Go using cgo, and I need to call a C function with a pointer to the underlying value in an Interface{} object. The value will be any of the atomic primitive types (not including complex64/complex128), or string.
I was hoping I'd be able to do something like this to get the address of ptr as an unsafe.Pointer:
unsafe.Pointer(reflect.ValueOf(ptr).UnsafeAddr())
But this results in a panic due to the value being unaddressable.
A similar question to this is Take address of value inside an interface, but this question is different, as in this case it is known that the value will always be one of the types specified above (which will be at most 64 bits), and I only need to give this value to a C function. Note that there are multiple C functions, and the one that will be called varies based off of a different unrelated parameter.
I also tried to solve this using a type switch statement, however I found myself unable to get the address of the values even after the type assertion was done. I was able to assign the values to temporary copies, then get the address of those copies, but I'd rather avoid making these copies if possible.
interface{} has own struct:
type eface struct {
typ *rtype
val unsafe.Pointer
}
You have no access to rtype directly or by linking, on the other hand, even though you'll copy whole rtype, it may be changed (deprecated) at future.
But thing is that you can replace pointer types with unsafe.Pointer (it may be anything else with same size, but pointer is much idiomatic, because each type has own pointer):
type eface struct {
typ, val unsafe.Pointer
}
So, now we can get value contained in eface:
func some_func(arg interface{}) {
passed_value := (*eface)(unsafe.Pointer(&arg)).val
*(*byte)(passed_value) = 'b'
}
some_var := byte('a')
fmt.Println(string(some_var)) // 'a'
some_func(some_var)
fmt.Println(string(some_var)) // 'a', it didn't changed, just because it was copied
some_func(&some_var)
fmt.Println(string(some_var)) // 'b'
You also might see some more usages at my repo:
https://github.com/LaevusDexter/fast-cast
Sorry for my poor English.
A package I am using, gosqlite, has a method with a variadic parameter where its type is the empty interface.
func (s *Stmt) Exec(args ...interface{}) os.Error
I can call this fine if explicitly pass individual parameters:
statement := blah()
error := statement.Exec("hello", 3.0, true) // works fine
However, as the variadic parameter corresponds to placeholders within the in operator of my SQL statement's select, the number of these placeholders is not known at compile time but dynamically changes at run time depending upon what the user is doing. E.g. I end up with SQL akin to the following if the user enters four values:
SELECT * FROM sky WHERE name IN (?,?,?,?)
So naturally I would like to call the Exec method with a slice of strings:
var values []string = getValuesFromUser()
statement := createStatementWithSufficientNumberOfPlaceholders(len(values))
_ := statement.Exec(values...) // compiler doesn't like this
This does not compile. I can get around this problem by creating an empty interface slice and copying the references over:
values2 := make([]interface{}, len(values))
for index, value := range values { values2[index] = value }
_ := statement.Exec(values2...) // compiler happy but I'm not
And this works fine but it feels a bit clunky. I was wondering if there was some trick to be able to pass values directly to this function or, failing that, a neater way of converting the string slice to an empty interface one?
Many thanks.
There is no way to pass a []string directly to a ...interface{} parameter. Doing this requires a linear time copy (with n + 1 allocations!). If the language hid this from you, it would be a significant hidden cost. Normally, passing a slice to a variadic argument just passes the slice into the function.
As for other ways of doing this, you could make it cleaner by writing a function that takes a []string and returns the corresponding []interface{}. Of course, you'll have to write it again for each []T -> []interface{} conversion you want to do, but its a rather short function, and all that changes is the signature. You could use reflection, which comes with an inherent runtime cost, to make the function "generic", such as in:
valuesVal := reflect.ValueOf(values)
...
for i := range values2 { values2[i] = valuesVal.Index(i).Interface() }
I don't have an answer. And I don't suppose there is one since even built-in and variadic copy and append have the same (or compatible concrete) element type "blockhead", but I have two obvious suggestions:
do not return []string from getValuesFromUser() (i.e. pass still unadorned []interface{}),
on the other type end wrap calls to statement.Exec() with a func making []string to []interface{} conversion.
Or on the same, third, obvious note extend type statement with Exec(args ...string).
P.S. I haven't made any benchmarks myself but I don't think this kind of conversion is highly expensive as interface{} feels like a reference type and compiler is probably doing some dirty trickstery behind the curtain... then again perhaps not, though, I'd be happy, too, to learn of an actual solution.
You need to pass a varargs slice of interface{} type like this to the method.
var paramArray []interface{}
paramArray = append(paramArray, "test1")
paramArray = append(paramArray, "test2")
varargsFunc(paramArray...)