Go allows conversion from rune to byte. But the underlying type for rune is int32 (because Go uses UTF-8) and for byte it is uint8, the conversion therefore results in a loss of information. However it is not possible to convert from a rune to []byte.
var b byte = '©'
bs := []byte(string('©'))
fmt.Println(b)
fmt.Println(bs)
// Output
169
[194 169]
Working example
Why does Go allow conversion from rune to byte instead of rune to []byte?
Go supports conversion from rune to byte as it does for all pairs of numeric types. It would be a surprising special case if int32 to byte conversion was not allowed.
But the underlying type for rune is int32 (because Go uses UTF-8)
This misses an important detail: rune is an alias for int32. They are the same type.
It's true that the underlying type is rune is int32, but that's because rune and int32 are the same type and the underlying type of a builtin type is the type itself.
The representation of Unicode code points as int32 values is unrelated to UTF-8 encoding.
the conversion therefore results in a loss of information
Yes, conversions between numeric types can result in loss of information. This is one reason why conversions in Go must be explicit.
Note that the statement var b byte = '©' does not do any conversions. The expression '#' is an untyped constant.
The compiler reports an error if the assignment of an untyped constant results in a loss of information. For example, the statement var b byte = '世' causes a compilation error.
All UTF-8 encoding functionality in the language is related to the string type. The UTF-8 aware conversions are all to or from the string type. The []byte(numericType) conversion could be supported, but that would bring UTF-8 encoding outside of the string type.
The Go authors regret including the string(numericType) conversion because it's not very useful in practice and the conversion is not what some people expect. A library function is a better place for the functionality.
Yes it is possible to convert from a rune to []byte (for example via a byte) and back again.
package main
import "fmt"
func main() {
var b byte = '©'
bs := []byte{b}
fmt.Printf("%T %v\n", b, b) // uint8 169
fmt.Printf("%T %v\n", bs, bs) // []uint8 [169]
s := string(bs[0]) // s := string(b) works too.
r2 := rune(s[0]) // r2 := rune(b) works too.
fmt.Printf("%T %v\n", s, s) // string ©
fmt.Printf("%T %v\n", r2, r2) // int32 169
}
The reason for this behaviour is the same reason why it's legal to do
var b int32
b = 1000000
fmt.Printf("%b\n", b)
fmt.Printf("%b", uint8(b))
// Output:
// 11110100001001000000
// 1000000
You should expect the conversion to loose data when you put data of a type with larger memory footprint into one with a smaller memory footprint.
Also, for encoding a rune you can use EncodeRune which indeed uses a []byte.
Related
I'm teaching myself Go from a C background.
The code below works as I expect (the first two Printf() will access bytes, the last two Printf() will access codepoints).
What I am not clear is if this involves any copying of data.
package main
import "fmt"
var a string
func main() {
a = "èe"
fmt.Printf("%d\n", a[0])
fmt.Printf("%d\n", a[1])
fmt.Println("")
fmt.Printf("%d\n", []rune(a)[0])
fmt.Printf("%d\n", []rune(a)[1])
}
In other words:
does []rune("string") create an array of runes and fill it with the runes corresponding to "string", or it's just the compiler that figures out how to get runes from the string bytes?
It is not possible to turn []uint8 (i.e. a string) into []int32 (an alias for []rune) without allocating an array.
Also, strings are immutable in Go but slices are not, so the conversion to both []byte and []rune must copy the string's bytes in some way or another.
It involves a copy because:
strings are immutable; if the conversion []rune(s) didn't make a copy, you would be able to index the rune slice and change the string contents
a string value is a "(possibly empty) sequence of bytes", where byte is an alias of uint8, whereas a rune is a "an integer value identifying a Unicode code point" and an alias of int32. The types are not identical and even the lengths may not be the same:
a = "èe"
r := []rune(a)
fmt.Println(len(a)) // 3 (3 bytes)
fmt.Println(len(r)) // 2 (2 Unicode code points)
Per this doc
type Point struct {
X, Y float64
_ struct{} // to prevent unkeyed literals
}
For Point{X: 1, Y: 1} everything will be fine, but for Point{1,1} you will get a compile error:
./file.go:1:11: too few values in &Pointer literal
Then I tried it in another data type _ byte and _ func() as below
type Pointer struct {
X, Y int
//_ byte // to prevent unkeyed literals
//_ func() // to prevent unkeyed literals
}
Both of them could prevent unkeyed literals. How does it prevent unkeyed literal? Is _ struct{} more efficient?
Unkeyed structs require you to specify all struct keys; it is an error if you don't specify the value for Y for example:
type Point struct {
X, Y float64
}
_ = Point{1}
// Output:
// ./main.go:8:8: too few values in Point{...}
The _ struct{} field doesn't really prevent unkeyed literals from the same package, as you can still do:
type Point struct {
X, Y float64
_ struct{} // to prevent unkeyed literals
}
_ = Point{1, 2, struct{}{}}
// Ugly and weird, but valid!
But in order to be able to assign values in struct fields from other packages they need to be "exported", that is, start with a capital letter, and _ doesn't, so this is an error:
_ = x.Point{1, 2, struct{}{}}
// Output
// ./main.go:6:28: implicit assignment of unexported field '_' in x.Point literal
There is nothing special about _; you can use anything else that doesn't start with a capital as well, such as noexport struct{} or whatnot.
Why struct{} and not byte or int? Well, those types allocate some amount of memory; for an int it's usually 8 bytes (or 4 bytes on a 32bit system), and byte is an alias for uint8 and allocates one byte.
struct{} on the other hand is an "empty" type (you can't assign anything to it) and won't use any memory. This is a very small optimisation, but if you're going to type something you might as well type struct{}.
Is all of this worth it? In my opinion it's not; if someone wants to use unkeyed struct literals with your library then that's their choice. Many lint tools will already warn on this, including the built-in go vet:
$ go vet main.go
./main.go:8:6: net/mail.Address composite literal uses unkeyed fields
How does it prevent unkeys literals works?
An unkeyed struct literal must specify all fields; by adding a field that cannot be specified from outside the package, it makes it impossible to use this format, so it requires a keyed literal. "Keyed" or "unkeyed" refers to whether the field names appear in the struct literal.
Does _ struct{} more efficient?
Yes, because it has a width of zero, so it doesn't consume any memory. All other types would increase the memory footprint of the struct unnecessarily.
I've got the following Go program:
package main
import (
"fmt"
)
func main() {
r := rune(249)
x := uint16(r)
fmt.Println(x)
}
If I have a rune r and call uint16(r) on it, is it going to be big or little endian encoded? Does Go default to one? Or does this depend on my hardware?
The rune type is an alias for int32, and the expression uint16(r) is a type conversion, which will retain the lowest 16 bits of the rune value. There is no encoding or serialization involved in here.
Little endian or Big endian comes into play when you serialize values into a series of bytes, but that does not happen here.
See this example:
r := rune(0x0000fafa)
fmt.Printf("%x\n", uint16(r))
i := uint32(0xfffffafa)
r = rune(i)
fmt.Printf("%x\n", uint16(r))
Output (try it on the Go Playground):
fafa
fafa
I was watching this talk given at FOSDEM '17 about implementing "tail -f" in Go => https://youtu.be/lLDWF59aZAo
In the author's initial example program, he creates a Reader using a file handle, and then uses the ReadString method with delimiter '\n' to read the file line by line and print its contents. I usually use Scanner, so this was new to me.
Program below | Go Playground Link
package main
import (
"bufio"
"fmt"
"log"
"os"
)
func main() {
fileHandle, err := os.Open("someFile.log")
if err != nil {
log.Fatalln(err)
return
}
defer fileHandle.Close()
reader := bufio.NewReader(fileHandle)
for {
line, err := reader.ReadString('\n')
if err != nil {
log.Fatalln(err)
break
}
fmt.Print(line)
}
}
Now, ReadString takes a byte as its delimiter argument[https://golang.org/pkg/bufio/#Reader.ReadString]
So my question is, how in the world did '\n', which is a rune, get converted into a byte? I am not able to get my head around this. Especially since byte is an alias for uint8, and rune is an alias for int32.
I asked the same question in Gophers slack, and was told that '\n' is not a rune, but an untyped constant. If we actually created a rune using '\n' and passed it in, the compilation would fail. This actually confused me a bit more.
I was also given a link to a section of the Go spec regarding Type Identity => https://golang.org/ref/spec#Type_identity
If the program is not supposed to compile if it were an actual rune, why does the compiler allow an untyped constant to go through? Isn't this unsafe behaviour?
My guess is that this works due to a rule in the Assignability section in the Go spec, which says
x is an untyped constant representable by a value of type T.
Since '\n' can indeed be assigned to a variable of type byte, it is therefore converted.
Is my reasoning correct?
TL;DR Yes you are correct but there's something more.
'\n' is an untyped rune constant. It doesn't have a type but a default type which is int32 (rune is an alias for int32). It holds a single byte representing the literal "\n", which is the numeric value 10:
package main
import (
"fmt"
)
func main() {
fmt.Printf("%T %v %c\n", '\n', '\n', '\n') // int32 10 (newline)
}
https://play.golang.org/p/lMjrTFDZUM
The part of the spec that answers your question lies in the § Calls (emphasis mine):
Given an expression f of function type F,
f(a1, a2, … an)
calls f with arguments a1, a2, … an. Except for one
special case, arguments must be single-valued expressions assignable
to the parameter types of F and are evaluated before the function is
called.
"assignable" is the key term here and the part of the spec you quoted explains what it means. As you correctly guessed, among the various rules of assignability, the one that applies here is the following:
x is an untyped constant representable by a value of type T.
In our case this translates to:
'\n' is an untyped (rune) constant representable by a value of type byte
The fact that '\n' is actually converted to a byte when calling ReadString() is more apparent if we try passing an untyped rune constant wider than 1 byte, to a function that expects a byte:
package main
func main() {
foo('α')
}
func foo(b byte) {}
https://play.golang.org/p/W0EUZppWHH
The code above fails with:
tmp/sandbox120896917/main.go:9: constant 945 overflows byte
That's because 'α' is actually 2 bytes, which means it cannot be converted to a value of type byte (the maximum integer a byte can hold is 255 while 'α' is actually 945).
All this is explained in the official blog post, Constants.
Yes, your reading is correct. Spec: Assignability section applies here as the value you want to pass must be assignable to the type of the parameter.
When you pass the value '\n', that is an untyped constant specified by a rune literal. It represents a number equal to the Unicode code of the '\n' character (which is 10 by the way). The rule you quoted applies here:
x is an untyped constant representable by a value of type T.
Constants have a default type, which will be used when a type is "missing" from the context where the value is used. Such an example is the short variable declaration:
r := '\n'
fmt.Printf("%T", r)
The default type of a rune literal is that: rune. The above code prints int32 because the rune type is an alias for int32 (they are "identical", interchangable). Try it on the Go Playground.
Now if you try to pass the variable r to a function which expects a value of type byte, it is a compile time error, because this case matches none of the assignability rules. You need explicit type conversion to make such a case work:
r := '\n'
line, err := reader.ReadString(byte(r))
See related blog posts and questions:
Spec: Constants
The Go Blog: Constants
Defining a variable in Go programming language
Custom type passed to function as a parameter
Why do these two float64s have different values?
Does go compiler's evaluation differ for constant expression and other expression
I came across a function posted online that used the rune() function in golang, but I am having a hard time looking up what it is. I am going through the tutorial and inexperienced with the docs so it is hard to find what I am looking for.
Specifically, I am trying to see why this fails...
fmt.Println(rune("foo"))
and this does not
fmt.Println([]rune("foo"))
rune is a type in Go. It's just an alias for int32, but it's usually used to represent Unicode points. rune() isn't a function, it's syntax for type conversion into rune. Conversions in Go always have the syntax type() which might make them look like functions.
The first bit of code fails because conversion of strings to numeric types isn't defined in Go. However conversion of strings to slices of runes/int32s is defined like this in language specification:
Converting a value of a string type to a slice of runes type yields a
slice containing the individual Unicode code points of the string.
[golang.org]
So your example prints a slice of runes with values 102, 111 and 111
As stated in #Michael's first-rate comment fmt.Println([]rune("foo")) is a conversion of a string to a slice of runes []rune. When you convert from string to []rune, each utf-8 char in that string becomes a Rune. See https://stackoverflow.com/a/51611567/12817546. Similarly, in the reverse conversion, when converted from []rune to string, each rune becomes a utf-8 char in the string. See https://stackoverflow.com/a/51611567/12817546. A []rune can also be set to a byte, float64, int or a bool.
package main
import (
. "fmt"
)
func main() {
r := []rune("foo")
c := []interface{}{byte(r[0]), float64(r[0]), int(r[0]), r, string(r), r[0] != 0}
checkType(c)
}
func checkType(s []interface{}) {
for k, _ := range s {
Printf("%T %v\n", s[k], s[k])
}
}
byte(r[0]) is set to “uint8 102”, float64(r[0]) is set to “float64 102”,int(r[0]) is set to “int 102”, r is the rune” []int32 [102 111 111]”, string(r) prints “string foo”, r[0] != 0 and shows “bool true”.
[]rune to string conversion is supported natively by the spec. See the comment in https://stackoverflow.com/a/46021588/12817546. In Go then a string is a sequence of bytes. However, since multiple bytes can represent a rune code-point, a string value can also contain runes. So, it can be converted to a []rune , or vice versa. See https://stackoverflow.com/a/19325804/12817546.
Note, there are only two built-in type aliases in Go, byte (alias of uint8) and rune (alias of int32). See https://Go101.org/article/type-system-overview.html. Rune literals are just 32-bit integer values. For example, the rune literal 'a' is actually the number "97". See https://stackoverflow.com/a/19311218/12817546. Quotes edited.