The package golang.org/x/text/message allows us to format numbers using national formats:
const n = 1222333.444555
prEn := message.NewPrinter(language.English)
prEn.Printf("%20.6f\n", n)
// Prints:
// 1,222,333.444555
prRu := message.NewPrinter(language.Russian)
prRu.Printf("%20.6f\n", n)
// Prints:
// 1 222 333,444555
Can I use it with math/big.Rat? That is, something like (doesn't work):
rat := big.NewRat(1222333444555, 1000000)
prEn.Printf("%20.6f\n", rat.FloatString(6))
// Should print:
// 1,222,333.444555
I know that I can wrap Rat in my own type and implement fmt.Formatter, but maybe there is a built-in way already?
Related
I want to parse a string like "12.49" into a *big.Int in Go. The resulting *big.Int should represent the amount of cents in the given value, in this case 1249. Here are some more examples of inputs and their expected outputs:
"3": 300
"3.1": 310
".19": 19
I already tried working with *big.Float and its Int function, but realized, that *big.Float does not provide arbitrary precision.
Right now I'm using this algorithm, but it seems fragile (Go Playground link):
func eurToCents(in string) *big.Int {
missingZerosUntilCents := 2
i := strings.Index(in, ".")
if i > -1 {
missingZerosUntilCents -= len(in) - i - 1
if missingZerosUntilCents < 0 {
panic("too many decimal places")
}
}
in = strings.Replace(in, ".", "", 1)
in += strings.Repeat("0", missingZerosUntilCents)
out, ok := big.NewInt(0).SetString(in, 10)
if !ok {
panic(fmt.Sprintf("could not parse '%s' as an interger", in))
}
return out
}
Is there a standard library function or other common way to parse currencies in Go? An external library is not an option.
PS: I'm parsing Nano cryptocurrency values, which have 30 decimal places and a maximum value of 133,248,297.0. That's why I'm asking for *big.Int and not uint64.
Update: Seems like this solution is still buggy, because an inaccurate result is reported after multiplication: https://play.golang.org/p/RS-DC6SeRwz
After revisiting the solution with *big.Float, I realized, that it does work perfectly fine. I think I forgot to use SetPrec on rawPerNano previously. I'm going to provide an example for the Nano cryptocurrency, because it requires many decimal places.
This works as expected (Go Playground link):
func nanoToRaw(in string) *big.Int {
f, _ := big.NewFloat(0).SetPrec(128).SetString(in)
rawPerNano, _ := big.NewFloat(0).SetPrec(128).SetString("1000000000000000000000000000000")
f.Mul(f, rawPerNano)
i, _ := f.Int(big.NewInt(0))
return i
}
Thanks #hymns-for-disco for nudging me in the right direction!
I have a process that needs to pack a large array of int16s to a protobuf every few milliseconds. Understanding the protobuf side of it isn't critical, since all I really need is a way to convert a bunch of int16s (160-16k of them) to []byte. It's a CPU-critical operation, so I don't want to do something like this:
for _, sample := range listOfIntegers {
protobufObject.ByteStream = append(protobufObject.Bytestream, byte(sample>>8))
protobufObject.ByteStream = append(protobufObject.Bytestream, byte(sample&0xff))
}
(If you're interested, this is the protobuf)
message ProtobufObject {
bytes byte_stream = 1;
... = 2;
etc.
}
There has to be a faster way to supply that list of ints as a block of memory to the protobuf. I've fiddled with the cgo library to get access to memcpy, but suspect I've been destroying an underlying go data structure because I get crashes in totally unrelated sections of code.
A faster version of the above code is:
protobufObject.ByteStream := make([]byte, len(listOfIntegers) * 2)
for i, n := range listOfIntegers {
j := i * 2
protobufObject.ByteStream[j+1] = byte(n)
protobufObject.ByteStream[j] = byte(n>>8)
}
You can avoid copying the data when running on a big-endian architecture.
Use the unsafe package to copy the []int16 header to the []byte header. Use the unsafe package again to get a pointer to the []byte header and adjust the length and capacity for the conversion.
b = *(*[]byte)(unsafe.Pointer(&listOfIntegers))
hdr := (*reflect.SliceHeader)(unsafe.Pointer(&b))
hdr.Len *= 2
hdr.Cap *= 2
protobufObject.ByteStream = b
I am using the Go flatbuffers interface for the first time. I find the instructions sparse.
I would like to write a vector of uint64s into a table. Ideally, I would like to store numbers directly in a vector without knowing how many there are up front (I'm reading them from sql.Rows iterator). I see the generated code for the table has functions:
func DatasetGridAddDates(builder *flatbuffers.Builder, dates flatbuffers.UOffsetT) {
builder.PrependUOffsetTSlot(2, flatbuffers.UOffsetT(dates), 0)
}
func DatasetGridStartDatesVector(builder *flatbuffers.Builder, numElems int) flatbuffers.UOffsetT {
return builder.StartVector(8, numElems, 8)
}
Can I first write the vector using (??), then use DatasetGridAddDates to record the resulting vector in the containing "DatasetGrid" table?
(caveat: I have not heard of FlatBuffers prior to reading your question)
If you do know the length in advance, storing a vector is done as explained in the tutorial:
name := builder.CreateString("hello")
q55310927.DatasetGridStartDatesVector(builder, len(myDates))
for i := len(myDates) - 1; i >= 0; i-- {
builder.PrependUint64(myDates[i])
}
dates := builder.EndVector(len(myDates))
q55310927.DatasetGridStart(builder)
q55310927.DatasetGridAddName(builder, name)
q55310927.DatasetGridAddDates(builder, dates)
grid := q55310927.DatasetGridEnd(builder)
builder.Finish(grid)
Now what if you don’t have len(myDates)? On a toy example I get exactly the same output if I replace StartDatesVector(builder, len(myDates)) with StartDatesVector(builder, 0). Looking at the source code, it seems like the numElems may be necessary for alignment and for growing the buffer. I imagine alignment might be moot when you’re dealing with uint64, and growing seems to happen automatically on PrependUint64, too.
So, try doing it without numElems:
q55310927.DatasetGridStartDatesVector(builder, 0)
var n int
for rows.Next() { // use ORDER BY to make them go in reverse order
var date uint64
if err := rows.Scan(&date); err != nil {
// ...
}
builder.PrependUint64(date)
n++
}
dates := builder.EndVector(n)
and see if it works on your data.
I'm using https://github.com/coocood/freecache to cache database results, but currently I need to dump bigger chunks on every delete, which costs multiple microseconds extra compared to targeted deletion. fmt.Sprintf("%d_%d_%d") for a pattern like #SUBJECT_#ID1_#ID2 also costs multiple microseconds. Even tho that doesn't sound like much, in the current ratio of the cache's response time that a multitude slower than it currently is.
I was thinking of using the library's SetInt/GetInt which works with int64 keys instead of strings.
So let's say I'm storing in a #SUBJECT_#ID1_#ID2 pattern. The Subject is a table or query-segment-range in my code (e.a. everything concern ACL or Productfiltering).
Let's take an example of Userright.id is #ID1 and User.id is #ID2 and Subject ACL. I would build it as something like this:
// const CACHE_SUBJECT_ACL = 0x1
// var userrightID int64 = 0x1
// var userID int64 = 0x1
var storeKey int64 = 0x1000000101
fmt.Println("Range: ", storeKey&0xff)
fmt.Println("ID1 : ", storeKey&0xfffffff00-0xff)
fmt.Println("ID2 : ", storeKey&0x1fffffff00000000-0xfffffffff)
How can I compile the CACHE_SUBJECT_ACL/userrightID/userID into the storeKey?
I know I can call userrightID 0x100000001, but it's a dynamic value so I'm not sure what's the best way to compile this without causing more overhead than formatting the string as a key.
The idea is that in a later state when I need to flush the cache I can call a small range of int64 calls instead of just dumping a whole partition (of maybe thousands of entries).
I was thinking of adding them to each other with bit shifting, like userID<<8, but I'm not sure if that's the safe route.
If I failed to supply enough information, please ask.
Packing numbers to an int64
If we can make sure the numbers we want to pack are not negative and they fit into the bit range we're reserving for them, then yes, this is a safe and efficient way to pack them.
An int64 has 64 bits, that's how many we can assign to the parts we want to pack into it. Often the sign bit is not used to avoid confusion, or an unsigned version uint64 is used.
For example, if we reserve 8 bits for subject, that leaves 64-8=56 bits for the rest, 28 bits for each ID.
| ID2 | ID1 |SUB|
Encoded key bits: |f f f f f f f|f f f f f f f|f f|
Note that when encoding, it's recommended to also use a bitmask with bitwise AND to make sure the numbers we pack do not overlap (arguable, because if the components are bigger, we're screwed anyway...).
Also note that if we're also using the sign bit (63th), we have to apply masking after the bitshift when decoding, as shifting right "brings in" the sign bit and not 0 (sign bit is 1 in case of negative numbers).
Since we used 28 bits for both ID1 and ID2, we can use the same mask for both IDs:
Use these short utility functions which get the job done:
const (
maskSubj = 0xff
maskId = 0xfffffff
)
func encode(subj, id1, id2 int64) int64 {
return subj&maskSubj | (id1&maskId)<<8 | (id2&maskId)<<36
}
func decode(key int64) (sub, id1, id2 int64) {
return key & maskSubj, (key >> 8) & maskId, (key >> 36) & maskId
}
Testing it:
key := encode(0x01, 0x02, 0x04)
fmt.Printf("%016x\n", key)
fmt.Println(decode(key))
Output (try it on the Go Playground):
0000004000000201
1 2 4
Sticking to string
Originally you explored packing into an int64 because fmt.Sprintf() was slow. Note that Sprintf() uses a format string, and it takes time to parse the format string and format the arguments according to the "rules" laid out in the format string.
But in your case we don't need this. We can simply get what you originally wanted like this:
id2, id1, subj := 0x04, 0x02, 0x01
key := fmt.Sprint(id2, "_", id1, "_", subj)
fmt.Println(key)
Output:
4_2_1
This one will be significantly faster as it doesn't have to process a format string, it will just concatenate the arguments.
We can even do better; if none of 2 arguments being next to each other are string values, a space is automatically inserted, so it's really enough to just list the numbers:
key = fmt.Sprint(id2, id1, subj)
fmt.Println(key)
Output:
4 2 1
Try these on the Go Playground.
Utilizing fmt.AppendInt()
We can improve it further by using fmt.AppendInt(). This function appends the textual representation of an integer to a byte slice. We can use base 16 so we'll have more compact representation, and also because the algorithm to convert a number to base 16 is faster than to base 10:
func encode(subj, id1, id2 int64) string {
b := make([]byte, 0, 20)
b = strconv.AppendInt(b, id2, 16)
b = append(b, '_')
b = strconv.AppendInt(b, id1, 16)
b = append(b, '_')
b = strconv.AppendInt(b, subj, 16)
return string(b)
}
Testing it:
id2, id1, subj := int64(0x04), int64(0x02), int64(0x01)
key := encode(subj, id1, id2)
fmt.Println(key)
Output (try it on the Go Playground):
4_2_1
Seemed to have figured it out:
const CacheSubjectACL = 1
var userrightID int64 = 8
var userID int64 = 2
storeKey := CacheSubjectACL + (userrightID << 8) + (userID << 36)
fmt.Println("storeKey: ", storeKey)
fmt.Println("Range : ", storeKey&0xff)
fmt.Println("ID1 : ", storeKey&0xfffffff00>>8)
fmt.Println("ID2 : ", storeKey&0x1ffffff000000000>>36)
Gives:
storeKey: 137438955521
Range : 1
ID1 : 8
ID2 : 2
storeKey builds the int64 masked. And the masking with a new shift the other way around fishes the old values out of the int64 again.
Because storeKey&0x1ffffff000000000>>36 runs to the end anyway, storeKey>>36 will suffice too as there are no bits on the further left.
I'm looking for something like the c++ function .clear() for the primitive type map.
Or should I just create a new map instead?
Update: Thank you for your answers. By looking at the answers I just realized that sometimes creating a new map may lead to some inconsistency that we don't want. Consider the following example:
var a map[string]string
var b map[string]string
func main() {
a = make(map[string]string)
b=a
a["hello"]="world"
a = nil
fmt.Println(b["hello"])
}
I mean, this is still different from the .clear() function in c++, which will clear the content in the object.
You should probably just create a new map. There's no real reason to bother trying to clear an existing one, unless the same map is being referred to by multiple pieces of code and one piece explicitly needs to clear out the values such that this change is visible to the other pieces of code.
So yeah, you should probably just say
mymap = make(map[keytype]valtype)
If you do really need to clear the existing map for whatever reason, this is simple enough:
for k := range m {
delete(m, k)
}
Unlike C++, Go is a garbage collected language. You need to think things a bit differently.
When you make a new map
a := map[string]string{"hello": "world"}
a = make(map[string]string)
the original map will be garbage-collected eventually; you don't need to clear it manually. But remember that maps (and slices) are reference types; you create them with make(). The underlying map will be garbage-collected only when there are no references to it.
Thus, when you do
a := map[string]string{"hello": "world"}
b := a
a = make(map[string]string)
the original array will not be garbage collected (until b is garbage-collected or b refers to something else).
// Method - I , say book is name of map
for k := range book {
delete(book, k)
}
// Method - II
book = make(map[string]int)
// Method - III
book = map[string]int{}
Go 1.18 and above
You can use maps.Clear. The function belongs to the package golang.org/x/exp/maps (experimental and not covered by the compatibility guarantee)
Clear removes all entries from m, leaving it empty.
Example usage:
func main() {
testMap := map[string]int{"gopher": 1, "badger": 2}
maps.Clear(testMap)
fmt.Println(testMap)
testMap["zebra"] = 2000
fmt.Println(testMap)
}
Playground: https://go.dev/play/p/qIdnGrd0CYs?v=gotip
If you don't want to depend on experimental packages, you can copy-paste the source, which is actually extremely simple:
func Clear[M ~map[K]V, K comparable, V any](m M) {
for k := range m {
delete(m, k)
}
}
IMPORTANT NOTE: just as with the builtin delete — which the implementation of maps.Clear uses —, this does not remove irreflexive keys from the map. The reason is that for irreflexive keys, by definition, x == x is false. Irreflexive keys are NaN floats and every other type that supports comparison operators but contains NaN floats somewhere.
See this code to understand what this entails:
func main() {
m := map[float64]string{}
m[1.0] = "foo"
k := math.NaN()
fmt.Println(k == k) // false
m[k] = "bar"
maps.Clear(m)
fmt.Printf("len: %d, content: %v\n", len(m), m)
// len: 1, content: map[NaN:bar]
a := map[[2]float64]string{}
a[[2]float64{1.0, 2.0}] = "foo"
h := [2]float64{1.0, math.NaN()}
fmt.Println(h == h) // false
a[h] = "bar"
maps.Clear(a)
fmt.Printf("len: %d, content: %v\n", len(a), a)
// len: 1, content: map[[1 NaN]:bar]
}
Playground: https://go.dev/play/p/LWfiD3iPA8Q
A clear builtin is being currently discussed (Autumn 2022) that, if added to next Go releases, will delete also irreflexive keys.
For the method of clearing a map in Go
for k := range m {
delete(m, k)
}
It only works if m contains no key values containing NaN.
delete(m, k) doesn't work for any irreflexive key (such as math.NaN()), but also structs or other comparable types with any NaN float in it. Given struct{ val float64 } with NaN val is also irreflexive (Quote by blackgreen comment)
To resolve this issue and support clearing a map in Go, one buildin clear(x) function could be available in the new release, for more details, please refer to add clear(x) builtin, to clear map, zero content of slice, ptr-to-array
If you are trying to do this in a loop, you can take advantage of the initialization to clear out the map for you. For example:
for i:=0; i<2; i++ {
animalNames := make(map[string]string)
switch i {
case 0:
animalNames["cat"] = "Patches"
case 1:
animalNames["dog"] = "Spot";
}
fmt.Println("For map instance", i)
for key, value := range animalNames {
fmt.Println(key, value)
}
fmt.Println("-----------\n")
}
When you execute this, it clears out the previous map and starts with an empty map. This is verified by the output:
$ go run maptests.go
For map instance 0
cat Patches
-----------
For map instance 1
dog Spot
-----------