I'm pulling in data that is in long hexadecimal string form which I need to convert into decimal notation, truncate 18 decimal places, and then serve up in JSON.
For example I may have the hex string:
"0x00000000000000000000000000000000000000000000d3c21bcecceda1000000"
At first I was attempting to use ParseUint(), however since the highest it supports is int64, my number ends up being way too big.
This example after conversion and truncation results in 10^6.
However there are instances where this number can be up to 10^12 (meaning pre truncation 10^30!).
What is the best strategy to attack this?
Use math/big for working with numbers larger than 64 bits.
From the Int.SetString example:
s := "d3c21bcecceda1000000"
i := new(big.Int)
i.SetString(s, 16)
fmt.Println(i)
https://play.golang.org/p/vf31ce93vA
The math/big types also support the encoding.TextMarshaler and fmt.Scanner interfaces.
For example
i := new(big.Int)
fmt.Sscan("0x000000d3c21bcecceda1000000", i)
Or
i := new(big.Int)
fmt.Sscanf("0x000000d3c21bcecceda1000000", "0x%x", i)
Related
I have the following code:
x := []byte("1575455669.4")
bits := binary.LittleEndian.Uint64(x)
f := math.Float64frombits(bits)
On calling fmt.Println(f) I expect 1.5754556694e+09. But instead I end up with 1.451098468672448e-47
When I try the same conversion through strconv.ParseFloat(string(x), 64) I get the correct result. What am I doing wrong here?
This:
x := []byte("1575455669.4")
will give you the (UTF-8 encoded) bytes of the "1575455669.4" string. This has nothing to do with the memory representation of the floating point number 1575455669.4 which uses the IEEE 754 standard. But what you do next would assume just that:
bits := binary.LittleEndian.Uint64(x)
f := math.Float64frombits(bits)
You have a number given as its base-10 string representation, you have to use strconv.ParseFloat(string(x), 64) to parse it and have it as a float64.
I have an int64 variable which contains a negative number and I wish to subtract it from an uint64 variable which contains a positive number:
var endTime uint64
now := time.Now().Unix()
endTime = uint64(now)
var interval int64
interval = -3600
endTime = endTime + uint64(interval)
The above code appears to work but I wonder if I can rely on this. I am surprised, being new to Go, that after casting a negative number to uint64 that it remains negative -- I had planned to subtract the now positive value (after casting) to get what I wanted.
Converting a signed number to an unsigned number will not remain negative, it can't, as the valid range of unsigned types doesn't include negative numbers. If you print uint(interval), you will certainly see a positive number printed.
What you experience is deterministic and you can rely on it (but it doesn't mean you should). It is the result of Go (and most other programming languages) storing signed integer types using the 2's completement representation.
What this means is that in case of negative numbers, using n bit, the value -x (where x is positive) is stored as the binary representation of the positive value 2^n - x. This has the advantage that numbers can be added bitwise, and the result will be correct regardless of whether they are negative or positive.
So when you have a signed negative number, it is basically stored in memory like if you would subtract its absolute value from 0. Which means that if you convert a negative, signed value to unsigned, and you add that to an unsigned value, the result will be correct because overflow will happen, in a useful way.
Converting a value of type int64 to uint64 does not change the memory layout, only the type. So what 8 bytes the int64 had, the converted uint64 will have those same 8 bytes. And as mentioned above, the representation stored in those 8 bytes is the bit pattern identical to the bit pattern of value 0 - abs(x). So the result of the conversion will be a number that you would get if you would subtract abs(x) from 0, in the unsigned world. Yes, this won't be negative (as the type is unsigned), instead it will be a "big" number, counting down from the max value of the uint64 type. But if you add an y number bigger than abs(x) to this "big" number, overflow will happen, and the result will be like y - abs(x).
See this simple example demonstrating what's happening (try it on the Go Playground):
a := uint8(100)
b := int8(-10)
fmt.Println(uint8(b)) // Prints 226 which is: 0 - 10 = 256 - 10
a = a + uint8(b)
fmt.Println(a) // Prints 90 which is: 100 + 226 = 326 = 90
// after overflow: 326 - 256 = 90
As mentioned above, you should not rely on this, as this may cause confusion. If you intend to work with negative numbers, then use signed types.
And if you work with a code base that already uses uint64 values, then do a subtraction instead of addition, using uint64 values:
interval := uint64(3600)
endTime -= interval
Also note that if you have time.Time values, you should take advantage of its Time.Add() method:
func (t Time) Add(d Duration) Time
You may specify a time.Duration to add to the time, which may be negative if you want to go back in time, like this:
t := time.Now()
t = t.Add(-3600 * time.Second)
time.Duration is more expressive: we see the value we specified above uses seconds explicitly.
Imagine for printing in a 12 fixed width table we need printing float64 numbers:
fmt.Printf("%12.6g\n", 9.405090880450127e+119) //"9.40509e+119"
fmt.Printf("%12.6g\n", 0.1234567890123) //" 0.123457"
fmt.Printf("%12.6g\n", 123456789012.0) //" 1.23457e+11"
We prefer 0.1234567890 to " 0.123457" we lose 6 significant digits.
We prefer 123456789012 to " 1.23457e+11" we lose 6 significant digits.
Is there any standard library to convert float64 to string with fix width with maximum number of significant digits?
Thanks in Advance.
Basically you have 2 output formats: either a scientific notation or a regular form. The turning point between those 2 formats is 1e12.
So you can branch if x >= 1e12. In both branches you may do a formatting with 0 fraction digits to see how long the number will be, so you can calculate how many fraction digits will fit in for 12 width, and so you can construct the final format string, using the calculated precision.
The pre-check is required in the scientific notation too (%g), because the width of exponent may vary (e.g. e+1, e+10, e+100).
Here is an example implementation. This is to get you started, but it does not mean to handle all cases, and it is not the most efficient solution (but relatively simple and does the job):
// format12 formats x to be 12 chars long.
func format12(x float64) string {
if x >= 1e12 {
// Check to see how many fraction digits fit in:
s := fmt.Sprintf("%.g", x)
format := fmt.Sprintf("%%12.%dg", 12-len(s))
return fmt.Sprintf(format, x)
}
// Check to see how many fraction digits fit in:
s := fmt.Sprintf("%.0f", x)
if len(s) == 12 {
return s
}
format := fmt.Sprintf("%%%d.%df", len(s), 12-len(s)-1)
return fmt.Sprintf(format, x)
}
Testing it:
fs := []float64{0, 1234.567890123, 0.1234567890123, 123456789012.0, 1234567890123.0,
9.405090880450127e+9, 9.405090880450127e+19, 9.405090880450127e+119}
for _, f := range fs {
fmt.Println(format12(f))
}
Output (try it on the Go Playground):
0.0000000000
0.1234567890
1234.5678901
123456789012
1.234568e+12
9405090880.5
9.405091e+19
9.40509e+119
I'm using the hash function murmur2 which returns me an uint64.
I want then to store it in PostgreSQL, which only support BIGINT (signed 64 bits).
As I'm not interested in the number itself, but just the binary value (as I use it as an id for detecting uniqueness (my set of values being of ~1000 values, a 64bit hash is enough for me) I would like to convert it into int64 by "just" changing the type.
How does one do that in a way that pleases the compiler?
You can simply use a type conversion:
i := uint64(0xffffffffffffffff)
i2 := int64(i)
fmt.Println(i, i2)
Output:
18446744073709551615 -1
Converting uint64 to int64 always succeeds: it doesn't change the memory representation just the type. What may confuse you is if you try to convert an untyped integer constant value to int64:
i3 := int64(0xffffffffffffffff) // Compile time error!
This is a compile time error as the constant value 0xffffffffffffffff (which is represented with arbitrary precision) does not fit into int64 because the max value that fits into int64 is 0x7fffffffffffffff:
constant 18446744073709551615 overflows int64
What is the correct way to store and do arithmetic on currency in Go? There doesn't seem to be a corresponding decimal type and using floats is a big no.
I'd say a way to go is to store amounts of money using properly sized integer type, normalized to the lowest possible amount. Say, if you need to store amounts in US dollars down to one cent, multiply your values by 100 and hence store them in full cents.
Another way is to implement a custom type which would model what is "decimal" in some other languages, that is, it would use two integer numbers to represent amount of money.
This seems like a great opportunity to create a type, which stores the value in a safe and precise integer-based way, but gives you extra behavior you'd want from a decimal type. For instance, a quick implementation might look like this (https://play.golang.org/p/nYbLiadQOc):
// USD represents US dollar amount in terms of cents
type USD int64
// ToUSD converts a float64 to USD
// e.g. 1.23 to $1.23, 1.345 to $1.35
func ToUSD(f float64) USD {
return USD((f * 100) + 0.5)
}
// Float64 converts a USD to float64
func (m USD) Float64() float64 {
x := float64(m)
x = x / 100
return x
}
// Multiply safely multiplies a USD value by a float64, rounding
// to the nearest cent.
func (m USD) Multiply(f float64) USD {
x := (float64(m) * f) + 0.5
return USD(x)
}
// String returns a formatted USD value
func (m USD) String() string {
x := float64(m)
x = x / 100
return fmt.Sprintf("$%.2f", x)
}
The given type behaves the way one might expect, especially given tricky use-cases.
fmt.Println("Product costs $9.09. Tax is 9.75%.")
f := 9.09
t := 0.0975
ft := f * t
fmt.Printf("Floats: %.18f * %.18f = %.18f\n", f, t, ft)
u := ToUSD(9.09)
ut := u.Multiply(t)
fmt.Printf("USD: %v * %v = %v\n", u, t, ut)
Product costs $9.09. Tax is 9.75%.
Floats: 9.089999999999999858 * 0.097500000000000003 = 0.886275000000000035
USD: $9.09 * 0.0975 = $0.89
Rational numbers are quite a good solution for representing money values. That is, a type that has a numerator and a denominator.
Often monetary data structures are overly complex - Java's BigDecimal being an example. A more mathematically-consistent approach is to define a type that handles rational numbers. When 64bit integers are used, a huge range of numbers can be accurately and efficiently represented. Errors and rounding issues are less of a problem than for any solution that needs to convert binary fractions to/from decimal fractions.
Edit: The Go standard library includes arbitrary-precision integers and rational numbers. The Rat type will work well for currency, especially for those cases that require arbitrary precision, e.g. foreign exchange. Here's an example.
Edit 2: I have used the decimal.Decimal Shopspring package extensively. Under the hood, this combines big.Int with an exponent to provide a fixed-point decimal with a nearly-unlimited range of values. The Decimal type is a rational number where the denominator is always a power of ten, which works very well in practice.
There are actually a few packages implementing a decimal type, though there's no clear leader among them.