I'm trying to generate a random field of stars using OpenSimplex noise, but I'm noticing repeating patterns appearing and now that I've noticed it, I can't stop noticing it. I think I've found a workaround but I still like to know why this is happening.
At the moment my code generates a 2D greyscale image using OpenSimplex noise, then sets every pixel below a threshold value to 0, so only a small, supposedly random set of "stars" are left.
Here is the generated image with a repeated pattern circled.
I'm currently using this package for the noise implementation: https://github.com/ojrac/opensimplex-go
package main
import (
"flag"
"fmt"
"image"
"image/color"
"image/png"
"math/rand"
"os"
"time"
"github.com/ojrac/opensimplex-go"
)
const (
startupMessage = "starfieldtest"
)
var (
flgSize int
flgSeed int64
flgCullThreshold float64
)
func init() {
rand.Seed(time.Now().UnixNano())
seed := rand.Int63()
flag.IntVar(&flgSize, "size", 2048, "size of the starfield")
flag.Int64Var(&flgSeed, "seed", seed, "random noise seed value")
flag.Float64Var(&flgCullThreshold, "cull", 0.998, "normalised threshold to cull values below")
}
func main() {
fmt.Println(startupMessage)
flag.Parse()
img := generate(flgSize, flgSeed, float32(flgCullThreshold))
outName := fmt.Sprintf("generated_%s.png", time.Now().Format("20060102150405"))
pngFile, err := os.OpenFile(outName, os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
panic(err)
}
defer pngFile.Close()
if err = png.Encode(pngFile, img); err != nil {
panic(err)
}
fmt.Printf("Done!\n")
}
func generate(size int, seed int64, threshold float32) *image.Gray {
noise := opensimplex.NewNormalized32(seed)
pix := image.NewGray(image.Rect(0, 0, size, size))
for y := 0; y < size; y++ {
for x := 0; x < size; x++ {
v := noise.Eval2(float32(x), float32(y))
if v < threshold {
v = 0
}
pix.SetGray(x, y, color.Gray{Y: uint8(v * 255.0)})
}
}
return pix
}
I think I can get around it by using the Eval3 function and changing depth after a certain number of pixels, but is this expected behaviour because it's only really pseudo-random, or is OpenSimplex noise not supposed to do this? I can't really find any statement by anyone knowledgeable one way or the other as to whether this is just a limitation of pseudo-random noise or a problem with the implementation.
EDIT: this still happens when I vary the depth, it seems like repeating patterns exist across all dimensions of the noise. This line however, does work (when I get rid of the dimensionality and treat it as a one long wrapped line of pixels):
v := noise.Eval2((float32(size)*float32(y))+float32(x), 0)
Related
Let's assume I have a very accurate input number(string format), and after math/big manipulation, I want to convert to string format again.
package main
import (
"fmt"
"math/big"
"os"
)
func check(e error) {
if e != nil {
panic(e)
}
}
func main() {
// edit
s := "3.1415926123456789123456789123456789123456789"
var n, _ = new(big.Float).SetString(s)
// var n = big.NewFloat(3.1415926123456789123456789123456789123456789)
fmt.Println(n) // 3.1415926123456788
N := n.String()
fmt.Println(N) // 3.141592612
d1 := []byte(N)
err := os.WriteFile("./dat1.txt", d1, 0644) // 3.141592612
check(err)
}
How to save a big Float like 3.1415926123456789123456789123456789123456789 into a file? I want to keep all the decimal points, or at least as much as possible
You can parse and store your input "precisely", but you must increase the precision (the default precision doesn't cover that). Use Float.SetPrec() for that (requires a bit-count).
When generating text representation, use Float.Text(), again, with sufficiently large precision (requires a decimal digit-count). If you don't know the required digit-precision, as per the doc, you may use a negative value to have the smallest number of decimal digits that is needed for the Float's mantissa bits.
For example:
s := "3.1415926123456789123456789123456789123456789"
fmt.Println(s)
n := big.NewFloat(0)
n.SetPrec(200)
n.SetString(s)
N := n.Text('f', 50)
fmt.Println(N)
N = n.Text('f', -1)
fmt.Println(N)
This will output (try it on the Go Playground):
3.1415926123456789123456789123456789123456789
3.14159261234567891234567891234567891234567890000000
3.1415926123456789123456789123456789123456789
I've just looked what String method does and its documentation (https://pkg.go.dev/math/big#Float.String).
String formats x like x.Text('g', 10)...
So let's go to that method doc.
func (x *Float) Text(format byte, prec int) string
Text converts the floating-point number x to a string according to the given format and precision prec.
The precision prec controls the number of digits ... A negative precision selects the smallest number of decimal digits necessary to identify the value x uniquely
All you need is just reading the doc.
Used a decimal point of 200 as the precision, I need to calculate a number from atto to decimal number similar screenshot.
To get the values at precision of nano and atto you can use %.9f and %.18f in fmt.Printf() respectively,I created a small program to get your value of 0.000000000000099707 as follows:
package main
import (
"fmt"
"math"
)
func main() {
powr := math.Pow(10, -18)
numb := 99707 * powr
fmt.Println("number", numb)
fmt.Printf("\nthe value in atto %.18f\n", numb)
}
Output:
number 9.970700000000001e-14
the value in atto 0.000000000000099707
You can use the github.com/shopspring/decimal package for this as well. This library can represents numbers up to 2^31 (2147483648) digits. Here is a simple code to do the calculation:
d := decimal.NewFromInt(99707)
d10 := decimal.NewFromInt(10)
dpow := decimal.NewFromInt(-18)
d10pow := d10.Pow(dpow)
dmul := d.Mul(d10pow)
fmt.Println(dmul)
This can simplified to:
d := decimal.NewFromInt(99707).Mul(decimal.NewFromInt(10).Pow(decimal.NewFromInt(-18)))
fmt.Println(d)
Output: 0.000000000000099707
See playground
I was interested in how to do this so I found the apd package from cockroach that handles arbitrary precision calculations. You can use it like this:
import (
"fmt"
"github.com/cockroachdb/apd"
)
func main() {
// 99707 * 10^(-18)
n1 := apd.New(99707, 0)
n2 := apd.New(10, 0)
n3 := apd.New(-18, 0)
c := apd.BaseContext.WithPrecision(200)
res := apd.New(0,0)
ctx, err := c.Pow(res, n2, n3)
if err != nil {
panic(err)
}
ctx, err = c.Mul(res, res, n1)
if err != nil {
panic(err)
}
fmt.Println(ctx.Inexact(), res.Text('f'))
}
And it will output:
false 0.000000000000099707
You will have to be careful with the loss of precision that may happen and look at the inexact field.
With following IDL files my intention is to measure the serialization speed of Flatbuffer . I am using golang for my analysis
namespace MyFlat;
struct Vertices {
x : double;
y :double;
}
table Polygon {
polygons : [Vertices];
}
table Layer {
polygons : [Polygon];
}
root_type Layer;
Here is the code I have written for calculation
package main
import (
"MyFlat"
"fmt"
"io/ioutil"
"log"
"strconv"
"time"
flatbuffers "github.com/google/flatbuffers/go"
)
func calculation(size int, vertices int) {
b := flatbuffers.NewBuilder(0)
var polyoffset []flatbuffers.UOffsetT
rawSize := ((16 * vertices) * size) / 1024
var vec1 flatbuffers.UOffsetT
var StartedAtMarshal time.Time
var EndedAtMarshal time.Time
StartedAtMarshal = time.Now()
for k := 0; k < size; k++ {
MyFlat.PolygonStartPolygonsVector(b, vertices)
for i := 0; i < vertices; i++ {
MyFlat.CreateVertices(b, 2.0, 2.4)
}
vec1 = b.EndVector(vertices)
MyFlat.PolygonStart(b)
MyFlat.PolygonAddPolygons(b, vec1)
polyoffset = append(polyoffset, MyFlat.PolygonEnd(b))
}
MyFlat.LayerStartPolygonsVector(b, size)
for _, offset := range polyoffset {
b.PrependUOffsetT(offset)
}
vec := b.EndVector(size)
MyFlat.LayerStart(b)
MyFlat.LayerAddPolygons(b, vec)
finalOffset := MyFlat.LayerEnd(b)
b.Finish(finalOffset)
EndedAtMarshal = time.Now()
SeElaprseTime := EndedAtMarshal.Sub(StartedAtMarshal).String()
mybyte := b.FinishedBytes()
file := "/tmp/myflat_" + strconv.Itoa(size) + ".txt"
if err := ioutil.WriteFile(file, mybyte, 0644); err != nil {
log.Fatalln("Failed to write address book:", err)
}
StartedAt := time.Now()
layer := MyFlat.GetRootAsLayer(mybyte, 0)
size = layer.PolygonsLength()
obj := &MyFlat.Polygon{}
layer.Polygons(obj, 1)
for i := 0; i < obj.PolygonsLength(); i++ {
objVertices := &MyFlat.Vertices{}
obj.Polygons(objVertices, i)
fmt.Println(objVertices.X(), objVertices.Y())
}
EndedAt := time.Now()
DeElapseTime := EndedAt.Sub(StartedAt).String()
fmt.Println(size, ",", vertices, ", ", SeElaprseTime, ",", DeElapseTime, ",", (len(mybyte) / 1024), ",", rawSize)
}
func main() {
data := []int{500000, 1000000, 1500000, 3000000, 8000000}
for _, size := range data {
//calculation(size, 5)
//calculation(size, 10)
calculation(size, 20)
}
}
Problem is I find it serialization is quite slow compared to protobuff with similar idl.
For 3M polygons serialization its taking almost 4.1167037s. Where in protobuf its taking half. Deserilization time for flatbuf is very less (in micro sec). In protobuf its quite high. But still if I add both flatbuf performance is lower.
Do you see any optimized way to serialize it. Flatbuffer is having a method createBinaryVector for byte vector but there is no direct way to serialize vector of polygon from a existing a user defined type vector.
I am adding protobuf code also
syntax = 'proto3';
package myproto;
message Polygon {
repeated double v_x = 1 ;
repeated double v_y = 2 ;
}
message CADData {
repeated Polygon polygon = 1;
string layer_name = 2;
}
Go Code with protobuf
package main
import (
"fmt"
"io/ioutil"
"log"
"math/rand"
"myproto"
"strconv"
"time"
"github.com/golang/protobuf/proto"
)
func calculation(size int, vertices int) {
var comp []*myproto.Polygon
var vx []float64
var vy []float64
for i := 0; i < vertices; i++ {
r := 0 + rand.Float64()*(10-0)
vx = append(vx, r)
vy = append(vy, r/2)
}
rawSize := ((16 * vertices) * size) / 1024
StartedAtMarshal := time.Now()
for i := 0; i < size; i++ {
comp = append(comp, &myproto.Polygon{
VX: vx,
VY: vy,
})
}
pfs := &myproto.CADData{
LayerName: "Layer",
Polygon: comp,
}
data, err := proto.Marshal(pfs)
if err != nil {
log.Fatal("marshaling error: ", err)
}
EndedAtMarshal := time.Now()
SeElaprseTime := EndedAtMarshal.Sub(StartedAtMarshal).String()
file := "/tmp/myproto_" + strconv.Itoa(size) + ".txt"
if err := ioutil.WriteFile(file, data, 0644); err != nil {
log.Fatalln("Failed to write address book:", err)
}
StartedAt := time.Now()
serialized := &myproto.CADData{}
proto.Unmarshal(data, serialized)
EndedAt := time.Now()
DeElapseTime := EndedAt.Sub(StartedAt).String()
fmt.Println(size, ",", vertices, ", ", SeElaprseTime, ",", DeElapseTime, ",", (len(data) / 1024), ",", rawSize)
}
func main() {
data := []int{500000, 1000000, 1500000, 3000000, 8000000}
for _, size := range data {
// calculation(size, 5)
//calculation(size, 10)
calculation(size, 20)
}
}
The time you give, is that for serialization, de-serialization, or both?
Your de-serialization code is likely entirely dominated by fmt.Println. Why don't you instead do sum += objVertices.X() + objVertices.Y() and print sum after timing is done? Can you pull objVertices := &MyFlat.Vertices{} outside of the loop?
You didn't post your protobuf code. Are you including in the timing the time to create the tree of objects which is being serialized (which is required for use in Protobuf but not in FlatBuffers)? Similarly, are you doing the timed (de-)serialization at least a 1000x or so, so you can include the cost of GC (Protobuf allocates a LOT of objects, FlatBuffers allocates few/none) in your comparison?
If after you do the above, it is still slower, post on the FlatBuffers github issues, the authors of the Go port may be able to help further. Make sure you post full code for both systems, and full timings.
Note generally: the design of FlatBuffers is such that it will create the biggest performance gap with Protobuf in C/C++. That said, it should still be a lot faster in Go also. There are unfortunate things about Go however that prevent it from maximizing the performance potential.
b := flatbuffers.NewBuilder(0)
I'm not sure what the "grows automatically" behavior is in Go for flatbuffers, but I'm pretty sure requiring the buffer to grow automatically is not the preferred pattern. Could you try doing your same timing comparison after initializing the buffer with flatbuffers.NewBuilder(moreBytesThanTheMessageNeeds)?
I'm trying to write a short one, which will read a PNG file, and swap one channel with the other (R,G,B) being the possible choices.
I can't find out however, how to extract the integer from the color.Color object returned by image.At(x,y) . Writing it back would probably easier with image.Set(x,y,color) once i can construct the new RGBA color with the swapped channels.
Here I am now (you can pretty much skip to the last loop):
package main
import (
"flag"
"fmt"
//"image"
"image/color"
"image/png"
"os"
)
type Choice struct {
value string
valid bool
}
func (c *Choice) validate() {
goodchoices := []string{"R", "G", "B"}
for _, v := range goodchoices {
if c.value == v {
c.valid = true
}
}
}
func main() {
var fname string
var c1 Choice
var c2 Choice
flag.StringVar(&c1.value, "c1", "", "The color channel to swap - R or G or B ")
flag.StringVar(&c2.value, "c2", "", "The color channel to swap with - R or G or B ")
flag.StringVar(&fname, "f", "", "A .png image (normal map)")
flag.Parse()
c1.validate()
c2.validate()
if c1.valid == true && c2.valid == true {
fmt.Println("We could proceed..")
fmt.Println("Swapping channels:", c1.value, "<->", c2.value, "In", fname) //for testing
} else {
fmt.Println("Invalid channel... Please use R, G or B.")
return
}
file, err := os.Open(fname)
if err != nil {
fmt.Println(err)
return
}
defer file.Close()
pic, err := png.Decode(file)
if err != nil {
fmt.Fprintf(os.Stderr, "%s: %v\n", fname, err)
return
}
b := pic.Bounds()
for y := b.Min.Y; y < b.Max.Y; y++ {
for x := b.Min.X; x < b.Max.X; x++ {
col := pic.At(x, y)
???? How do I swap the channels in col ????
}
}
}
I'm really new to Go and programming in general, so please consider it in your answer. Thank You.
Hmmm, that was harder than I thought it would be - I wonder if anyone can come up with a better idea!
The problem is that you don't know the concrete type that png.Decode returns - it may return any of the image types. You only have an image.Image interface which doesn't have a Set method.
To get round that, first define an interface which all the Image types which can set pixels satisfies
type ImageSet interface {
Set(x, y int, c color.Color)
}
Next see whether pic implements that interface (go will panic if it doesn't - use the picSet, ok form if that bothers you)
// Get an interface which can set pixels
picSet := pic.(ImageSet)
Now your loop looks like this - I only swapped red and green so you can see the idea.
for y := b.Min.Y; y < b.Max.Y; y++ {
for x := b.Min.X; x < b.Max.X; x++ {
col := pic.At(x, y)
r, g, b, a := col.RGBA()
// Swap green and red
newCol := color.RGBA{uint8(g>>8), uint8(r>>8), uint8(b>>8), uint8(a>>8)}
picSet.Set(x, y, newCol)
}
}
I suspect that a high performing version of this would have to use a type switch to determine which image type it was, then have a customized code for each one with uint8s for 24 bit images and uint16s for 48 bit images etc.
Here is the complete working example if you want to have a go. It doesn't work in the playground though - you'll have to download it.
Update: Just noticed your comment. If you know that you have an RGBA image, then you can use a type assertion to get the underlying image which makes things a whole lot easier.
// Get an image.RGBA if it is one
rgba, ok := pic.(*image.RGBA)
if !ok {
fmt.Println("That wasn't an RGBA!")
return
}
for y := b.Min.Y; y < b.Max.Y; y++ {
for x := b.Min.X; x < b.Max.X; x++ {
// Note type assertion to get a color.RGBA
col := rgba.At(x, y).(color.RGBA)
// Swap green and red
col.G, col.R = col.R, col.G
rgba.Set(x, y, col)
}
}
I am trying to generate random numbers (integers) in Go, to no avail. I found the rand package in crypto/rand, which seems to be what I want, but I can't tell from the documentation how to use it. This is what I'm trying right now:
b := []byte{}
something, err := rand.Read(b)
fmt.Printf("something = %v\n", something)
fmt.Printf("err = %v\n", err)
But unfortunately this always outputs:
something = 0
err = <nil>
Is there a way to fix this so that it actually generates random numbers? Alternatively, is there a way to set the upper bound on the random numbers this generates?
Depending on your use case, another option is the math/rand package. Don't do this if you're generating numbers that need to be completely unpredictable. It can be helpful if you need to get results that are reproducible, though -- just pass in the same seed you passed in the first time.
Here's the classic "seed the generator with the current time and generate a number" program:
package main
import (
"fmt"
"math/rand"
"time"
)
func main() {
rand.Seed(time.Now().Unix())
fmt.Println(rand.Int())
}
crypto/rand provides only binary stream of random data, but you can read integers from it using encoding/binary:
package main
import "encoding/binary"
import "crypto/rand"
func main() {
var n int32
binary.Read(rand.Reader, binary.LittleEndian, &n)
println(n)
}
As of 1 april 2012, after the release of the stable version of the lang, you can do the following:
package main
import "fmt"
import "time"
import "math/rand"
func main() {
rand.Seed(time.Now().UnixNano()) // takes the current time in nanoseconds as the seed
fmt.Println(rand.Intn(100)) // this gives you an int up to but not including 100
}
You can also develop your own random number generator, perhaps based upon a simple "desert island PRNG", a Linear Congruential Generator. Also, look up L'Ecuyer (1999), Mersenne Twister, or Tausworthe generator...
https://en.wikipedia.org/wiki/Pseudorandom_number_generator
(Avoid RANDU, it was popular in the 1960's, but the random numbers generated fall on 15 hyperplanes in 3-space).
package pmPRNG
import "errors"
const (
Mersenne31 = 2147483647 // = 2^31-1
Mersenne31Inv = 1.0 / 2147483647.0 // = 4.656612875e-10
// a = 16807
a = 48271
)
// Each stream gets own seed
type PRNGStream struct {
state int
}
func PRNGStreamNew(seed int) *PRNGStream {
prng := (&PRNGStream{})
prng.SetSeed(seed)
return prng
}
// enforce seed in [1, 2^31-1]
func (r*PRNGStream) SetSeed(seed int) error {
var err error
if seed < 1 || seed > Mersenne31 {
err = errors.New("Seed OOB")
}
if seed > Mersenne31 { seed = seed % Mersenne31 }
if seed < 1 { seed = 1 }
r.state = seed
return err
}
// Dig = Park-Miller DesertIslandGenerator
// integer seed in [1, 2^31-1]
func (r*PRNGStream) Dig(seed int) float32 {
xprev := r.state // x[i-1]
xnext := (a * xprev) % Mersenne31 // x[i] = (a*x[i-1])%m
r.state = xnext // x[i-1] = x[i]
Ri := float32(xnext) * Mersenne31Inv // convert Ui to Ri
return Ri
}
func (r*PRNGStream) Rand() float32 {
r.state = (uint64_t)*r.state * Multby % 0x7fffffff
return float32(r.state) * Mersenne31Inv
}
A few relevant links:
https://en.wikipedia.org/wiki/Lehmer_random_number_generator
You might use this function to update your x[i+1], instead of the one above,
val = ((state * 1103515245) + 12345) & 0x7fffffff
(basically, different values of a, c, m)
https://www.redhat.com/en/blog/understanding-random-number-generators-and-their-limitations-linux
https://www.iro.umontreal.ca/~lecuyer/myftp/papers/handstat.pdf
https://www.math.utah.edu/~alfeld/Random/Random.html
https://learn.microsoft.com/en-us/archive/msdn-magazine/2016/august/test-run-lightweight-random-number-generation