Advise on panic error writing buffer to io.Writer - go

I'm having some issue with writing []float32 data in stream order to a io.Writer buffer in Go.
I have the following 16 bit per pixel image noise16.jpeg that I open, and read as follows in my test:
func TestNewFITSImageFrom2DDataWriteFloatData(t *testing.T) {
f, err := os.Open("../../images/noise16.jpeg")
if err != nil {
t.Errorf("Error opening image: %s", err)
}
defer f.Close()
img, err := jpeg.Decode(f)
if err != nil {
t.Errorf("Error decoding image: %s", err)
}
bounds := img.Bounds()
ex := make([][]uint32, bounds.Dx())
for x := 0; x < bounds.Dx(); x++ {
col := make([]uint32, bounds.Dy())
ex[x] = col
}
for j := 0; j < bounds.Dy(); j++ {
for i := 0; i < bounds.Dx(); i++ {
r, g, b, _ := img.At(i, j).RGBA()
lum := 0.299*float64(r) + 0.587*float64(g) + 0.114*float64(b)
ex[i][j] = uint32(lum)
}
}
var fit = NewFITSImageFrom2DData(ex, 16, 2, int32(bounds.Dx()), int32(bounds.Dy()))
var w io.Writer
err = writeFloat32Array(w, fit.Data, true)
if err != nil {
t.Errorf("Error writing float32 array: %s", err)
}
}
I essentially open the image, and read in each pixel to an "exposure" array (the []float32 data). I then pass the data to the following method, with an instantiated writer:
// Writes FITS binary body data in network byte order.
// Optionally replaces NaNs with zeros for compatibility with other software
func writeFloat32Array(w io.Writer, data []float32, replaceNaNs bool) error {
bufLen := 16 * 1024
buff := make([]byte, bufLen)
// Process the data in chunks of BUFFER_LENGTH bytes:
for block := 0; block < len(data); block += (bufLen >> 2) {
size := len(data) - block
if size > (bufLen >> 2) {
size = (bufLen >> 2)
}
for offset := 0; offset < size; offset++ {
d := data[block+offset]
if replaceNaNs && math.IsNaN(float64(d)) {
d = 0
}
val := math.Float32bits(d)
buff[(offset<<2)+0] = byte(val >> 24)
buff[(offset<<2)+1] = byte(val >> 16)
buff[(offset<<2)+2] = byte(val >> 8)
buff[(offset<<2)+3] = byte(val)
}
_, err := w.Write(buff[:(size << 2)])
if err != nil {
return err
}
}
// Complete the last partial block, for strictly FITS compliant software
bytesWritten := len(data) << 2
lastPartialBlock := bytesWritten % 2880
if lastPartialBlock != 0 {
sb := strings.Builder{}
for i := lastPartialBlock; i < 2880; i++ {
sb.WriteRune(' ')
}
_, err := w.Write([]byte(sb.String()))
if err != nil {
fmt.Println(err)
return err
}
}
return nil
}
My testing is coming back with a Panic error:
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x119101f]
Which signifies that this line is throwing an error from inside the writeFloat32Array function in the first iteration of the for block := 0; block < len(data); block += (bufLen >> 2) loop:
_, err := w.Write(buff[:(size << 2)])
Could anyone shed some light on what I have possibly done wrong? My instincts are telling me something around bufLen is incorrect, this should probably be the number of bytes in the image, rather than 16384 bytes?
However, when I set a very large number ... I still see the same error?
I have also tried as #jdizzle suggested, and encoded with the binary package but I am still seeing the panic error:
// Writes FITS binary body data in network byte order.
// Optionally replaces NaNs with zeros for compatibility with other software
func writeFloat32Array(w io.Writer, data []float32, replaceNaNs bool) error {
buf := new(bytes.Buffer)
err := binary.Write(buf, binary.BigEndian, data)
if err != nil {
return err
}
_, err = w.Write(buf.Bytes())
if err != nil {
return err
}
return nil
}

Related

Slice automatically be sorted?

While I want to create my own pipeline to practice with goroutines, there's something particularly weird.
I use the random perm function to generate some int numbers, randomly obviously, I write them to IO writer and then read them form IO reader, cuz its binary source so I print them and they are sorted!!
Here's the code:
func RandomSource(tally int) chan int {
out := make(chan int)
sli := rand.Perm(tally)
fmt.Println(sli)
go func() {
for num := range sli {
out <- num
}
close(out)
}()
return out
}
func ReaderSource(reader io.Reader) chan int {
out := make(chan int)
go func() {
buffer := make([]byte, 8)
for ; ; {
n, err := reader.Read(buffer)
if n > 0 {
v := int(binary.BigEndian.Uint64(buffer))
out <- v
}
if err != nil {
break
}
}
close(out)
}()
return out
}
func WriterSink(writer io.Writer, in chan int) {
for v := range in {
buffer := make([]byte, 8)
binary.BigEndian.PutUint64(
buffer, uint64(v))
writer.Write(buffer)
}
}
func main() {
fileName := "small.in"
file, err := os.Create(fileName)
if err != nil {
log.Fatal(err)
}
defer file.Close()
p := RandomSource(500)
WriterSink(file, p)
file, err = os.Open(fileName)
if err != nil {
log.Fatal(err)
}
defer file.Close()
p = ReaderSource(file)
for v := range p {
fmt.Println(v)
}
}
range returns an index as the first value for an array or slice, which always goes from 0 up to len - 1. Use for _, num := range sli { if you want to iterate over the values themselves rather than the set of indices.

Image.At returns nil

I'm having an issue. Here is my code:
package main
import (
"math/rand"
"image/draw"
"image/png"
"image/color"
"strconv"
"os"
"time"
"fmt"
)
func genSites(width, height int) ([][]int) {
rand.Seed(time.Now().Unix())
l, err := strconv.Atoi(os.Args[len(os.Args)-2])
if err != nil {
panic(err)
}
sites := make([][]int, l)
for i := range sites {
sites[i] = make([]int, 2)
sites[i][0] = rand.Intn(width)
sites[i][1] = rand.Intn(height)
}
return sites
}
func main() {
inputF, err := os.Open(os.Args[len(os.Args)-3])
if err != nil {
panic(err)
}
defer inputF.Close()
inputR, err := png.Decode(inputF)
if err != nil {
panic(err)
}
input := inputR.(draw.Image)
minx, miny := input.Bounds().Min.X, input.Bounds().Min.Y
maxx, maxy := input.Bounds().Max.X-1, input.Bounds().Max.Y-1
sites := genSites(maxx-minx, maxy-miny)
siteColors := make([][]color.Color, len(sites))
//todo: figure out something better than this
for i := range siteColors {
siteColors[i] = make([]color.Color, (maxx-minx)*(maxy-miny))
}
siteBelongs := make([][]int, maxx - minx)
for x := range siteBelongs {
siteBelongs[x] = make([]int, maxy - miny)
for y := range siteBelongs[x] {
dmin := (maxx-minx)*(maxx-minx) + (maxy-miny)*(maxy-miny)
var smin int
for i, s := range sites {
d := (s[0]-x)*(s[0]-x) + (s[1]-y)*(s[1]-y)
if d > dmin {
smin = i
dmin = d
}
}
siteBelongs[x][y] = smin
siteColors[smin] = append(siteColors[smin], input.At(x+minx, y+miny))
}
}
siteAvgColors := make([]color.Color, len(sites))
for i := range siteAvgColors {
var sR, sG, sB, sA int
for _, val := range siteColors[i] {
fmt.Println(val)
r, g, b, a := val.RGBA()
sR += int(r)
sG += int(g)
sB += int(b)
sA += int(a)
}
siteAvgColors[i] = color.RGBA{
uint8(sR/len(siteColors[i])),
uint8(sG/len(siteColors[i])),
uint8(sB/len(siteColors[i])),
uint8(sA/len(siteColors[i]))}
}
for x := range siteBelongs {
for y := range siteBelongs[x] {
input.Set(minx + x, miny + y, siteAvgColors[siteBelongs[x][y]])
}
}
output, err := os.Create(os.Args[len(os.Args)-1])
if err != nil {
panic(err)
}
defer output.Close()
err = png.Encode(output, input)
if err != nil {
panic(err)
}
}
The error is this:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4b46e1]
goroutine 1 [running]:
main.main()
/home/victor/programs/goprograms/src/v/imagerandvornoi/main.go:71 +0x7a1
Line 71 is the one that says r, g, b, a := val.RGBA(). That val is inserted at line 63, or siteColors[smin] = append(siteColors[smin], input.At(x+minx, y+miny)), which means that input.At is returning nil. Why? How can I fix this?
This has to do with the semantics of the make() builtin within Go. make is special in that it can take two or three arguments:
make(Type, length) or make(Type, length, capacity). For the former, the length and capacity are set to the same value. If you're going to be assigning values to the slice using append, you're going to want to use the latter form. This is because the two argument form (make(Type, 10)) builds a slice with 10 elements in it already with the zero value, when you use append() it becomes the 11th item.
You're hitting a nil derefernece here because you're using the two-argument version of make with append, and so the first item in the slice is nil.

reader.ReadLine() doesn't advance after a scanner.Scan() call

The code below reads its values from this file:
2 3\n
1.0 2.0 3.0\n
-1.0 -2.0 -3.0\n
And should print:
[ {1 2 3}, {-1 -2 -3} ]
But instead I get this:
[{2 [31 2 3]} {0 []}] strconv.ParseFloat: parsing "3.0-1.0": invalid syntax
It seems that the reader.ReadLine() stays at the same location. Is there a simpler way to scan lines, then values inside each line?
package main
import (
"bufio"
"bytes"
"fmt"
"os"
"strconv"
"strings"
)
type Example struct {
classLabel int
attributes []float64
}
func NewExample(classLabel int, attributes []float64) *Example {
return &Example{classLabel, attributes}
}
func readFile(path string) ([]Example, error) {
var (
result []Example
err error
file *os.File
part []byte
size int
attributeNum int
)
if file, err = os.Open(path); err != nil {
return result, err
}
defer file.Close()
reader := bufio.NewReader(file)
buffer := bytes.NewBuffer(make([]byte, 0))
if part, _, err = reader.ReadLine(); err != nil {
return result, err
}
buffer.Write(part)
newLine := buffer.String()
fmt.Println("newLine=" + newLine)
r := strings.NewReader(newLine)
scanner := bufio.NewScanner(r)
scanner.Split(bufio.ScanWords)
if scanner.Scan() {
size, err = strconv.Atoi(scanner.Text())
if err != nil {
return result, err
}
}
fmt.Println("size=" + strconv.Itoa(size))
if scanner.Scan() {
attributeNum, err = strconv.Atoi(scanner.Text())
if err != nil {
return result, err
}
}
fmt.Println("attributeNum=" + strconv.Itoa(attributeNum))
result = make([]Example, size)
var classLabel int
var attributes []float64
for k := 0; k < size; k++ {
if part, _, err = reader.ReadLine(); err != nil {
return result, err
}
buffer.Write(part)
newLine := buffer.String()
fmt.Println("newLine=" + newLine)
r := strings.NewReader(newLine)
scanner := bufio.NewScanner(r)
scanner.Split(bufio.ScanWords)
if scanner.Scan() {
classLabel, err = strconv.Atoi(scanner.Text())
if err != nil {
return result, err
}
}
fmt.Println("classLabel=" + strconv.Itoa(classLabel))
for i := 0; i < attributeNum; i++ {
var attribute float64
if scanner.Scan() {
attribute, err = strconv.ParseFloat(scanner.Text(), 64)
if err != nil {
return result, err
}
attributes = append(attributes, attribute)
fmt.Println("attribute=" + strconv.FormatFloat(attribute, 'f', -1, 64))
}
}
result[k] = *NewExample(classLabel, attributes)
}
return result, scanner.Err()
}
func main() {
example, err := readFile("test.txt")
fmt.Println(example, err)
}
When you do this inside the for loop:
buffer.Write(part)
newLine := buffer.String()
fmt.Println("newLine=" + newLine)
The next line gets appended to buffer.
That is,
before the loop begins, buffer contains 2 3,
and then after reading 1.0 2.0 3.0,
it gets appended to buffer,
so the content becomes 2 31.0 2.0 3.0,
which you store in newLine.
That's where things start to go sideways.
You probably want to clear the buffer before reading each new line:
buffer.Reset()
buffer.Write(part)
newLine := buffer.String()
fmt.Println("newLine=" + newLine)
But then you will have further problems still, here:
if scanner.Scan() {
classLabel, err = strconv.Atoi(scanner.Text())
if err != nil {
return result, err
}
}
Since the line contains 1.0 2.0 3.0, the strconf.Atoi is going to fail.
I don't understand the purpose of this snippet,
perhaps you can delete it (or comment out).
With the above fixed, you will still have one more problem, on this line:
attributes = append(attributes, attribute)
Since attributes is never reset, it keeps growing.
That is, after the first line, it will contain 1 2 3,
and after the second line it will contain 1 2 3 -1 -2 -3.
You could correct that by moving the declaration of attributes without the outer loop, like this:
var attributes []float64
for i := 0; i < attributeNum; i++ {
var attribute float64
if scanner.Scan() {
attribute, err = strconv.ParseFloat(scanner.Text(), 64)
if err != nil {
return result, err
}
attributes = append(attributes, attribute)
fmt.Println("attribute=" + strconv.FormatFloat(attribute, 'f', -1, 64))
}
}

Go: deep copy slices

I would like to read a slice of strings representing hexadecimal numbers, and decode them to a slice of byte slices ([]string --> [][]byte). This is my code so far:
func (self *algo_t) decode_args(args []string) ([][]byte, error) {
var data [][]byte
for i := uint32(0); i < self.num_args; i++ {
data = make([][]byte, self.num_args)
tmp, err := hex.DecodeString(args[i])
fmt.Printf("i = %d\ttmp = %x\n", i, tmp)
data[i] = make([]byte, len(tmp))
copy(data[i], tmp)
if err != nil {
fmt.Fprintf(os.Stderr, "Error decoding hex string %s: %s\n", args[i], err.Error())
return nil, err
}
}
fmt.Printf("line 69\tdata[0] = %x\tdata[1] = %x\tdata[2] = %x\n",data[0], data[1], data[2])
return data, nil
}
calling this code and passing args = []string{"010203","040506","070809"} yields the following output:
i = 0 tmp = 010203
i = 1 tmp = 040506
i = 3 tmp = 070809
line 69 data[0] = data[1] = data[2] = 070809
Presumably the function returns [][]byte{[]byte{}, []byte{}, []byte{0x07, 0x08, 0x09}}.
I understand that this is because of the pointer behavior of Go; what is the best practice for doing a deep copy of this kind?
For example,
package main
import (
"encoding/hex"
"fmt"
)
// Decode hex []string to [][]byte
func decode(s []string) ([][]byte, error) {
b := make([][]byte, len(s))
for i, ss := range s {
h, err := hex.DecodeString(ss)
if err != nil {
err = fmt.Errorf(
"Error decoding hex string %s: %s\n",
ss, err.Error(),
)
return nil, err
}
b[i] = h
}
return b, nil
}
func main() {
s := []string{"010203", "040506", "070809"}
fmt.Println(s)
b, err := decode(s)
if err != nil {
fmt.Println(err)
} else {
fmt.Println(b)
}
s = []string{"ABCDEF", "012345", "09AF"}
fmt.Println(s)
b, err = decode(s)
if err != nil {
fmt.Println(err)
} else {
fmt.Println(b)
}
s = []string{"01", "123XYZ"}
fmt.Println(s)
b, err = decode(s)
if err != nil {
fmt.Println(err)
} else {
fmt.Println(b)
}
}
Output:
[010203 040506 070809]
[[1 2 3] [4 5 6] [7 8 9]]
[ABCDEF 012345 09AF]
[[171 205 239] [1 35 69] [9 175]]
[01 123XYZ]
Error decoding hex string 123XYZ: encoding/hex: invalid byte: U+0058 'X'
There is a package built specifically to handle deep copy: http://godoc.org/code.google.com/p/rog-go/exp/deepcopy
You can look at the source here: https://code.google.com/p/rog-go/source/browse/exp/deepcopy/deepcopy.go. It covers copying slices and pointers, so it should cover your case.

limitation on bytes.Buffer?

I am trying to gzip a slice of bytes using the package "compress/gzip". I am writing to a bytes.Buffer and I am writing 45976 bytes, when I am trying to uncompress the content using a gzip.reader and then reader function - I find that the not all of the content is recovered. Is there some limitations to bytes.buffer? and is it a way to by pass or alter this? here is my code (edit):
func compress_and_uncompress() {
var buf bytes.Buffer
w := gzip.NewWriter(&buf)
i,err := w.Write([]byte(long_string))
if(err!=nil){
log.Fatal(err)
}
w.Close()
b2 := make([]byte, 80000)
r, _ := gzip.NewReader(&buf)
j, err := r.Read(b2)
if(err!=nil){
log.Fatal(err)
}
r.Close()
fmt.Println("Wrote:", i, "Read:", j)
}
output from testing (with a chosen string as long_string) would give
Wrote: 45976, Read 32768
Continue reading to get the remaining 13208 bytes. The first read returns 32768 bytes, the second read returns 13208 bytes, and the third read returns zero bytes and EOF.
For example,
package main
import (
"bytes"
"compress/gzip"
"fmt"
"io"
"log"
)
func compress_and_uncompress() {
var buf bytes.Buffer
w := gzip.NewWriter(&buf)
i, err := w.Write([]byte(long_string))
if err != nil {
log.Fatal(err)
}
w.Close()
b2 := make([]byte, 80000)
r, _ := gzip.NewReader(&buf)
j := 0
for {
n, err := r.Read(b2[:cap(b2)])
b2 = b2[:n]
j += n
if err != nil {
if err != io.EOF {
log.Fatal(err)
}
if n == 0 {
break
}
}
fmt.Println(len(b2))
}
r.Close()
fmt.Println("Wrote:", i, "Read:", j)
}
var long_string string
func main() {
long_string = string(make([]byte, 45976))
compress_and_uncompress()
}
Output:
32768
13208
Wrote: 45976 Read: 45976
Use ioutil.ReadAll. The contract for io.Reader says it doesn't have to return all the data and there is a good reason for it not to to do with sizes of internal buffers. ioutil.ReadAll works like io.Reader but will read until EOF.
Eg (untested)
import "io/ioutil"
func compress_and_uncompress() {
var buf bytes.Buffer
w := gzip.NewWriter(&buf)
i,err := w.Write([]byte(long_string))
if err!=nil {
log.Fatal(err)
}
w.Close()
r, _ := gzip.NewReader(&buf)
b2, err := ioutil.ReadAll(r)
if err!=nil {
log.Fatal(err)
}
r.Close()
fmt.Println("Wrote:", i, "Read:", len(b2))
}
If the read from gzip.NewReader does not return the whole expected slice. You can just keep re-reading until you have recieved all the data in the buffer.
Regarding you problem where if you re-read the subsequent reads did not append to the end of the slice, but instead at the beginning; the answer can be found in the implementation of gzip's Read function, which includes
208 z.digest.Write(p[0:n])
This will result in an "append" at the beginning of the string.
This can be solves in this manner
func compress_and_uncompress(long_string string) {
// Writer
var buf bytes.Buffer
w := gzip.NewWriter(&buf)
i,err := w.Write([]byte(long_string))
if(err!=nil){
log.Fatal(err)
}
w.Close()
// Reader
var j, k int
b2 := make([]byte, 80000)
r, _ := gzip.NewReader(&buf)
for j=0 ; ; j+=k {
k, err = r.Read(b2[j:]) // Add the offset here
if(err!=nil){
if(err != io.EOF){
log.Fatal(err)
} else{
break
}
}
}
r.Close()
fmt.Println("Wrote:", i, "Read:", j)
}
The result will be:
Wrote: 45976 Read: 45976
Also after testing with a string of 45976 characters i can confirm that the output is in exactly the same manner as the input, where the second part is correctly appended after the first part.
Source for gzip.Read: http://golang.org/src/pkg/compress/gzip/gunzip.go?s=4633:4683#L189

Resources