Since golang is a systems language, it should be capable of doing some user space hardware driver job then.
I want to mmap "/dev/mem" and do some read or writes to some memory mapped pci registers.
Since golang's syscall.Mmap return a byte array. How could I possibly do a 32-bit(for example) read or writes to the registers?
byte by byte access is not approperiate since some registers only support 32 bit access.
You can do this by using unsafe and getting a *uint32 pointer into the block at the right offset. Here's an example for a regular byte array, but an mmap'ed one should work the same.
package main
import (
"fmt"
"unsafe"
)
func main() {
a := make([]byte, 30)
p := (*uint32)(unsafe.Pointer(&a[8]))
*p = 0xabcd0123
fmt.Println(a)
}
Here is an example. You are welcome. ;)
func Readu32(baseAddress int64, offset int64) uint32 {
var value uint32 = 0xFFFFFFFF
const bufferSize int = 4096
file, err := os.Open("/dev/mem")
if err != nil {
log.Fatal(err)
}
defer file.Close()
mmap, err := syscall.Mmap(int(file.Fd()), baseAddress, bufferSize, syscall.PROT_READ, syscall.MAP_SHARED)
if err != nil {
log.Fatal(err)
}
value = *(*uint32)(unsafe.Pointer(&mmap[offset]))
err = syscall.Munmap(mmap)
if err != nil {
log.Fatal(err)
}
return value
}
Related
I want to generate 64-bit random integer using the secure crypto/rand package. I found this online:
package main
import (
"crypto/rand"
"encoding/base64"
)
// GenerateRandomBytes returns securely generated random bytes.
// It will return an error if the system's secure random
// number generator fails to function correctly, in which
// case the caller should not continue.
func GenerateRandomBytes(n int) ([]byte, error) {
b := make([]byte, n)
_, err := rand.Read(b)
// Note that err == nil only if we read len(b) bytes.
if err != nil {
return nil, err
}
return b, nil
}
But it seems to generate random bytes instead. I want a random 64-bit int. Namely, I want something like var i uint64 = rand(). Any ideas how to achieve this?
You can generate a random number with crypto.Rand, and then convert those bytes to an int64 using the binary package:
func randint64() (int64, error) {
var b [8]byte
if _, err := rand.Read(b[:]); err != nil {
return 0, err
}
return int64(binary.LittleEndian.Uint64(b[:])), nil
}
https://play.golang.org/p/2Q8tvttqbJ (result is cached)
If you look at the source code for LittleEndian.Uint64, you can see it's simply performing a few bit operations on the data; something that you could implemented for yourself.
You also use rand.Int in crypto/rand package
func randint64() (int64, error) {
val, err := rand.Int(rand.Reader, big.NewInt(int64(math.MaxInt64)))
if err != nil {
return 0, err
}
return val.Int64(), nil
}
https://play.golang.org/p/fqoQxpmjOSu
I'm trying to use go's net/rpc package to send data structures. The data structure includes a pointer to uint64. The pointer is never nil, but the value may be 0. I'm finding that when the value is 0, the receiver sees a nil pointer. When the value is non-0, the receives sees a non-nil pointer that points to a proper value. This is problematic, because it means that the RPC is breaking an invariant of my data structure: the pointer will never be nil.
I have a go playground that demonstrates this behavior here: https://play.golang.org/p/Un3bTe5F-P
package main
import (
"bytes"
"encoding/gob"
"fmt"
"log"
)
type P struct {
Zero, One int
Ptr *int
}
func main() {
// Initialize the encoder and decoder. Normally enc and dec would be
// bound to network connections and the encoder and decoder would
// run in different processes.
var network bytes.Buffer // Stand-in for a network connection
enc := gob.NewEncoder(&network) // Will write to network.
dec := gob.NewDecoder(&network) // Will read from network.
// Encode (send) the value.
var p P
p.Zero = 0
p.One = 1
p.Ptr = &p.Zero
fmt.Printf("p0: %s\n", p)
err := enc.Encode(p)
if err != nil {
log.Fatal("encode error:", err)
}
// Decode (receive) the value.
var q P
err = dec.Decode(&q)
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Printf("q0: %s\n", q)
p.Ptr = &p.One
fmt.Printf("p1: %s\n", p)
err = enc.Encode(p)
if err != nil {
log.Fatal("encode error:", err)
}
err = dec.Decode(&q)
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Printf("q1: %s\n", q)
}
The output from this code is:
p0: {%!s(int=0) %!s(int=1) %!s(*int=0x1050a780)}
q0: {%!s(int=0) %!s(int=1) %!s(*int=<nil>)}
p1: {%!s(int=0) %!s(int=1) %!s(*int=0x1050a784)}
q1: {%!s(int=0) %!s(int=1) %!s(*int=0x1050aba8)}
So when Ptr points to a 0, it becomes nil on the receiver side. When Ptr points to 1, it is passed through normally.
Is this a bug? Is there a way around this problem? I want to avoid having to unmarshall my detastructure on the receiver side to fix all the unexpected nil pointers...
This behaviour is a limitation of the gob protocol according the defect raised back in 2013 - see https://github.com/golang/go/issues/4609
Bear in mind that gob doesn't send pointers, the pointer is dereferenced and the value is passed. As such when the p.Ptr is set to &p.One, you'll find that q.Ptr != &q.One
I am trying to read a buffered stream of signed 16 bit integers (wav format), but the bufio.Read method only accepts an array of bytes. My question is a 2-parter:
Can I preformat the byte stream into a buffered int16 array?
If I can't, whats the best way of post-processing the byte array into int16 array? My initial thought is to use tmp arrays and keep pushing/processing them, but I was curious if there was a more idiomatic way of doing this?
package main
import (
"bufio"
"io"
"log"
"os/exec"
)
func main() {
app := "someapp"
cmd := exec.Command(app)
stdout, err := cmd.StdoutPipe()
r := bufio.NewReader(stdout)
if err != nil {
log.Fatal(err)
}
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
//"someapp" outputs signed 16bit integers (little endian))
buf := make([]byte, 0, 4*1024)
for {
n, err := r.Read(buf[:cap(buf)]) //r.Read only accepts type []byte
buf = buf[:n]
if n == 0 {
if err == nil {
continue
}
if err == io.EOF {
break
}
log.Fatal(err)
}
log.Printf("%x\n", buf)
//process buf here
if err != nil && err != io.EOF {
log.Fatal(err)
}
}
}
When working with IO, you always work with []bytes, there's no way to substitute that with []int16, or pre-format that as int16s, it's always a stream of bytes.
You can look at the encoding/binary package to decode this stream.
// to get the first uint16 as i
i := binary.LittleEndian.Uint16(buf[:2])
You can then iterate through the buf as needed.
You can also use binary.Read to read directly from the io.Reader.
var i uint16
for {
err := binary.Read(r, binary.LittleEndian, &i)
if err != nil {
log.Println(err)
break
}
fmt.Println(i)
}
It may worth noting the simplicity of what needs to be done. Each uint16 is created via:
func (littleEndian) Uint16(b []byte) uint16 {
return uint16(b[0]) | uint16(b[1])<<8
}
You can use encoding/binary.Read to fill an []int16 directly from your reader, although technically the answer to your first question is still no (check the source of binary.Read, it reads the data to a []byte first).
I am trying to parse a file that annoying consists of many separately zipped segments. I have parsed these segments one at a time into a slice of bytes and I want to uncompress them as I go.
Here is my current code that does the decompressing, which doesn't work. from and to are just set at the top as an example, in reality they are set by the code. data is the byte array containing the entire file. I don't want to seek it while it's on disk because its location on another server, so it's only realistic for me to load the entire file to []byte first and then parse it.
from, to := 0, 1000;
b := bytes.NewReader(data[from:from+to])
z, err := zlib.NewReader(b)
CheckErr(err)
defer z.Close()
p := make([]byte,0,1024)
z.Read(p)
fmt.Println(string(p))
So how is it so massively difficult just to unzip a slice of bytes? Anyway...
The problem appears to with how I am reading it out. Where it says z.Read, that doesn't seem to do anything.
How can I read the entire thing in one go into a slice of bytes?
Here's an outline for you. Note: In Go, CHECK FOR ERRORS!
package main
import (
"bytes"
"compress/zlib"
"fmt"
"io/ioutil"
)
func readSegment(data []byte, from, to int) ([]byte, error) {
b := bytes.NewReader(data[from : from+to])
z, err := zlib.NewReader(b)
if err != nil {
return nil, err
}
defer z.Close()
p, err := ioutil.ReadAll(z)
if err != nil {
return nil, err
}
return p, nil
}
func main() {
from, to := 0, 1000
data := make([]byte, from+to)
// ** parse input segments into data **
p, err := readSegment(data, from, to)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(p))
}
Use ReadAll(r io.Reader) ([]byte, error) from the io/ioutil package.
p, err := ioutil.ReadAll(b)
fmt.Println(string(p))
Read only reads up to the length of the given slice (1024 bytes in your case).
To read in chunks of 1024 bytes:
p := make([]byte,1024)
for {
numBytes, err := l.Read(p)
if err == io.EOF {
// you are done, numBytes might be less than len(p)
break
}
// do what you want with p
}
If you are getting the data from a webserver, you might even do
import (
"net/http"
"io/ioutil"
)
...
resp, errGet := http.Get("http://example.com/somefile")
// do error handling
z, errZ := zlib.NewReader(resp.Body)
// do error handling
resp.Body.Close()
p, err := ioutil.ReadAll(b)
// do error handling
since resp.Body happens to be an io.Reader as most io related types.
I'm wondering if it's possible to count and print the number of bytes downloaded while the file is being downloaded.
out, err := os.Create("file.txt")
defer out.Close()
if err != nil {
fmt.Println(fmt.Sprint(err))
panic(err)
}
resp, err := http.Get("http://example.com/zip")
defer resp.Body.Close()
if err != nil {
fmt.Println(fmt.Sprint(err))
panic(err)
}
n, er := io.Copy(out, resp.Body)
if er != nil {
fmt.Println(fmt.Sprint(err))
}
fmt.Println(n, "bytes ")
If I understand you correctly, you wish to display the number of bytes read, while the data is transferring. Presumably to maintain some kind of a progress bar or something. In which case, you can use Go's compositional data structures to wrap the reader or writer in a custom io.Reader or io.Writer implementation.
It simply forwards the respective Read or Write call to the underlying stream, while doing some additional work with the (int, error) values returned by them. Here is an example you can run on the Go playground.
package main
import (
"bytes"
"fmt"
"io"
"os"
"strings"
)
// PassThru wraps an existing io.Reader.
//
// It simply forwards the Read() call, while displaying
// the results from individual calls to it.
type PassThru struct {
io.Reader
total int64 // Total # of bytes transferred
}
// Read 'overrides' the underlying io.Reader's Read method.
// This is the one that will be called by io.Copy(). We simply
// use it to keep track of byte counts and then forward the call.
func (pt *PassThru) Read(p []byte) (int, error) {
n, err := pt.Reader.Read(p)
pt.total += int64(n)
if err == nil {
fmt.Println("Read", n, "bytes for a total of", pt.total)
}
return n, err
}
func main() {
var src io.Reader // Source file/url/etc
var dst bytes.Buffer // Destination file/buffer/etc
// Create some random input data.
src = bytes.NewBufferString(strings.Repeat("Some random input data", 1000))
// Wrap it with our custom io.Reader.
src = &PassThru{Reader: src}
count, err := io.Copy(&dst, src)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
fmt.Println("Transferred", count, "bytes")
}
The output it generates is this:
Read 512 bytes for a total of 512
Read 1024 bytes for a total of 1536
Read 2048 bytes for a total of 3584
Read 4096 bytes for a total of 7680
Read 8192 bytes for a total of 15872
Read 6128 bytes for a total of 22000
Transferred 22000 bytes
The stdlib now provides something like jimt's PassThru: io.TeeReader. It helps simplify things a bit:
// WriteCounter counts the number of bytes written to it.
type WriteCounter struct {
Total int64 // Total # of bytes transferred
}
// Write implements the io.Writer interface.
//
// Always completes and never returns an error.
func (wc *WriteCounter) Write(p []byte) (int, error) {
n := len(p)
wc.Total += int64(n)
fmt.Printf("Read %d bytes for a total of %d\n", n, wc.Total)
return n, nil
}
func main() {
// ...
// Wrap it with our custom io.Reader.
src = io.TeeReader(src, &WriteCounter{})
// ...
}
playground
The grab Go package implements progress updates (and many other features) for file downloads.
An example of printing progress updates while a download is in process is included in the following walkthrough: http://cavaliercoder.com/blog/downloading-large-files-in-go.html
You can basically call grab.GetAsync which downloads in a new Go routine and then monitor the BytesTransferred or Progress of the returned grab.Response from the calling thread.
Other answers have explained about PassThru. Just provide a full example with callback function base on Dave Jack's answer.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
)
// writeCounter counts the number of bytes written to it.
type writeCounter struct {
total int64 // total size
downloaded int64 // downloaded # of bytes transferred
onProgress func(downloaded int64, total int64)
}
// Write implements the io.Writer interface.
//
// Always completes and never returns an error.
func (wc *writeCounter) Write(p []byte) (n int, e error) {
n = len(p)
wc.downloaded += int64(n)
wc.onProgress(wc.downloaded, wc.total)
return
}
func newWriter(size int64, onProgress func(downloaded, total int64)) io.Writer {
return &writeCounter{total: size, onProgress: onProgress}
}
func main() {
client := http.DefaultClient
url := "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerFun.mp4"
saveTo := "/Users/tin/Desktop/ForBiggerFun.mp4"
download(client, url, saveTo, func(downloaded, total int64) {
fmt.Printf("Downloaded %d bytes for a total of %d\n", downloaded, total)
})
}
func download(client *http.Client, url, filePath string, onProgress func(downloaded, total int64)) (err error) {
// Create file writer
file, err := os.Create(filePath)
if err != nil {
return
}
defer file.Close()
// Determinate the file size
resp, err := client.Head(url)
if err != nil {
return
}
contentLength := resp.Header.Get("content-length")
length, err := strconv.Atoi(contentLength)
if err != nil {
return
}
// Make request
resp, err = client.Get(url)
if err != nil {
return
}
defer resp.Body.Close()
// pipe stream
body := io.TeeReader(resp.Body, newWriter(int64(length), onProgress))
_, err = io.Copy(file, body)
return err
}
Base #Dave Jack
I add progress value and receiving file data from NC (direct TCP data transfer）
// WriteCounter counts the number of bytes written to it.
type WriteCounter struct {
Total int64 // Total # of bytes transferred
Last int64
LastUpdate time.Time
}
// Write implements the io.Writer interface.
//
// Always completes and never returns an error.
func (wc *WriteCounter) Write(p []byte) (int, error) {
n := len(p)
wc.Total += int64(n)
now := time.Now()
duration := now.Sub(wc.LastUpdate).Seconds()
if duration > 1 {
wc.LastUpdate = now
rate := float64(wc.Total-wc.Last) / (duration) / 1024.0
wc.Last = wc.Total
fmt.Printf("Read %d bytes for a total of %d , Rate %.1fKb/s \n", n, wc.Total, rate)
}
return n, nil
}
func Server(dest string) {
outputFile, err := os.Create(dest)
if err != nil {
fmt.Println(err)
}
defer outputFile.Close()
fileWriter := bufio.NewWriter(outputFile)
serverListener, err := net.Listen("tcp", "0.0.0.0:"+PORT)
if err != nil {
fmt.Println(err)
}
defer serverListener.Close()
serverConn, err := serverListener.Accept()
if err != nil {
fmt.Println(err)
}
defer serverConn.Close()
wc := &WriteCounter{}
reader := io.TeeReader(serverConn, wc)
serverConnReader := bufio.NewReaderSize(reader, 32*1024*1024)
io.Copy(fileWriter, serverConnReader)
fileWriter.Flush()
outputFile.Sync()
fmt.Println("Done: Writer")
}