I try to do something very simple in Go but I do not manage to find any resources.
I receive an hexadump and I want to write it to a file but the content of both files (src and dst) do not match at all. Currently the only way I have find it's to manually add \x every 2 characters.
I tried to loop over my string and add \x the string looks identical but output is very different.
This code manually works:
binary.Write(f, binary.LittleEndian, []byte("\x00\x00\x00\x04\x0A\xFA\x64\xA7\x00\x03\x31\x30"))
But I did not manage to make it from string "000000040afa64a700033130"...
What i currently do (this is what I do in python3):
text := "000000040afa64a700033130"
j := 0
f, _ := os.OpenFile("gotest", os.O_WRONLY|os.O_CREATE, 0600)
for i := 0; i < len(text); i += 2 {
if (i + 2) <= len(text) {
j = i + 2
}
value, _ := strconv.ParseInt(hex, 16, 8)
binary.Write(f, binary.LittleEndian,value)
s = append(s, value)
}
If your hex data is in the from of a string and you want to write the raw bytes you'll have to convert it first, the easier way would be to use hex.Decode.
import (
"encoding/hex"
"io/ioutil"
)
func foo() {
stringData := []byte("48656c6c6f20476f7068657221")
hexData := make([]byte, hex.DecodedLen(len(stringData)))
_, err := hex.Decode(stringData, hexData)
// handle err
err := ioutil.WriteFile("filename", hexData, 0644)
// handle err
}
Based on your use you could swap over to using ioutil.WriteFile. It writes the given byte slice to a file, creating the file if it doesn't exist or truncating it in the case it already exists.
Related
I write a file uploader with Go. I would like to have md5 of the file as a file name when I save it to the disk.
What is the best way to solve this problem?
I save a file this way:
reader, _ := r.MultipartReader()
p, _ := reader.NextPart()
f, _ := os.Create("./filename") // here I need md5 as a file name
defer f.Close()
lmt := io.LimitReader(p, maxSize + 1)
written, _ := io.Copy(f, lmt)
if written > maxSize {
os.Remove(f.Name())
}
here is an example using io.TeeReader to perform both computation and copy at same time
https://play.golang.org/p/IJJQiaeTOBh
package main
import (
"crypto/sha256"
"fmt"
"io"
"os"
"strings"
)
func main() {
var s io.Reader = strings.NewReader("some data")
// maxSize := 4096
// s = io.LimitReader(s, maxSize + 1)
h := sha256.New()
tr := io.TeeReader(s, h)
io.Copy(os.Stdout, tr)
fmt.Printf("\n%x", h.Sum(nil))
}
// Output:
//some data
//1307990e6ba5ca145eb35e99182a9bec46531bc54ddf656a602c780fa0240dee
And the comparison test for correctness
$ echo -n "some data" | sha256sum -
1307990e6ba5ca145eb35e99182a9bec46531bc54ddf656a602c780fa0240dee -
Instead of using io.TeeReader I have used io.MultiWriter to create 2 buffers (I will use the first buffer to calculate md5 and the second to write to a file with md5 name)
lmt := io.LimitReader(buf, maxSize + 1)
hash := md5.New()
var buf1, buf2 bytes.Buffer
w := io.MultiWriter(&buf1, &buf2)
if _, err := io.Copy(w, lmt); err != nil {
log.Fatal(err)
}
if _, err := io.Copy(hash, &buf1); err != nil {
log.Fatal(err)
}
fmt.Println("md5 is: ", hex.EncodeToString(hash.Sum(nil)))
// Now we can create file with os.Openfile passing md5 name as an argument + write &buf2 to this file
I liked the solution with TeeReader here, but simplified it like this:
type HashReader struct {
io.Reader
hash.Hash
}
func NewHashReader(r io.Reader, h hash.Hash) HashReader {
return HashReader{io.TeeReader(r, h), h}
}
func NewMD5Reader(r io.Reader) HashReader {
return NewHashReader(r, md5.New())
}
func main() {
dataReader := bytes.NewBufferString("Hello, world!")
hashReader := NewMD5Reader(dataReader)
resultBytes := make([]byte, dataReader.Len())
_, err := hashReader.Read(resultBytes)
if err != nil {
fmt.Println(err)
}
fmt.Println(hex.EncodeToString(hashReader.Sum(nil)))
}
hex-encoded string of md5 looks more familiar to me, but feel free to encode result byte array of hashReader.Sum(nil) as you wish.
P.S. One more thing on the playground example. They assign md5 result on EOF, but definitely not all consumers read until EOF. Since Hash object stores current hash calculation, it is enough to call hashReader.Sum after consumption finishes and use the result.
Consider a text file like this:
Some text
here.
---
More text
another line.
---
Third part of text.
I want to split it into three parts, divided by the --- separator. The parts should be stored in a map.
Now, the exact same programs with different types.
When I use string, everything works fine:
KEY: 0
Some text
here.
KEY: 1
More text
another line.
KEY: 2
Third part of text.
https://play.golang.org/p/IcGdoUNcTEe
When I use []byte, things gets messed up:
KEY: 0
Third part of teKEY: 1
Third part of text.
ne.
KEY: 2
Third part of text.
https://play.golang.org/p/jqLhCrqsvOs
Why?
Program 1 (string):
func main() {
parts := parseParts([]byte(input))
for k, v := range parts {
fmt.Printf("KEY: %d\n%s", k, v)
}
}
func parseParts(input []byte) map[int]string {
parts := map[int]string{}
s := bufio.NewScanner(bytes.NewReader(input))
buf := bytes.Buffer{}
i := 0
for s.Scan() {
if s.Text() == "---" {
parts[i] = buf.String()
buf.Reset()
i++
continue
}
buf.Write(s.Bytes())
buf.WriteString("\n")
}
parts[i] = buf.String()
return parts
}
Program 2 ([]byte):
func main() {
parts := parseParts([]byte(input))
for k, v := range parts {
fmt.Printf("KEY: %d\n%s", k, v)
}
}
func parseParts(input []byte) map[int]string {
parts := map[int]string{}
s := bufio.NewScanner(bytes.NewReader(input))
buf := bytes.Buffer{}
i := 0
for s.Scan() {
if s.Text() == "---" {
parts[i] = buf.String()
buf.Reset()
i++
continue
}
buf.Write(s.Bytes())
buf.WriteString("\n")
}
parts[i] = buf.String()
return parts
}
In the string version,
parts[i] = buf.String()
sets parts[i] to a new string every time. In the []byte version,
parts[i] = buf.Bytes()
sets parts[i] to a byte slice backed by the same array every time. The contents of the backing array are the same for all three slices, but the lengths match the length when created, which is why all three slices show the same content but cut off at different places.
You could replace the byte slice line
parts[i] = buf.Bytes()
with something like this:
bb := buf.Bytes()
b := make([]byte, len(bb))
copy(b, bb)
parts[i] = b
in order to get the behavior to match the string version. But the string version is easier and better matches what you seem to be trying to do.
The difference is that bytes.Buffer.String copies the memory, while bytes.Buffer.Bytes does not. Quoting the documentation,
The slice is valid for use only until the next buffer modification (that is, only until the next call to a method like Read, Write, Reset, or Truncate).
I had seen several blurbs on the interwebs which had loosely talked about why one should use bufio.Scanner instead of bufio.Reader.
I don't know if my test case is relevant, but I decided to test one vs the other when it comes to reading 1,000,000 lines from a text file:
package main
import (
"fmt"
"strconv"
"bufio"
"time"
"os"
//"bytes"
)
func main() {
fileName := "testfile.txt"
// Create 1,000,000 integers as strings
numItems := 1000000
startInitStringArray := time.Now()
var input [1000000]string
//var input []string
for i:=0; i < numItems; i++ {
input[i] = strconv.Itoa(i)
//input = append(input,strconv.Itoa(i))
}
elapsedInitStringArray := time.Since(startInitStringArray)
fmt.Printf("Took %s to populate string array.\n", elapsedInitStringArray)
// Write to a file
fo, _ := os.Create(fileName)
for i:=0; i < numItems; i++ {
fo.WriteString(input[i] + "\n")
}
fo.Close()
// Use reader
openedFile, _ := os.Open(fileName)
startReader := time.Now()
reader := bufio.NewReader(openedFile)
for i:=0; i < numItems; i++ {
reader.ReadLine()
}
elapsedReader := time.Since(startReader)
fmt.Printf("Took %s to read file using reader.\n", elapsedReader)
openedFile.Close()
// Use scanner
openedFile, _ = os.Open(fileName)
startScanner := time.Now()
scanner := bufio.NewScanner(openedFile)
for i:=0; i < numItems; i++ {
scanner.Scan()
scanner.Text()
}
elapsedScanner := time.Since(startScanner)
fmt.Printf("Took %s to read file using scanner.\n", elapsedScanner)
openedFile.Close()
}
A pretty average output I receive on the timings looks like this:
Took 44.1165ms to populate string array.
Took 17.0465ms to read file using reader.
Took 23.0613ms to read file using scanner.
I am curious, when is it better to use a reader vs. a scanner, and is it based on performance, or functionality?
It's a flawed benchmark. They are not doing the same thing.
func (b *Reader) ReadLine() (line []byte, isPrefix bool, err error)
returns []byte.
func (s *Scanner) Text() string
returns string([]byte)
To be comparable, use,
func (s *Scanner) Bytes() []byte
It's a flawed benchmark. It reads short strings, the integers from "0\n" to "999999\n". What real-world data set looks like that?
In the real world we read Shakespeare: http://www.gutenberg.org/ebooks/100: Plain Text UTF-8: pg100.txt.
Took 2.973307ms to read file using reader. size: 5340315 lines: 124787
Took 2.940388ms to read file using scanner. size: 5340315 lines: 124787
I'm trying to figure out how to (or if it's possible to) combine multiple assignment and ranges in Golang
ex pseudo code of what I'd like to do
files := [2]*os.File{}
for i, _, fileName := 0, range os.Args[1:3] {
files[i], _ = os.Open(fileName)
}
The idea being I want to have both an iteration counter (i) and the filenames (fileName). I know this can be achieved by using the key from range and some math (key -1), thats not the point of the example.
Edit:
Upon debugging the above example, I learned that i will range 0-1 in that example; Because os.Args[1:2] is a slice and that slice has indexing 0-1 . Therefore I dont need "some math" to properly index the keys.
** EDIT 2: **
This post is also a must read as to why the above [2]*os.File{} is not idiomatic go, instead it should not have a size specified (files := []*os.File{}) so that files is of type slice of *os.File
There are a lot of different issues here. First, range already does what you want. There's no need for even math.
for i, fileName := range os.Args[1:] {
i will range from 0 to 1 here, just like you want. Ranging over a slice always starts at index 0 (it's relative to the start of the slice). (http://play.golang.org/p/qlVM6Y7yPD)
Note that os.Args[1:2] is just one element. You probably meant it to be two.
In any case, this is likely what you really meant:
http://play.golang.org/p/G4yfkKrEe7
files := make([]*os.File, 0)
for _, fileName := range os.Args[1:] {
f, err := os.Open(fileName)
if err != nil {
log.Fatalf("Could not open file: %v", err)
}
files = append(files, f)
}
fmt.Printf("%v\n", files)
Fixed-length arrays are very uncommon in Go. Generally you want a slice, created with make.
For example,
so.go:
package main
import (
"fmt"
"os"
)
func main() {
files := [2]*os.File{}
for i, fileName := range os.Args[1:] {
if i >= len(files) {
break
}
var err error
files[i], err = os.Open(fileName)
if err != nil {
// handle error
}
}
fmt.Println(files)
}
Output:
$ go build so.go && ./so no.go so.go to.go
[<nil> 0xc820030020]
$
I'm trying to parse a string from WebSockets connection in Go language. I'm implementing both sides of the connection, so the specification of data format is depending only on me.
As this is a simple app (generally for learning purposes), I've come up with ActionId Data, where ActionId is a uint8. BackendHandler is a handler for every request in WebSocket Connection.
Platform information
kuba:~$ echo {$GOARCH,$GOOS,`6g -V`}
amd64 linux 6g version release.r60.3 9516
code:
const ( // Specifies ActionId's
SabPause = iota
)
func BackendHandler(ws *websocket.Conn) {
buf := make([]byte, 512)
_, err := ws.Read(buf)
if err != nil { panic(err.String()) }
str := string(buf)
tmp, _ := strconv.Atoi(str[:0])
data := str[2:]
fmt.Println(tmp, data)
switch tmp {
case SabPause:
// Here I get `parsing "2": invalid argument`
// when passing "0 2" to websocket connection
minutes, ok := strconv.Atoui(data)
if ok != nil {
panic(ok.String())
}
PauseSab(uint8(minutes))
default:
panic("Unmatched input for BackendHandler")
}
}
All the output: (note the Println that I used for inspecting)
0 2
panic: parsing "2": invalid argument [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
I couldn't find the code from which this error is launch, only where the error code is defined (dependent on platform). I'd appreciate general ideas for improving my code, but mainly I just want to solve the conversion problem.
Is this related to my buffer -> string conversion and slice-manipulation(I didn't want to use SplitAfter methods)?
Edit
This code reproduces the problem:
package main
import (
"strconv"
"io/ioutil"
)
func main() {
buf , _ := ioutil.ReadFile("input")
str := string(buf)
_, ok := strconv.Atoui(str[2:])
if ok != nil {
panic(ok.String())
}
}
The file input has to contain 0 2\r\n (depending on the file ending, it may look different on other OSes). This code can be fixed by adding the ending index for reslice, this way:
_, ok := strconv.Atoui(str[2:3])
You didn't provide a small compilable and runnable program to illustrate your problem. Nor did you provide full and meaningful print diagnostic messages.
My best guess is that you have a C-style null-terminated string. For example, simplifying your code,
package main
import (
"fmt"
"strconv"
)
func main() {
buf := make([]byte, 512)
buf = []byte("0 2\x00") // test data
str := string(buf)
tmp, err := strconv.Atoi(str[:0])
if err != nil {
fmt.Println(err)
}
data := str[2:]
fmt.Println("tmp:", tmp)
fmt.Println("str:", len(str), ";", str, ";", []byte(str))
fmt.Println("data", len(data), ";", data, ";", []byte(data))
// Here I get `parsing "2": invalid argument`
// when passing "0 2" to websocket connection
minutes, ok := strconv.Atoui(data)
if ok != nil {
panic(ok.String())
}
_ = minutes
}
Output:
parsing "": invalid argument
tmp: 0
str: 4 ; 0 2 ; [48 32 50 0]
data 2 ; 2 ; [50 0]
panic: parsing "2": invalid argument
runtime.panic+0xac /home/peter/gor/src/pkg/runtime/proc.c:1254
runtime.panic(0x4492c0, 0xf840002460)
main.main+0x603 /home/peter/gopath/src/so/temp.go:24
main.main()
runtime.mainstart+0xf /home/peter/gor/src/pkg/runtime/amd64/asm.s:78
runtime.mainstart()
runtime.goexit /home/peter/gor/src/pkg/runtime/proc.c:246
runtime.goexit()
----- goroutine created by -----
_rt0_amd64+0xc9 /home/peter/gor/src/pkg/runtime/amd64/asm.s:65
If you add my print diagnostic statements to your code, what do you see?
Note that your tmp, _ := strconv.Atoi(str[:0]) statement is probably wrong, since str[:0] is equivalent to str[0:0], which is equivalent to the empty string "".
I suspect that your problem is that you are ignoring the n return value from ws.Read. For example (including diagnostic messages), I would expect,
buf := make([]byte, 512)
buf = buf[:cap(buf)]
n, err := ws.Read(buf)
if err != nil {
panic(err.String())
}
fmt.Println(len(buf), n)
buf = buf[:n]
fmt.Println(len(buf), n)
Also, try using this code to set tmp,
tmp, err := strconv.Atoi(str[:1])
if err != nil {
panic(err.String())
}