Deleting bytes from a io.ReadWriteSeeker(via file) - go

Let's say I have an io.ReadWriteSeeker that is reading an writing from a certain file.
At some point I decide I want to remove some bytes from the file (specifically the end) and reduce its length.
Whats the right way to go about this?
I can't just overwrite it by seeking because I want to end up with a shorter file.
io.CopyN will similarly not work (as src is smaller than dest)

If the io.ReadWriteSeeker value also supports a Truncate method, like an *os.File value.
For example,
package main
import (
"fmt"
"io"
"io/ioutil"
"os"
)
func truncate(rws io.ReadWriteSeeker, size int64) error {
type Truncater interface {
Truncate(size int64) error
}
t, ok := rws.(Truncater)
if !ok {
return fmt.Errorf("truncate: unable to truncate")
}
return t.Truncate(size)
}
func main() {
filename := `/tmp/truncate.test.file`
f, err := os.Create(filename)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
defer f.Close()
n, err := f.Write([]byte("test data"))
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
data, err := ioutil.ReadFile(filename)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
fmt.Println(string(data))
// truncate io.ReadWriteSeeker
err = truncate(f, int64(n-1))
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
data, err = ioutil.ReadFile(filename)
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
fmt.Println(string(data))
os.Remove(filename)
}
Playground: https://play.golang.org/p/pp4IUSoKo4M
Output:
test data
test dat

Related

Creating/Writing/Reading CSV file in Golang

The following code should create a CSV file and write the employee data to it, then read it out in the terminal. The code will create the file and not print the contents in the terminal. If I take the last chunk of code out to print the contents, it will work as an independent program as long as the CSV file is present. Why does the code not work when written all as one?
package main
import (
"encoding/csv"
"fmt"
"os"
"sort"
)
type Employee struct {
Name string
Age int
Salary float64
}
func main() {
employees := []Employee{
{"Fred", 44, 56000.00},
{"Amy", 25, 65000.00},
{"Zack", 29, 20400.00},
{"Jerry", 73, 120500.00},
}
sort.Slice(employees, func(i, j int) bool {
return employees[i].Name < employees[j].Name
})
file, err := os.Create("employees.csv")
if err != nil {
panic(err)
}
defer file.Close()
writer := csv.NewWriter(file)
defer writer.Flush()
header := []string{"Name", "Age", "Salary"}
if err := writer.Write(header); err != nil {
panic(err)
}
for _, employee := range employees {
record := []string{employee.Name, fmt.Sprintf("%d", employee.Age), fmt.Sprintf("%.2f", employee.Salary)}
if err := writer.Write(record); err != nil {
panic(err)
}
}
file, err = os.Open("employees.csv")
if err != nil {
panic(err)
}
defer file.Close()
reader := csv.NewReader(file)
records, err := reader.ReadAll()
if err != nil {
panic(err)
}
for _, record := range records {
fmt.Println(record)
}
}
//This code will print the contents of the CSV correctly when ran separately
package main
import (
"encoding/csv"
"fmt"
"os"
)
func main() {
file, err := os.Open("employees.csv")
if err != nil {
fmt.Println("Error opening file:", err)
return
}
defer file.Close()
reader := csv.NewReader(file)
reader.FieldsPerRecord = -1
records, err := reader.ReadAll()
if err != nil {
fmt.Println("Error reading CSV data:", err)
return
}
for _, record := range records {
fmt.Println(record)
}
}
The first program will create the CSV file with the expected cells, but instead of printing the contents to the terminal, the program completes.

Loading CSV file into bigquery after os.Create() doesn't load data

I'm trying to run the following flow:
Get data from somewhere
Create new local CSV file, write the data into that file
Upload the CSV to Bigquery
Delete the local file
But it seems to load empty data.
This is the code:
func (c *Client) Do(ctx context.Context) error {
bqClient, err := bigquerypkg.NewBigQueryUtil(ctx, "projectID", "datasetID")
if err != nil {
return err
}
data, err := c.GetSomeData(ctx)
if err != nil {
return err
}
file, err := os.Create("example.csv")
if err != nil {
return err
}
defer file.Close()
// also file need to be delete
writer := csv.NewWriter(file)
defer writer.Flush()
timestamp := time.Now().UTC().Format("2006-01-02 03:04:05.000000000")
for _, d := range data {
csvRow := []string{
d.ID,
d.Name,
timestamp,
}
err = writer.Write(csvRow)
if err != nil {
log.Printf("error writing data to CSV: %v\n", err)
}
}
source := bigquery.NewReaderSource(file)
source.Schema = bigquery.Schema{
{Name: "id", Type: bigquery.StringFieldType},
{Name: "name", Type: bigquery.StringFieldType},
{Name: "createdAt", Type: bigquery.TimestampFieldType},
}
if _, err = bqClient.LoadCsv(ctx, "tableID", source); err != nil {
return err
}
return nil
}
LoadCSV() looks like this:
func (c *Client) LoadCsv(ctx context.Context, tableID string, src bigquery.LoadSource) (string, error) {
loader := c.bigQueryClient.Dataset(c.datasetID).Table(tableID).LoaderFrom(src)
loader.WriteDisposition = bigquery.WriteTruncate
job, err := loader.Run(ctx)
if err != nil {
return "", err
}
status, err := job.Wait(ctx)
if err != nil {
return job.ID(), err
}
if status.Err() != nil {
return job.ID(), fmt.Errorf("job completed with error: %v", status.Err())
}
return job.ID(), nil
}
After running this, bigquery does create the schema but with no data.
If I'm changing os.Create() to os.Open() and the file already exist, everything work. It's like when loading the CSV the file data is not yet written (?)
What's the reason?
The problem I see here is that you don't rewind the file handle's cursor to the beginning of the file. Thus, the next read will be at the end of the file, and will be a 0 byte read. That explains why it seems like there's no content in the file.
https://pkg.go.dev/os#File.Seek can handle this for you.
Actually, the Flush is not relevant, because you're using the same file handle to read the file than you did to write it, so you'll see your own written bytes even without a flush. This would not be the case if the file was opened by a different process or was reopened.
Edit: OP Claims this flush was necessary in their case and I cannot provide evidence to disagree. Flush will not hurt things either.
Demonstration:
package main
import (
"fmt"
"io"
"os"
)
func main() {
f, err := os.CreateTemp("", "data.csv")
if err != nil {
panic(err)
} else {
defer f.Close()
defer os.Remove(f.Name())
}
fmt.Fprintf(f, "hello, world")
fmt.Fprintln(os.Stderr, "Before rewind: ")
if _, err := io.Copy(os.Stderr, f); err != nil {
panic(err)
}
f.Seek(0, io.SeekStart)
fmt.Fprintln(os.Stderr, "\nAfter rewind: ")
if _, err := io.Copy(os.Stderr, f); err != nil {
panic(err)
}
fmt.Fprintln(os.Stderr, "\n")
}
% go run t.go
Before rewind:
After rewind:
hello, world

reading golang websocket returns random bytes

My program:
package main
import (
"fmt"
"io"
"log"
"net"
"github.com/gobwas/ws"
)
func HandleConn(conn net.Conn) {
for {
header, err := ws.ReadHeader(conn)
if err != nil {
log.Fatal(err)
}
buf := make([]byte, header.Length)
_, err = io.ReadFull(conn, buf)
if err != nil {
log.Fatal(err)
}
fmt.Println(buf)
fmt.Println(string(buf))
}
}
func main() {
ln, err := net.Listen("tcp", "localhost:8080")
if err != nil {
log.Fatal(err)
}
for {
conn, err := ln.Accept()
if err != nil {
log.Fatal(err)
}
_, err = ws.Upgrade(conn)
if err != nil {
log.Fatal(err)
}
go HandleConn(conn)
}
}
I do in browser console:
let socket = new WebSocket("ws://127.0.0.1:8080")
socket.send("Hello world")
I see random bytes in the my terminal. Each call to socket.send("Hello world") return different bytes. But the length of the byte array is always equal to the length of the string. Where does golang get these random bytes? How can I fix this? My program is an example from the docs.
If you are going to not use the wsutil you need to unmask the payload:
buff := make([]byte, header.Length)
_, err = io.ReadFull(conn, buff)
if err != nil {
// handle error
}
if header.Masked {
ws.Cipher(buff, header.Mask, 0)
}
fmt.Println(string(buff))

Golang: file extracted from tar throws permissions error

I've written the following code to tar a file, code works but strangely if I untar the archive the file permissions are gone so I can't read it unless I then chmod the file:
package main
import (
"archive/tar"
"io/ioutil"
"log"
"os"
)
func main() {
c, err := os.Create("/path/to/tar/file/test.tar")
if err != nil {
log.Fatalln(err)
}
tw := tar.NewWriter(c)
f, err := os.Open("sample.txt")
if err != nil {
log.Fatalln(err)
}
fi, err := f.Stat()
if err != nil {
log.Fatalln(err)
}
hdr := &tar.Header{Name: f.Name(),
Size: fi.Size(),
}
if err := tw.WriteHeader(hdr); err != nil {
log.Fatalln(err)
}
r, err := ioutil.ReadFile("sample.txt")
if err != nil {
log.Fatalln(err)
}
if _, err := tw.Write(r); err != nil {
log.Fatalln(err)
}
if err := tw.Close(); err != nil {
log.Fatalln(err)
}
}
Any idea what I'm doing wrong?
You're not preserving the original permissions of the file. You're manually creating a header, and specifying only the name and size. Instead, use tar.FileInfoHeader to build the header.
package main
import (
"archive/tar"
"io/ioutil"
"log"
"os"
)
func main() {
c, err := os.Create("/path/to/tar/file/test.tar")
if err != nil {
log.Fatalln(err)
}
tw := tar.NewWriter(c)
f, err := os.Open("sample.txt")
if err != nil {
log.Fatalln(err)
}
fi, err := f.Stat()
if err != nil {
log.Fatalln(err)
}
// create header from FileInfo
hdr, err := tar.FileInfoHeader(fi, "")
if err != nil {
log.Fatalln(err)
}
if err := tw.WriteHeader(hdr); err != nil {
log.Fatalln(err)
}
// instead of reading the whole file into memory, prefer io.Copy
r, err := io.Copy(tw, f)
if err != nil {
log.Fatalln(err)
}
log.Printf("Wrote %d bytes\n", r)
}
Also note that I used io.Copy to copy data from the file (an io.Reader) to the tar writer (an io.Writer). This will work much better for larger files.
Also - pay special attention to this note from the docs:
Because os.FileInfo's Name method returns only the base name of the file it describes, it may be necessary to modify the Name field of the returned header to provide the full path name of the file.
In this simple example, you're just using sample.txt so you shouldn't run into trouble. If you wanted to preserve a directory structure in your tar, you may have to modify the Name field in the header.

How can I efficiently download a large file using Go?

Is there a way to download a large file using Go that will store the content directly into a file instead of storing it all in memory before writing it to a file? Because the file is so big, storing it all in memory before writing it to a file is going to use up all the memory.
I'll assume you mean download via http (error checks omitted for brevity):
import ("net/http"; "io"; "os")
...
out, err := os.Create("output.txt")
defer out.Close()
...
resp, err := http.Get("http://example.com/")
defer resp.Body.Close()
...
n, err := io.Copy(out, resp.Body)
The http.Response's Body is a Reader, so you can use any functions that take a Reader, to, e.g. read a chunk at a time rather than all at once. In this specific case, io.Copy() does the gruntwork for you.
A more descriptive version of Steve M's answer.
import (
"os"
"net/http"
"io"
)
func downloadFile(filepath string, url string) (err error) {
// Create the file
out, err := os.Create(filepath)
if err != nil {
return err
}
defer out.Close()
// Get the data
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
// Check server response
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("bad status: %s", resp.Status)
}
// Writer the body to file
_, err = io.Copy(out, resp.Body)
if err != nil {
return err
}
return nil
}
The answer selected above using io.Copy is exactly what you need, but if you are interested in additional features like resuming broken downloads, auto-naming files, checksum validation or monitoring progress of multiple downloads, checkout the grab package.
Here is a sample. https://github.com/thbar/golang-playground/blob/master/download-files.go
Also I give u some codes might help you.
code:
func HTTPDownload(uri string) ([]byte, error) {
fmt.Printf("HTTPDownload From: %s.\n", uri)
res, err := http.Get(uri)
if err != nil {
log.Fatal(err)
}
defer res.Body.Close()
d, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Fatal(err)
}
fmt.Printf("ReadFile: Size of download: %d\n", len(d))
return d, err
}
func WriteFile(dst string, d []byte) error {
fmt.Printf("WriteFile: Size of download: %d\n", len(d))
err := ioutil.WriteFile(dst, d, 0444)
if err != nil {
log.Fatal(err)
}
return err
}
func DownloadToFile(uri string, dst string) {
fmt.Printf("DownloadToFile From: %s.\n", uri)
if d, err := HTTPDownload(uri); err == nil {
fmt.Printf("downloaded %s.\n", uri)
if WriteFile(dst, d) == nil {
fmt.Printf("saved %s as %s\n", uri, dst)
}
}
}

Resources