I have stored the last one hour data into file. So I've to upload the previous data to openTSTB.
So, the code is as follows:
go func() {
file, err := os.Open("/var/lib/agent/agent.db")
if err != nil {
fmt.Println(err, "Err")
}
scanner := bufio.NewScanner(file)
for scanner.Scan() {
arr := []byte(scanner.Text())
url := "http://192.168.2.40:4242/api/put"
req, err := http.NewRequest("POST", url, bytes.NewBuffer(arr))
req.Header.Set("Content-Type", "")
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
}
}()
The above code pushes the last one hour data to openTSTB.
Current data is also pushed to openTSTB using another GoRoutine.
The code is as follows:
// Regular run
go func() {
timeStamp, _ := strconv.ParseInt(strconv.FormatInt(time.Now().UnixNano()/1e9, 10), 10, 64)
err := opentsdb.Put(
MetricName,
4,
timeStamp,
opentsdb.Tag{"host", hostname},
)
}()
The problem is if last record is 4, my previous record has been uploaded with the old data [Ex: 4+4].
If I run single GoRoutine, it is working correctly. If I go with old and current data, the result is wrong.
How to fix this? Any help is greatly appreciated. Thanks in advance.
Related
I want to read the response from a net/smtp connection like this:
c, err := smtp.Dial(mailHost + ":25")
if err != nil {
return or, err
}
defer c.Quit()
expectedCode := 2
_, banner, err := c.Text.ReadResponse(expectedCode)
However, readResponse() never returns. When I debug the code with delve, I can see that the buffer has the response, padded with NULs.
Does it still wait for data? How can I fix this?
Thanks!
Is it possible to, in a goroutine, stream a file as it is being written to by a subprocess command? The goal here is to capture the output as both a file and stream it live. I have:
cmd := exec.CommandContext(ctx, c.Bin, args...)
// CANT USE NON FILE!!
// https://github.com/golang/go/issues/23019
tempout, err := ioutil.TempFile("", "workerout")
if err != nil {
return "", err
}
tempoutName := tempout.Name()
defer os.Remove(tempoutName) // clean up
temperr, err := ioutil.TempFile("", "workererr")
if err != nil {
return "", err
}
temperrName := temperr.Name()
defer os.Remove(temperrName) // clean up
cmd.Stdout = tempout
cmd.Stderr = temperr
err = cmd.Start()
// Stream the logs
// Does not work. Flushing issue???
/*
ro := bufio.NewReader(tempout)
go func() {
line, _, _ := ro.ReadLine()
logger.Debug(line)
}()
re := bufio.NewReader(temperr)
go func() {
line, _, _ := re.ReadLine()
logger.Error(line)
}()
*/
cmd.Wait()
return tempout.Read(... // read the file into a string and return it
The commented out section of the code seems to show the logs only once the command exits (either by ctx being cancelled, or it finishes), in that it does not log in real time. Is there a way to make this log in real time?
If I'm opening a file inside a for loop and will be finished with it at the end of that iteration, should I call Close immediately or trick Defer using a closure?
I have a series of filenames being read in from a chan string which have data to be copied into a zipfile. This is all being processed in a go func.
go func(fnames <-chan string, zipfilename string) {
f, _ := os.Create(zipfilename) // ignore error handling for this example
defer f.Close()
zf := zip.NewWriter(f)
defer zf.Close()
for fname := range fnames {
r, _ := os.Open(fname)
w, _ := zf.Create(r.Name())
io.Copy(w, r)
w.Close()
r.Close()
}(files, "some name.zip")
Inside my for loop, would it be more idiomatic Go to write:
for fname := range fnames {
func(){
r, _ := os.Open(fname)
defer r.Close()
w, _ := zf.Create(r.Name())
defer w.Close()
io.Copy(w, r)
}()
}
or should I continue with my code as-written?
You should be checking your errors. I know this is meant to just be an example, but in this case it is important. If all you do is defer Close(), you can't actually check if there was an error during defer.
The way I would write this is to create a helper function:
func copyFileToZip(zf *zip.Writer, filename string) error {
r, err := os.Open(filename)
if err != nil {
return err
}
defer r.Close()
w, err := zf.Create(r.Name())
if err != nil {
return err
}
defer w.Close()
_, err = io.Copy(w, r)
if err != nil {
return err
}
return w.Close()
}
Once you add in all that error handling, the function is big enough to make it a named function. It also has the added benefit of checking the error when closing the writer. Checking the reader's error is unnecessary since that won't affect if the data was written.
I am trying to stream out bytes of a zip file using io.Pipe() function in golang. I am using pipe reader to read the bytes of each file in the zip and then stream those out and use the pipe writer to write the bytes in the response object.
func main() {
r, w := io.Pipe()
// go routine to make the write/read non-blocking
go func() {
defer w.Close()
bytes, err := ReadBytesforEachFileFromTheZip()
err := json.NewEncoder(w).Encode(bytes)
handleErr(err)
}()
This is not a working implementation but a structure of what I am trying to achieve. I don't want to use ioutil.ReadAll since the file is going to be very large and Pipe() will help me avoid bringing all the data into memory. Can someone help with a working implementation using io.Pipe() ?
I made it work using golang io.Pipe().The Pipewriter writes byte to the pipe in chunks and the pipeReader reader from the other end. The reason for using a go-routine is to have a non-blocking write operation while simultaneous reads happen form the pipe.
Note: It's important to close the pipe writer (w.Close()) to send EOF on the stream otherwise it will not close the stream.
func DownloadZip() ([]byte, error) {
r, w := io.Pipe()
defer r.Close()
defer w.Close()
zip, err := os.Stat("temp.zip")
if err != nil{
return nil, err
}
go func(){
f, err := os.Open(zip.Name())
if err != nil {
return
}
buf := make([]byte, 1024)
for {
chunk, err := f.Read(buf)
if err != nil && err != io.EOF {
panic(err)
}
if chunk == 0 {
break
}
if _, err := w.Write(buf[:chunk]); err != nil{
return
}
}
w.Close()
}()
body, err := ioutil.ReadAll(r)
if err != nil {
return nil, err
}
return body, nil
}
Please let me know if someone has another way of doing it.
I am learning go and I have the following code which works fine:
resp, err := http.Get(url) // get the html
...
doc, err := html.Parse(resp.Body) // parse the html page
Now I want to print out the html first then do the parsing:
resp, err := http.Get(url)
...
b, err := ioutil.ReadAll(resp.Body) // this line is added, not working now...
doc, err := html.Parse(resp.Body)
I guess the reason is resp.Body is a reader, I can not call the read twice? Any idea how can I do this correctly? Copy the resp.Body?
Because the client streams the response body from the network, it's not possible to read the body twice.
Read the response to a []byte as you are already doing. Create a io.Reader on the bytes for the HTML parser using bytes.NewReader.
resp, err := http.Get(url)
...
b, err := io.ReadAll(resp.Body)
doc, err := html.Parse(bytes.NewReader(b))