I am learning go and I have the following code which works fine:
resp, err := http.Get(url) // get the html
...
doc, err := html.Parse(resp.Body) // parse the html page
Now I want to print out the html first then do the parsing:
resp, err := http.Get(url)
...
b, err := ioutil.ReadAll(resp.Body) // this line is added, not working now...
doc, err := html.Parse(resp.Body)
I guess the reason is resp.Body is a reader, I can not call the read twice? Any idea how can I do this correctly? Copy the resp.Body?
Because the client streams the response body from the network, it's not possible to read the body twice.
Read the response to a []byte as you are already doing. Create a io.Reader on the bytes for the HTML parser using bytes.NewReader.
resp, err := http.Get(url)
...
b, err := io.ReadAll(resp.Body)
doc, err := html.Parse(bytes.NewReader(b))
Related
I want to read the response from a net/smtp connection like this:
c, err := smtp.Dial(mailHost + ":25")
if err != nil {
return or, err
}
defer c.Quit()
expectedCode := 2
_, banner, err := c.Text.ReadResponse(expectedCode)
However, readResponse() never returns. When I debug the code with delve, I can see that the buffer has the response, padded with NULs.
Does it still wait for data? How can I fix this?
Thanks!
I've got a (hopefully) simple problem. I'm trying to write the results of a HTTP request to a Gzip file in S3. However when downloading the resultant file from S3, it's just in plain text and not compressed. Below is a snippet of the code (sans bootstrapping). The code builds, lints and runs without error, so I'm not sure where I'm going wrong...any pointers would be greatly appreciated!
r, w := io.Pipe()
gw := gzip.NewWriter(w)
go func() {
defer w.Close()
defer gw.Close()
_, err := gw.Write(httpResponse)
if err != nil {
fmt.Println(“error”)
}
}()
cfg, _ := config.LoadDefaultConfig(context.TODO())
s3Client := s3.NewFromConfig(cfg)
ul := manager.NewUploader(s3Client)
_, err := ul.Upload(context.TODO(), &s3.PutObjectInput{
Bucket: aws.String(bucket),
ContentEncoding: aws.String("gzip"),
Key: aws.String(fileName),
Body: r,
})
if err != nil {
fmt.Println(“error”)
}
This is a side effect of downloading via the browser, it'll decode the gzip but leave the .gz extension (which is frankly confusing). If you use the AWS cli or API to download the file, it will remain as GZIP.
See: https://superuser.com/questions/940605/chromium-prevent-unpacking-tar-gz?noredirect=1&lq=1
I am trying to stream out bytes of a zip file using io.Pipe() function in golang. I am using pipe reader to read the bytes of each file in the zip and then stream those out and use the pipe writer to write the bytes in the response object.
func main() {
r, w := io.Pipe()
// go routine to make the write/read non-blocking
go func() {
defer w.Close()
bytes, err := ReadBytesforEachFileFromTheZip()
err := json.NewEncoder(w).Encode(bytes)
handleErr(err)
}()
This is not a working implementation but a structure of what I am trying to achieve. I don't want to use ioutil.ReadAll since the file is going to be very large and Pipe() will help me avoid bringing all the data into memory. Can someone help with a working implementation using io.Pipe() ?
I made it work using golang io.Pipe().The Pipewriter writes byte to the pipe in chunks and the pipeReader reader from the other end. The reason for using a go-routine is to have a non-blocking write operation while simultaneous reads happen form the pipe.
Note: It's important to close the pipe writer (w.Close()) to send EOF on the stream otherwise it will not close the stream.
func DownloadZip() ([]byte, error) {
r, w := io.Pipe()
defer r.Close()
defer w.Close()
zip, err := os.Stat("temp.zip")
if err != nil{
return nil, err
}
go func(){
f, err := os.Open(zip.Name())
if err != nil {
return
}
buf := make([]byte, 1024)
for {
chunk, err := f.Read(buf)
if err != nil && err != io.EOF {
panic(err)
}
if chunk == 0 {
break
}
if _, err := w.Write(buf[:chunk]); err != nil{
return
}
}
w.Close()
}()
body, err := ioutil.ReadAll(r)
if err != nil {
return nil, err
}
return body, nil
}
Please let me know if someone has another way of doing it.
Sorry if the question is too newbie, as i just started to learn go yesterday.
I try to convert publishEvent into bytes, and compiler shown error like following:
cannot convert publishEvent (type *common.MapStr) to type []byte
Can anyone show me the way ?
Thank You.
var parsed map[string]interface{}
bytes := []byte(publishEvent) --->Error occur here
err := json.Unmarshal(bytes, &parsed)
if err != nil{
fmt.Println("error: ", err)
}
I assume the struct you are working with is common.MapStr from https://github.com/elastic/libbeat
common.MapStr is already a map[string]interface{} so I'm not sure why you are turing it into JSON, and then parsing it back into the same kind of structure, but if thats what you really want to do, replacing the error line with:
bytes, err := json.Marshal(publishEvent)
should work. You will get an error on the next line about redeclaring err so change it to:
err = json.Unmarshal(bytes, &parsed)
Resulting in the following code (also added another error check):
var parsed map[string]interface{}
bytes, err := json.Marshal(publishEvent)
if err != nil{
fmt.Println("error: ", err)
// you'll want to exit or return here since we can't parse `bytes`
}
err = json.Unmarshal(bytes, &parsed)
if err != nil{
fmt.Println("error: ", err)
}
I have stored the last one hour data into file. So I've to upload the previous data to openTSTB.
So, the code is as follows:
go func() {
file, err := os.Open("/var/lib/agent/agent.db")
if err != nil {
fmt.Println(err, "Err")
}
scanner := bufio.NewScanner(file)
for scanner.Scan() {
arr := []byte(scanner.Text())
url := "http://192.168.2.40:4242/api/put"
req, err := http.NewRequest("POST", url, bytes.NewBuffer(arr))
req.Header.Set("Content-Type", "")
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
}
}()
The above code pushes the last one hour data to openTSTB.
Current data is also pushed to openTSTB using another GoRoutine.
The code is as follows:
// Regular run
go func() {
timeStamp, _ := strconv.ParseInt(strconv.FormatInt(time.Now().UnixNano()/1e9, 10), 10, 64)
err := opentsdb.Put(
MetricName,
4,
timeStamp,
opentsdb.Tag{"host", hostname},
)
}()
The problem is if last record is 4, my previous record has been uploaded with the old data [Ex: 4+4].
If I run single GoRoutine, it is working correctly. If I go with old and current data, the result is wrong.
How to fix this? Any help is greatly appreciated. Thanks in advance.