I have a go code that creates an excel file (using "github.com/tealeg/xlsx") on a daily basis.
The file is then encoded to base64 and sent as an attachment via email, using go-simple-mail.
This works fine 99% of the time, but once in a while the email is sent as a NOOP, with the xlsx file as a string in the email's body.
Here's my code:
func (n *EmailNotifier) verifyServerConnection() error {
err := n.smtpClient.Noop()
if err != nil {
n.smtpClient, err = n.server.Connect()
if err != nil {
return err
}
}
return nil
}
func (n *EmailNotifier) SendFileByEmail(ctx context.Context, addresses []string, file, filename, mailSubject, body string) error {
mailFile := mail.File{
Name: filename,
B64Data: file,
}
err := n.SendEmail(ctx, addresses, mailSubject, body, &mailFile)
if err != nil {
return err
}
return nil
}
func (n *EmailNotifier) SendEmail(ctx context.Context, to []string, mailSubject string, body string, mailFile *mail.File) error {
err := n.verifyServerConnection()
if err != nil {
nl.Log(ctx).WithValues().Error("Error connecting to SMTP server")
return err
}
email := mail.NewMSG()
email.SetFrom(n.server.Username).AddBcc(to...).SetSubject(mailSubject)
email.SetBody(mail.TextPlain, body)
if mailFile != nil {
email.Attach(mailFile)
if email.Error != nil {
nl.Log(ctx).WithError(email.Error).WithValues("attachment error", "noop")
}
}
err = email.Send(n.smtpClient)
if err != nil {
nl.Log(ctx).WithValues("subject", to).WithError(err).Error("Error sending email to")
return err
}
return nil
}
And here is the problematic mail:
NOOP
Subject: xxxxx
Content-Type: multipart/mixed;
boundary=259435eabdfa9a8b5c048944afa23eb1b547d42281bd7269b93eeb559924
Date: Wed, 18 Jan 2023 04:00:14 +0000
Mime-Version: 1.0
From: mailto:xxx#yyy.com
--259435eabdfa9a8b5c048944afa23eb1b547d42281bd7269b93eeb559924
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8
--259435eabdfa9a8b5c048944afa23eb1b547d42281bd7269b93eeb559924
Content-Disposition: attachment;
filename="xxx.xlsx"
Content-Transfer-Encoding: base64
Content-Type: application/octet-stream;
name="xxx.xlsx"
UEsDBBQACAAIAAAAAAAAAAAAAAAAAAAAAAARAAAAZG9jUHJvcHMvY29yZS54bWxs0M9KxDAQx/G7
T1Fy30y6gkhpuzdPCoIKXkMydoP5M2RG23172aJVcM/zyRfy6w9Lis0nVg4lD6rVRjWYXfEhT4N6
eb7b3aqGxWZvY8k4qBOyOoxXvaPOlYqPtRBWCcjNkmLmztGgjiLUAbA7YrKsC2FeUnwrNVlhXeoE
ZN27nRD2xtxAQrHeioVzcEdbUX0nvduS9FHjGvAOMGLCLAytbuHXpiAnwosvfo5/tGBNfBGvl00u
HDY1z7Oer1e3N6aF14f7p/Wru5DPUzlUYw//Bhq/AgAA//9QSwcIO+98GtgAAABsAQAAUEsDBBQA....
Decoding this string to an xlsx file generates a valid file. I also tried logging any attachment errors - nothing in the logs.
also - no open issues about this in go-simple-mail
idea-
I am using NOOP to check the server connection, but I am setting the attachment and the recipient only later in my code. Is there any chance that somehow the recipient is set before the NOOP in verifyServerConnection and it messes it up?
Any other suggestions on how to further explore this issue will be welcomed
Related
When I send a request to get the email body, the Gmail API returns everything but the body data on the payload object.
Things I've tried so far
The "Watch" method is already implemented and working fine
As you can see from the screenshot, the response shows the "snipped", which means that the message get is working, but the body data and the "raw" field is still empty.
I am using the history id correctly (saving the current one to use for subsequent requests)
upgrade all the dependencies to the latest stable version
Am I missing anything?
func GetEmail(srv *gmail.Service, historyId uint64) (string, string) {
hist := getHistory(srv, historyId)
for _, h := range hist.History {
for _, m := range h.MessagesAdded {
id := m.Message.Id
mailContent, err := srv.Users.Messages.Get("me", id).Format("full").Do()
if err != nil {
log.Println("error when getting mail content: ", err)
}
if mailContent != nil {
if mailContent.Payload != nil {
payload := mailContent.Payload.Body
data, err := b64.RawURLEncoding.DecodeString(payload.Data)
if err != nil {
log.Println("error b64 decoding: ", err)
}
body := string(data)
if len(body) > 0 {
subject := getSubject(mailContent)
log.Println("subject ", subject)
return body, subject
}
}
}
}
}
return "No email to process, something's wrong - GetEmail func", ""
}
If you want the RAW message data then you need to use Format("RAW")
func GetEmail(srv *gmail.Service, messageId string) {
gmailMessageResposne, err := srv.Users.Messages.Get("me", messageId).Format("RAW").Do()
if err != nil {
log.Println("error when getting mail content: ", err)
}
if gmailMessageResposne != nil {
decodedData, err := base64.RawURLEncoding.DecodeString(gmailMessageResposne.Raw)
if err != nil {
log.Println("error b64 decoding: ", err)
}
fmt.Printf("- %s\n", decodedData)
}
}
Good question that was fun 😁
How to read a gmail email body with Go?
I'm writing a Go client to create backups via a REST-API. The REST-API Response with a multipart form data to a GET-Request. So the content of the response (type *http.Response) body looks like this:
--1ceb25134a5967272c26c9f3f543e7d26834a5967272c26c9f3f595caf08
Content-Disposition: form-data; name="configuration"; filename="test.gz"
Content-Type: application/x-gzip
...
--1ceb25134a5967272c26c9f3f543e7d26834a5967272c26c9f3f595caf08--
How can I extract the zip file from the response body?
I tried to use the builtin (net/http) methods but these requires an Request struct.
Use the mime/multipart package. Assuming that resp is the *http.Response, use the following code to iterate through the parts.
contentType := resp.Header.Get("Content-Type")
mediaType, params, err := mime.ParseMediaType(contentType)
if err != nil {
log.Fatal(err)
}
if strings.HasPrefix(mediaType, "multipart/") {
mr := multipart.NewReader(resp.Body, params["boundary"])
for {
p, err := mr.NextPart()
if err == io.EOF {
return
}
if err != nil {
log.Fatal(err)
}
// p.FormName() is the name of the element.
// p.FileName() is the name of the file (if it's a file)
// p is an io.Reader on the part
// The following code prints the part for demonstration purposes.
slurp, err := ioutil.ReadAll(p)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Part %q, %q: %q\n", p.FormName(), p.FileName(), slurp)
}
}
The code in the answer handles errors by calling log.Fata. Adjust the error handling to meet the needs of your application.
I'm attempting to use Go to write a utility that authenticates and uploads a file by making a multipart http request to our server. Everything seems to go okay, except the file is not arriving on the server. Looking at it further it appears the multipart in the request is empty. Code and request output below. What am I missing in my Go code?
The Code: (I've changed the URL...)
package main
import (
"net/http"
"mime/multipart"
"strings"
"fmt"
"io/ioutil"
"io"
"os"
"bytes"
"flag"
"encoding/json"
)
var (
filename = flag.String("filename", "", "file to upload")
name = flag.String("name", "", "name to give file on server")
username = flag.String("username", "", "username for authentication")
password = flag.String("password", "", "password for authentication")
)
func main() {
flag.Parse()
// Create multipart
var b bytes.Buffer
w := multipart.NewWriter(&b)
f, _ := os.Open(*filename) //open file to send
defer f.Close()
fw, err := w.CreateFormFile("file", *name) //give file a name
if err != nil {
fmt.Println(err)
}
if _, err := io.Copy(fw, f); err != nil { //copy the file to the multipart buffer
fmt.Println(err)
}
w.Close()
// print the head of the multipart data
bs := b.Bytes()
fmt.Printf("%+v\n\n", string(bs[:1000]))
// Send authentication/login
r, e := http.Post("https://mysite/login", "application/json", strings.NewReader(fmt.Sprintf("{\"username\":\"%s\",\"password\":\"%s\"}", *username, *password)))
if e != nil {
fmt.Println(e)
} else {
// Get the token from the body
type Body struct {
Token string
}
// convert json to get the token
body, _ := ioutil.ReadAll(r.Body)
bd := bytes.NewBuffer(body)
dec := json.NewDecoder(bd)
var m Body
dec.Decode(&m)
// Upload file
req, err := http.NewRequest("POST", "https://mysite/api/apps", &b)
if err != nil {
fmt.Printf("%v\n", err)
}
req.Header.Set("Authentication", fmt.Sprintf("Bearer: %s", m.Token))
req.Header.Set("Content-Type", w.FormDataContentType())
client := &http.Client{}
res, err := client.Do(req)
if err != nil {
fmt.Printf("%v\n", err)
}
// print status and request body
fmt.Println(res.Status)
fmt.Printf("%+v\n", res.Request)
}
}
The first thing I print is the byte buffer, b, which contains the multipart data, everything looks good from here. (it was an xml file)
--83451b003d8e5cc38c0e8f60ad318e522cad4818cf293745c84ec36d26d5
Content-Disposition: form-data; name="file"; filename="snapshot-162224-820-99"
Content-Type: application/octet-stream
<manifest>
<projects>
<project name=........
Next I print the status of the request.
200 OK
Then I printed out the request structure, here is where I saw the MultipartForm was empty.
&{Method:GET URL:https://mysite/home/ Proto: ProtoMajor:0 ProtoMinor:0 Header:map[Authentication:[Bearer: DY0LCJL0g] Content-Type:[multipart/form-data; boundary=83451b003d8e5cc38c0e8f60ad318e522cad4818cf293745c84ec36d26d5] Referer:[http://mysite/home/]] Body:<nil> GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host: Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:0xc42018a360 ctx:<nil>}
I highly doubt if the server really received nothing. The behavior of the printed body having a nil body is expected and documented in http.Response
// Request is the request that was sent to obtain this Response.
// Request's Body is nil (having already been consumed).
// This is only populated for Client requests.
Request *Request
If you want to debug the request body sented, you should either use a mock server or a proxy.
On another note, your code's attempt to login is not going to work. It does not maintain the cookie of login info, so the later requests cannot utilize them.
this is my actual code :
func cURL(cURL string, follow bool) (*string, error) {
var err error
var resp *http.Response
var req *http.Request
var u *url.URL
if u, err = url.Parse(cURL); nil != err {
if logLevel >= 30 {
log.Print("ERROR: ", err)
}
return nil, err
}
if req, err = http.NewRequest("GET", u.Scheme+"://"+u.Host+u.Path, nil); err != nil {
if logLevel >= 40 {
log.Print("ERROR: ", err)
}
return nil, err
}
req.URL.RawQuery = u.RawQuery
client := &http.Client{
Timeout: 3 * time.Second, //timeout connexion
}
if resp, err = client.Do(req); nil != err {
if logLevel >= 50 {
log.Print("ERROR: ", err)
}
return nil, err
}
defer resp.Body.Close()
var body []byte
if body, err = ioutil.ReadAll(resp.Body); nil != err {
if logLevel >= 30 {
log.Print("ERROR: ", err)
}
return nil, err
}
var html string
html = string(body)
return &html, nil
}
same time print this error in terminal, i want to hide this :
2017/09/29 18:19:28 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nServer: AkamaiGHost\r\nMime-Version: 1.0\r\nDate: Fri, 29 Sep 2017 16:18:21 GMT\r\nContent-Type: text/html\r\nContent-Length: 218\r\nExpires: Fri, 29 Sep 2017 16:18:21 GMT\r\n\r\n<HTML><HEAD>\n<TITLE>Request Timeout</TITLE>\n</HEAD><BODY>\n<H1>Request Timeout</H1>\nThe server timed out while waiting for the browser's request.<P>\nReference 
i want to hide this error, not print in terminal, how i can do this ? Where im wrong ? i use this function for check large list urls if have specific keyword, the list is really big 50 milions urls.
That message comes from http.Transport within the http.Client, specifically here
log.Printf("Unsolicited response received on idle HTTP channel starting with %q; err=%v", buf, peekErr)
Unfortunately, unlike http.Server which has an ErrorLog field for overriding the default logger, http.Transport always writes to the standard logger.
To prevent the message from being printed you can use log.SetOutput(io.Discard) or log.SetOutput(ioutil.Discard) before 1.16. Keep in mind this will discard everything written to the standard logger. As JimB mentions, if you still want to log in other places, you can create a custom logger.
I am using net/http library in 'Go' to make an HTTP GET request. In the response, i get 12 headers. But when i run the exact same query through postman, i get 16 headers. One of those missing is 'Content-Encoding'. I understand this must be a CORS issue.
But since i have not set the header Accept-Encoding: gzip in my request, and i am still getting the gzip encoding in response, the Go transport is not automatically decompressing the response for me. So, i need to be able to manually detect the encoding and then decompress it. But, i cannot detect if the 'Content-Encoding' header is missing in the response.
Here is my code where i try to do this:
func calcDistanceAndDurationWithUberApi(originLat float64, originLon float64, destinationLat float64, destinationLon float64) (float64, float64, error) {
endpoint := "https://api.uber.com/v1.2/estimates/price"
parameters := fmt.Sprintf("?start_latitude=%v&start_longitude=%v&end_latitude=%v&end_longitude=%v", originLat, originLon, destinationLat, destinationLon)
req, err := http.NewRequest("GET", endpoint + parameters, nil)
if err != nil {
return 0, 0, err
}
req.Header.Add("Authorization", "Token " + getUberApiKey())
req.Header.Add("Accept-Language", "en_US")
req.Header.Add("Content-Type", "application/json")
httpClient := &http.Client{}
resp, err := httpClient.Do(req)
if err != nil {
return 0, 0, err
}
if resp.StatusCode != 200 {
return 0, 0, errors.NotFound("Response: %v", resp.StatusCode)
}
defer resp.Body.Close()
pretty.Println("- REQUEST: ")
pretty.Println(req)
// Check if server sent gzipped response. Decompress if yes.
var respReader io.ReadCloser
switch resp.Header.Get("Content-Encoding") {
case "gzip":
fmt.Println("Content-Encoding is gzip")
respReader, err = gzip.NewReader(resp.Body)
defer respReader.Close()
default:
fmt.Println("Content-Encoding is Not gzip")
respReader = resp.Body
}
pretty.Println("- RESPONSE HEADER: ")
pretty.Println(resp.Header)
pretty.Println("- RESPONSE BODY: ")
pretty.Println(respReader)
return 0, 0, nil
}
The response status is '200 OK'. Here is the output (Response):
- RESPONSE HEADER:
http.Header{
"Content-Language": {"en"},
"Cache-Control": {"max-age=0"},
"X-Uber-App": {"uberex-nonsandbox", "optimus"},
"Strict-Transport-Security": {"max-age=604800", "max-age=2592000"},
"X-Content-Type-Options": {"nosniff"},
"Date": {"Fri, 19 May 2017 07:52:17 GMT"},
"Content-Geo-System": {"wgs-84"},
"Connection": {"keep-alive"},
"X-Frame-Options": {"SAMEORIGIN"},
"X-Xss-Protection": {"1; mode=block"},
"Server": {"nginx"},
"Content-Type": {"application/json"},
}
- RESPONSE BODY:
&http.gzipReader{
body: &http.bodyEOFSignal{
body: &http.body{
src: &internal.chunkedReader{
r: &bufio.Reader{
buf: {0x48, 0x54, .......... }
I gave in to the stubbornness of the uber api and added another request header, req.Header.Add("Accept-Encoding", "gzip").
Now i am getting the response header "Content-Encoding": "gzip", although i am still getting an undecipherable response body, but that's beyond the scope of this question.
If you don't disable compression [1], and you don't manually request compression with Accept-Encoding: gzip, then what I call "automatic mode" is used. With automatic mode, Go automatically adds Accept-Encoding: gzip, then if server responds Content-Encoding: gzip, Go wrap the response body in a Gzip reader, and removes the Content-Encoding and Content-Length response headers [2]. I disagree with this practice, as the end user is essentially being lied to about what the true response was. Contrast this with cURL, which gives you the pure response, regardless of what you do:
PS C:\> curl -v --compressed https://github.com/manifest.json
< content-encoding: gzip
< content-length: 345
To deal with this, I wrote a wrapper for http.Transport:
package mech
import (
"compress/gzip"
"io"
"net/http"
"strings"
)
type Transport struct { http.Transport }
func (t Transport) RoundTrip(req *http.Request) (*http.Response, error) {
if !t.DisableCompression {
req.Header.Set("Accept-Encoding", "gzip")
}
res, err := t.Transport.RoundTrip(req)
if err != nil {
return nil, err
}
if strings.EqualFold(res.Header.Get("Content-Encoding"), "gzip") {
gz, err := gzip.NewReader(res.Body)
if err != nil {
return nil, err
}
res.Body = readCloser{gz, res.Body}
}
return res, nil
}
type readCloser struct {
io.Reader
io.Closer
}
https://golang.org/pkg/net/http#Transport.DisableCompression
https://github.com/golang/go/blob/go1.16.5/src/net/http/transport.go#L2186-L2192