I'm attempting to retrieve the full message body of messages using the Gmail API in Go. Currently when I do so, I only get the first three characters of the message body which are "<ht". I'm pretty sure my issue lies with the decoding of the message body but I can't seem to figure out what I'm doing wrong.
I've looked at examples in several other languages and have tried to translate them to Go with no success. The encoded message body is rather large so I'm fairly certain some data is getting lost somewhere.
Here is an (abridged) code snippet illustrating how I've been attempting to go about this:
req := svc.Users.Messages.List("me").Q("from:someone#somedomain.com,label:inbox")
r, _ := req.Do()
for _, m := range r.Messages {
msg, _ := svc.Users.Messages.Get("me", m.Id).Format("full").Do()
for _, part := range msg.Payload.Parts {
if part.MimeType == "text/html" {
data, _ := base64.StdEncoding.DecodeString(part.Body.Data)
html := string(data)
fmt.Println(html)
}
}
}
Need to use Base64 URL encoding (slightly different alphabet than standard Base64 encoding).
Using the same base64 package, you should use:
base64.URLEncoding.DecodeString instead of
base64.StdEncoding.DecodeString.
To get URL Base64 from standard Base64, replace:
+ to - (char 62, plus to dash)
/ to _ (char 63, slash to underscore)
= to * padding
from body string (source here: Base64 decoding of MIME email not working (GMail API) and here: How to send a message successfully using the new Gmail REST API?).
Related
I have a program that is sending a base64url encoded string, but I read in places that the '\' character isn't supported by base64. My purpose is to send emails with the Gmail API in Go. The body part consists of the following:
"Name: \n\nThis is the body of the email\n\nSincerely,\nSenderName"
When I send emails through the Gmail API, I need to pass it a base64url string.
I have the following function to handle that:
func encodeWeb64String(b []byte) string {
s := base64.URLEncoding.EncodeToString(b)
var i = len(s) - 1
for s[i] == '=' {
i--
}
return s[0 : i+1]
}
I have already added the header information to msg following this post, and have set the content type to text/html; charset=\"utf-8\". I then create the Gmail message using this:
gmsg := gmail.Message{
Raw: encodeWeb64String([]byte(msg)),
}
When the email comes through, it looks like this:
Name: This is the body of the email Sincerely, SenderName
But I want each '\n' to put in actual newline. Thanks for any help, I am new with the Gmail API for Go.
I finally fixed it. I had to change the content type from text/html to text/plain, and now the newlines show properly on the email client
I am trying to send an email message with both an email body and a file attachment (a CSV file) in Go (Golang).
I am following the mime standard of a multi-part message, however I am not very familiar with the structure of the messages following that standard. I am vaguely following a Python code snippet from a colleague as a guide which is using the Python library email (I think this is from the standard library) e.g. MIMEText and MIMEMultipart.
The email message body is not showing up when executing the following Go code:
What is wrong with it?
How can I send an email message with both that file attachment and an email message body?
This function should return a byte slice to be used as a parameter for the call to smtp.SendMail from the Go standard library. See the comments below explaining what's happening to the received email message (the THIS DOES NOT SHOW UP [...] and the THIS ALSO DOES NOT SHOW UP [...]).
func msgWithAttachment(subject, filePath string) ([]byte, error) {
// this is the separator used for the various parts of the MIME message structure
// identified as "boundary"
bPlaceholder := "our-custom-separator"
// the message setup of the common/standard initial part
mime := bytes.NewBuffer(nil)
mime.WriteString(fmt.Sprintf("Subject: %s\r\nMIME-Version: 1.0\r\n", subject))
mime.WriteString(fmt.Sprintf("Content-Type: multipart/mixed; boundary=%s\r\n", bPlaceholder))
// THIS DOES NOT SHOW UP AS THE BODY OF THE EMAIL...
// mime.WriteString("\r\n")
// mime.WriteString(fmt.Sprintf("--%s\r\n", bPlaceholder))
// mime.WriteString("This should be the email message body (v1)...")
// mime.WriteString("\r\n")
// THIS ALSO DOES NOT SHOW UP AS THE BODY OF THE EMAIL...
// BUT IS NEEDED OTHERWISE THE EMAIL MESSAGE SEEMS TO CONTAIN AS ATTACHMENT THE EMAIL MESSAGE ITSELF
// (CONTAINING ITSELF THE REAL ATTACHMENT)
mime.WriteString(fmt.Sprintf("--%s\r\n", bPlaceholder))
mime.WriteString("Content-Type: text/plain; charset=utf-8\r\n")
mime.WriteString("This should be the email message body (v2)...")
// attach a file from the filesystem
_, msgFilename := filepath.Split(filePath)
mime.WriteString(fmt.Sprintf("\n--%s\r\n", bPlaceholder))
mime.WriteString("Content-Type: application/octet-stream\r\n")
mime.WriteString("Content-Description: " + msgFilename + "\r\n")
mime.WriteString("Content-Transfer-Encoding: base64\r\n")
mime.WriteString("Content-Disposition: attachment; filename=\"" + msgFilename + "\"\r\n\r\n")
fileContent, err := ioutil.ReadFile(filePath) // read and encode the content of the file
if err != nil {
return nil, err
}
b := make([]byte, base64.StdEncoding.EncodedLen(len(fileContent)))
base64.StdEncoding.Encode(b, fileContent)
mime.Write(b)
// footer of the email message
mime.WriteString("\r\n--" + bPlaceholder + "--\r\n\r\n")
return mime.Bytes(), nil
}
Coincidentally I had a similar issue the other day. I needed a blank line between the body content type and the beginning of the body itself. Below is the updated lines for this section of your code:
mime.WriteString("Content-Type: text/plain; charset=utf-8\r\n")
mime.WriteString("\r\nThis should be the email message body (v2)...")
For clarity, this newline (\r\n) doesn't have to be exactly here, it can be appended to the content-type line above. It just needs to see a blank line between the content-type and the beginning of the body.
I'm assuming the attachment is being attached without problems, correct? My assumption is this is because you have the double new-line at the end of the content-disposition line before the attachment data is added.
Reading the RFC spec helped me: https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html
Note that the encapsulation boundary must occur at the beginning of a line, i.e., following a CRLF, and that that initial CRLF is considered to be part of the encapsulation boundary rather than part of the preceding part. The boundary must be followed immediately either by another CRLF and the header fields for the next part, or by two CRLFs, in which case there are no header fields for the next part (and it is therefore assumed to be of Content-Type text/plain).
I need to display image with HTTP GET but the thing is i can only use String as the response body.
So for example (headers:image/png, body:Aeacxxffsaf(encoded representation or whatever) )
It's more or less like this web https://codebeautify.org/base64-to-image-converter, but i want the string to output into image when using http GET.
Some code snippets explanations here:
//string that is generated from image (encoded)
encString := "iVBORw0KGgoAAAANSUhEUgAAANIAAAAzCAYAAADigVZl..."
//set http headers to png
//and assign the encString to the body
Is there any way for that? By using string only to serve image
Sorry if my question is a bit confusing but it is the best i can describe it, i have been searching for the answer since several days ago
You do that just like with any other content, just decode the base64 first.
func handler(w http.ResponseWriter, r *http.Request) {
encString := "iVBORw0KGgoAAAANSUhEUgAAANIAAAAzCAYAAADigVZl..."
bytes, err := base64.StdEncoding.DecodeString(encString)
if err != nil {
// todo
}
w.Header().Set("Content-Type", "image/png")
_, err = w.Write(bytes)
if err != nil {
// todo
}
}
However, if you want to display it in browser without decoding, then you will have to do some client-side hacking.
It depends on how you interpret it on the other side.
The encoded b64, b32, byte array, etc. string all represent the same pattern of bytes.
To get it to display as an image on the other side all depends on how you interpret it from the other side.
EDIT:
I see what you mean now. Have a look at the image/jpeg package.
Sadly I don't have a code snippet to share with you right now, but with this you should be able to load your image onto a buffer and decode it to get an image.Image object.
Afterwards you can use that image.Image object to write it into your response body. Sadly I can't provide a code snippet right this second, but do let me know if it works for you.
remember to set your writer's appropriate header.
w.Header().Set("Content-Type", "image/jpeg")
I have created a simple reverse proxy in my main.go as follows:
reverseproxy := httputil.NewSingleHostReverseProxy("https://someurl/someuri")
this is working fine, however, I would like to log the response that comes back from the server. I can also do this utilizing the following code inside the standard RoundTrip method recommended by golang:
response, err := http.DefaultTransport.RoundTrip(request)
dumpresp, err := httputil.DumpResponse(response, true)
if err != nil {
return nil, err
}
log.Printf("%s", dumpresp)
All the above works as expected aside from one thing, the response, when Content-Encoding: gzip, the string appears to the logs as non-utf8 gzip characters. it seems to be a golang oversight but maybe there is something i have missed in the documentation, which I have read to its completion a few times. I can't post the logs here because the characters are non utf8 so they would not display on this site anyway. So, I know what your thinking, just grab the gzip content and use a method to remove the gzip compression. That would be great if the response from DumpResponse was not partly gzip and partly standard utf8 with no way to separate the sections from each other.
So I know what your going to say, why not just take the raw response like the following and gzip decode, and "not" use DumpResponse. Well I can do that as well, but there is an issue with the following:
var reader io.Reader
startreader := httputility.NewChunkedReader(reader)
switch response.Header.Get("Content-Encoding") {
case "gzip":
startreader, err = gzip.NewReader(response.Body)
log.Println("Response body gzip: ")
buf := new(bytes.Buffer)
buf.ReadFrom(startreader)
b := buf.Bytes()
s := *(*string)(unsafe.Pointer(&b))
for name, value := range response.Header {
var hvaluecomp string = ""
for i := 0; i < len(value); i++ {
hvaluecomp += value[i]
}
response.Header.Add(name,hvaluecomp)
}
log.Printf("%s", s)
default:
startreader = response.Body
buf := new(bytes.Buffer)
buf.ReadFrom(startreader)
b := buf.Bytes()
s := *(*string)(unsafe.Pointer(&b))
for name, value := range response.Header {
fmt.Printf("%v: %v\n", name, value)
}
log.Printf("%s", s)
}
The issue with the above is, responses can only be read one time via the reader, after that the response cannot be read by the reverse proxy anymore and it makes the proxy response back to the browser nil =), so again I was met with failure. I cant imagine that the folks coding golang would have missed decoding gzip, just strange, it seems to be so simple.
So in the end, my question is, does DumpResponse give me the ability to decompress gzip so I can log the actual response instead of non utf8 characters? This does the log reader no good when debugging an issue for the production product. This would render the built in golang reverse proxy useless in my eyes and I would start development on my own.
The answer is to copy the stream, then you can use the second variable to re-post the stream back to the request.body object so you don't lose any data.
buf, _ := ioutil.ReadAll(response.Body)
responseuse1 := ioutil.NopCloser(bytes.NewBuffer(buf))
responsehold := ioutil.NopCloser(bytes.NewBuffer(buf))
Method to pull logging: extractLogging(responseuse1)
Push the hold back to the body so its untouched: response.Body = responsehold
return response
does DumpResponse give me the ability to decompress gzip
No it does not.
To me it seems as if the major problem is neither the reverse proxy nor DumpResponse but that you are trying to "log" binary data: Be it gzip, or other binary data like images. Just fix your logging logic: If the raw body is binary you should render some kind of representation or transformation of it. For gziped stuff gunzip it first (but this might still be binary "non utf8" data unsuitable for logging). Focus on the real problem: How to "log" binary data.
After using the protogen tool, I have a message type for sending messages:
type File struct {
Info string `protobuf:"bytes,1,opt,name=info,json=info" json:"info,omitempty"`
BytesValues []byte `protobuf:"bytes,2,opt,name=bytes_values,json=bytesValues,proto3" json:"bytes_values,omitempty"`
}
I am trying to send some binary data using the BytesValues field like so:
filePath := filepath.Join("test", "myfile.bin")
f, _ := ioutil.ReadFile(filePath) // error return value ignored for brevity
msg := File{BytesValues: f}
body, _ := proto.Marshal(msg) // encode
The server seems to have problems decoding the message I am sending to it. Is this the correct way to send binary data using a []byte field with protocol buffers?
In my case, the problem was actually the server not reading the raw bytes from the correct field.
The correct way to send raw bytes is to just set the bytes to the field. There is no need to encode the bytes in any way because protocol buffers is a binary format.
filePath := filepath.Join("test", "myfile.bin")
f, _ := ioutil.ReadFile(filePath) // error return value ignored for brevity
msg := File{BytesValues: f}
body, _ := proto.Marshal(msg) // encode