I need to display image with HTTP GET but the thing is i can only use String as the response body.
So for example (headers:image/png, body:Aeacxxffsaf(encoded representation or whatever) )
It's more or less like this web https://codebeautify.org/base64-to-image-converter, but i want the string to output into image when using http GET.
Some code snippets explanations here:
//string that is generated from image (encoded)
encString := "iVBORw0KGgoAAAANSUhEUgAAANIAAAAzCAYAAADigVZl..."
//set http headers to png
//and assign the encString to the body
Is there any way for that? By using string only to serve image
Sorry if my question is a bit confusing but it is the best i can describe it, i have been searching for the answer since several days ago
You do that just like with any other content, just decode the base64 first.
func handler(w http.ResponseWriter, r *http.Request) {
encString := "iVBORw0KGgoAAAANSUhEUgAAANIAAAAzCAYAAADigVZl..."
bytes, err := base64.StdEncoding.DecodeString(encString)
if err != nil {
// todo
}
w.Header().Set("Content-Type", "image/png")
_, err = w.Write(bytes)
if err != nil {
// todo
}
}
However, if you want to display it in browser without decoding, then you will have to do some client-side hacking.
It depends on how you interpret it on the other side.
The encoded b64, b32, byte array, etc. string all represent the same pattern of bytes.
To get it to display as an image on the other side all depends on how you interpret it from the other side.
EDIT:
I see what you mean now. Have a look at the image/jpeg package.
Sadly I don't have a code snippet to share with you right now, but with this you should be able to load your image onto a buffer and decode it to get an image.Image object.
Afterwards you can use that image.Image object to write it into your response body. Sadly I can't provide a code snippet right this second, but do let me know if it works for you.
remember to set your writer's appropriate header.
w.Header().Set("Content-Type", "image/jpeg")
Related
I'm writing a webhook in Go that parses a JSON payload. I'm attempting to log the raw payload and then decode it immediately after but it fails when I try. If I perform the actions separately, they both work fine independently.
Can someone explain why I can't use ioutil.ReadAll and json.NewDecoder together?
func webhook(w http.ResponseWriter, r *http.Request) {
body, _ := ioutil.ReadAll(r.Body)
log.Printf("incoming message - %s", body)
var p payload
decoder := json.NewDecoder(r.Body)
err := decoder.Decode(&p)
if err != nil {
// Returns EOF
log.Printf("invalid payload - %s", err)
}
defer r.Body.Close()
}
Can someone explain why I can't use ioutil.ReadAll and json.NewDecoder
together?
The request body is an io.ReadCloser that reads bytes, more or less, directly from a network connection. The contents of the Body aren't stored in memory by default. That's why after the first time you've read the Body the next time you try to read it you'll get EOF.
So if you need to process the request Body more than once, you yourself will have to store the contents into memory, which is what you are already doing with:
body, _ := ioutil.ReadAll(r.Body)
You can then reuse body as many times as you like, and since you have the Body contents at your disposal as a []byte value, you can use json.Unmarshal instead of json.NewDecoder(...).Decode.
This is unrelated to your question, but please do not ignore the error returned from ioutil.ReadAll.
Also you can drop the defer r.Body.Close() line, because you do not have to close the request body in your server handlers. (emphasis mine)
For server requests the Request Body is always non-nil but will return
EOF immediately when no body is present. The Server will close the
request body. The ServeHTTP Handler does not need to.
r.Body is meant to be read exactly once.
When you use the ioutil.ReadAll function you do read all the data from the body. That's why the decoder which also relies on r.Body in fact gets nothing to decode.
Minor additional point about json.Decoder and json.Unmarshal: at first glance it looks like the only difference between the two is just that the former operates on a stream and the latter on a []byte, but they actually have different semantics.
json.Unmarshal will return an error if the data contains more than one json object. So, for example, it will parse {}, but it will not parse {}{}.
json.Decoder parses one complete object per call to Decode, so if you give it {}{}, it will parse those two objects and then the third call will return io.EOF and it's More method will return false.
In a normal http body, you probably only want a single object, so you'd want to use Unmarshal if you're not worried about loading all the data into memory at once. You can also use Decoder and manually check that there is only one object if you care to do so.
I have created a simple reverse proxy in my main.go as follows:
reverseproxy := httputil.NewSingleHostReverseProxy("https://someurl/someuri")
this is working fine, however, I would like to log the response that comes back from the server. I can also do this utilizing the following code inside the standard RoundTrip method recommended by golang:
response, err := http.DefaultTransport.RoundTrip(request)
dumpresp, err := httputil.DumpResponse(response, true)
if err != nil {
return nil, err
}
log.Printf("%s", dumpresp)
All the above works as expected aside from one thing, the response, when Content-Encoding: gzip, the string appears to the logs as non-utf8 gzip characters. it seems to be a golang oversight but maybe there is something i have missed in the documentation, which I have read to its completion a few times. I can't post the logs here because the characters are non utf8 so they would not display on this site anyway. So, I know what your thinking, just grab the gzip content and use a method to remove the gzip compression. That would be great if the response from DumpResponse was not partly gzip and partly standard utf8 with no way to separate the sections from each other.
So I know what your going to say, why not just take the raw response like the following and gzip decode, and "not" use DumpResponse. Well I can do that as well, but there is an issue with the following:
var reader io.Reader
startreader := httputility.NewChunkedReader(reader)
switch response.Header.Get("Content-Encoding") {
case "gzip":
startreader, err = gzip.NewReader(response.Body)
log.Println("Response body gzip: ")
buf := new(bytes.Buffer)
buf.ReadFrom(startreader)
b := buf.Bytes()
s := *(*string)(unsafe.Pointer(&b))
for name, value := range response.Header {
var hvaluecomp string = ""
for i := 0; i < len(value); i++ {
hvaluecomp += value[i]
}
response.Header.Add(name,hvaluecomp)
}
log.Printf("%s", s)
default:
startreader = response.Body
buf := new(bytes.Buffer)
buf.ReadFrom(startreader)
b := buf.Bytes()
s := *(*string)(unsafe.Pointer(&b))
for name, value := range response.Header {
fmt.Printf("%v: %v\n", name, value)
}
log.Printf("%s", s)
}
The issue with the above is, responses can only be read one time via the reader, after that the response cannot be read by the reverse proxy anymore and it makes the proxy response back to the browser nil =), so again I was met with failure. I cant imagine that the folks coding golang would have missed decoding gzip, just strange, it seems to be so simple.
So in the end, my question is, does DumpResponse give me the ability to decompress gzip so I can log the actual response instead of non utf8 characters? This does the log reader no good when debugging an issue for the production product. This would render the built in golang reverse proxy useless in my eyes and I would start development on my own.
The answer is to copy the stream, then you can use the second variable to re-post the stream back to the request.body object so you don't lose any data.
buf, _ := ioutil.ReadAll(response.Body)
responseuse1 := ioutil.NopCloser(bytes.NewBuffer(buf))
responsehold := ioutil.NopCloser(bytes.NewBuffer(buf))
Method to pull logging: extractLogging(responseuse1)
Push the hold back to the body so its untouched: response.Body = responsehold
return response
does DumpResponse give me the ability to decompress gzip
No it does not.
To me it seems as if the major problem is neither the reverse proxy nor DumpResponse but that you are trying to "log" binary data: Be it gzip, or other binary data like images. Just fix your logging logic: If the raw body is binary you should render some kind of representation or transformation of it. For gziped stuff gunzip it first (but this might still be binary "non utf8" data unsuitable for logging). Focus on the real problem: How to "log" binary data.
I have a client application which reads in the full body of a http response into a buffer and performs some processing on it:
body, _ = ioutil.ReadAll(containerObject.Resp.Body)
The problem is that this application runs on an embedded device, so responses that are too large fill up the device RAM, causing Ubuntu to kill the process.
To avoid this, I check the content-length header and bypass processing if the document is too large. However, some servers (I'm looking at you, Microsoft) send very large html responses without setting content-length and crash the device.
The only way I can see of getting around this is to read the response body up to a certain length. If it reaches this limit, then a new reader could be created which first streams the in-memory buffer, then continues reading from the original Resp.Body. Ideally, I would assign this new reader to the containerObject.Resp.Body so that callers would not know the difference.
I'm new to GoLang and am not sure how to go about coding this. Any suggestions or alternative solutions would be greatly appreciated.
Edit 1: The caller expects a Resp.Body object, so the solution needs to be compatible with that interface.
Edit 2: I cannot parse small chunks of the document. Either the entire document is processed or it is passed unchanged to the caller, without loading it into memory.
If you need to read part of the response body, then reconstruct it in place for other callers, you can use a combination of an io.MultiReader and ioutil.NopCloser
resp, err := http.Get("http://google.com")
if err != nil {
return err
}
defer resp.Body.Close()
part, err := ioutil.ReadAll(io.LimitReader(resp.Body, maxReadSize))
if err != nil {
return err
}
// do something with part
// recombine the buffered part of the body with the rest of the stream
resp.Body = ioutil.NopCloser(io.MultiReader(bytes.NewReader(part), resp.Body))
// do something with the full Response.Body as an io.Reader
If you can't defer resp.Body.Close() because you intend to return the response before it's read in its entirety, you will need to augment the replacement body so that the Close() method applies to the original body. Rather than using the ioutil.NopCloser as the io.ReadCloser, create your own that refers to the correct method calls.
type readCloser struct {
io.Closer
io.Reader
}
resp.Body = readCloser{
Closer: resp.Body,
Reader: io.MultiReader(bytes.NewReader(part), resp.Body),
}
I searched around and as far as I can tell, POST form requests are already limited to 10MB (http://golang.org/src/net/http/request.go#L721).
If I were to go about reducing this in my ServeHTTP method, I'm not sure how to properly do it. I would try something like this:
r.Body = http.MaxBytesReader(w, r.Body, MaxFileSize)
err := r.ParseForm()
if err != nil {
//redirect to some error page
return
}
But would returning upon error close the connection as well? How would I prevent having to read everything? I found this: https://stackoverflow.com/a/26393261/2202497, but what if content length is not set and in the middle of reading I realize that the file is too big.
I'm using this as a security measure to prevent someone from hogging my server's resources.
The correct way to limit the size of the request body is to do as you suggested:
r.Body = http.MaxBytesReader(w, r.Body, MaxFileSize)
err := r.ParseForm()
if err != nil {
// redirect or set error status code.
return
}
MaxBytesReader sets a flag on the response when the limit is reached. When this flag is set, the server does not read the remainder of the request body and the server closes the connection on return from the handler.
If you are concerned about malicious clients, then you should also set Server.ReadTimeout, Server.WriteTimeout and possibly Server.MaxHeaderBytes.
If you want to set the request body limit for all of your handlers, then wrap root handler with a handler that sets the limit before delegating to the root handler:
type maxBytesHandler struct {
h http.Handler
n int64
}
func (h *maxBytesHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, h.n)
h.h.ServeHTTP(w, r)
}
Wrap the root handler when calling ListenAndServe:
log.Fatal(http.ListenAndServe(":8080", &maxBytesHandler{h:mux, n:4096))
or when configuring a server:
s := http.Server{
Addr: ":8080",
Handler: &maxBytesReader{h:mux, n:4096},
}
log.Fatal(s.ListenAndServe())
There's no need for a patch as suggested in another answer. MaxBytesReader is the official way to limit the size of the request body.
Edit: As others cited MaxByteReader is the supported way. It is interesting that the default reader is instead, limitreader after type asserting for max byte reader.
Submit a patch to the Go source code and make it configurable! You are working with an open source project after all. Adding a setter to http.Request and some unit tests for it is probably only 20 minutes worth of work. Having a hardcoded value here is a bit clunky, give back and fix it :).
You can of course implement your own ParseForm(r *http.Request) method if you really need to override this. Go is essentially BSD, so you can copy paste the library ParseForm and change the limit, but thats a bit ugly no?
I am building a web application.
On one of the pages there is an upload form, where user can upload a file. After the upload is done, I want to check on the server if the uploaded file is an image.
Is it possible to check this beyond simple file extension checking (i.e. not assuming that a *.png filename is actually a PNG image)?
For example, if I edit a JPEG image adding/editing a byte in a random place to make an invalid JPEG file, I want to detect that it is not a JPEG image anymore. I used to do such type of thing via PHP some time ago, using a GD library.
I would like to know if it is possible to do with Go?
DetectContentType is way better than a manual magic number checking. The use is simple:
clientFile, _, _ := r.FormFile("img") // or get your file from a file system
defer clientFile.Close()
buff := make([]byte, 512) // docs tell that it take only first 512 bytes into consideration
if _, err = clientFile.Read(buff); err != nil {
fmt.Println(err) // do something with that error
return
}
fmt.Println(http.DetectContentType(buff)) // do something based on your detection.
Using this method you need to know that you still are not guaranteed to have a correct file. So I would recommend to do some image manipulation with that file (like resize it to make sure this is really an image).
The http package can do this for you:
func DetectContentType(data []byte) string
DetectContentType implements the algorithm described at
http://mimesniff.spec.whatwg.org/ to determine the Content-Type of the
given data. It considers at most the first 512 bytes of data.
DetectContentType always returns a valid MIME type: if it cannot
determine a more specific one, it returns "application/octet-stream".
Code: https://golang.org/src/net/http/sniff.go
What is usually done is checking if the file has the right magic number for the image file format you want. While this test is not super accurate, it is usually good enough. You can use code like this:
package foo
import "strings"
// image formats and magic numbers
var magicTable = map[string]string{
"\xff\xd8\xff": "image/jpeg",
"\x89PNG\r\n\x1a\n": "image/png",
"GIF87a": "image/gif",
"GIF89a": "image/gif",
}
// mimeFromIncipit returns the mime type of an image file from its first few
// bytes or the empty string if the file does not look like a known file type
func mimeFromIncipit(incipit []byte) string {
incipitStr := []byte(incipit)
for magic, mime := range magicTable {
if strings.HasPrefix(incipitStr, magic) {
return mime
}
}
return ""
}