Convert base64 string to JPG - go
I am getting the image as base64 string ( dataurl ), Below is my function that converts the dataurl into the image,
Now if the image is png then png library will create the image and jpg library will throw an error and vice versa.
Problem is when i upload png it works well but when i upload jpg file
returns this error
invalid JPEG format: missing SOI marker
func (ProfileController) SetProfilePic(w http.ResponseWriter, r *http.Request) {
session, _ := session.UserSession.Get(r, "mvc-user-session")
var fieldMapForBasic map[string]*json.RawMessage
var f *os.File
data, _ := ioutil.ReadAll(r.Body)
json.Unmarshal(data, &fieldMapForBasic)
image, _ := json.Marshal(fieldMapForBasic["image"])
coI := strings.Index(string(image), ",")
rawImage := string(image)[coI+1:]
// Encoded Image DataUrl //
unbased, _ := base64.StdEncoding.DecodeString(string(rawImage))
res := bytes.NewReader(unbased)
path, _ := os.Getwd()
// Path to store the image //
newpath := filepath.Join(path + "/storage", strconv.Itoa(session.Values["id"].(int)))
os.MkdirAll(newpath, os.ModePerm)
uid := uuid.NewV4()
fmt.Println(newpath)
// if image is png this function will create the image from dataurl string //
pngI, errPng := png.Decode(res)
if errPng == nil {
f, _ = os.OpenFile(newpath + "/" + uid.String() + ".png", os.O_WRONLY | os.O_CREATE, 0777)
png.Encode(f, pngI)
fmt.Println("Png generated")
user := model.User{}
user.ProfilePic = "storage/" + strconv.Itoa(session.Values["id"].(int)) + "/" + uid.String()
session.Values["profile_pic"] = user.ProfilePic
session.Save(r, w)
database.Connection.Id(session.Values["id"].(int)).Update(&user)
} else {
fmt.Println(errPng.Error())
}
// If image is Jpg this will generate the Image //
jpgI, errJpg := jpeg.Decode(res)
if errJpg == nil {
f, _ = os.OpenFile(newpath + "/" + uid.String() + ".jpg", os.O_WRONLY | os.O_CREATE, 0777)
jpeg.Encode(f, jpgI, &jpeg.Options{Quality: 75})
fmt.Println("Jpg created")
} else {
fmt.Println(errJpg.Error())
}
defer func() {
f.Close()
}()
render.Json(w, "ok")
}
Input dataurl is:
"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAIBAQIBAQICAgICAgICAwUDAwMDAwYEBAMFBwYHBwcGBwcICQsJCAgKCAcHCg0KCgsMDAwMBwkODw0MDgsMDAz/2wBDAQICAgMDAwYDAwYMCAcIDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAz/wAARCADeAN4DASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDtKKKK/cD8rCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAP/Z"
This might help
You're passing your io.Reader to png.Decode(), which begins consuming the reader, only to discover that the input is not a valid PNG, so returns an error.
Then your partly-consumed reader is passed to jpeg.Decode(), which reads the data not yet read, which is not a valid JPEG, and returns the error you observe.
You need to create a new reader for each decoder:
pngI, errPng := png.Decode(bytes.NewReader(unbased))
// ...
jpgI, errJpg := jpeg.Decode(bytes.NewReader(unbased))
Or better yet, consider the MIME type, and only call the proper decoder:
switch strings.TrimSuffix(image[5:coI], ";base64") {
case "image/png":
pngI, err = png.Decode(res)
// ...
case "image/jpeg":
jpgI, err = jpeg.Decode(res)
// ...
}
Here's something which may help, demonstrates quite a few other things too (you need to have two files: flower.jpg and mouse.png to make this work, available in the repository https://github.com/rravishankar/golangtraining/tree/master/test/jpegencode):
package main
import (
"bufio"
"encoding/base64"
"fmt"
"image"
"image/jpeg"
"image/png"
"strings"
"log"
"os"
)
func main() {
createJpg()
createPng()
base64toPng()
base64toJpg(getJPEGbase64("flower.jpg"))
}
//Take an existing jpg srcFileName and decode/encode it
func createJpg() {
srcFileName := "flower.jpg"
dstFileName := "newFlower.jpg"
// Decode the JPEG data. If reading from file, create a reader with
reader, err := os.Open(srcFileName)
if err != nil {
log.Fatal(err)
}
defer reader.Close()
//Decode from reader to image format
m, formatString, err := image.Decode(reader)
if err != nil {
log.Fatal(err)
return
}
fmt.Println("Got format String", formatString)
fmt.Println(m.Bounds())
//Encode from image format to writer
f, err := os.OpenFile(dstFileName, os.O_WRONLY|os.O_CREATE, 0777)
if err != nil {
log.Fatal(err)
return
}
err = jpeg.Encode(f, m, &jpeg.Options{Quality: 75})
if err != nil {
log.Fatal(err)
return
}
fmt.Println("Jpg file", dstFileName, "created")
}
//Take an existing png srcFileName and decode/encode it
func createPng() {
srcFileName := "mouse.png"
dstFileName := "newMouse.png"
reader, err := os.Open(srcFileName)
if err != nil {
log.Fatal(err)
}
defer reader.Close()
//Decode from reader to image format
m, formatString, err := image.Decode(reader)
if err != nil {
log.Fatal(err)
return
}
fmt.Println("Got format String", formatString)
fmt.Println(m.Bounds())
//Encode from image format to writer
f, err := os.OpenFile(dstFileName, os.O_WRONLY|os.O_CREATE, 0777)
if err != nil {
log.Fatal(err)
return
}
err = png.Encode(f, m)
if err != nil {
log.Fatal(err)
return
}
fmt.Println("Png file", dstFileName, "created")
}
//Converts pre-existing base64 data (found in example of https://golang.org/pkg/image/#Decode) to test.png
func base64toPng() {
const data = `
/9j/4AAQSkZJRgABAQIAHAAcAAD/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdA
SFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2Nj
Y2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wAARCABnAJYDASIAAhEBAxEB/8QA
HwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIh
MUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVW
V1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXG
x8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQF
BgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAV
YnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOE
hYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq
8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDlwKMD0pwzSiuK57QzGDxS7D6in8Y5ximnAPUfSlcq4m3ilUYp
2OKXHvRcVxnTtS7c07HNFK4DQPakC4PNOA+tOx70XAjK/So5gBGP94fzqfvUVx/qxx/EP51UXqRP4WSE
cmgjilP3jSEZqS0IO/NGDnpUiocDg/McDjvV6HTPOdVWYgsM5KcfzzQ2JySM2jp6VYu7SWzmMUwG4cgj
kMPUVBjjtTGtRu0Zopw+lFFxhinrGzuqqMsxAA9yaXFSRv5cqSEcIwYj6GpuZ30O30fSLKzhUpbpNMv3
5XGTn29BV28jt7pPLuIVljPBBFVreYx+VbqAjycgt3x14zRcNOxGyVFHQkIc/wA61exyKLbuzjdZ046d
ftEuTEw3Rk9SPT8P8Kpbea3tchbyVae4JkjbbGpGdwOM89Af6ViFTWUtGdcXoM2+woK1JtpNtTcoZt+l
Jt7ZqTbRtouFyPFRXI/c9D94fzqzioLsfuD/ALw/nVReqIn8LJCOTSY+tSMOTmkIpXLRu+F0t5pJxPHG
wjjUAuBjJJz1+laD6Pai+WaK9SBX6puzn6ZP+NV/Dkdtc6ZNbyAFwxLAHDYPv6VoQ21nPNEEiQGEFRtk
Gf0NaWTOeW7Of8QwGG4MRZnEbYXPJwRnOR0zWNXW+KrqBLUWi5EjbWCgcAA9c/gRXKYqZaGlK/LqMH0F
FLtHvRSNiYD2pSDTgpp6p0ywUHoTULXYxcktzrdCf7Xo8LP/AKyEmMNjJ46dfbFWJ5TDGNwB9lFUvDV9
YrbfYGbyrjcWG88S57g+vtV26ZIvMlumKwwjLZ6V0WfU54yTvYwtbubea2WNWbzg4bYQeBgj8OtYeKhj
u4y2HQxqxOD1xzxmrWAQCCGB6EGsaikndmsJxeiYzBo280/Z7UbayuaXGY5oIp+2lx9KLjIsVDeD/Rj/
ALy/zq1t96r3y4tT/vL/ADq4P3kRP4WSleTSFKkkKoCW4GaqNcMxIjXj1pxjKT0FKrGC1Nrw3vGrKkYz
5kTAr6455/HH510UdwPtRgWCbzF5+YYUf4Vwun39xpmoR3qASMmQUJwGU9Rnt/8AWrpbrxhb8/ZdOmaQ
gAGZwFH5ZJrpVKVlY5ZYhN6kXiu2eO/ikZlIljAAB5yM549OawSOOlPuLqe+umuLqTfM4OSOAo7ADsKh
hl/cRsTuJHPv7mlKi3sVTxNtGP20VJhThgSQaK52mnZnUqsWrpkyeUrr5pABOAPU1AGaXUCWJISHGPfP
P8qL7BiKnsMg46H3qrbzupbj5mPTPTpXVSglG551SpzSsXJ4/MBUgYIxyKpySyGBYJriV1D7kRpCVH4V
bSeNJ4xchni3DeqnBI+td7F4b0mKIRjT45VbktJlzk455+n6VtYzv2PNwFZWBHBGKVJDGVC54/nXQeMN
NttLNkba1jgWVWDmM8bhg4/nzXLSSbXVj6fyNKUdNRp21RtIRJGrjuM0u3FQ2DbodvcEkfQmrW2vLqLl
k0ejCXNFMj2/jQV9qkxSYNRcsZiq2oI32N2CkhWXJxwOe9XMcVt6hoPn6dFaW0wgRpNzvKDlz6+/0rai
ryv2Jm9LHJai+ZRGCBjnr71ErdAxAY9B611t1Y2cunbbaOQ3FvKZI3UqGlZMbiWwfcfhV231iwvLSM3U
lt5Uq52TuZG+hGMA12xXJGxxzjzybOQtNOvb5j9ktZJhnBIHyg+5PFX38JayqK/2eLJIBUTgkDA9q7ex
itrSHFpGsUbndhRgc+g7VNIyfZJAoJZUbb3I46CtFJMylBo8sdWhmYMuCnylc9wef5VUT7+1chc5NS7h
sUZO5RtIPUH3pkBDOxxxmqM9TQtn+WilhHfHaik43KTG3Z4IyPyrNVjGCsZ+dmwv6V3cXhSG8sYpJLud
JJIwxChdoJGcYx/Wkg8DafA4knvLiQr/ALqj+VQpKw3FtnFFfvbiSMgZJ6/jXp2n3d9cQRBTFsKD96EP
oOxPU/8A68VVtbbRtMVntbePKDLTSHJH/Aj/AEqHTvE66rq72VugMMcbSGTnL4wMAfjT5n0HyW3L+s6b
baxaJBdzN+7bcrxkAhun0rz3VNCv7e7lgigknWI43xLu6jjIHTjtXqfkpPGVYsBkghTikgsYIN/lhgXb
cxLkknp/ShczQ7xtY8vtEmhkj8yGRBuCnehUcnHcVtmwfJ/fQ8e7f/E12txZW91C0U6b42xlST2OR/Ko
Bo1gM/uW55/1jf41nOipu7LhV5FZHIGzI6zwj/vr/Ck+yr3uYf8Ax7/CutbQdMb71tn/ALaN/jSf8I/p
X/PoP++2/wAan6rAr6wzkWt0II+1Rc/7Lf4Vd1eeCSKBbdZDdShYoiZNoyfY10P/AAj2lf8APmP++2/x
oPh/SjKspsozIuNrZORjp3qo0FHYPb3OZt7ae3SzjuItsiRSAgnccl/UA+3Q1yNjKLR4ZZYY5VD7tkv3
WwO/+e1evPp9nI257aJm6bioz1z1+tY+s6Hplnot9PbWMMcqwOFcLyOO1bJWMZSTOPHi+9w3mosrlyd2
9lCj02g9P/1e9a3hzxAbl2ikZRcdQueHHt7j864Y8Z4I4oRzG6urFWU5BHBB7HNJxTFGbR6he6Vpmtgm
eLy5zwZI/lb8fX8azIvBUUTHdfSFP4QsYB/HNZ+k+KEnRY75hHOvAk6K/v7H9K6yyvlnQBmDZ6GsnzR0
N0oy1RzOtaN/Y1tHNFO06u+zYy4I4Jzx9KKveJblXuordSGES5b6n/62PzorKVdp2LjQTVyWz8UWEWlq
jSgyxfJt6EgdDzWTdeLIZGO7zHI/hVajGmWWP+PWL8qwlAIURrhpMAHHJA71pRcZrToZzcoEuo6heakA
GHk245CZ6/X1qPTLq40q+W5t2QybSpDAkEEc55/zilk5k2r91eKhLDzWz2rpsczbbuemeD76fUNG865I
MiysmQMZAAwa3a5j4ftu0ByP+fh/5CulkLLG7INzhSVHqe1Fh3uOoqn9qQQxyhndmHIxwOmSR2xQ13KD
KoiBZOV9JBnt707MVy5RWdNdy7wRGf3bfMinnO1jg+vY03WXLaJO3mhQ20b0zwpYf0qlG7S7icrJs08U
VwumgC+YiQyeVtZH567hzj8aSL949oGhE/2v5pJCDkksQwBHC4/+vXQ8LZ2uYxxCavY7us/xCcaBfn0h
b+VP0bnSrb94ZMJgOecj1rl/GfidUE2k2gy5+SeQjgA/wj3rlas2jdao48qrjLAGkSKPk4Gc1WMj92I+
lIJnU8OfxPWo5inBokmtQTmM4OOh71b0q6vbFmWCbaxHyqQGAP0PT8KhSTzVyo5ocSKA5VfTOTmqsmRd
pl99XjPzThzK3zOeOSeveirNmkgg/fIpYsTkYORxRXmzlTjJqx6EVUcU7mhkKCzdAK59QI9zYxtG1fYU
UVtgtmY4nZEa8Ak9aqFv3rfSiiu1nMeifDv/AJF+T/r4f+QrqqKKQwzQenNFFMCOKFIgNuThdoJ5OPSk
ubeK6t3gnXdG4wwziiii/UTKMOg6dbzJLFE4dSCP3rEdeOM8805tDsGMvySgSsS6rM6gk9eAcUUVftZt
3uyVGNthuq3Eei6DK8H7sRR7YuMgHtXkc8rzTNLM26RyWY+p70UVnLY0iEsUipG7rhZBlDkc1HgYoorM
0HwyBXGeRjmrcUhMg2ghezd//rUUVcTKW5s2jZtY/QDaOKKKK8ip8bPRj8KP/9k=
`
reader := base64.NewDecoder(base64.StdEncoding, strings.NewReader(data))
m, formatString, err := image.Decode(reader)
if err != nil {
log.Fatal(err)
}
bounds := m.Bounds()
fmt.Println(bounds, formatString)
//Encode from image format to writer
pngFilename := "test.png"
f, err := os.OpenFile(pngFilename, os.O_WRONLY|os.O_CREATE, 0777)
if err != nil {
log.Fatal(err)
return
}
err = png.Encode(f, m)
if err != nil {
log.Fatal(err)
return
}
fmt.Println("Png file", pngFilename, "created")
}
//Given a base64 string of a JPEG, encodes it into an JPEG image test.jpg
func base64toJpg(data string) {
reader := base64.NewDecoder(base64.StdEncoding, strings.NewReader(data))
m, formatString, err := image.Decode(reader)
if err != nil {
log.Fatal(err)
}
bounds := m.Bounds()
fmt.Println("base64toJpg", bounds, formatString)
//Encode from image format to writer
pngFilename := "test.jpg"
f, err := os.OpenFile(pngFilename, os.O_WRONLY|os.O_CREATE, 0777)
if err != nil {
log.Fatal(err)
return
}
err = jpeg.Encode(f, m, &jpeg.Options{Quality: 75})
if err != nil {
log.Fatal(err)
return
}
fmt.Println("Jpg file", pngFilename, "created")
}
//Gets base64 string of an existing JPEG file
func getJPEGbase64(fileName string) string {
imgFile, err := os.Open(fileName)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
defer imgFile.Close()
// create a new buffer base on file size
fInfo, _ := imgFile.Stat()
var size = fInfo.Size()
buf := make([]byte, size)
// read file content into buffer
fReader := bufio.NewReader(imgFile)
fReader.Read(buf)
imgBase64Str := base64.StdEncoding.EncodeToString(buf)
//fmt.Println("Base64 string is:", imgBase64Str)
return imgBase64Str
}
You may also want to read the example in https://golang.org/pkg/image/#Decode
Hope this helps.
Related
convert pdf to jpg using imagick in golang.but using memory is too much in lunix. how to solve it?
import "gopkg.in/gographics/imagick.v3/imagick" func LoadParsePdf(lessonName string, validPageNos []int, localFolder string, pdfName string, imgPrefix string, imgSuffix string) (int, error) { support.Debug("begin convert pdf %s to img ", pdfName) imagick.Initialize() defer imagick.Terminate() mw := imagick.NewMagickWand() defer mw.Destroy() if err := mw.SetResourceLimit(imagick.RESOURCE_MEMORY, 50*1024*1024); err != nil { return 0, err } if err := mw.SetResolution(50, 50); err != nil { return 0, err } support.Debug("lesson %s , begin read pdf ", lessonName) if err := mw.ReadImage(util.JoinPath(localFolder, pdfName)); err != nil { return 0, err } support.Debug("end read pdf ") .... ... } I try to use “mw.SetResourceLimit(imagick.RESOURCE_MEMORY, xxx)” to solve this memory.In windows,it's ok,using 50M. But in lunix ,it is failed. who can help me to solve it? Thank you
Golang multipart file form request
I'm writing an API client against Mapbox, uploading a batch of svg images to a custom map. The api they provide for this is documented with an example cUrl call that works fine: curl -F images=#include/mapbox/sprites_dark/aubergine_selected.svg "https://api.mapbox.com/styles/v1/<my_company>/<my_style_id>/sprite?access_token=$MAPBOX_API_KEY" --trace-ascii /dev/stdout When attemting to do the same from golang I quickly came across that the multiform library is very limited, and wrote some code to make the request as similar to the cUrl request mentioned above. func createMultipartFormData(fileMap map[string]string) (bytes.Buffer, *multipart.Writer) { var b bytes.Buffer var err error w := multipart.NewWriter(&b) var fw io.Writer for fileName, filePath := range fileMap { h := make(textproto.MIMEHeader) h.Set("Content-Disposition", fmt.Sprintf(`form-data; name="%s"; filename="%s"`, "images", fileName)) h.Set("Content-Type", "image/svg+xml") if fw, err = w.CreatePart(h); err != nil { fmt.Printf("Error creating form File %v, %v", fileName, err) continue } fileContents, err := ioutil.ReadFile(filePath) fileContents = bytes.ReplaceAll(fileContents, []byte("\n"), []byte(".")) blockSize := 64 remainder := len(fileContents) % blockSize iterations := (len(fileContents) - remainder) / blockSize newBytes := []byte{} for i := 0; i < iterations; i++ { start := i * blockSize end := i*blockSize + blockSize newBytes = append(newBytes, fileContents[start:end]...) newBytes = append(newBytes, []byte("\n")...) } if remainder > 0 { newBytes = append(newBytes, fileContents[iterations*blockSize:]...) newBytes = append(newBytes, []byte("\n")...) } if err != nil { fmt.Printf("Error reading svg file: %v: %v", filePath, err) continue } _, err = fw.Write(newBytes) if err != nil { log.Debugf("Could not write file to multipart: %v, %v", fileName, err) continue } } w.Close() return b, w } Along with setting the headers in the actual request: bytes, formWriter := createMultipartFormData(filesMap) req, err := http.NewRequest("Post", fmt.Sprintf("https://api.mapbox.com/styles/v1/%v/%v/sprite?access_token=%v", "my_company", styleID, os.Getenv("MAPBOX_API_KEY")), &bytes) if err != nil { return err } req.Header.Set("User-Agent", "curl/7.64.1") req.Header.Set("Accept", "*/*") req.Header.Set("Content-Length", fmt.Sprintf("%v", len(bytes.Bytes()))) req.Header.Set("Content-Type", formWriter.FormDataContentType()) byts, _ := httputil.DumpRequest(req, true) fmt.Println(string(byts)) res, err := http.DefaultClient.Do(req) Even want as far to limit the line length and replicate the encoding used by cUrl but so far no luck. Does anyone with experience know why this works from cUrl but not golang?
Well, I admit that all the parts of the "puzzle" to solve your task can be found on the 'net in abundance, there are two problems with this: They quite often miss certain interesting details. Sometimes, they give outright incorrect advice. So, here's a working solution. package main import ( "bytes" "fmt" "io" "io/ioutil" "mime" "mime/multipart" "net/http" "net/textproto" "net/url" "os" "path/filepath" "strconv" "strings" ) func main() { const ( dst = "https://api.mapbox.com/styles/v1/AcmeInc/Style_001/sprite" fname = "path/to/a/sprite/image.svg" token = "an_invalid_token" ) err := post(dst, fname, token) if err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } } func post(dst, fname, token string) error { u, err := url.Parse(dst) if err != nil { return fmt.Errorf("failed to parse destination url: %w", err) } form, err := makeRequestBody(fname) if err != nil { return fmt.Errorf("failed to prepare request body: %w", err) } q := u.Query() q.Set("access_token", token) u.RawQuery = q.Encode() hdr := make(http.Header) hdr.Set("Content-Type", form.contentType) req := http.Request{ Method: "POST", URL: u, Header: hdr, Body: ioutil.NopCloser(form.body), ContentLength: int64(form.contentLen), } resp, err := http.DefaultClient.Do(&req) if err != nil { return fmt.Errorf("failed to perform http request: %w", err) } defer resp.Body.Close() _, _ = io.Copy(os.Stdout, resp.Body) return nil } type form struct { body *bytes.Buffer contentType string contentLen int } func makeRequestBody(fname string) (form, error) { ct, err := getImageContentType(fname) if err != nil { return form{}, fmt.Errorf( `failed to get content type for image file "%s": %w`, fname, err) } fd, err := os.Open(fname) if err != nil { return form{}, fmt.Errorf("failed to open file to upload: %w", err) } defer fd.Close() stat, err := fd.Stat() if err != nil { return form{}, fmt.Errorf("failed to query file info: %w", err) } hdr := make(textproto.MIMEHeader) cd := mime.FormatMediaType("form-data", map[string]string{ "name": "images", "filename": fname, }) hdr.Set("Content-Disposition", cd) hdr.Set("Contnt-Type", ct) hdr.Set("Content-Length", strconv.FormatInt(stat.Size(), 10)) var buf bytes.Buffer mw := multipart.NewWriter(&buf) part, err := mw.CreatePart(hdr) if err != nil { return form{}, fmt.Errorf("failed to create new form part: %w", err) } n, err := io.Copy(part, fd) if err != nil { return form{}, fmt.Errorf("failed to write form part: %w", err) } if int64(n) != stat.Size() { return form{}, fmt.Errorf("file size changed while writing: %s", fd.Name()) } err = mw.Close() if err != nil { return form{}, fmt.Errorf("failed to prepare form: %w", err) } return form{ body: &buf, contentType: mw.FormDataContentType(), contentLen: buf.Len(), }, nil } var imageContentTypes = map[string]string{ "png": "image/png", "jpg": "image/jpeg", "jpeg": "image/jpeg", "svg": "image/svg+xml", } func getImageContentType(fname string) (string, error) { ext := filepath.Ext(fname) if ext == "" { return "", fmt.Errorf("file name has no extension: %s", fname) } ext = strings.ToLower(ext[1:]) ct, found := imageContentTypes[ext] if !found { return "", fmt.Errorf("unknown file name extension: %s", ext) } return ct, nil } Some random notes on implementation to help you understands the concepts: To construct the request's payload (body), we use a bytes.Buffer instance. It has a nice property in that a pointer to it (*bytes.Buffer) implements both io.Writer and io.Reader and hence can be easily composed with other parts of the Go stdlib which deal with I/O. When preparing a multipart form for sending, we do not slurp the whole file's contents into memory but instead "pipe" them right into the "mutipart form writer". We have a lookup table which maps the extension of a file name to submit to its MIME type; I have no idea whether this is needed by the API or not; if it's not really required, the part of the code which prepares a form's field containing a file could be simplified a lot, but cURL send it, so do we.
Just to be curious, What is this for? fileContents = bytes.ReplaceAll(fileContents, []byte("\n"), []byte(".")) blockSize := 64 remainder := len(fileContents) % blockSize iterations := (len(fileContents) - remainder) / blockSize newBytes := []byte{} for i := 0; i < iterations; i++ { start := i * blockSize end := i*blockSize + blockSize newBytes = append(newBytes, fileContents[start:end]...) newBytes = append(newBytes, []byte("\n")...) } if remainder > 0 { newBytes = append(newBytes, fileContents[iterations*blockSize:]...) newBytes = append(newBytes, []byte("\n")...) } if err != nil { fmt.Printf("Error reading svg file: %v: %v", filePath, err) continue } Reading a entire file into memory is rarely a good idea (ioutil.ReadFile). As #muffin-top says, how about those three lines of code? for fileName, filePath := range fileMap { // h := ... fw, _ := w.CreatePart(h) // TODO: handle error f, _ := os.Open(filePath) // TODO: handle error io.Copy(fw, f) // TODO: handle error f.Close() // TODO: handle error }
Trouble getting content type of file in Go
I have a function in which I take in a base64 string and get the content of it (PDF or JPEG). I read in the base64 content, convert it to bytes and decode it into the file that it is. I then create a file where I will output the decoded file (JPEG or PDF). Then I write the bytes to it. Then I call my GetFileContentType on it and it returns to me an empty string. If I run the functions separately, as in I first the first function to create the decoded file, and end it. And then call the second function to get the content type, it works and returns it as JPEG or PDF. What am I doing wrong here? And is there a better way to do this? func ConvertToJPEGBase64( src string, dst string, ) error { b, err := ioutil.ReadFile(src) if err != nil { return err } str := string(b) byteArray, err := base64.StdEncoding.DecodeString(str) if err != nil { return err } f, err := os.Create(dst) if err != nil { return err } if _, err := f.Write(byteArray); err != nil { return err } f.Sync() filetype, err := client.GetFileContentType(f) if err != nil { return err } if strings.Contains(filetype, "jpeg") { // do something } else { // do something else } return nil } // GetFileContentType tells us the type of file func GetFileContentType(out *os.File) (string, error) { // Only the first 512 bytes are used to sniff the content type. buffer := make([]byte, 512) _, err := out.Read(buffer) if err != nil { return "", err } contentType := http.DetectContentType(buffer) return contentType, nil }
The problem is that GetFileContentType reads from the end of the file. Fix this be seeking back to the beginning of the file before calling calling GetFileContentType: if _, err := f.Seek(io.SeekStart, 0); err != nil { return err } A better fix is to use the file data that's already in memory. This simplifies the code to the point where there's no need for the GetFileContentType function. func ConvertToJPEGBase64( src string, dst string, ) error { b, err := ioutil.ReadFile(src) if err != nil { return err } str := string(b) byteArray, err := base64.StdEncoding.DecodeString(str) if err != nil { return err } f, err := os.Create(dst) if err != nil { return err } defer f.Close() // <-- Close the file on return. if _, err := f.Write(byteArray); err != nil { return err } fileType := http.DetectContentType(byteArray) // <-- use data in memory if strings.Contains(fileType, "jpeg") { // do something } else { // do something else } return nil } More code can be eliminated by using ioutil.WriteFile: func ConvertToJPEGBase64(src, dst string) error { b, err := ioutil.ReadFile(src) if err != nil { return err } byteArray, err := base64.StdEncoding.DecodeString(string(b)) if err != nil { return err } if err := ioutil.WriteFile(dst, byteArray, 0666); err != nil { return err } fileType := http.DetectContentType(byteArray) if strings.Contains(fileType, "jpeg") { // do something } else { // do something else } return nil }
How to write RIFF chunk header when store image from url?
I just tried to download webp image from url, but I found something different when I try to process the stored image. If I download the image from the browser, it can be decoded using x/image/webp package, but if I store the image using http.Get() then create a new file then io.Copy() the image, it says: "missing RIFF chunk header" I assume that I need to write some RIFF chunk header when I store it using golang code. func main(){ response, e := http.Get(URL) if e != nil { log.Fatal(e) } defer response.Body.Close() //open a file for writing file, err := os.Create('tv.webp') if err != nil { log.Fatal(err) } defer file.Close() // Use io.Copy to just dump the response body to the file. This supports huge files _, err = io.Copy(file, response.Body) if err != nil { log.Fatal(err) } fmt.Println("Success!") imgData, err := os.Open("tv.webp") if err != nil { fmt.Println(err) return } log.Printf("%+v", imgData) image, err := webp.Decode(imgData) if err != nil { fmt.Println(err) return } fmt.Println(image.Bounds()) } Here is the URL IMG URL
download file is not webp type. it's png. package main import ( "fmt" "image" "io" "log" "net/http" "os" _ "image/png" ) func main() { response, e := http.Get("https://www.sony.com/is/image/gwtprod/0abe7672ff4c6cb4a0a4d4cc143fd05b?fmt=png-alpha") if e != nil { log.Fatal(e) } defer response.Body.Close() file, err := os.Create("dump") if err != nil { log.Fatal(err) } defer file.Close() _, err = io.Copy(file, response.Body) if err != nil { log.Fatal(err) } fmt.Println("Success!") imageFile, err := os.Open("dump") if err != nil { panic(err) } m, name, err := image.Decode(imageFile) if err != nil { panic(err) } fmt.Println("image type is ", name, m.Bounds()) }
How to covert a []byte object to an image and store it as a jpeg image on disk
I am trying to convert an []byte object to an image and save it as a jpeg in Golang. I tried to use Decode function of image but it always returns <nil>. func saveFrames(imgByte []byte) { img, _, _ := image.Decode(bytes.NewReader(imgByte)) out, err := os.Create("./img.jpeg") if err != nil { fmt.Println(err) os.Exit(1) } err = jpeg.Encode(out, img) if err != nil { fmt.Println(err) os.Exit(1) } }
You are not passing Options,to jpeg.Encode, you can also set it to nil. func serveFrames(imgByte []byte) { img, _, err := image.Decode(bytes.NewReader(imgByte)) if err != nil { log.Fatalln(err) } out, _ := os.Create("./img.jpeg") defer out.Close() var opts jpeg.Options opts.Quality = 1 err = jpeg.Encode(out, img, &opts) //jpeg.Encode(out, img, nil) if err != nil { log.Println(err) } } Don't forget to close any file, if opened. You can use log.Fatalln(...), if you want to print error message and quit in-case of any error.