How to write RIFF chunk header when store image from url? - go

I just tried to download webp image from url, but I found something different when I try to process the stored image.
If I download the image from the browser, it can be decoded using x/image/webp package, but if I store the image using http.Get() then create a new file then io.Copy() the image, it says:
"missing RIFF chunk header"
I assume that I need to write some RIFF chunk header when I store it using golang code.
func main(){
response, e := http.Get(URL)
if e != nil {
log.Fatal(e)
}
defer response.Body.Close()
//open a file for writing
file, err := os.Create('tv.webp')
if err != nil {
log.Fatal(err)
}
defer file.Close()
// Use io.Copy to just dump the response body to the file. This supports huge files
_, err = io.Copy(file, response.Body)
if err != nil {
log.Fatal(err)
}
fmt.Println("Success!")
imgData, err := os.Open("tv.webp")
if err != nil {
fmt.Println(err)
return
}
log.Printf("%+v", imgData)
image, err := webp.Decode(imgData)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(image.Bounds())
}
Here is the URL IMG URL

download file is not webp type. it's png.
package main
import (
"fmt"
"image"
"io"
"log"
"net/http"
"os"
_ "image/png"
)
func main() {
response, e := http.Get("https://www.sony.com/is/image/gwtprod/0abe7672ff4c6cb4a0a4d4cc143fd05b?fmt=png-alpha")
if e != nil {
log.Fatal(e)
}
defer response.Body.Close()
file, err := os.Create("dump")
if err != nil {
log.Fatal(err)
}
defer file.Close()
_, err = io.Copy(file, response.Body)
if err != nil {
log.Fatal(err)
}
fmt.Println("Success!")
imageFile, err := os.Open("dump")
if err != nil {
panic(err)
}
m, name, err := image.Decode(imageFile)
if err != nil {
panic(err)
}
fmt.Println("image type is ", name, m.Bounds())
}

Related

Having issues with multipart.NewWriter using io.pipe

I'm having issues with requests being sent randomly empty. It doesn't always happen, but sometimes out of the blue, it will not send any of the multipart fields. I thought it might have to do with the upload server, so I created a local upload server to print out the request that's being sent, and it comes out empty.
I have added error checks everywhere, but no errors are being returned.
I tried to run the code with -race, but no race condition has been reported.
Edit: Update the code to use CloseWithError()
package main
import (
"fmt"
"io"
"log"
"mime/multipart"
"net/http"
"os"
)
var (
upload_url string = "https://upload.imagekit.io/api/v1/files/upload"
file_name string = "favicon-516140983.ico"
api_secret_key string = "PRIVATE_KEY"
)
func UploadMultipartFile(client *http.Client, uri, key, path string) (*http.Response, error) {
body, writer := io.Pipe()
req, err := http.NewRequest(http.MethodPost, uri, body)
if err != nil {
log.Println(err)
return nil, err
}
mwriter := multipart.NewWriter(writer)
req.Header.Add("Content-Type", mwriter.FormDataContentType())
req.SetBasicAuth(api_secret_key, "")
go func() {
var err error
defer func() {
if err != nil {
writer.CloseWithError(err)
} else {
writer.Close()
}
}()
var file *os.File
file, err = os.Open(path)
if err != nil {
return
}
defer file.Close()
if err = mwriter.WriteField("fileName", file_name); err != nil {
return
}
var w io.Writer
w, err = mwriter.CreateFormFile("file", path)
if err != nil {
return
}
var written int64
if written, err = io.Copy(w, file); err != nil {
err = fmt.Errorf("error copying %s (%d bytes written): %v", path, written, err)
return
}
if err = mwriter.Close(); err != nil {
return
}
}()
resp, err := client.Do(req)
if err != nil {
return nil, err
}
return resp, nil
}
func main() {
path, _ := os.Getwd()
path += "/" + file_name
client := &http.Client{}
resp, err := UploadMultipartFile(client, upload_url, "file", path)
if err != nil {
log.Println(err)
} else {
fmt.Println(resp.StatusCode)
fmt.Println(resp.Header)
_, err := io.Copy(os.Stdout, resp.Body)
if err != nil {
log.Fatal(err)
}
resp.Body.Close()
}
}

Convert protobuf serialized messages to JSON without precompiling Go code

I want to convert protobuf serialized messages into a human readable JSON format. The major problem I face is that I need to do this without compiling the proto descriptor into Go code beforehand. I have access to the .proto files at runtime, but not at compile time.
I had the impression that the new Protobuf API v2 (https://github.com/protocolbuffers/protobuf-go) supports dynamic deserialization (see package types/dynamicpb), but I couldn't figure out how to use it apparently:
func readDynamically(in []byte) {
// How do I load the required descriptor (for NewMessage()) from my `addressbook.proto` file?)
descriptor := ??
msg := dynamicpb.NewMessage(descriptor)
err := protojson.Unmarshal(in, msg)
if err != nil {
panic(err)
}
}
Above code is annotated with my problem: How can I get the required descriptor for the dynamicpb.NewMessage() from a .proto file?
Should work like this with the dynamicpb package.
func readDynamically(in []byte) {
registry, err := createProtoRegistry(".", "addressbook.proto")
if err != nil {
panic(err)
}
desc, err := registry.FindFileByPath("addressbook.proto")
if err != nil {
panic(err)
}
fd := desc.Messages()
addressBook := fd.ByName("AddressBook")
msg := dynamicpb.NewMessage(addressBook)
err = proto.Unmarshal(in, msg)
jsonBytes, err := protojson.Marshal(msg)
if err != nil {
panic(err)
}
fmt.Println(string(jsonBytes))
if err != nil {
panic(err)
}
}
func createProtoRegistry(srcDir string, filename string) (*protoregistry.Files, error) {
// Create descriptors using the protoc binary.
// Imported dependencies are included so that the descriptors are self-contained.
tmpFile := filename + "-tmp.pb"
cmd := exec.Command("./protoc/protoc",
"--include_imports",
"--descriptor_set_out=" + tmpFile,
"-I"+srcDir,
path.Join(srcDir, filename))
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err := cmd.Run()
if err != nil {
return nil, err
}
defer os.Remove(tmpFile)
marshalledDescriptorSet, err := ioutil.ReadFile(tmpFile)
if err != nil {
return nil, err
}
descriptorSet := descriptorpb.FileDescriptorSet{}
err = proto.Unmarshal(marshalledDescriptorSet, &descriptorSet)
if err != nil {
return nil, err
}
files, err := protodesc.NewFiles(&descriptorSet)
if err != nil {
return nil, err
}
return files, nil
}
This question is kind of interesting. I have done some works on protobuf plugs. As far as i can tell, additional cli is needed because we don't want to "reinvent the wheel".
Step one, we need protoc to translate ".proto" file to some format so we can get "protoreflect.MessageDescriptor" easily.
This plug is to get raw bytes which protoc sends to other plugs as input.
package main
import (
"fmt"
"io/ioutil"
"os"
)
func main() {
if len(os.Args) == 2 && os.Args[1] == "--version" {
// fmt.Fprintf(os.Stderr, "%v %v\n", filepath.Base(os.Args[0]), version.String())
os.Exit(0)
}
in, err := ioutil.ReadAll(os.Stdin)
if err != nil {
fmt.Printf("error: %v", err)
return
}
ioutil.WriteFile("./out.pb", in, 0755)
}
build and rename it as protoc-gen-raw, then generate protoc --raw_out=./pb ./server.proto, you will get out.pb. Forget your ".proto" file from now on, and put this "out.pb" where you intend to put ".proto". And what we get is official support with this .pb file.
Step 2: Deserialize a protobuf serialized message into JSON.
package main
import (
"fmt"
"io/ioutil"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/compiler/protogen"
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/types/dynamicpb"
"google.golang.org/protobuf/types/pluginpb"
)
func main() {
in, err := ioutil.ReadFile("./out.pb")
if err != nil {
fmt.Printf("failed to read proto file: %v", err)
return
}
req := &pluginpb.CodeGeneratorRequest{}
if err := proto.Unmarshal(in, req); err != nil {
fmt.Printf("failed to unmarshal proto: %v", err)
return
}
gen, err := protogen.Options{}.New(req)
if err != nil {
fmt.Printf("failed to create new plugin: %v", err)
return
}
// serialize protobuf message "ServerConfig"
data := &ServerConfig{
GameType: 1,
ServerId: 105,
Host: "host.host.host",
Port: 10024,
}
raw, err := data.Marshal()
if err != nil {
fmt.Printf("failed to marshal protobuf: %v", err)
return
}
for _, f := range gen.Files {
for _, m := range f.Messages {
// "ServerConfig" is the message name of the serialized message
if m.GoIdent.GoName == "ServerConfig" {
// m.Desc is MessageDescriptor
msg := dynamicpb.NewMessage(m.Desc)
// unmarshal []byte into proto message
err := proto.Unmarshal(raw, msg)
if err != nil {
fmt.Printf("failed to Unmarshal protobuf data: %v", err)
return
}
// marshal message into json
jsondata, err := protojson.Marshal(msg)
if err != nil {
fmt.Printf("failed to Marshal to json: %v", err)
return
}
fmt.Printf("out: %v", string(jsondata))
}
}
}
}
// the output is:
// out: {"gameType":1, "serverId":105, "host":"host.host.host", "port":10024}

Upload a file with POST request golang

I'm new to golang and I'm trying to write a function that uploads a file with a post request to telegram for a bot I'm writing.
I've tried with this code but the error I'm getting from telegram is
Bad Request: there is no photo in the request.
I've searched on the net for how to do that, but none of what I found helped me through the problem.
func SendPostRequest (url string, filename string) []byte {
file, err := os.Open(filename)
if err != nil {
log.Fatal(err)
}
defer file.Close()
response, err := http.Post(url, "binary/octet-stream", file)
if err != nil {
log.Fatal(err)
}
defer response.Body.Close()
content, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal(err)
}
return content
}
The function I'm calling the SendPostRequest from is
func (e Engine) SendPhoto (filename string, chatId int64) APIResponse {
var url = fmt.Sprintf("%ssendPhoto?chat_id=%d", e.baseUrl, chatId)
var content []byte = SendPostRequest(url, filename)
var response APIResponse
json.Unmarshal(content, &response)
return response
}
EDIT:
The link to the Telegram bot api I'm using in the code is https://core.telegram.org/bots/api
And the api method is https://core.telegram.org/bots/api#sendphoto
After some digging I figured it out with this
import (
"bytes"
"io"
"mime/multipart"
"net/http"
"path/filepath"
)
// content is a struct which contains a file's name, its type and its data.
type content struct {
fname string
ftype string
fdata []byte
}
func sendPostRequest(url string, files ...content) ([]byte, error) {
var (
buf = new(bytes.Buffer)
w = multipart.NewWriter(buf)
)
for _, f := range files {
part, err := w.CreateFormFile(f.ftype, filepath.Base(f.fname))
if err != nil {
return []byte{}, err
}
_, err = part.Write(f.fdata)
if err != nil {
return []byte{}, err
}
}
err := w.Close()
if err != nil {
return []byte{}, err
}
req, err := http.NewRequest("POST", url, buf)
if err != nil {
return []byte{}, err
}
req.Header.Add("Content-Type", w.FormDataContentType())
client := &http.Client{}
res, err := client.Do(req)
if err != nil {
return []byte{}, err
}
defer res.Body.Close()
cnt, err := io.ReadAll(res.Body)
if err != nil {
return []byte{}, err
}
return cnt, nil
}

Golang: file extracted from tar throws permissions error

I've written the following code to tar a file, code works but strangely if I untar the archive the file permissions are gone so I can't read it unless I then chmod the file:
package main
import (
"archive/tar"
"io/ioutil"
"log"
"os"
)
func main() {
c, err := os.Create("/path/to/tar/file/test.tar")
if err != nil {
log.Fatalln(err)
}
tw := tar.NewWriter(c)
f, err := os.Open("sample.txt")
if err != nil {
log.Fatalln(err)
}
fi, err := f.Stat()
if err != nil {
log.Fatalln(err)
}
hdr := &tar.Header{Name: f.Name(),
Size: fi.Size(),
}
if err := tw.WriteHeader(hdr); err != nil {
log.Fatalln(err)
}
r, err := ioutil.ReadFile("sample.txt")
if err != nil {
log.Fatalln(err)
}
if _, err := tw.Write(r); err != nil {
log.Fatalln(err)
}
if err := tw.Close(); err != nil {
log.Fatalln(err)
}
}
Any idea what I'm doing wrong?
You're not preserving the original permissions of the file. You're manually creating a header, and specifying only the name and size. Instead, use tar.FileInfoHeader to build the header.
package main
import (
"archive/tar"
"io/ioutil"
"log"
"os"
)
func main() {
c, err := os.Create("/path/to/tar/file/test.tar")
if err != nil {
log.Fatalln(err)
}
tw := tar.NewWriter(c)
f, err := os.Open("sample.txt")
if err != nil {
log.Fatalln(err)
}
fi, err := f.Stat()
if err != nil {
log.Fatalln(err)
}
// create header from FileInfo
hdr, err := tar.FileInfoHeader(fi, "")
if err != nil {
log.Fatalln(err)
}
if err := tw.WriteHeader(hdr); err != nil {
log.Fatalln(err)
}
// instead of reading the whole file into memory, prefer io.Copy
r, err := io.Copy(tw, f)
if err != nil {
log.Fatalln(err)
}
log.Printf("Wrote %d bytes\n", r)
}
Also note that I used io.Copy to copy data from the file (an io.Reader) to the tar writer (an io.Writer). This will work much better for larger files.
Also - pay special attention to this note from the docs:
Because os.FileInfo's Name method returns only the base name of the file it describes, it may be necessary to modify the Name field of the returned header to provide the full path name of the file.
In this simple example, you're just using sample.txt so you shouldn't run into trouble. If you wanted to preserve a directory structure in your tar, you may have to modify the Name field in the header.

How can I efficiently download a large file using Go?

Is there a way to download a large file using Go that will store the content directly into a file instead of storing it all in memory before writing it to a file? Because the file is so big, storing it all in memory before writing it to a file is going to use up all the memory.
I'll assume you mean download via http (error checks omitted for brevity):
import ("net/http"; "io"; "os")
...
out, err := os.Create("output.txt")
defer out.Close()
...
resp, err := http.Get("http://example.com/")
defer resp.Body.Close()
...
n, err := io.Copy(out, resp.Body)
The http.Response's Body is a Reader, so you can use any functions that take a Reader, to, e.g. read a chunk at a time rather than all at once. In this specific case, io.Copy() does the gruntwork for you.
A more descriptive version of Steve M's answer.
import (
"os"
"net/http"
"io"
)
func downloadFile(filepath string, url string) (err error) {
// Create the file
out, err := os.Create(filepath)
if err != nil {
return err
}
defer out.Close()
// Get the data
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
// Check server response
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("bad status: %s", resp.Status)
}
// Writer the body to file
_, err = io.Copy(out, resp.Body)
if err != nil {
return err
}
return nil
}
The answer selected above using io.Copy is exactly what you need, but if you are interested in additional features like resuming broken downloads, auto-naming files, checksum validation or monitoring progress of multiple downloads, checkout the grab package.
Here is a sample. https://github.com/thbar/golang-playground/blob/master/download-files.go
Also I give u some codes might help you.
code:
func HTTPDownload(uri string) ([]byte, error) {
fmt.Printf("HTTPDownload From: %s.\n", uri)
res, err := http.Get(uri)
if err != nil {
log.Fatal(err)
}
defer res.Body.Close()
d, err := ioutil.ReadAll(res.Body)
if err != nil {
log.Fatal(err)
}
fmt.Printf("ReadFile: Size of download: %d\n", len(d))
return d, err
}
func WriteFile(dst string, d []byte) error {
fmt.Printf("WriteFile: Size of download: %d\n", len(d))
err := ioutil.WriteFile(dst, d, 0444)
if err != nil {
log.Fatal(err)
}
return err
}
func DownloadToFile(uri string, dst string) {
fmt.Printf("DownloadToFile From: %s.\n", uri)
if d, err := HTTPDownload(uri); err == nil {
fmt.Printf("downloaded %s.\n", uri)
if WriteFile(dst, d) == nil {
fmt.Printf("saved %s as %s\n", uri, dst)
}
}
}

Resources