How to load local assets within a GCP cloud function? - go

I'm building a tiny GCP cloud function in Golang that returns a generated PNG file when calling it via HTTP. I deployed my code via ZIP-Upload in the Google Cloud Console.
Currently it gets called properly and the code gets compiled etc. But in my code I have to load several local files - a font and a .png image.
I bundled those in the ZIP I uploaded and the files are visible in the Source-View in GCP. All files (images, fonts and go-files) are within the same directory.
When calling the cloud function the log states the following:
2019/01/21 14:59:31 open /english.png: no such file or directory
I tried to change the way i build the path to the file in go. I already used /german.png statically, used several attempts to build the path dynamically.
I'm not 100 percent sure if this is the way to go, but it is my first experiment with "serverless" and i am willing to get it done the "right" way.
import "github.com/fogleman/gg"
func main() {
ex, err := os.Executable()
if err != nil {
panic(err)
}
executableDir := filepath.Dir(ex)
img, err :=gg.LoadPNG(path.Join(executableDir, "./english.png"))
if err != nil {
log.Fatal(err)
}
}
Currently the file can not be found in any attempt i made. Maybe the path the images get "deployed" into are different from the ones i tried - i have not found any note on that in the documentation.
I'd obviously expect it to be loaded properly.

For the Go 1.13 according to the documentation, as of today (Jul 2020), source code is located in the ./serverless_function_source_code directory.
A good reference to follow is the buildpack.

I created http functions with the following structure:
api
|--test.txt
|--api.go
And wrote simple function to reply with file content:
package api
import (
"io/ioutil"
"net/http"
)
// FileTest func
func FileTest(w http.ResponseWriter, r *http.Request) {
content, err := ioutil.ReadFile("./test.txt")
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
w.Write([]byte(err.Error()))
return
}
w.Write(content)
}
It returns the file content without any problems. https://us-central1-clickshield24m.cloudfunctions.net/api
So in your case I would try change the path to gg.LoadPNG("./english.png")

./serverless_function_source_code didn't work for me using go113 runtime with vendoring. I wanted to locate a file in the vendor directory.
My file structure is:
myfn
|- main.go
|- go.mod
|- .gcloudignore
My deployment steps are:
cd myfn
go mod vendor
gcloud functions deploy MyFunction --runtime go113 --set-env-vars "PATH_FILES=./src/myfn/vendor/static"
I found this out by deploying a function that listed all files to a certain depth starting with "." and didn't see a serverless_function_source_code directory. But there was src/myfn.

Related

Portable Go module with bundled files that must be read [duplicate]

This question already has answers here:
What's the best way to bundle static resources in a Go program?
(5 answers)
Closed last year.
I'm trying to create a portable Go module that will be re-used in many different projects. This module, in order to function, needs to be able to read non-Go files that are bundled as part of the module. In this case, they're some certificate bundles. The selection of which file to load is dynamic and based on input arguments.
What is the best way to specify the path to these files when loading them? I can't seem to find any Go functions to get a path relative to the module (vs. relative to the executable that is using this module). For example, if my module structure looks like this:
mymodule/
go.mod
go.sum
loadcerts.go
certs/
cert_A.pem
cert_B.pem
And I need to do something like this:
// loadcerts.go
package mymodule
func LoadCerts(useB bool) error {
newCertPool := x509.NewCertPool()
// This is just to show that the selection is dynamic, and since
// there are many different potential files to load, we can't
// embed them all
bundleName := "cert_A.pem"
if useB {
bundleName = "cert_B.pem"
}
pem, err := ioutil.ReadFile(fmt.Sprintf("./certs/%s", bundleName))
if err != nil {
return err
}
if ok := newCertPool.AppendCertsFromPEM(pem); !ok {
return err
}
...
}
Referencing this file with a relative path (./certs/cert1.pem) doesn't work, since Go uses the executable's working directory for relative paths, and this imported module is somewhere entirely different.
How can I load this .pem file that is bundled with the portable module, regardless of where this module is being imported to?
Embed the files in the executable as a file system:
//go:embed certs
var f embed.FS
Read the files from the file system:
pem, err := f.ReadFile(fmt.Sprintf("certs/%s", bundleName))

gcp cloud function returns 404 when deployed from command line

If I deploy the example hello world within the console, the url/trigger works. If I deploy from command line it looks to be exact same code / attributes in cloud functions console, but the url is 404. I can't spot the difference/issue.
The deployed trigger/url shows - "404 page not found" for the below hello world example if deployed this way from command line.
gcloud functions deploy hellogo --entry-point=HelloWorld --trigger-http --region=us-central1 --memory=128MB --runtime=go116 --allow-unauthenticated
// Package p contains an HTTP Cloud Function.
package p
import (
"encoding/json"
"fmt"
"html"
"io"
"log"
"net/http"
)
// HelloWorld prints the JSON encoded "message" field in the body
// of the request or "Hello, World!" if there isn't one.
func HelloWorld(w http.ResponseWriter, r *http.Request) {
var d struct {
Message string `json:"message"`
}
if err := json.NewDecoder(r.Body).Decode(&d); err != nil {
switch err {
case io.EOF:
fmt.Fprint(w, "Hello World!")
return
default:
log.Printf("json.NewDecoder: %v", err)
http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
return
}
}
if d.Message == "" {
fmt.Fprint(w, "Hello World!")
return
}
fmt.Fprint(w, html.EscapeString(d.Message))
}
Tried to reproduce your error but couldn't replicate on my end. I've use the same command as yours and I can successfully access the url or call via HTTP trigger the deployed hello world cloud function.
gcloud functions deploy hellogo --entry-point=HelloWorld --trigger-http --region=us-central1 --memory=128MB --runtime=go116 --allow-unauthenticated
Output of successful curl:
I suggest you check the url you're trying to access since according to this GCP doc:
If you attempt to invoke a function that does not exist, Cloud Functions responds with an HTTP/2 302 redirect which takes you to the Google account login page. This is incorrect. It should respond with an HTTP/2 404 error response code. The problem is being addressed.
The solution
Make sure you specify the name of your function correctly. You can
always check using gcloud functions call which returns the correct 404
error for a missing function.
You can also refer to this complete guide to quickstart your CF creation and deployment using the Go runtime.
Thanks all, my project where I got stuck has more code in it than just this function. I went down this path after trying to deploy a single function/file within larger project. If I simplify down to a folder with just a hello.go and go.mod indeed it works :-/ to deploy it from command line:
gcloud functions deploy hellogo --entry-point=HelloWorld --trigger-http --region=us-central1 --memory=128MB --runtime=go116 --allow-unauthenticated
// go.mod
module github.com/nickfoden/hello
go 1.16
Thank you for the fast replies and assistance. Rather than try to create a single function within existing project with larger go.sum, multiple folders, existing server/api etc. I am going to start from here having a single file with a cloud function and build on top of it and see at what point/if I get stuck again.

Handling Viper Config File Path During Go Tests

So I have a pretty basic configuration with Viper reading a .env file from my base directory. I fatal kill the process if there's no .env file. All goes well when running my app normally.
When I run my tests with go test -v ./.., the test framework seems to step into each file's directory, and calls my config init() function each time, so the viper.AddConfigPath(".") is pointing to the wrong location.
this is my directory structure:
/
/restapi
items.go
items_test.go
/util
env.go
main.go
.env
env.go
package util
imports...
// global variables available via util package
var (
Port int
DbURI string
)
func init() {
viper.SetDefault(PORT, 8080)
viper.SetConfigFile(".env")
viper.AddConfigPath(".")
viper.AutomaticEnv()
fmt.Println("---------to see in test printout")
cwd, _ := os.Getwd()
fmt.Println(cwd)
fmt.Println("---------")
if err := viper.ReadInConfig(); err != nil {
log.Fatal("no environment file!")
}
Port = viper.GetInt("PORT")
DbURI = viper.GetString("DB_URI")
}
Every package basically relies on my util package and this init function therefore runs for every test. Is there some way to have viper always pull the .env file from the base directory even when there are tests running? I've tried a few different AddConfigPath() calls. Kinda new to Go. Or is this structure setup for environment variables not going to work since it fails my tests each time?
So apparently the viper.SetConfigFile() call does not respect the viper.AddConfigPath() call... I modified this to using viper.SetConfigName(".env") and it would actually pick up the calls to AddConfigPath, so I could then add config paths for the current directory and parent.
The problem is the path you are giving to the viper.AddConfigPath(".") method, but your env file relative path is not on the test file based on the folder structure tree you shared, it must be this: viper.AddConfigPath("./../util").

Flutter web with Golang Server

Is it possible to run flutter web build using golang server? Golang has facility to serve html file and flutter web gives output as index.html and js files.
if it is possible then how golang code should look like?
as the friendly doc mentions it, i believe you got to build your app.
https://flutter.dev/docs/get-started/web#build
Run the following command to generate a release build:
flutter build web
This populates a build/web directory with built files, including an assets directory, which need to be served together.
how golang code should look like?
like any other regular HTTP golang server.
http.Handle("/build/web/", http.StripPrefix("/build/web/", http.FileServer(http.Dir("build/web"))))
http.ListenAndServe(":8080", nil)
package main
import (
"flag"
"log"
"net/http"
)
func main() {
port := flag.String("p", "8181", "port to serve on")
directory := flag.String("d", "web", "the directory of static file to host")
flag.Parse()
http.Handle("/", http.FileServer(http.Dir(*directory)))
log.Printf("Serving %s on HTTP port: %s\n", *directory, *port)
log.Fatal(http.ListenAndServe(":"+*port, nil))
}
Just copy web folder to your's go app.

Unable to use the same relative path in my program AND my unit tests

In my Go Project I use a function that opens a specific file and returns its content. The file is stored in another directory but still inside my project directory.
package infrastructure
func openKey() ([]byte, error) {
if path, err := filepath.Abs("../security/key.rsa"); err != nil {
return nil, err
}
return ioutil.ReadFile(path)
}
This function works if I call it from a unit test. But if I call the same function in my program, I've this error:
2015/08/13 15:47:54 open
/me/go/src/github.com/myaccount/security/key.rsa: no such file
or directory
The correct absolute path of the file is:
/me/go/src/github.com/myaccount/myrepo/security/key.rsa
Both code that use the openKey function (from my program and unit test) are in the same package: infrastructure
Here is how I execute my program:
go install && ../../../../bin/myproject
And how I execute my unit tests:
go test ./...
And finally the directory structure of my project:
go/src/github.com/myaccount/myrepo/:
- main.go
- security:
- key.rsa // The file that I want to open
- ...
- infrastructure
- openFile.go // The file with the func `openKey``
- server.go // The file with the func that call the func `openKey`
- openFile_test.go // The unit test that calls the func `openKey`
Edit:
Here are the absolute paths of where the binary of my program is located:
/Users/me/Documents/DeĢveloppement/Jean/go/bin
And where my unit tests are located:
/var/folders/tj/8ywtc7pj3rs_j0y6zzldwh5h0000gn/T/go-build221890578/github.com/myaccount/myrepo/infrastructure/_test
Any suggestion?
Thanks!
First, you shouldn't use the same files when running your tests than when running your application in production. Because the test files are accessible to everyone that has access to the repository, which is a security fail.
As said in the comments, the problem is that when running your tests, the working directory is these of the source code (in fact, go copy the whole bunch into a temp directory prior to running the tests), while when you run the program for real, the working directory is the one you are running the command from, hence the wrong relative path.
What I would advise is to use a configuration option to get a the file from which load your file (or a base directory to use with your paths). Either using an environment variable (I strongly encourage you to do that, see the 12factors manofesto for details), a configuration file, a command-line flag, etc.
The go test ./... command changes the current directory to go/src/github.com/myaccount/myrepo/infrastructure before running the tests in that directory. So all relative paths in the tests are relative to the infrastructure directory.
A simple, robust way to allow testing would be to change the signature of your function.
func openKey(relPath string) ([]byte, error) {
if path, err := filepath.Abs(relPath); err != nil {
return nil, err
}
return ioutil.ReadFile(path)
}
This gives you freedom to place key.rsa anywhere during testing, you can for example make a copy of it and store it in infrastructure
An even more elegant and robust way would be to use io.Reader
func openKey(keyFile io.Reader) ([]byte, error) {
return ioutil.ReadAll(keyFile)
}
since you can pass it a strings.Reader for testing. This means that you can test your code without having to rely on the filesystem at all. But that's probably overkill in this case.

Resources