I use viper to load runtime environment specific property files (located under ./configs/*.conf). I am looking to see how I can embed these files in the binary.
Following snippet that loads the files
viper.AddConfigPath("./configs")
viper.SetConfigName("app_dev.conf")
viper.ReadInConfig()
I have tried using the following embed directive
//go:embed configs/*.conf
var resources embed.FS
However getting an error that it cannot load the property files. It works, as expected, if I add the config folder in the same location as the binary.
I realized that I can use io.reader to load Viper config
data, _ := _configFile.ReadFile("configs/app_dev.conf")
err = viper.ReadConfig(strings.NewReader(string(data)))
Related
This question already has answers here:
What's the best way to bundle static resources in a Go program?
(5 answers)
Closed last year.
I'm trying to create a portable Go module that will be re-used in many different projects. This module, in order to function, needs to be able to read non-Go files that are bundled as part of the module. In this case, they're some certificate bundles. The selection of which file to load is dynamic and based on input arguments.
What is the best way to specify the path to these files when loading them? I can't seem to find any Go functions to get a path relative to the module (vs. relative to the executable that is using this module). For example, if my module structure looks like this:
mymodule/
go.mod
go.sum
loadcerts.go
certs/
cert_A.pem
cert_B.pem
And I need to do something like this:
// loadcerts.go
package mymodule
func LoadCerts(useB bool) error {
newCertPool := x509.NewCertPool()
// This is just to show that the selection is dynamic, and since
// there are many different potential files to load, we can't
// embed them all
bundleName := "cert_A.pem"
if useB {
bundleName = "cert_B.pem"
}
pem, err := ioutil.ReadFile(fmt.Sprintf("./certs/%s", bundleName))
if err != nil {
return err
}
if ok := newCertPool.AppendCertsFromPEM(pem); !ok {
return err
}
...
}
Referencing this file with a relative path (./certs/cert1.pem) doesn't work, since Go uses the executable's working directory for relative paths, and this imported module is somewhere entirely different.
How can I load this .pem file that is bundled with the portable module, regardless of where this module is being imported to?
Embed the files in the executable as a file system:
//go:embed certs
var f embed.FS
Read the files from the file system:
pem, err := f.ReadFile(fmt.Sprintf("certs/%s", bundleName))
So I have a pretty basic configuration with Viper reading a .env file from my base directory. I fatal kill the process if there's no .env file. All goes well when running my app normally.
When I run my tests with go test -v ./.., the test framework seems to step into each file's directory, and calls my config init() function each time, so the viper.AddConfigPath(".") is pointing to the wrong location.
this is my directory structure:
/
/restapi
items.go
items_test.go
/util
env.go
main.go
.env
env.go
package util
imports...
// global variables available via util package
var (
Port int
DbURI string
)
func init() {
viper.SetDefault(PORT, 8080)
viper.SetConfigFile(".env")
viper.AddConfigPath(".")
viper.AutomaticEnv()
fmt.Println("---------to see in test printout")
cwd, _ := os.Getwd()
fmt.Println(cwd)
fmt.Println("---------")
if err := viper.ReadInConfig(); err != nil {
log.Fatal("no environment file!")
}
Port = viper.GetInt("PORT")
DbURI = viper.GetString("DB_URI")
}
Every package basically relies on my util package and this init function therefore runs for every test. Is there some way to have viper always pull the .env file from the base directory even when there are tests running? I've tried a few different AddConfigPath() calls. Kinda new to Go. Or is this structure setup for environment variables not going to work since it fails my tests each time?
So apparently the viper.SetConfigFile() call does not respect the viper.AddConfigPath() call... I modified this to using viper.SetConfigName(".env") and it would actually pick up the calls to AddConfigPath, so I could then add config paths for the current directory and parent.
The problem is the path you are giving to the viper.AddConfigPath(".") method, but your env file relative path is not on the test file based on the folder structure tree you shared, it must be this: viper.AddConfigPath("./../util").
I'm building a tiny GCP cloud function in Golang that returns a generated PNG file when calling it via HTTP. I deployed my code via ZIP-Upload in the Google Cloud Console.
Currently it gets called properly and the code gets compiled etc. But in my code I have to load several local files - a font and a .png image.
I bundled those in the ZIP I uploaded and the files are visible in the Source-View in GCP. All files (images, fonts and go-files) are within the same directory.
When calling the cloud function the log states the following:
2019/01/21 14:59:31 open /english.png: no such file or directory
I tried to change the way i build the path to the file in go. I already used /german.png statically, used several attempts to build the path dynamically.
I'm not 100 percent sure if this is the way to go, but it is my first experiment with "serverless" and i am willing to get it done the "right" way.
import "github.com/fogleman/gg"
func main() {
ex, err := os.Executable()
if err != nil {
panic(err)
}
executableDir := filepath.Dir(ex)
img, err :=gg.LoadPNG(path.Join(executableDir, "./english.png"))
if err != nil {
log.Fatal(err)
}
}
Currently the file can not be found in any attempt i made. Maybe the path the images get "deployed" into are different from the ones i tried - i have not found any note on that in the documentation.
I'd obviously expect it to be loaded properly.
For the Go 1.13 according to the documentation, as of today (Jul 2020), source code is located in the ./serverless_function_source_code directory.
A good reference to follow is the buildpack.
I created http functions with the following structure:
api
|--test.txt
|--api.go
And wrote simple function to reply with file content:
package api
import (
"io/ioutil"
"net/http"
)
// FileTest func
func FileTest(w http.ResponseWriter, r *http.Request) {
content, err := ioutil.ReadFile("./test.txt")
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
w.Write([]byte(err.Error()))
return
}
w.Write(content)
}
It returns the file content without any problems. https://us-central1-clickshield24m.cloudfunctions.net/api
So in your case I would try change the path to gg.LoadPNG("./english.png")
./serverless_function_source_code didn't work for me using go113 runtime with vendoring. I wanted to locate a file in the vendor directory.
My file structure is:
myfn
|- main.go
|- go.mod
|- .gcloudignore
My deployment steps are:
cd myfn
go mod vendor
gcloud functions deploy MyFunction --runtime go113 --set-env-vars "PATH_FILES=./src/myfn/vendor/static"
I found this out by deploying a function that listed all files to a certain depth starting with "." and didn't see a serverless_function_source_code directory. But there was src/myfn.
I have a Go project with the following structure and Im struggling to read config file which is located in my project,I need to read the config yaml (which inside the root project) and I should read it inside other package under sub root directory and I got error of not found
myproject
- config.yaml
- cmd
--com
---ftp
----fs.go
Inside the fs.go I need to read the config.yaml and in not having success with it. I try with os.Getwd and also ex, err := os.Executable() and also "../../../" without success, any idea ?
#VonC - suggested to use https://github.com/gobuffalo/packr which can help I guess but the problem is that I need to call it inside the fs.go file and I need to pass this as parameter from the main.go file, is there a better approach ? because I need to pass this parameter in lots of functions...
does viper can help? https://github.com/spf13/viper
My program is CLI program which will be used as bin.
2018: If the binary is built in GOPATH/bin, while your sources are in GOPATH/src, then the relative path would be (at runtime) ../src/myproject.
But a cleaner way would be to embed that file in your binary.
See for instance gobuffalo/packr.
Update Q1 2021: with Go 1.16, you would use the embed package
Go source files that import "embed" can use the //go:embed directive to initialize a variable of type string, []byte, or FS with the contents of files read from the package directory or subdirectories at compile time.
//go:embed hello.txt
var f embed.FS
data, _ := f.ReadFile("hello.txt")
print(string(data))
This question is a follow up to an earlier question of mine. I've closed the question so I hope its okay that I ask a fresh but related question here. Go: embed static files in binary
How do I serve JS files with go-bindata? Do I pass it into html like this
hi.html
<script>{{.Bindata}}></script>
Doesn't seem to work even though I have no compile or JS errors.
Using https://github.com/elazarl/go-bindata-assetfs
Assuming you have the following structure:
myprojectdirectory
├───api
├───cmd
├───datastores
└───ui
├───css
└───js
Where ui is the directory structure you'd like to wrap up and pack into your app...
Generate a source file
The go-bindata-assetfs tool is pretty simple. It will look at the directories you pass to it and generate a source file with variables that can contain the binary data in those files. So make sure your static files are there, and then run the following command from myprojectdirectory:
go-bindata-assetfs ./ui/...
Now, by default, this will create a source file in the package main. Sometimes, this is ok. In my case, it isn't. You can generate a file with a different package name if you'd like:
go-bindata-assetfs.exe -pkg cmd ./ui/...
Put the source file in the correct location
In this case, the generated file bindata_assetfs.go is created in the myprojectdirectory directory (which is incorrect). In my case, I just manually move the file to the cmd directory.
Update your application code
In my app, I already had some code that served files from a directory:
import (
"net/http"
"github.com/gorilla/mux"
)
// Create a router and setup routes
var Router = mux.NewRouter()
Router.PathPrefix("/ui").Handler(http.StripPrefix("/ui", http.FileServer(http.Dir("./ui"))))
// Start listening
http.ListenAndServe("127.0.0.1:3000", Router)
Make sure something like this works properly, first. Then it's trivial to change the FileServer line to:
Router.PathPrefix("/ui").Handler(http.StripPrefix("/ui", http.FileServer(assetFS())))
Compile the app
Now you have a generated source file with your static assets in them. You can now safely remove the 'ui' subdirectory structure. Compile with
go install ./...
And you should have a binary that serves your static assets properly.
Use https://github.com/elazarl/go-bindata-assetfs
From the readme:
go-bindata-assetfs data/...
In your code setup a route with a file server
http.Handle("/", http.FileServer(assetFS()))
Got my answer here: Unescape css input in HTML
var safeCss = template.CSS(`body {background-image: url("paper.gif");}`)