text/template issue Parse() vs. ParseFiles() - go

I'm trying to do some simple work with the text/template package. The sample given at the top of template is what I'm working with.
How do I write the 'parsed' file so template.ParseFiles() properly reads and executes it?
package main
import (
"text/template"
"os"
)
type Inventory struct {
Material string
Count uint
}
func main() {
sweaters := Inventory{"wool", 17}
tmpl, err := template.New("test").Parse("{{.Count}} items are made of {{.Material}}")
// tmpl, err := template.New("test").ParseFiles("file.txt")
if err != nil { panic(err) }
err = tmpl.Execute(os.Stdout, sweaters)
if err != nil { panic(err) }
}
/*
Contents of file.txt:
{{.Count}} items are made of {{.Material}}
Error thrown:
panic: template: test:1: "test" is an incomplete or empty template
goroutine 1 [running]:
main.main()
/tmp/templates/t.go:19 +0x21a
goroutine 2 [syscall]:
created by runtime.main
/var/tmp/portage/dev-lang/go-1.0.1/work/go/src/pkg/runtime/proc.c:221
*/
I have a copy of this code posted at the golang playground here
Edit #1:
I've been doing some research on this issue... since it's the Execute() method that actually throws the exception, and not the ParseFiles() part, I checked the method definition:
// Execute applies a parsed template to the specified data object,
// and writes the output to wr.
func (t *Template) Execute(wr io.Writer, data interface{}) (err error) {
defer errRecover(&err)
value := reflect.ValueOf(data)
state := &state{
tmpl: t,
wr: wr,
line: 1,
vars: []variable{{"$", value}},
}
if t.Tree == nil || t.Root == nil {
state.errorf("%q is an incomplete or empty template", t.name)
}
state.walk(value, t.Root)
return
}
So, on a hunch, I dumped the value of t.Tree for the inline 'non-file' style, tmpl is: &parse.Tree{Name:"test", Root:(*parse.ListNode)(0xf840030700), funcs:[]map[string]interface {}(nil), lex:(*parse.lexer)(nil), token:[2]parse.item{parse.item{typ:6, val:""}, parse.item{typ:9, val:"{{"}}, peekCount:1, vars:[]string(nil)} and
when ran with ParseFiles(), tmpl is: (*parse.Tree)(nil). I find it odd that one is a dereference, and one value is a pointer. This may help solve the riddle

sweaters := Inventory{"wool", 17}
tmpl, err := template.ParseFiles("file.txt")
if err != nil {
panic(err)
}
err = tmpl.ExecuteTemplate(os.Stdout, "file.txt", sweaters)
if err != nil {
panic(err)
}
If you have many files, you can use ParseGlob:
tmpl, err := template.ParseGlob("*.txt")
if err != nil {
panic(err)
}
err = tmpl.ExecuteTemplate(os.Stdout, "file.txt", sweaters)
if err != nil {
panic(err)
}
err = tmpl.ExecuteTemplate(os.Stdout, "file2.txt", sweaters)
if err != nil {
panic(err)
}

There is a little trick in Go template parseFiles.
func parseFiles(t *Template, filenames ...string) (*Template, error) {
if len(filenames) == 0 {
// Not really a problem, but be consistent.
return nil, fmt.Errorf("template: no files named in call to ParseFiles")
}
for _, filename := range filenames {
b, err := ioutil.ReadFile(filename)
if err != nil {
return nil, err
}
s := string(b)
name := filepath.Base(filename)
// First template becomes return value if not already defined,
// and we use that one for subsequent New calls to associate
// all the templates together. Also, if this file has the same name
// as t, this file becomes the contents of t, so
// t, err := New(name).Funcs(xxx).ParseFiles(name)
// works. Otherwise we create a new template associated with t.
var tmpl *Template
if t == nil {
t = New(name)
}
if name == t.Name() {
tmpl = t
} else {
tmpl = t.New(name)
}
_, err = tmpl.Parse(s)
if err != nil {
return nil, err
}
}
return t, nil
}
Only the template with same name will be reuse, otherwise create new one.
as your sample:
tmpl, err := template.New("test").ParseFiles("file.txt")
tmpl is the template named "test", and associated another template named "file.txt", you call Execute on "test" template, this template is a empty template, so raise the error "test is an incomplete or empty template".
It worked when you change the template name to file.txt
tmpl, err := template.New("file.txt").ParseFiles("file.txt")

Related

Loading CSV file into bigquery after os.Create() doesn't load data

I'm trying to run the following flow:
Get data from somewhere
Create new local CSV file, write the data into that file
Upload the CSV to Bigquery
Delete the local file
But it seems to load empty data.
This is the code:
func (c *Client) Do(ctx context.Context) error {
bqClient, err := bigquerypkg.NewBigQueryUtil(ctx, "projectID", "datasetID")
if err != nil {
return err
}
data, err := c.GetSomeData(ctx)
if err != nil {
return err
}
file, err := os.Create("example.csv")
if err != nil {
return err
}
defer file.Close()
// also file need to be delete
writer := csv.NewWriter(file)
defer writer.Flush()
timestamp := time.Now().UTC().Format("2006-01-02 03:04:05.000000000")
for _, d := range data {
csvRow := []string{
d.ID,
d.Name,
timestamp,
}
err = writer.Write(csvRow)
if err != nil {
log.Printf("error writing data to CSV: %v\n", err)
}
}
source := bigquery.NewReaderSource(file)
source.Schema = bigquery.Schema{
{Name: "id", Type: bigquery.StringFieldType},
{Name: "name", Type: bigquery.StringFieldType},
{Name: "createdAt", Type: bigquery.TimestampFieldType},
}
if _, err = bqClient.LoadCsv(ctx, "tableID", source); err != nil {
return err
}
return nil
}
LoadCSV() looks like this:
func (c *Client) LoadCsv(ctx context.Context, tableID string, src bigquery.LoadSource) (string, error) {
loader := c.bigQueryClient.Dataset(c.datasetID).Table(tableID).LoaderFrom(src)
loader.WriteDisposition = bigquery.WriteTruncate
job, err := loader.Run(ctx)
if err != nil {
return "", err
}
status, err := job.Wait(ctx)
if err != nil {
return job.ID(), err
}
if status.Err() != nil {
return job.ID(), fmt.Errorf("job completed with error: %v", status.Err())
}
return job.ID(), nil
}
After running this, bigquery does create the schema but with no data.
If I'm changing os.Create() to os.Open() and the file already exist, everything work. It's like when loading the CSV the file data is not yet written (?)
What's the reason?
The problem I see here is that you don't rewind the file handle's cursor to the beginning of the file. Thus, the next read will be at the end of the file, and will be a 0 byte read. That explains why it seems like there's no content in the file.
https://pkg.go.dev/os#File.Seek can handle this for you.
Actually, the Flush is not relevant, because you're using the same file handle to read the file than you did to write it, so you'll see your own written bytes even without a flush. This would not be the case if the file was opened by a different process or was reopened.
Edit: OP Claims this flush was necessary in their case and I cannot provide evidence to disagree. Flush will not hurt things either.
Demonstration:
package main
import (
"fmt"
"io"
"os"
)
func main() {
f, err := os.CreateTemp("", "data.csv")
if err != nil {
panic(err)
} else {
defer f.Close()
defer os.Remove(f.Name())
}
fmt.Fprintf(f, "hello, world")
fmt.Fprintln(os.Stderr, "Before rewind: ")
if _, err := io.Copy(os.Stderr, f); err != nil {
panic(err)
}
f.Seek(0, io.SeekStart)
fmt.Fprintln(os.Stderr, "\nAfter rewind: ")
if _, err := io.Copy(os.Stderr, f); err != nil {
panic(err)
}
fmt.Fprintln(os.Stderr, "\n")
}
% go run t.go
Before rewind:
After rewind:
hello, world

template: is an incomplete or empty template

I have a following piece of code in which i am trying to send an email using gopkg.in/gomail.v2. I am perfectly able to send the email when the email template is placed in the root directory of the project like this
./
main.go
template.html
// Info defines
type Info struct {
Age int
Name string
}
func (i Info) sendMail() {
fp := filepath.Join("template.html")
t := template.New(fp)
var err error
t, err = t.ParseFiles(fp)
if err != nil {
log.Println(err)
}
var tpl bytes.Buffer
if err := t.Execute(&tpl, i); err != nil {
log.Println(err)
}
result := tpl.String()
// ... email sending logic
}
func main() {
info := &Info{
Name: "name 1",
Age: 20,
}
info.sendMail()
}
but when i change the template directory to emails/template.html and change the filepath to
fp := filepath.Join("emails", "template.html")
then I get the error from t.Execute()
template: "emails/template.html" is an incomplete or empty template
I have also tried
fp, _ := filepath.Abs("emails/template.html")
but got error
template: "/mnt/data/go/test/emails/template.html" is an incomplete or empty template
the path mentioned is correct though.
I changed
if err := t.Execute(&tpl, i); err != nil {
log.Println(err)
}
to
if err := t2.ExecuteTemplate(&tpl, "template.html", i); err != nil {
log.Println(err)
}
and it worked
If I want to use t.Execute(&tpl, i) instead, then I have to specify the templates name as filename while creating the template
t := template.New("template.html")

Trouble getting content type of file in Go

I have a function in which I take in a base64 string and get the content of it (PDF or JPEG).
I read in the base64 content, convert it to bytes and decode it into the file that it is.
I then create a file where I will output the decoded file (JPEG or PDF).
Then I write the bytes to it.
Then I call my GetFileContentType on it and it returns to me an empty string.
If I run the functions separately, as in I first the first function to create the decoded file, and end it. And then call the second function to get the content type, it works and returns it as JPEG or PDF.
What am I doing wrong here?
And is there a better way to do this?
func ConvertToJPEGBase64(
src string,
dst string,
) error {
b, err := ioutil.ReadFile(src)
if err != nil {
return err
}
str := string(b)
byteArray, err := base64.StdEncoding.DecodeString(str)
if err != nil {
return err
}
f, err := os.Create(dst)
if err != nil {
return err
}
if _, err := f.Write(byteArray); err != nil {
return err
}
f.Sync()
filetype, err := client.GetFileContentType(f)
if err != nil {
return err
}
if strings.Contains(filetype, "jpeg") {
// do something
} else {
// do something else
}
return nil
}
// GetFileContentType tells us the type of file
func GetFileContentType(out *os.File) (string, error) {
// Only the first 512 bytes are used to sniff the content type.
buffer := make([]byte, 512)
_, err := out.Read(buffer)
if err != nil {
return "", err
}
contentType := http.DetectContentType(buffer)
return contentType, nil
}
The problem is that GetFileContentType reads from the end of the file. Fix this be seeking back to the beginning of the file before calling calling GetFileContentType:
if _, err := f.Seek(io.SeekStart, 0); err != nil {
return err
}
A better fix is to use the file data that's already in memory. This simplifies the code to the point where there's no need for the GetFileContentType function.
func ConvertToJPEGBase64(
src string,
dst string,
) error {
b, err := ioutil.ReadFile(src)
if err != nil {
return err
}
str := string(b)
byteArray, err := base64.StdEncoding.DecodeString(str)
if err != nil {
return err
}
f, err := os.Create(dst)
if err != nil {
return err
}
defer f.Close() // <-- Close the file on return.
if _, err := f.Write(byteArray); err != nil {
return err
}
fileType := http.DetectContentType(byteArray) // <-- use data in memory
if strings.Contains(fileType, "jpeg") {
// do something
} else {
// do something else
}
return nil
}
More code can be eliminated by using ioutil.WriteFile:
func ConvertToJPEGBase64(src, dst string) error {
b, err := ioutil.ReadFile(src)
if err != nil {
return err
}
byteArray, err := base64.StdEncoding.DecodeString(string(b))
if err != nil {
return err
}
if err := ioutil.WriteFile(dst, byteArray, 0666); err != nil {
return err
}
fileType := http.DetectContentType(byteArray)
if strings.Contains(fileType, "jpeg") {
// do something
} else {
// do something else
}
return nil
}

multiple-value in single-value context no return func

I have a func in Go that simply writes to a buffer. I have no return type set on the func so I am not sure why I am seeing this error. Here is my code:
func Write(buffer *bytes.Buffer, values ...string) {
for _, val := range values
_, err := *buffer.WriteString(val)
if err != nil {
// print error
}
}
_, err := *buffer.WriteString(" ")
if err != nil {
// print error
}
}
It complains at both lines where I have buffer.WriteString. This leads me to believe it has something to do with the return types of the WriteString method on the buffer but I am not experienced enough in Go to know for sure.
Any help would be appreciated.
Edit: Updated code.
You don't need to dereference pointers to call methods in Go. The * operator before buffer.WriteString is applied to the returned values. To dereference buffer you would need to write (*buffer).WriteString, but that's not needed at all:
func Write(buffer *bytes.Buffer, values ...string) {
for _, val := range values {
_, err := buffer.WriteString(val)
if err != nil {
// print error
}
}
_, err := buffer.WriteString(" ")
if err != nil {
// print error
}
}

pass interface pointer and assignment value

I want to write a file cache in Go. I am using gob encoding, and saving to a file, but my get function has some problem:
package main
import (
"encoding/gob"
"fmt"
"os"
)
var (
file = "tmp.txt"
)
type Data struct {
Expire int64
D interface{}
}
type User struct {
Id int
Name string
}
func main() {
user := User{
Id: 1,
Name: "lei",
}
err := set(file, user, 10)
if err != nil {
fmt.Println(err)
return
}
user = User{}
err = get(file, &user)
if err != nil {
fmt.Println(err)
return
}
//user not change.
fmt.Println(user)
}
func set(file string, v interface{}, expire int64) error {
f, err := os.OpenFile(file, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0600)
if err != nil {
return err
}
defer f.Close()
//wrapper data
//save v in data.D
data := Data{
Expire: expire,
D: v,
}
gob.Register(v)
enc := gob.NewEncoder(f)
err = enc.Encode(data)
if err != nil {
return err
}
return nil
}
func get(file string, v interface{}) error {
f, err := os.OpenFile(file, os.O_RDONLY, 0600)
if err != nil {
return err
}
defer f.Close()
var data Data
dec := gob.NewDecoder(f)
err = dec.Decode(&data)
if err != nil {
return err
}
//get v
v = data.D
fmt.Println(v)
return nil
}
The get function passes interface type and I want to change the value, but not change.
http://play.golang.org/p/wV7rBH028o
In order to insert an unknown value into v of type interface{}, you need to use reflection. This is somewhat involved, but if you want to support this in full, you can see how its done by walking through the decoding process in some of the encoding packages (json, gob).
To get you started, here's a basic version of your get function using reflection. This skips a number of checks, and will only decode something that was encoded as a pointer.
func get(file string, v interface{}) error {
f, err := os.OpenFile(file, os.O_RDONLY, 0600)
if err != nil {
return err
}
defer f.Close()
rv := reflect.ValueOf(v)
if rv.Kind() != reflect.Ptr || rv.IsNil() {
panic("need a non nil pointer")
}
var data Data
dec := gob.NewDecoder(f)
err = dec.Decode(&data)
if err != nil {
return err
}
dv := reflect.ValueOf(data.D)
if dv.Kind() != reflect.Ptr {
panic("didn't decode a pointer")
}
rv.Elem().Set(dv.Elem())
return nil
}
I would actually suggest an easier way to handle this in your own code, which is to have the Get function return an interface{}. Since you will know what the possible types are at that point, you can use a type switch to assert the correct value.
An alternative approach is to return directly the value from the file:
func get(file string) (interface{}, error) {
f, err := os.OpenFile(file, os.O_RDONLY, 0600)
if err != nil {
return nil, err
}
defer f.Close()
var data Data
dec := gob.NewDecoder(f)
err = dec.Decode(&data)
if err != nil {
return nil,err
}
fmt.Println(data.D)
return data.D,nil
}
full working example: http://play.golang.org/p/178U_LVC5y

Resources