I'm writing a parser HTML in Go. I need to get HTML and pass it to another function.
I did it so:
Can`t pass "doc" to another function
receivedURL, err := http.Get("http://lavillitacafe.com/")
doc, err := goquery.NewDocumentFromReader(receivedURL.Body)
//"linkScrape" this is another function
contactURL := linkScrape(doc)
and
HTML is transferred in parts to another function.
resp, err := http.Get("http://lavillitacafe.com/")
if err != nil {
fmt.Println(err)
return
}
defer resp.Body.Close()
for true {
bs := make([]byte, 1014)
n, err := resp.Body.Read(bs)
contactURL := linkScrape(bs[:n])
if n == 0 || err != nil{
break
}
}
How do I do it right?
Here's the basic goquery example adjusted to your use case:
package main
import (
"fmt"
"log"
"strings"
"github.com/PuerkitoBio/goquery"
)
func findHeader(d *goquery.Document) string {
header := d.Find("h1").Text()
return header
}
func main() {
// create from a string
data := `
<html>
<head>
<title>My document</title>
</head>
<body>
<h1>Header</h1>
</body>
</html>`
doc, err := goquery.NewDocumentFromReader(strings.NewReader(data))
if err != nil {
log.Fatal(err)
}
fmt.Println(findHeader(doc))
}
Related
Trying to move my golang html templates from files to using embed
Works fine:
func loadTemplates() multitemplate.Render {
r := multitemplate.New()
layouts, err := filepath.Glob("templates/layouts/*.tmpl")
if err != nil {
panic(err.Error())
}
includes, err := filepath.Glob("templates/includes/*.tmpl")
if err != nil {
panic(err.Error())
}
// Generate our templates map from our layouts/ and includes/ directories
for _, layout := range layouts {
files := append(includes, layout)
r.Add(filepath.Base(layout), template.Must(template.ParseFiles(files...)))
log.Println(filepath.Base(layout) + ": " + files[0])
}
return r
}
Very similar code returns blank page, no errors:
//go:embed templates/*
var f embed.FS
func loadTemplates() multitemplate.Render {
r := multitemplate.New()
// Generate our templates map from our layouts/ and includes/ directories
layouts, err := embed.FS.ReadDir(f, "templates/layouts")
if err != nil {
panic(err.Error())
}
for _, layout := range layouts {
embeddedTemplate, err := template.ParseFS(f, "templates/layouts/"+layout.Name(), "templates/includes/base.tmpl")
if err != nil {
log.Println(err)
}
r.Add(layout.Name(), embeddedTemplate)
log.Println(layout.Name() + " loaded")
}
return r
}
I confirmed in the debugger that all templates contain no errors and their respective content. Other embedded files such as static assets work fine and get served ok. Even other templates loaded from a database work fine. Just those from embed end up blank.
Any hints what's happening here?
Thanks!
Edit: Full example:
main.go
package main
import (
"embed"
"html/template"
"log"
"path/filepath"
"github.com/gin-contrib/multitemplate"
"github.com/gin-gonic/gin"
)
//go:embed templates/*
var f embed.FS
func main() {
router := gin.Default()
router.HTMLRender = loadTemplates()
router.GET("/embed", HomeHandlerEmbed(router))
router.GET("/file", HomeHandlerFile(router))
router.Run(":8080")
}
func loadTemplates() multitemplate.Render {
r := multitemplate.New()
//load same template from embed FS
embeddedTemplate, err := template.ParseFS(f, "templates/layouts/home.tmpl", "templates/includes/base.tmpl")
if err != nil {
log.Println(err)
}
r.Add("homeEmbed.tmpl", embeddedTemplate)
log.Println("homeEmbed.tmpl" + " loaded from embed FS")
// load same template from real file system
layoutsFile, err := filepath.Glob("templates/layouts/*.tmpl")
if err != nil {
panic(err.Error())
}
includes, err := filepath.Glob("templates/includes/*.tmpl")
if err != nil {
panic(err.Error())
}
for _, layout := range layoutsFile {
files := append(includes, layout)
r.Add(filepath.Base(layout), template.Must(template.ParseFiles(files...)))
log.Println(filepath.Base(layout) + ": " + files[0])
}
return r
}
func HomeHandlerEmbed(r *gin.Engine) gin.HandlerFunc {
return gin.HandlerFunc(func(c *gin.Context) {
c.HTML(200, "homeEmbed.tmpl", nil)
})
}
func HomeHandlerFile(r *gin.Engine) gin.HandlerFunc {
return gin.HandlerFunc(func(c *gin.Context) {
c.HTML(200, "home.tmpl", nil)
})
}
templates/includes/base.tmpl
<!DOCTYPE html>
<html>
<head>
{{template "head" .}}
</head>
<body>
{{template "body" .}}
</body>
</html>
templates/layouts/home.tmpl
{{define "head"}}<title>Test</title>{{end}}
{{define "body"}}
Body
{{end}}
/file works fine, /embed comes up blank
In function loadTemplates() just fix this line:
embeddedTemplate, err := template.ParseFS(f, "templates/includes/base.tmpl", "templates/layouts/home.tmpl")
In your example patterns will be presented in this sequence:
first: "templates/layouts/home.tmpl"
second: "templates/includes/base.tmpl"
But if I understood correctly, the sequence of patterns is important for the function template.ParseFS, because base.tmpl will be included in all you templates.
The function template.ParseFS reads the templates in the process and tries to generate them.
I have created a simple scraper that takes the top 10 news from a website and returns a JSON with the title and the score. I want to pass the title and the score as HTML template so I can generate a webpage. I'm not familiar with the templating Go language and I don't know how to pass the values for each of the links. Here is the HTML code that I should use and my implementation for now:
<!DOCTYPE html>
<html>
<head><linkrel="stylesheet" href="https://unpkg.com/mvp.css"
/>
</head>
<body>
<h1>{{.PageTitle}}</h1>
<ul>
{{range .Links}}
<li>{{.Title}}: {{.Score}}</li>
{{end}}
</ul>
</body>
</html>
My code:
package main
import (
"encoding/json"
"html/template"
"log"
"net/http"
"strconv"
)
type TopStories struct {
Title string `json:"title"`
Score int `json:"score"`
}
type TopStoriesPayload struct {
TopStories []TopStories
}
type NewsScraper struct {
url string
Data []TopStories
}
type templateData struct {
PageTitle string
Data []TopStories
}
func NewNewsScraper(url string) *NewsScraper {
return &NewsScraper{url: url}
}
func Top10Stories() []string {
req, err := http.NewRequest("GET", "https://hacker-news.firebaseio.com/v0/topstories.json", nil)
if err != nil {
log.Fatal(err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
var IDs []int
json.NewDecoder(resp.Body).Decode(&IDs)
IDs = IDs[:10]
var IDsString []string
for _, id := range IDs {
IDsString = append(IDsString, strconv.Itoa(id))
}
return IDsString
}
func (n *NewsScraper) GetTopStories() {
req, err := http.NewRequest("GET", n.url, nil)
if err != nil {
log.Fatal(err)
}
for _, id := range Top10Stories() {
req.URL.Path = "/v0/item/" + id + ".json"
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
var topStory TopStories
json.NewDecoder(resp.Body).Decode(&topStory)
n.Data = append(n.Data, topStory)
}
}
//create html template handler for top stories
func HTMLHandler(w http.ResponseWriter, r *http.Request) {
scraper := NewNewsScraper("https://hacker-news.firebaseio.com")
scraper.GetTopStories()
tmpl:= template.Must(template.ParseFiles("template.html"))
data := templateData{
PageTitle: "Top Stories",
Data :[]TopStories{
//what should I put here?
},
}
tmpl.Execute(w, data)
}
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/top", HTMLHandler)
http.ListenAndServe(":8080", mux)
}
I see three issues with your code:
a) The template.html file should have space between link & rel
<linkrel="stylesheet" href="https://unpkg.com/mvp.css"/>
to
<link rel="stylesheet" href="https://unpkg.com/mvp.css"/>
b) The template.html file should contain .Data instead of .Links.
c) The go code should be replaced from the below
Data :[]TopStories{
//what should I put here?
},
to
Data : scraper.Data,
I have inserted dummy data into the database.
The sample print function prints 13 and its square as per the code in the console. But I need to pass the entire data to the HTML template(index) where I can see a table of all the numbers passed with their respective square number.
How to pass this data in index HTML?
Number x Number = SquareNumber
1 x 1 = 1
2 x 2 = 4
3 x 3 = 9
...
and so on.
func main() {
db, err := sql.Open("mysql", "root:#/godb")
if err != nil {
panic(err.Error())
}
defer db.Close()
stmtIns, err := db.Prepare("INSERT INTO squarenum VALUES(?, ?, ? )") // ? =
placeholder
if err != nil {
panic(err.Error()) }
defer stmtIns.Close()
stmtOut, err := db.Prepare("SELECT squareNum FROM squareNum WHERE number =
?")
if err != nil {
panic(err.Error())
}
defer stmtOut.Close()
for i := 1; i < 50; i++ {
_, err = stmtIns.Exec(0,i, (i * i))
if err != nil {
panic(err.Error())
}
}
err = stmtOut.QueryRow(13).Scan(&squareNum) // WHERE number = 13
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
tRes:=pageData{}
tRes.SquareNum=squareNum
fmt.Printf("The square number of 13 is: %d \n", squareNum)
// Query another number.. 1 maybe?
err = stmtOut.QueryRow(1).Scan(&squareNum) // WHERE number = 1
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
fmt.Printf("The square number of 1 is: %d \n", squareNum)
http.HandleFunc("/",idx)
http.ListenAndServe(":8888", nil)
}
func idx(w http.ResponseWriter, r *http.Request) {
pd := pageData{
SquareNum: squareNum,
}
err := tpl.ExecuteTemplate(w, "index.html", pd)
if err != nil {
log.Println("LOGGED", err)
http.Error(w, "Internal server error", http.StatusInternalServerError)
return
}
}
package main
import (
"html/template"
"os"
)
// here is your template
const tplString = `
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="ie=edge">
<title>{{.Title}}</title>
</head>
<body>
{{range .Dummies}}
<div>Square of {{.Number}} is {{.Square}}</div>
{{end}}
</body>
</html>
`
var (
tpl *template.Template
err error
)
// the dummydata you talked about
type DummyData struct {
Number int
Square int
}
//some PageData just with dummies and a title
type PageData struct {
Title string
Dummies []*DummyData
}
//here you would be using your sql queries
func createSomeDummies(amount int) []*DummyData {
dummies := make([]*DummyData, amount)
for i := 0; i < amount; i++ {
dd := new(DummyData)
dd.Number = i
dd.Square = i * i
dummies[i] = dd
}
return dummies
}
func main() {
pd := new(PageData)
pd.Title = "Hello Dummies"
pd.Dummies = createSomeDummies(10)
tpl = template.New("index")
tpl, err = tpl.Parse(tplString)
if err != nil {
panic(err)
}
err = tpl.Execute(os.Stdout, pd)
if err != nil {
panic(err)
}
}
This Snippet creates a PageData struct to hold the array of dummydata entries and a Title for the webpage.
type PageData struct {
Title string
Dummies []*DummyData
}
It then uses a function to create 10 dummydata structs.
This array is then assigned to the Dummmies field of the PageData.
pd.Dummies = createSomeDummies(10)
This function is a placeholder for your sql query function you just have to loop over your sql rows instead of manually creating them like i do.
func createSomeDummies(amount int) []*DummyData {
dummies := make([]*DummyData, amount)
for i := 0; i < amount; i++ {
dd := new(DummyData)
dd.Number = i
dd.Square = i * i
dummies[i] = dd
}
return dummies
}
The Template itself inserts the title like so:
<title>{{.Title}}</title>
The Dummies themselves are inserted by a range template directive that works like a for iterator.
The one thing to be careful about is that inside this loop all data points to the DummyData item and not the PageData
{{range .Dummies}}
<div>Square of {{.Number}} is {{.Square}}</div>
{{end}}
Then the template is parsed. Errors will halt execution and print the error message.
tpl = template.New("index")
tpl, err = tpl.Parse(tplString)
if err != nil {
panic(err)
}
And finally the template will be rendered to Stdout. For use in a http Handler you would have to use the http.ResponseWriter instead.
Once again errors halt execution and print the error message.
err = tpl.Execute(os.Stdout, pd)
if err != nil {
panic(err)
}
Working Example here:
Go Playground
How can I execute function in templates that returns no value? Here is example:
func main() {
u, err := url.Parse("http://example.com/test?param1=true¶m2=true")
if err != nil {
log.Fatal(err)
}
m := u.Query()
m.Del("param1") // param1 successful deleted!
u.RawQuery = m.Encode()
fmt.Println(u.RawQuery)
const tmpl = `
{{$m := .Query}}
{{$m.Del "param2"}} <!-- failed to delete param2! -->
{{.RawQuery}}
`
t := template.Must(template.New("").Parse(tmpl))
err = t.Execute(os.Stdout, u)
if err != nil {
log.Println("executing template:", err)
}
}
see this code in play.golang.org
I know that in templates shouldn't be much logic, but ignorance of running function that returns no value seems to me interesting issue.
Templates in Go are not like those in other languages (e.g. PHP). Use template.FuncMap to create custom functions for your templates.
package main
import (
"fmt"
"log"
"net/url"
"os"
"text/template"
)
func main() {
funcMap := template.FuncMap{
"delete": deleteMap,
}
u, err := url.Parse("http://example.com/test?param1=true¶m2=true")
if err != nil {
log.Fatal(err)
}
u = deleteMap(u, "param1") // works in regular code and templates
fmt.Println(u.RawQuery)
const tmpl = `
{{$m := delete . "param2"}} <!-- WORKS! -->
{{$m.RawQuery}}
`
t := template.New("").Funcs(funcMap)
t = template.Must(t.Parse(tmpl))
err = t.Execute(os.Stdout, u)
if err != nil {
log.Println("executing template:", err)
}
}
func deleteMap(u *url.URL, key string) *url.URL {
m := u.Query()
m.Del(key) // key successful deleted!
u.RawQuery = m.Encode()
return u
}
Or, try the playground version.
I'm rather new to Stackoverflow. I'm stuck at this problem. I'm trying to make a map.
package main
import (
"encoding/json"
"fmt"
"html/template"
"io/ioutil"
"log"
"net/http"
"net/url"
"os"
)
func main() {
http.HandleFunc("/", handler)
http.HandleFunc("/showimage", showimage)
fmt.Println("listening...")
err := http.ListenAndServe(GetPort(), nil)
if err != nil {
log.Fatal("ListenAndServe: ", err)
}
}
func GetPort() string {
var port = os.Getenv("PORT")
if port == "" {
port = "4747"
fmt.Println("INFO: No PORT environment variable detected, defaulting to " + port)
}
return ":" + port
}
func handler (w http.ResponseWriter, r *http.Request) {
fmt.Fprint(w, rootForm)
}
const rootForm =
`<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Flickr photos</title>
</head>
<body>
<h1>Flickr photos</h1>
<p>Find photos by tags!</p>
<form action="/showimage" method="post" accept-charset="utf-8">
<input type="text" name="str" value="Type Tags..." id="str">
<input type="submit" value=".. and see the images!">
</form>
</body>
</html>`
var upperTemplate = template.Must(template.New("showimage").Parse(upperTemplateHTML)) //irrelevant to issue here
func showimage(w http.ResponseWriter, r *http.Request) {
tag := r.FormValue("str")
safeTag := url.QueryEscape(tag)
fullUrl := fmt.Sprintf("https://api.instagram.com/v1/users/search?q=%s&access_token=ACCESS-TOKEN&count=1", safeTag)
client := &http.Client{}
req, err := http.NewRequest("GET", fullUrl, nil)
if err != nil {
log.Fatal("NewRequest: ", err)
return
}
resp, requestErr := client.Do(req)
if requestErr != nil {
log.Fatal("Do: ", requestErr)
return
}
defer resp.Body.Close()
body, dataReadErr := ioutil.ReadAll(resp.Body)
if dataReadErr != nil {
log.Fatal("ReadAll: ", dataReadErr)
return
}
res := make(map[string][]map[string]interface{})
However, when I try to put data into the interface
json.Unmarshal(body, &res)
userid, _ := res["data"][0]["username"]
queryUrl := fmt.Sprintf("http://instagram.com/%s", userid)
I get the error
http: panic serving [::1]:63089: runtime error: index out of range
goroutine 28 [running]:
any idea why? This error is resolved if I remove the [] in res:= and userid :=, but I won't be able to access the data I want.
It is wrong with "res"'s using. "res" is a map(key is string, value is a slice),so res["data"] may be a nil in your code, and it will panic when you use res["data"][0] .You should do like this:
json.Unmarshal(body, &res)
s, ok := res["data"]
if ok {
if len(s)>0{
userid , ok := s[0]["username"]
if ok{
queryUrl := fmt.Sprintf("http://instagram.com/%s", userid)
}
}
}