I tried to use i18n but it can't work
i18n is below.
http://beego.me/docs/module/i18n.md
routers/init.go:
// Initialized language type list.
langs := strings.Split(models.Cfg.MustValue("lang", "types"), "|")
names := strings.Split(models.Cfg.MustValue("lang", "names"), "|")
langTypes = make([]*langType, 0, len(langs))
for i, v := range langs {
langTypes = append(langTypes, &langType{
Lang: v,
Name: names[i],
})
}
for _, lang := range langs {
beego.Trace("Loading language: " + lang)
if err := i18n.SetMessage(lang, "conf/"+"locale_"+lang+".ini"); err != nil {
beego.Error("Fail to set message file: " + err.Error())
return
}
}
routers/router.go
// Initialized language type list.
langs := strings.Split(models.Cfg.MustValue("lang", "types"), "|")
names := strings.Split(models.Cfg.MustValue("lang", "names"), "|")
langTypes = make([]*langType, 0, len(langs))
for i, v := range langs {
langTypes = append(langTypes, &langType{
Lang: v,
Name: names[i],
})
}
for _, lang := range langs {
beego.Trace("Loading language: " + lang)
if err := i18n.SetMessage(lang, "conf/"+"locale_"+lang+".ini"); err != nil {
beego.Error("Fail to set message file: " + err.Error())
return
}
}
I just copied and paste.
It is my full code.
https://github.com/shinriyo/sample_i18n
I'd like to know minimum code.
it is error
bee run
2015/05/24 17:48:56 [INFO] Uses 'sample_i18n' as 'appname'
2015/05/24 17:48:56 [INFO] Initializing watcher...
2015/05/24 17:48:56 [TRAC] Directory(/Users/shinriyo/src/sample_i18n/controllers)
2015/05/24 17:48:56 [TRAC] Directory(/Users/shinriyo/src/sample_i18n)
2015/05/24 17:48:56 [TRAC] Directory(/Users/shinriyo/src/sample_i18n/routers)
2015/05/24 17:48:56 [TRAC] Directory(/Users/shinriyo/src/sample_i18n/tests)
2015/05/24 17:48:56 [INFO] Start building...
main.go:4:2:
routers/init.go:2:1: expected 'package', found 'IDENT' langs
2015/05/24 17:48:56 [ERRO] ============== Build failed ===================
The best way I found was that:
Create translation files inside conf/ folder
Configure your translation list at conf/app.conf. My languages are en-US and pt-BR
lang_types = "en-US|pt-BR"
Create file baseController.go inside controller/baseController/ folder
with this code
package controllers
import (
"github.com/astaxie/beego"
"github.com/beego/i18n"
"strings"
)
var langTypes []string // Languages that are supported.
// baseController represents base router for all other app routers.
// It implemented some methods for the same implementation;
// thus, it will be embedded into other routers.
type BaseController struct {
beego.Controller // Embed struct that has stub implementation of the interface.
i18n.Locale // For i18n usage when process data and render template.
}
func init() {
beego.AddFuncMap("i18n", i18n.Tr)
// Initialize language type list.
langTypes = strings.Split(beego.AppConfig.String("lang_types"), "|")
// Load locale files according to language types.
for _, lang := range langTypes {
beego.Trace("Loading language: " + lang)
if err := i18n.SetMessage(lang, "conf/"+"locale_"+lang+".ini"); err != nil {
beego.Error("Fail to set message file:", err)
return
}
}
}
// Prepare implemented Prepare() method for baseController.
// It's used for language option check and setting.
func (this *BaseController) Prepare() {
// Reset language option.
this.Lang = "" // This field is from i18n.Locale.
beego.Trace("running prepare")
// 1. Get language information from 'Accept-Language'.
al := this.Ctx.Request.Header.Get("Accept-Language")
if len(al) > 4 {
al = al[:5] // Only compare first 5 letters.
if i18n.IsExist(al) {
this.Lang = al
}
}
beego.Trace("Accept-Language is " + al)
// 2. Default language is English.
if len(this.Lang) == 0 {
this.Lang = "en-US"
}
// Set template level language option.
this.Data["Lang"] = this.Lang
}
Use your custom BaseController in other controller.
package controllers
import (
bc "github.com/myFirstApp/controllers/baseController"
)
type LoginController struct {
bc.BaseController
}
func (c *LoginController) Get() {
c.TplNames = "login.tpl"
}
Use i18n in templates
{{i18n .Lang "login.capitalized"}}
My locale_en-US.ini
login = login
[login]
uppercase = LOGIN
capitalized = Login
package controllers
import (
"github.com/astaxie/beego"
"github.com/beego/i18n"
"os"
"strings"
)
type BaseController struct {
beego.Controller
}
func (b *BaseController) settingLocales() {
// load locales with locale_LANG.ini files
langs := "en-US|zh-CN"
for _, lang := range strings.Split(langs, "|") {
lang = strings.TrimSpace(lang)
files := []string{"conf/" + "locale_" + lang + ".ini"}
if fh, err := os.Open(files[0]); err == nil {
fh.Close()
} else {
files = nil
}
if err := i18n.SetMessage(lang, "conf/"+"locale_"+lang+".ini", files...); err != nil {
beego.Error("Fail to set message file: " + err.Error())
os.Exit(2)
}
}
Las := i18n.ListLangs()
beego.Debug("langs:", Las)
}
func (b *BaseController) Get() {
b.settingLocales()
if i18n.IsExist("en-US") {
b.Data["json"] = i18n.Tr("en-US", "hi")
} else {
b.Data["json"] = s
}
b.ServeJson()
}
create file locale_en-US.ini and locale_zh-CN.ini in conf/
in locale_en-US.ini like:
hi = hello
bye = goodbye
Related
I have a struct which contains a type based on an enum. I am trying to render it to a user friendly string. Here's minimum viable code:
package main
import (
"fmt"
"gopkg.in/yaml.v3"
)
type Job struct {
Engine Engine `json:"Engine" yaml:"Engine"`
}
//go:generate stringer -type=Engine --trimprefix=Engine
type Engine int
const (
engineUnknown Engine = iota // must be first
EngineDocker
engineDone // must be last
)
func main() {
j := Job{Engine: EngineDocker}
fmt.Printf("%+v\n\n", j)
out, _ := yaml.Marshal(j)
fmt.Println(string(out))
}
Here's the generated code:
// Code generated by "stringer -type=Engine --trimprefix=Engine"; DO NOT EDIT.
package main
import "strconv"
func _() {
// An "invalid array index" compiler error signifies that the constant values have changed.
// Re-run the stringer command to generate them again.
var x [1]struct{}
_ = x[engineUnknown-0]
_ = x[EngineDocker-1]
_ = x[engineDone-2]
}
const _Engine_name = "engineUnknownDockerengineDone"
var _Engine_index = [...]uint8{0, 13, 19, 29}
func (i Engine) String() string {
if i < 0 || i >= Engine(len(_Engine_index)-1) {
return "Engine(" + strconv.FormatInt(int64(i), 10) + ")"
}
return _Engine_name[_Engine_index[i]:_Engine_index[i+1]]
}
Here's the output:
{Engine:1}
Engine: 1
Here's what I'd like the output to be:
{Engine:Docker}
Engine: Docker
I thought the String() in the generated file would accomplish this. Is there any way to do this? Thanks!
yaml marshaler doesn't use String method. Instead YAML uses encoding.TextMarshaler and encoding.TextUnmarshaler interfaces. Actually, all other codec schemes - JSON, XML, TOML, etc. - use those interfaces to read/write the values. So, if you implement those methods for your type, you will receive all other codecs for free.
Here is an example how to make a human-readable encoding for your enum: https://go.dev/play/p/pEcBmAM-oZJ
type Engine int
const (
engineUnknown Engine = iota // must be first
EngineDocker
engineDone // must be last
)
var engineNames []string
var engineNameToValue map[string]Engine
func init() {
engineNames = []string{"Unknown", "Docker"}
engineNameToValue = make(map[string]Engine)
for i, name := range engineNames {
engineNameToValue[strings.ToLower(name)] = Engine(i)
}
}
func (e Engine) String() string {
if e < 0 || int(e) >= len(engineNames) {
panic(fmt.Errorf("Invalid engine code: %d", e))
}
return engineNames[e]
}
func ParseEngine(text string) (Engine, error) {
i, ok := engineNameToValue[strings.ToLower(text)]
if !ok {
return engineUnknown, fmt.Errorf("Invalid engine name: %s", text)
}
return i, nil
}
func (e Engine) MarshalText() ([]byte, error) {
return []byte(e.String()), nil
}
func (e *Engine) UnmarshalText(text []byte) (err error) {
name := string(text)
*e, err = ParseEngine(name)
return
}
How it works:
func main() {
j := Job{Engine: EngineDocker}
fmt.Printf("%#v\n\n", j)
out, err := yaml.Marshal(j)
if err != nil {
panic(err)
}
fmt.Printf("YAML: %s\n", string(out))
var jj Job
err = yaml.Unmarshal(out, &jj)
if err != nil {
panic(err)
}
fmt.Printf("%#v\n\n", jj)
// == JSON ==
out, err = json.Marshal(j)
if err != nil {
panic(err)
}
fmt.Printf("JSON: %s\n", string(out))
var jjs Job
err = json.Unmarshal(out, &jjs)
if err != nil {
panic(err)
}
fmt.Printf("%#v\n\n", jjs)
}
the output
main.Job{Engine:1}
YAML: Engine: Docker
main.Job{Engine:1}
JSON: {"Engine":"Docker"}
main.Job{Engine:1}
See? It writes and reads strings to both YAML and JSON without any extra effort.
I'd like to write a method that will populate a Go Language array with the common timezones that are accepted by the time.Format() call, for use in an HTML template (Form select to allow them to read and choose their timezone). Is there a common way to do this?
To get a list of time zones, you can use something like:
package main
import (
"fmt"
"io/ioutil"
"strings"
)
var zoneDirs = []string{
// Update path according to your OS
"/usr/share/zoneinfo/",
"/usr/share/lib/zoneinfo/",
"/usr/lib/locale/TZ/",
}
var zoneDir string
func main() {
for _, zoneDir = range zoneDirs {
ReadFile("")
}
}
func ReadFile(path string) {
files, _ := ioutil.ReadDir(zoneDir + path)
for _, f := range files {
if f.Name() != strings.ToUpper(f.Name()[:1]) + f.Name()[1:] {
continue
}
if f.IsDir() {
ReadFile(path + "/" + f.Name())
} else {
fmt.Println((path + "/" + f.Name())[1:])
}
}
}
output:
Africa/Abidjan
Africa/Accra
Africa/Addis_Ababa
Africa/Algiers
Africa/Asmara
Africa/Asmera
Africa/Bamako
Africa/Bangui
...
Go's time pkg uses a timezone database.
You can load a timezone location like this:
loc, err := time.LoadLocation("America/Chicago")
if err != nil {
// handle error
}
t := time.Now().In(loc)
The Format function is not related to setting the time zone, this function takes a fixed reference time that allows you to format the date how you would like. Take a look at the time pkg docs.
For instance:
fmt.Println(t.Format("MST")) // outputs CST
Here is a running example
Here is an example: https://play.golang.org/p/KFGQiW5A1P-
package main
import (
"fmt"
"io/ioutil"
"strings"
"unicode"
)
func main() {
fmt.Println(GetOsTimeZones())
}
func GetOsTimeZones() []string {
var zones []string
var zoneDirs = []string{
// Update path according to your OS
"/usr/share/zoneinfo/",
"/usr/share/lib/zoneinfo/",
"/usr/lib/locale/TZ/",
}
for _, zd := range zoneDirs {
zones = walkTzDir(zd, zones)
for idx, zone := range zones {
zones[idx] = strings.ReplaceAll(zone, zd+"/", "")
}
}
return zones
}
func walkTzDir(path string, zones []string) []string {
fileInfos, err := ioutil.ReadDir(path)
if err != nil {
return zones
}
isAlpha := func(s string) bool {
for _, r := range s {
if !unicode.IsLetter(r) {
return false
}
}
return true
}
for _, info := range fileInfos {
if info.Name() != strings.ToUpper(info.Name()[:1])+info.Name()[1:] {
continue
}
if !isAlpha(info.Name()[:1]) {
continue
}
newPath := path + "/" + info.Name()
if info.IsDir() {
zones = walkTzDir(newPath, zones)
} else {
zones = append(zones, newPath)
}
}
return zones
}
I have some code for handling a YAML config file that's getting a little out-of-control w/ type assertions and I feel like there must be a better way to do this.
Here's the relevant snippet from my config file:
plugins:
taxii20:
default: default
api_roots:
default:
auth:
- ldap
- mutualtls
collections:
all:
selector: g.V().Save("<type>").Save("<created>").All()
selector_query_lang: gizmo
And here's my parsing code:
func parseTaxiiConfig() {
plg.ConfigMutex.Lock()
taxiiConfig := plg.ConfigData.Plugins["taxii20"].(map[interface{}]interface{})
ConfigData = &Config{}
if taxiiConfig["default"] != nil {
ConfigData.DefaultRoot = taxiiConfig["default"].(string)
}
if taxiiConfig["api_roots"] != nil {
ConfigData.APIRoots = make([]model.APIRoot, 0)
iroots := taxiiConfig["api_roots"].(map[interface{}]interface{})
for iname, iroot := range iroots {
root := model.APIRoot{Name: iname.(string)}
authMethods := iroot.(map[interface{}]interface{})["auth"].([]interface{})
root.AuthMethods = make([]string, 0)
for _, method := range authMethods {
root.AuthMethods = append(root.AuthMethods, method.(string))
}
collections := iroot.(map[interface{}]interface{})["collections"].(map[interface{}]interface{})
root.Collections = make([]model.Collection, 0)
for icolName, icollection := range collections {
collection := model.Collection{Name: icolName.(string)}
collection.Selector = icollection.(map[interface{}]interface{})["selector"].(string)
collection.SelectorQueryLang = icollection.(map[interface{}]interface{})["selector_query_lang"].(string)
root.Collections = append(root.Collections, collection)
}
ConfigData.APIRoots = append(ConfigData.APIRoots, root)
}
}
plg.ConfigMutex.Unlock()
// debug
fmt.Println(ConfigData)
}
The code works as intended, but there's just so many type assertions here and I can't shake the feeling that I'm missing a better way.
One possible critical item of note, as the config implies, this is configuration for a Caddy-style plugin system, so the main config parser cannot know ahead of time what the shape of the plugin config will look like. It has to delegate processing of the plugin's portion of the config file to the plugin itself.
Here's what I came up with instead. Much more readable.
// Config represents TAXII 2.0 plugin structure
type Config struct {
DefaultRoot string
APIRoots []model.APIRoot
}
// Intermediate config for mapstructure
type configRaw struct {
DefaultRoot string `mapstructure:"default"`
APIRoots map[string]apiRootRaw `mapstructure:"api_roots"`
}
type apiRootRaw struct {
AuthMethods []string `mapstructure:"auth"`
Collections map[string]collectionRaw `mapstructure:"collections"`
}
type collectionRaw struct {
Selector string `mapstructure:"selector"`
SelectorQueryLang string `mapstructure:"selector_query_lang"`
}
func parseTaxiiConfig() error {
plg.ConfigMutex.Lock()
defer plg.ConfigMutex.Unlock()
taxiiConfig := plg.ConfigData.Plugins["taxii20"].(map[interface{}]interface{})
fmt.Println(taxiiConfig)
ConfigData = &Config{}
raw := &configRaw{}
err := mapstructure.Decode(taxiiConfig, raw)
if err != nil {
return err
}
ConfigData.DefaultRoot = raw.DefaultRoot
ConfigData.APIRoots = make([]model.APIRoot, 0)
for name, root := range raw.APIRoots {
apiRoot := model.APIRoot{Name: name}
apiRoot.AuthMethods = root.AuthMethods
apiRoot.Collections = make([]model.Collection, 0)
for colName, col := range root.Collections {
collection := model.Collection{Name: colName}
collection.Selector = col.Selector
collection.SelectorQueryLang = col.SelectorQueryLang
apiRoot.Collections = append(apiRoot.Collections, collection)
}
ConfigData.APIRoots = append(ConfigData.APIRoots, apiRoot)
}
return nil
}
Is there a native way for inplace url parameters in native Go?
For Example, if I have a URL: http://localhost:8080/blob/123/test I want to use this URL as /blob/{id}/test.
This is not a question about finding go libraries. I am starting with the basic question, does go itself provide a basic facility to do this natively.
There is no built in simple way to do this, however, it is not hard to do.
This is how I do it, without adding a particular library. It is placed in a function so that you can invoke a simple getCode() function within your request handler.
Basically you just split the r.URL.Path into parts, and then analyse the parts.
// Extract a code from a URL. Return the default code if code
// is missing or code is not a valid number.
func getCode(r *http.Request, defaultCode int) (int, string) {
p := strings.Split(r.URL.Path, "/")
if len(p) == 1 {
return defaultCode, p[0]
} else if len(p) > 1 {
code, err := strconv.Atoi(p[0])
if err == nil {
return code, p[1]
} else {
return defaultCode, p[1]
}
} else {
return defaultCode, ""
}
}
Well, without external libraries you can't, but may I recommend two excellent ones:
httprouter - https://github.com/julienschmidt/httprouter - is extremely fast and very lightweight. It's faster than the standard library's router, and it creates 0 allocations per call, which is great in a GCed language.
Gorilla Mux - http://www.gorillatoolkit.org/pkg/mux -
Very popular, nice interface, nice community.
Example usage of httprouter:
func Hello(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
fmt.Fprintf(w, "hello, %s!\n", ps.ByName("name"))
}
func main() {
router := httprouter.New()
router.GET("/hello/:name", Hello)
log.Fatal(http.ListenAndServe(":8080", router))
}
What about trying using regex, and find a named group in your url, like playground:
package main
import (
"fmt"
"net/url"
"regexp"
)
var myExp = regexp.MustCompile(`/blob/(?P<id>\d+)/test`) // use (?P<id>[a-zA-Z]+) if the id is alphapatic
func main() {
s := "http://localhost:8080/blob/123/test"
u, err := url.Parse(s)
if err != nil {
panic(err)
}
fmt.Println(u.Path)
match := myExp.FindStringSubmatch(s) // or match := myExp.FindStringSubmatch(u.Path)
result := make(map[string]string)
for i, name := range myExp.SubexpNames() {
if i != 0 && name != "" {
result[name] = match[i]
}
}
fmt.Printf("id: %s\n", result["id"])
}
output
/blob/123/test
id: 123
Below full code to use it with url, that is receiving http://localhost:8000/hello/John/58 and returning http://localhost:8000/hello/John/58:
package main
import (
"fmt"
"net/http"
"regexp"
"strconv"
)
var helloExp = regexp.MustCompile(`/hello/(?P<name>[a-zA-Z]+)/(?P<age>\d+)`)
func hello(w http.ResponseWriter, req *http.Request) {
match := helloExp.FindStringSubmatch(req.URL.Path)
if len(match) > 0 {
result := make(map[string]string)
for i, name := range helloExp.SubexpNames() {
if i != 0 && name != "" {
result[name] = match[i]
}
}
if _, err := strconv.Atoi(result["age"]); err == nil {
fmt.Fprintf(w, "Hello, %v year old named %s!", result["age"], result["name"])
} else {
fmt.Fprintf(w, "Sorry, not accepted age!")
}
} else {
fmt.Fprintf(w, "Wrong url\n")
}
}
func main() {
http.HandleFunc("/hello/", hello)
http.ListenAndServe(":8090", nil)
}
How about writing your own url generator (extend net/url a little bit) as below.
// --- This is how does it work like --- //
url, _ := rest.NewURLGen("http", "stack.over.flow", "1234").
Pattern(foo/:foo_id/bar/:bar_id).
ParamQuery("foo_id", "abc").
ParamQuery("bar_id", "xyz").
ParamQuery("page", "1").
ParamQuery("offset", "5").
Do()
log.Printf("url: %s", url)
// url: http://stack.over.flow:1234/foo/abc/bar/xyz?page=1&offset=5
// --- Your own url generator would be like below --- //
package rest
import (
"log"
"net/url"
"strings"
"straas.io/base/errors"
"github.com/jinzhu/copier"
)
// URLGen generates request URL
type URLGen struct {
url.URL
pattern string
paramPath map[string]string
paramQuery map[string]string
}
// NewURLGen new a URLGen
func NewURLGen(scheme, host, port string) *URLGen {
h := host
if port != "" {
h += ":" + port
}
ug := URLGen{}
ug.Scheme = scheme
ug.Host = h
ug.paramPath = make(map[string]string)
ug.paramQuery = make(map[string]string)
return &ug
}
// Clone return copied self
func (u *URLGen) Clone() *URLGen {
cloned := &URLGen{}
cloned.paramPath = make(map[string]string)
cloned.paramQuery = make(map[string]string)
err := copier.Copy(cloned, u)
if err != nil {
log.Panic(err)
}
return cloned
}
// Pattern sets path pattern with placeholder (format `:<holder_name>`)
func (u *URLGen) Pattern(pattern string) *URLGen {
u.pattern = pattern
return u
}
// ParamPath builds path part of URL
func (u *URLGen) ParamPath(key, value string) *URLGen {
u.paramPath[key] = value
return u
}
// ParamQuery builds query part of URL
func (u *URLGen) ParamQuery(key, value string) *URLGen {
u.paramQuery[key] = value
return u
}
// Do returns final URL result.
// The result URL string is possible not escaped correctly.
// This is input for `gorequest`, `gorequest` will handle URL escape.
func (u *URLGen) Do() (string, error) {
err := u.buildPath()
if err != nil {
return "", err
}
u.buildQuery()
return u.String(), nil
}
func (u *URLGen) buildPath() error {
r := []string{}
p := strings.Split(u.pattern, "/")
for i := range p {
part := p[i]
if strings.Contains(part, ":") {
key := strings.TrimPrefix(p[i], ":")
if val, ok := u.paramPath[key]; ok {
r = append(r, val)
} else {
if i != len(p)-1 {
// if placeholder at the end of pattern, it could be not provided
return errors.Errorf("placeholder[%s] not provided", key)
}
}
continue
}
r = append(r, part)
}
u.Path = strings.Join(r, "/")
return nil
}
func (u *URLGen) buildQuery() {
q := u.URL.Query()
for k, v := range u.paramQuery {
q.Set(k, v)
}
u.RawQuery = q.Encode()
}
With net/http the following would trigger when calling localhost:8080/blob/123/test
http.HandleFunc("/blob/", yourHandlerFunction)
Then inside yourHandlerFunction, manually parse r.URL.Path to find 123.
Note that if you don't add a trailing / it won't work. The following would only trigger when calling localhost:8080/blob:
http.HandleFunc("/blob", yourHandlerFunction)
As of 19-Sep-22, with go version 1.19, instance of http.request URL has a method called Query, which will return a map, which is a parsed query string.
func helloHandler(res http.ResponseWriter, req *http.Request) {
// when request URL is `http://localhost:3000/?first=hello&second=world`
fmt.Println(req.URL.Query()) // outputs , map[second:[world] first:[hello]]
res.Write([]byte("Hello World Web"))
}
No way without standard library. Why you don't want to try some library? I think its not so hard to use it, just go get bla bla bla
I use Beego. Its MVC style.
how about a simple utility function ?
func withURLParams(u url.URL, param, val string) url.URL{
u.Path = strings.ReplaceAll(u.Path, param, val)
return u
}
you can use it like this:
u, err := url.Parse("http://localhost:8080/blob/:id/test")
if err != nil {
return nil, err
}
u := withURLParams(u, ":id","123")
// now u.String() is http://localhost:8080/blob/123/test
If you need a framework and you think it will be slow because it's 'bigger' than a router or net/http, then you 're wrong.
Iris is the fastest go web framework that you will ever find, so far according to all benchmarks.
Install by
go get gopkg.in/kataras/iris.v6
Django templates goes easy with iris:
import (
"gopkg.in/kataras/iris.v6"
"gopkg.in/kataras/iris.v6/adaptors/httprouter"
"gopkg.in/kataras/iris.v6/adaptors/view" // <-----
)
func main() {
app := iris.New()
app.Adapt(iris.DevLogger())
app.Adapt(httprouter.New()) // you can choose gorillamux too
app.Adapt(view.Django("./templates", ".html")) // <-----
// RESOURCE: http://127.0.0.1:8080/hi
// METHOD: "GET"
app.Get("/hi", hi)
app.Listen(":8080")
}
func hi(ctx *iris.Context){
ctx.Render("hi.html", iris.Map{"Name": "iris"})
}
I'm trying to run a go program using LiteIDE x22 but I get the message
C:/Go/bin/go.exe build [C:/Users/admins/Desktop/desktp/worm_scraper-master]
worm_scraper.go:11:2: cannot find package "github.com/codegangsta/cli" in any of:
C:\Go\src\pkg\github.com\codegangsta\cli (from $GOROOT)
C:\users\admins\gostuff\src\github.com\codegangsta\cli (from $GOPATH)
worm_scraper.go:12:2: cannot find package "github.com/puerkitobio/goquery" in any of:
C:\Go\src\pkg\github.com\puerkitobio\goquery (from $GOROOT)
C:\users\admins\gostuff\src\github.com\puerkitobio\goquery (from $GOPATH)
Error: process exited with code 1.
I think this means it's looking for it on my harddrive instead of online right? (btw I'm pretty clueless about programming just trying to something some else wrote)
how to I get it to access the web?
here's the full code
package main
import (
"errors"
"fmt"
"os"
"os/exec"
"regexp"
"strings"
"github.com/codegangsta/cli"
"github.com/puerkitobio/goquery"
)
const (
MainSite = "https://parahumans.wordpress.com/"
TableOfContents = "https://parahumans.wordpress.com/table-of-contents/"
)
type Arc struct {
Identifier string
Title string
Chapters []Chapter
}
type Chapter struct {
Title string
Url string
Tags []string
Paragraphs []Paragraph
Retries int
DatePosted string
}
type Paragraph string
// Format the paragraph
func (p *Paragraph) Format() {
s := string(*p)
// Handle emphasis
s = strings.Replace(s, "<em>", "*", -1)
s = strings.Replace(s, "</em>", "*", -1)
s = strings.Replace(s, "<i>", "*", -1)
s = strings.Replace(s, "</i>", "*", -1)
// Handle bold
s = strings.Replace(s, "<strong>", "**", -1)
s = strings.Replace(s, "</strong>", "**", -1)
s = strings.Replace(s, "<b>", "**", -1)
s = strings.Replace(s, "</b>", "**", -1)
// Remove new lines
s = strings.Replace(s, "\n", "", -1)
// And random double spaces
s = strings.Replace(s, ". ", ". ", -1)
*p = Paragraph(s)
}
// Return the Arc that the given chapter belongs to
func (ch *Chapter) WhichArc(arcList []*Arc) (*Arc, error) {
for _, arc := range arcList {
if strings.Replace(ch.Title[:2], ".", "", -1) == arc.Identifier {
return arc, nil
}
}
return &Arc{}, errors.New("chapter '" + ch.Title + "' did not match any Arcs")
}
// Parse a chapter and return it
func (ch *Chapter) Parse(done chan bool) {
if ch.Retries > 3 {
panic("Chapter url '" + ch.Url + "' has timed out too many times")
}
// Get the chapter
if strings.HasPrefix(ch.Url, "http") == false {
// Make sure it begins with http so goquery can use it
ch.Url = "https://" + ch.Url
}
doc, err := goquery.NewDocument(ch.Url)
if err != nil {
// Try again
ch.Retries++
go ch.Parse(done)
return
}
// Set the new chapter title
ch.Title = doc.Find("h1.entry-title").Text()
// Set the tags
doc.Find(".entry-meta a[rel=tag]").Each(func(_ int, s *goquery.Selection) {
ch.Tags = append(ch.Tags, s.Text())
if len(ch.Tags) == 0 {
ch.Tags = append(ch.Tags, "NONE")
}
})
// Get the date it was posted
ch.DatePosted = doc.Find("time.entry-date").Text()
// Now we'll get all the paragraphs
doc.Find(".entry-content > p").Each(func(_ int, s *goquery.Selection) {
// Check for the previous/next links
if len(s.Find("a").Nodes) > 0 {
return
}
// Get the paragraph HTML
st, _ := s.Html()
para := Paragraph("")
// Get the actual paragraph
if val, exists := s.Attr("padding-left"); exists && val == "30px" {
// Check to see if the paragraph is special (indented) block
para = Paragraph(" " + st)
} else if val, exists := s.Attr("text-align"); exists && val == "center" {
// Otherwise check to see if it's a separator paragraph
para = Paragraph("----------")
} else {
// It's just a normal paragraph in this case
para = Paragraph(st)
}
// And add the paragraph to the chapter
para.Format()
ch.Paragraphs = append(ch.Paragraphs, para)
})
// Finally, let's signal a success
done <- true
}
// Return a slice of Arcs extracted from the table of contents
func ParseArcs(s string) []*Arc {
arcs := []*Arc{}
r, _ := regexp.Compile(`[0-9]+`)
for _, line := range strings.Split(s, "\n") {
line = strings.TrimSpace(line)
if strings.HasPrefix(line, "Arc") {
arcs = append(arcs, &Arc{
Identifier: r.FindString(line),
Title: line,
})
} else if strings.HasPrefix(line, "Epilogue") {
arcs = append(arcs, &Arc{
Identifier: "E",
Title: line,
})
}
}
return arcs
}
func main() {
// Define the app
app := cli.NewApp()
app.Name = "Worm Scraper"
app.Usage = "A tool to let you get an updated EPUB copy of the serial web novel Worm, by Wildbow"
app.Version = "1.0"
app.Author = "Benjamin Harris"
// Define the application flags
app.Flags = []cli.Flag{
cli.BoolFlag{"pdf", "Save the book as a PDF instead of an EPUB, if possible"},
cli.BoolFlag{"with-link", "Include a link to the chapter online"},
cli.BoolFlag{"with-tags", "Include the tags each chapter was posted under"},
cli.BoolFlag{"with-date", "Include the date each chapter was posted"},
}
// The heart of the application
app.Action = func(context *cli.Context) {
// Starting the program
fmt.Println("Starting to scrape Worm")
// Get the list of arcs from the table of contents
fmt.Println("Gathering links from table of contents...")
contents, err := goquery.NewDocument(TableOfContents)
if err != nil {
panic("Failed to get the table of contents! " + err.Error())
}
// Parse the arcs
arcs := ParseArcs(contents.Find(".entry-content").Text())
// Now get the links for the arc chapters
contents.Find(".entry-content a:not([class*=share-icon])").Each(func(_ int, s *goquery.Selection) {
ch := Chapter{}
ch.Title = strings.Replace(strings.TrimSpace(s.Text()), "\n", "", -1)
ch.Url, _ = s.Attr("href")
if ch.Title == "" {
return
}
arc, _ := ch.WhichArc(arcs)
arc.Chapters = append(arc.Chapters, ch)
})
// Manually add missing chapter in Epilogue
c := Chapter{
Title: "E.2",
Url: "https://parahumans.wordpress.com/2013/11/05/teneral-e-2/",
}
a, _ := c.WhichArc(arcs)
a.Chapters = append(a.Chapters, c)
copy(a.Chapters[1+1:], a.Chapters[1:])
a.Chapters[1] = c
// Now start getting the chapters
chapters := 0
done := make(chan bool)
for _, arc := range arcs {
for i, _ := range arc.Chapters {
chapters++
go arc.Chapters[i].Parse(done)
}
}
fmt.Println("Starting to parse", chapters, "chapters")
fmt.Print("Finished: ")
totalChapters := chapters
for {
select {
case <-done:
chapters--
fmt.Print(totalChapters-chapters, ",")
}
if chapters == 0 {
// We're done with all the chapters
close(done)
fmt.Println()
break
}
}
// And let's write all this stuff to a file now
fmt.Println("Saving results to file...")
f, err := os.OpenFile("Worm.md", os.O_RDWR|os.O_CREATE|os.O_EXCL, 0666)
if err != nil {
panic(err)
}
defer f.Close()
// Define pagebreak
PageBreak := "\n\n"
// Write the cover
f.WriteString("# Worm\n\n")
f.WriteString("By Wildbow\n\n")
f.WriteString("Website: " + MainSite)
// Now loop through the Arcs
for _, arc := range arcs {
f.WriteString(PageBreak + "# " + arc.Title)
for _, chapter := range arc.Chapters {
f.WriteString("\n\n")
f.WriteString("## " + chapter.Title + "\n\n")
if context.Bool("with-tags") {
f.WriteString("**Tags:** " + strings.Join(chapter.Tags, ", ") + " ")
}
if context.Bool("with-date") {
f.WriteString("**Date:** " + chapter.DatePosted + " ")
}
if context.Bool("with-link") {
f.WriteString("**Link:** " + chapter.Url + " ")
}
f.WriteString("\n\n")
// Now save the chapter's paragraphs
for _, p := range chapter.Paragraphs {
f.WriteString(string(p) + "\n\n")
}
}
}
// Now let's try to convert the markdown file into an ebook format (epub, pdf)
fmt.Print("Attempting to convert Markdown file... ")
cmdText := []string{"-S", "Worm.md", "--epub-chapter-level", "2", "-o", "Worm.epub"}
if context.Bool("pdf") {
cmdText = []string{"Worm.md", "-o", "Worm.pdf"}
PageBreak = `<div style="page-break-after: always;"></div>`
}
cmd := exec.Command("pandoc", cmdText...)
err = cmd.Run()
if err != nil {
fmt.Println("Conversion failed! Make sure you've installed Pandoc (http://johnmacfarlane.net/pandoc/installing.html) if you want to convert the generated Markdown file to an ebook compatible format. In the meantime, we've left you the Markdown file.")
} else {
_ = os.Remove("Worm.md")
fmt.Println("Completed!")
}
}
// Run the application
app.Run(os.Args)
}
oh also would it be possible to modify it to output as .txt or .mobi? if not I'll just convert using Calibre. Thanks in advance.
Oh if it matters I'm using windows 7 64-bit
The go compiler doesn't import the libraries directly from the internet but it does know how to fetch them for you. When you import something like github.com/codegangsta/cli it doesn't look for it on that URL but instead it looks for it on your GOPATH/src folder.
The go get command can fetch the library for you in it's URL and download it to your GOPATH.
If you have already setup your GOPATH (if not, read How to Write Go Code) then before running your code run the command go get library for the go tool to download it for you. In your example you should run the following commands:
go get github.com/codegangsta/cli
go get github.com/puerkitobio/goquery
That would download the libraries to GOPATH/src/github.com/codegangsta/cli and GOPATH/src/github.com/puerkitobio/goquery respectively.