Here's my code ( part of it ) :
type SitemapIndex struct {
// Locations []Location `xml:"sitemap"`
Locations []string `xml:"sitemap>loc"`
}
~~~ SNIP ~~~
func main(){
var s SitemapIndex
resp, _ := http.Get("https://www.washingtonpost.com/news-sitemaps/index.xml")
bytes, _ := ioutil.ReadAll(resp.Body)
xml.Unmarshal(bytes, &s)
for _, Location := range s.Locations {
fmt.Printf("%s\n", Location)
resp, err := http.Get(Location)
if err != nil {
log.Fatal(err)
} else {
bytes, _ := ioutil.ReadAll(resp.Body)
xml.Unmarshal(bytes, &n)
for idx := range n.Titles {
newsMap[n.Titles[idx]] = NewsMap{n.Keywords[idx], n.Locations[idx]}
}
}
for idx, data := range newsMap {
fmt.Println("\n\n\n", idx)
fmt.Println("\n", data.Keyword)
fmt.Println("\n", data.Location)
}
}
Now, when I run this code I get this output :
https://www.washingtonpost.com/news-sitemaps/politics.xml
2019/01/28 02:37:13 parse
https://www.washingtonpost.com/news-sitemaps/politics.xml
: first path segment in URL cannot contain colon
exit status 1
I read a few posts and did some experiment myself, like I made another file with the following code
package main
import ("fmt"
"net/url")
func main(){
fmt.Println(url.Parse("https://www.washingtonpost.com/news-sitemaps/politics.xml"))
}
And it didn't throw any error, so I understand the error is not with the url .
Now, I just started learning Go using sentdex's tutorials , a few hours ago and so don't have much idea as of now. Here's the video link
Thanks and regards.
Temporarya
The problem here is that Location has whitespace prefix and suffix so string is not valid URL. Unfortunately, error message does not help to see that.
How to detect:
I typically use %q fmt helper that wraps string into parentheses:
fmt.Printf("%q", Location)
Will be printed as "\nhttps://www.washingtonpost.com/news-sitemaps/politics.xml\n"
How to fix:
add this line before using Location in code:
Location = strings.TrimSpace(Location)
Another reason to get this error is when you use an IP address without specifying the protocol before it.
Example for cases you'll get this error:
parsedUrl, err := url.Parse("127.0.0.1:3213")
How to fix it:
parsedUrl, err := url.Parse("http://127.0.0.1:3213")
Poor documentation, unfortunately.
Related
I would like to parse a package and output all of the strings in the code. The specific use case is to collect sql strings and run them through a sql parser, but that's a separate issue.
Is the best way to do this to just parse this line by line? Or is it possible to regex this or something? I imagine that some cases might be nontrivial, such as multiline strings:
str := "This is
the full
string"
// want > This is the full string
Use the go/scanner package to scan for strings in Go source code:
src, err := os.ReadFile(fname)
if err != nil {
/// handle error
}
// Create *token.File to scan.
fset := token.NewFileSet()
file := fset.AddFile(fname, fset.Base(), len(src))
var s scanner.Scanner
s.Init(file, src, nil, 0)
for {
pos, tok, lit := s.Scan()
if tok == token.EOF {
break
}
if tok == token.STRING {
s, _ := strconv.Unquote(lit)
fmt.Printf("%s: %s\n", fset.Position(pos), s)
}
}
https://go.dev/play/p/849QsbqVhho
I think is something that I miss theoretically from the passing by reference topic but I can't find a way to read the ID without using the support networkInterfaceReference
package main
import (
"context"
"fmt"
"github.com/Azure/azure-sdk-for-go/profiles/preview/resources/mgmt/resources"
"github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2021-03-01/compute"
"github.com/Azure/azure-sdk-for-go/services/subscription/mgmt/2020-09-01/subscription"
"github.com/Azure/go-autorest/autorest/azure/auth"
"github.com/ktr0731/go-fuzzyfinder"
)
var selectedSub subscription.Model
var selectedRG resources.Group
var selectedVM compute.VirtualMachine
func main() {
selectedSub = GetSubscription()
selectedRG = GetResourceGroup()
selectedVM = GetVM()
fmt.Printf("Sub: %s\nRG: %s\nVM: %s\n", *selectedSub.DisplayName, *selectedRG.Name, *selectedVM.Name)
// THIS WORK
networkInterfaceReference := *selectedVM.NetworkProfile.NetworkInterfaces
fmt.Printf("%s", *networkInterfaceReference[0].ID)
// THIS DOESN'T WORK
fmt.Printf("%s", *selectedVM.NetworkProfile.NetworkInterfaces[0].ID)
}
...
...
...
func GetVM() compute.VirtualMachine {
vmClient := compute.NewVirtualMachinesClient(*selectedSub.SubscriptionID)
authorizer, err := auth.NewAuthorizerFromCLI()
if err == nil {
vmClient.Authorizer = authorizer
}
vmList, err := vmClient.List(context.TODO(), *selectedRG.Name)
if err != nil {
panic(err)
}
idx, err := fuzzyfinder.Find(vmList.Values(), func(i int) string {
return *vmList.Values()[i].Name
})
if err != nil {
panic(err)
}
return vmList.Values()[idx]
}
Hovering to the error showed the following error message:
field NetworkProfile *[]compute.NetworkProfile
(compute.VirtualMachineProperties).NetworkProfile on pkg.go.dev
NetworkProfile - Specifies the network interfaces of the virtual machine.
invalid operation: cannot index selectedVM.NetworkProfile.NetworkInterfaces (variable of type *[]compute.NetworkInterfaceReference)compiler (NonIndexableOperand)
If you want the 2nd way to work:
// WORKS
networkInterfaceReference := *selectedVM.NetworkProfile.NetworkInterfaces
fmt.Printf("%s", *networkInterfaceReference[0].ID)
// THIS DOESN'T WORK
fmt.Printf("%s", *selectedVM.NetworkProfile.NetworkInterfaces[0].ID)
studying the compilation error you are getting (P.S. please don't post screen-shots of code/errors) - the error is failing because you are trying to index a pointer which is not allowed in Go. You can only index maps, arrays or slices.
The fix is simple, since you do two (2) pointer dereferences in the working version, you need to do two (2) same in the single expression - but also you need to ensure the lexical binding order so the indexing is done after the pointer dereference:
fmt.Printf("%s", *(*selectedVM.NetworkProfile.NetworkInterfaces)[0].ID)
Finally, there is no pass-by-reference in Go. Everything is by value. If you want to change a value, pass a pointer to it - but that pointer is a still just a value that is copied.
I'm trying to find the path of the helloworld.java file so that I can pass it down to a compiler function.
What I have:
I'm expecting this to return the path, of type []byte and then stringify it, of the only helloworld.java file in this directory and then pass it down to Java() function.
filePath, _ := exec.Command("find", "./helloworld/workspace", "-name", "*.java").Output()
Java(string(filePath))
The Problem is that cmd := exec.Command("javac", filePath) in my java() function is not recognizing the file path therefore not compiling it.
But if I hardcode the path that I get from exec.Command("find) like this:
This works fine
cmd := exec.Command("javac", "./helloworld/workspace/src/main/java/com/coveros/demo/helloworld/HelloWorld.java")
err := cmd.Run()
But this does not work:
What am I missing, How do I fix this?
func Java(filePath string) {
fmt.Println("compiler start")
cmd := exec.Command("javac", filePath)
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
fmt.Println("compiler End")
}
I think the result from find is returning multiple possible paths which are separated by a newline "\n". The newline will be "hidden" if it is printed to the command line. You can try this fmt.Println(stringPath + "hello, am I on a new line?") to show the stringPath has a new line in it.
See the below which uses a similar version of find looking for json files, then splits the string by newlines and then loops through these paths. If the path is blank (which it can be) it skips over it.
package main
import (
"fmt"
"os/exec"
"strings"
)
func main() {
filePath, err := exec.Command("find", ".", "-name", "*.json").Output()
if err != nil {
panic(err)
}
stringPath := string(filePath)
paths := strings.Split(stringPath, "\n")
CatFile(paths)
}
func CatFile(filePaths []string) {
for _, path := range filePaths {
if len(path) == 0 {
continue
}
output, err := exec.Command("cat", path).Output()
if err != nil {
fmt.Println("Error!")
fmt.Println(err)
}
fmt.Println(string(output))
}
}
See this related question, which discusses this: Go lang differentiate "\n" and line break
I need to automate the input of a code segment like bellow where the inputs of the ReadString are distinct.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
buf := bufio.NewReader(os.Stdin)
value, err := buf.ReadString('\n')
if err != nil {
fmt.Println(err)
} else {
fmt.Println(value)
}
buf = bufio.NewReader(os.Stdin)
value, err = buf.ReadString('\n')
if err != nil {
fmt.Println(err)
} else {
fmt.Println(value)
}
}
I have tried several formats like the bellow one following answers of this question, unfortunately, none worked.
>> echo "data1
data2" | go run main.go
output: data1
EOF
Here data1 and data2 and input of the separate ReadString methods. I don't have control over the source code. So, I can only try changing the bash input. How to resolve this issue?
This is happening because the second string does not end with a newline. Looking at the documentation for ReadString:
If ReadString encounters an error before finding a delimiter, it returns the data read before the error and the error itself (often io.EOF).
So, even though error is non-nil, you have the data. The following change should work for this specific case:
buf = bufio.NewReader(os.Stdin)
value, err = buf.ReadString('\n')
if value!="" {
fmt.Println(value)
}
if err != nil {
fmt.Println(err)
}
In general, even if you get an error from ReadString, you may still have nonempty data returned from the function.
I'm new to Golang, starting out with some examples. Currently, what I'm trying to do is reading a file line by line and replace it with another string in case it meets a certain condition.
The file is use for testing purposes contains four lines:
one
two
three
four
The code working on that file looks like this:
func main() {
file, err := os.OpenFile("test.txt", os.O_RDWR, 0666)
if err != nil {
panic(err)
}
reader := bufio.NewReader(file)
for {
fmt.Print("Try to read ...\n")
pos,_ := file.Seek(0, 1)
log.Printf("Position in file is: %d", pos)
bytes, _, _ := reader.ReadLine()
if (len(bytes) == 0) {
break
}
lineString := string(bytes)
if(lineString == "two") {
file.Seek(int64(-(len(lineString))), 1)
file.WriteString("This is a test.")
}
fmt.Printf(lineString + "\n")
}
file.Close()
}
As you can see in the code snippet, I want to replace the string "two" with "This is a test" as soon as this string is read from the file.
In order to get the current position within the file I use Go's Seek method.
However, what happens is that always the last line gets replaced by This is a test, making the file looking like this:
one
two
three
This is a test
Examining the output of the print statement which writes the current file position to the terminal, I get that kind of output after the first line has been read:
2016/12/28 21:10:31 Try to read ...
2016/12/28 21:10:31 Position in file is: 19
So after the first read, the position cursor already points to the end of my file, which explains why the new string gets appended to the end. Does anyone understand what is happening here or rather what is causing that behavior?
The Reader is not controller by the file.Seek. You have declared the reader as: reader := bufio.NewReader(file) and then you read one line at a time bytes, _, _ := reader.ReadLine() however the file.Seek does not change the position that the reader is reading.
Suggest you read about the ReadSeeker in the docs and switch over to using that. Also there is an example using the SectionReader.
Aside from the incorrect seek usage, the difficulty is that the line you're replacing isn't the same length as the replacement. The standard approach is to create a new (temporary) file with the modifications. Assuming that is successful, replace the original file with the new one.
package main
import (
"bufio"
"io"
"io/ioutil"
"log"
"os"
)
func main() {
// file we're modifying
name := "text.txt"
// open original file
f, err := os.Open(name)
if err != nil {
log.Fatal(err)
}
defer f.Close()
// create temp file
tmp, err := ioutil.TempFile("", "replace-*")
if err != nil {
log.Fatal(err)
}
defer tmp.Close()
// replace while copying from f to tmp
if err := replace(f, tmp); err != nil {
log.Fatal(err)
}
// make sure the tmp file was successfully written to
if err := tmp.Close(); err != nil {
log.Fatal(err)
}
// close the file we're reading from
if err := f.Close(); err != nil {
log.Fatal(err)
}
// overwrite the original file with the temp file
if err := os.Rename(tmp.Name(), name); err != nil {
log.Fatal(err)
}
}
func replace(r io.Reader, w io.Writer) error {
// use scanner to read line by line
sc := bufio.NewScanner(r)
for sc.Scan() {
line := sc.Text()
if line == "two" {
line = "This is a test."
}
if _, err := io.WriteString(w, line+"\n"); err != nil {
return err
}
}
return sc.Err()
}
For more complex replacements, I've implemented a package which can replace regular expression matches. https://github.com/icholy/replace
import (
"io"
"regexp"
"github.com/icholy/replace"
"golang.org/x/text/transform"
)
func replace2(r io.Reader, w io.Writer) error {
// compile multi-line regular expression
re := regexp.MustCompile(`(?m)^two$`)
// create replace transformer
tr := replace.RegexpString(re, "This is a test.")
// copy while transforming
_, err := io.Copy(w, transform.NewReader(r, tr))
return err
}
OS package has Expand function which I believe can be used to solve similar problem.
Explanation:
file.txt
one
two
${num}
four
main.go
package main
import (
"fmt"
"os"
)
var FILENAME = "file.txt"
func main() {
file, err := os.ReadFile(FILENAME)
if err != nil {
panic(err)
}
mapper := func(placeholderName string) string {
switch placeholderName {
case "num":
return "three"
}
return ""
}
fmt.Println(os.Expand(string(file), mapper))
}
output
one
two
three
four
Additionally, you may create a config (yml or json) and
populate that data in the map that can be used as a lookup table to store placeholders as well as their replacement strings and modify mapper part to use this table to lookup placeholders from input file.
e.g map will look like this,
table := map[string]string {
"num": "three"
}
mapper := func(placeholderName string) string {
if val, ok := table[placeholderName]; ok {
return val
}
return ""
}
References:
os.Expand documentation: https://pkg.go.dev/os#Expand
Playground