I would like to parse a package and output all of the strings in the code. The specific use case is to collect sql strings and run them through a sql parser, but that's a separate issue.
Is the best way to do this to just parse this line by line? Or is it possible to regex this or something? I imagine that some cases might be nontrivial, such as multiline strings:
str := "This is
the full
string"
// want > This is the full string
Use the go/scanner package to scan for strings in Go source code:
src, err := os.ReadFile(fname)
if err != nil {
/// handle error
}
// Create *token.File to scan.
fset := token.NewFileSet()
file := fset.AddFile(fname, fset.Base(), len(src))
var s scanner.Scanner
s.Init(file, src, nil, 0)
for {
pos, tok, lit := s.Scan()
if tok == token.EOF {
break
}
if tok == token.STRING {
s, _ := strconv.Unquote(lit)
fmt.Printf("%s: %s\n", fset.Position(pos), s)
}
}
https://go.dev/play/p/849QsbqVhho
Related
I am working on learning how to use and how golang's ast library works. I am parsing https://github.com/modern-go/concurrent, avoiding the test files and the go_below_19.go since it causes errors.
My problem is with the parsing of these lines in the file unbounded_executor.go,
var HandlePanic = func(recovered interface{}, funcName string) {
ErrorLogger.Println(fmt.Sprintf("%s panic: %v", funcName, recovered))
ErrorLogger.Println(string(debug.Stack()))
}
The ast.Ident for ErrorLogger in both instances have a nil obj.
But, I believe that it should not be nil and should reference these lines from log.go,
// ErrorLogger is used to print out error, can be set to writer other than stderr
var ErrorLogger = log.New(os.Stderr, "", 0)
Am I wrong, or is there a problem with the parser? I've followed several references on parsing files and reuse a *token.FileSet across each of the files and use ParseComments as the mode.
edit:
There is a large code base surrounding this, so the code demonstrating this will include snippets.
This is performed with the same fset across all non-test go files, without build restrictions that would stop the code from being used with 1.16
parsedFile, parseErr := parser.ParseFile(fset, filePath, nil, parser.ParseComments)
Call ast.NewPackage to resolve identifiers in the AST:
fset := token.NewFileSet()
files := make(map[string]*ast.File)
for _, name := range []string{"unbounded_executor.go", "log.go"} {
f, err := parser.ParseFile(fset, name, nil, parser.ParseComments)
if err != nil {
log.Fatal(err)
}
files[name] = f
}
ast.NewPackage(fset, files, nil, nil)
ast.Inspect(files["unbounded_executor.go"], func(n ast.Node) bool {
if n, ok := n.(*ast.Ident); ok && n.Name == "ErrorLogger" {
fmt.Println(n.Obj)
}
return true
})
Because a proper importer is not provided and the list of files does not include all files in the package, NewPackage returns unresolved symbol errors.
I am trying to figure out why my code is not working. I wish to take a slice of numbers and strings, and separate it into three slices. For each element in the slice, if it is a string, append it to the strings slice, and if it is a positive number, append it to the positive numbers, and likewise with negative. Yet, here is the output
Names:
EvTremblay
45.39934611083154
-75.71148292845268
[Crestview -75.73795670904249
BellevueManor -75.73886856878032
Dutchie'sHole -75.66809864107668 ...
Positives:[45.344387632924054 45.37223315413918 ... ]
Negatives: []
Here is my code. Can someone tell me what is causing the Negatives array to not have any values?
func main() {
fmt.Printf("%q\n", strings.Split("a,b,c", ","))
var names []string
var positives, negatives []float64
bs, err := ioutil.ReadFile("poolss.txt")
if err != nil {
return
}
str := string(bs)
fmt.Println(str)
tokens := strings.Split(str, ",")
for _, token := range tokens {
if num, err := strconv.ParseFloat(token, 64); err == nil {
if num > 0 {
positives = append(positives, num)
} else {
negatives = append(negatives, num)
}
} else {
names = append(names, token)
}
fmt.Println(token)
}
fmt.Println(fmt.Sprintf("Strings: %v",names))
fmt.Println(fmt.Sprintf("Positives: %v", positives))
fmt.Println(fmt.Sprintf("Negatives: %v",negatives))
for i := range names{
fmt.Println(names[i])
fmt.Println(positives[i])
fmt.Println(negatives[i])
}
}
Your code has strings as a variable name:
var strings []string
and strings as a package name:
tokens := strings.Split(str, ",")
Don't do that!
strings.Split undefined (type []string has no field or method Split)
Playground: https://play.golang.org/p/HfZGj0jOT-P
Your problem above I think lies with the extra \n attached to each float probably - you get no negative entries if you end in a linefeed or you would get one if you have no linefeed at the end. So insert a printf so that you can see the errors you're getting from strconv.ParseFloat and all will become clear.
Some small points which may help:
Check errors, and don't depend on an error to be of only one type (this is what is confusing you here) - always print the error if it arrives, particularly when debugging
Don't use the name of a package for a variable (strings), it won't end well
Use a datastructure which reflects your data
Use the CSV package to read CSV data
So for example for storing the data you might want:
type Place struct {
Name string
Latitude int64
Longitude int64
}
Then read the data into that, depending on the fact that cols are in a given order, and store it in a []Place.
Here's what I tried, it works now! Thanks for the help, everyone!
func main() {
findRoute("poolss.csv", 5)
}
func findRoute( filename string, num int) []Edge {
var route []Edge
csvFile, err := os.Open(filename)
if err != nil {
return route
}
reader := csv.NewReader(bufio.NewReader(csvFile))
var pools []Pool
for {
line, error := reader.Read()
if error == io.EOF {
break
} else if error != nil {
log.Fatal(error)
}
lat, err := strconv.ParseFloat(line[1], 64)
long, err := strconv.ParseFloat(line[2], 64)
if err == nil {
pools = append(pools, Pool{
name: line[0],
latitude: lat,
longitude: long,
})
}
}
return route
}
I'm new to Golang, starting out with some examples. Currently, what I'm trying to do is reading a file line by line and replace it with another string in case it meets a certain condition.
The file is use for testing purposes contains four lines:
one
two
three
four
The code working on that file looks like this:
func main() {
file, err := os.OpenFile("test.txt", os.O_RDWR, 0666)
if err != nil {
panic(err)
}
reader := bufio.NewReader(file)
for {
fmt.Print("Try to read ...\n")
pos,_ := file.Seek(0, 1)
log.Printf("Position in file is: %d", pos)
bytes, _, _ := reader.ReadLine()
if (len(bytes) == 0) {
break
}
lineString := string(bytes)
if(lineString == "two") {
file.Seek(int64(-(len(lineString))), 1)
file.WriteString("This is a test.")
}
fmt.Printf(lineString + "\n")
}
file.Close()
}
As you can see in the code snippet, I want to replace the string "two" with "This is a test" as soon as this string is read from the file.
In order to get the current position within the file I use Go's Seek method.
However, what happens is that always the last line gets replaced by This is a test, making the file looking like this:
one
two
three
This is a test
Examining the output of the print statement which writes the current file position to the terminal, I get that kind of output after the first line has been read:
2016/12/28 21:10:31 Try to read ...
2016/12/28 21:10:31 Position in file is: 19
So after the first read, the position cursor already points to the end of my file, which explains why the new string gets appended to the end. Does anyone understand what is happening here or rather what is causing that behavior?
The Reader is not controller by the file.Seek. You have declared the reader as: reader := bufio.NewReader(file) and then you read one line at a time bytes, _, _ := reader.ReadLine() however the file.Seek does not change the position that the reader is reading.
Suggest you read about the ReadSeeker in the docs and switch over to using that. Also there is an example using the SectionReader.
Aside from the incorrect seek usage, the difficulty is that the line you're replacing isn't the same length as the replacement. The standard approach is to create a new (temporary) file with the modifications. Assuming that is successful, replace the original file with the new one.
package main
import (
"bufio"
"io"
"io/ioutil"
"log"
"os"
)
func main() {
// file we're modifying
name := "text.txt"
// open original file
f, err := os.Open(name)
if err != nil {
log.Fatal(err)
}
defer f.Close()
// create temp file
tmp, err := ioutil.TempFile("", "replace-*")
if err != nil {
log.Fatal(err)
}
defer tmp.Close()
// replace while copying from f to tmp
if err := replace(f, tmp); err != nil {
log.Fatal(err)
}
// make sure the tmp file was successfully written to
if err := tmp.Close(); err != nil {
log.Fatal(err)
}
// close the file we're reading from
if err := f.Close(); err != nil {
log.Fatal(err)
}
// overwrite the original file with the temp file
if err := os.Rename(tmp.Name(), name); err != nil {
log.Fatal(err)
}
}
func replace(r io.Reader, w io.Writer) error {
// use scanner to read line by line
sc := bufio.NewScanner(r)
for sc.Scan() {
line := sc.Text()
if line == "two" {
line = "This is a test."
}
if _, err := io.WriteString(w, line+"\n"); err != nil {
return err
}
}
return sc.Err()
}
For more complex replacements, I've implemented a package which can replace regular expression matches. https://github.com/icholy/replace
import (
"io"
"regexp"
"github.com/icholy/replace"
"golang.org/x/text/transform"
)
func replace2(r io.Reader, w io.Writer) error {
// compile multi-line regular expression
re := regexp.MustCompile(`(?m)^two$`)
// create replace transformer
tr := replace.RegexpString(re, "This is a test.")
// copy while transforming
_, err := io.Copy(w, transform.NewReader(r, tr))
return err
}
OS package has Expand function which I believe can be used to solve similar problem.
Explanation:
file.txt
one
two
${num}
four
main.go
package main
import (
"fmt"
"os"
)
var FILENAME = "file.txt"
func main() {
file, err := os.ReadFile(FILENAME)
if err != nil {
panic(err)
}
mapper := func(placeholderName string) string {
switch placeholderName {
case "num":
return "three"
}
return ""
}
fmt.Println(os.Expand(string(file), mapper))
}
output
one
two
three
four
Additionally, you may create a config (yml or json) and
populate that data in the map that can be used as a lookup table to store placeholders as well as their replacement strings and modify mapper part to use this table to lookup placeholders from input file.
e.g map will look like this,
table := map[string]string {
"num": "three"
}
mapper := func(placeholderName string) string {
if val, ok := table[placeholderName]; ok {
return val
}
return ""
}
References:
os.Expand documentation: https://pkg.go.dev/os#Expand
Playground
For logging purposes I want to be able to quickly write a slice of any type, whether it be ints, strings, or custom structs, to a file in Go. For instance, in C#, I can do the following in 1 line:
File.WriteAllLines(filePath, myCustomTypeList.Select(x => x.ToString());
How would I go about doing this in Go? The structs implement the Stringer interface.
Edit: I in particular would like the output to be printed to a file and one line per item in the slice
Use the fmt package format values as strings and print to a file:
func printLines(filePath string, values []interface{}) error {
f, err := os.Create(filePath)
if err != nil {
return err
}
defer f.Close()
for _, value := range values {
fmt.Fprintln(f, value) // print values to f, one per line
}
return nil
}
fmt.Fprintln will call Stringer() on your struct type. It will also print int values and string values.
playground example
Use the reflect package to write any slice type:
func printLines(filePath string, values interface{}) error {
f, err := os.Create(filePath)
if err != nil {
return err
}
defer f.Close()
rv := reflect.ValueOf(values)
if rv.Kind() != reflect.Slice {
return errors.New("Not a slice")
}
for i := 0; i < rv.Len(); i++ {
fmt.Fprintln(f, rv.Index(i).Interface())
}
return nil
}
If you have variable values of type myCustomList, then you can call it like this: err := printLines(filePath, values)
playground example
I'm trying to figure out how to (or if it's possible to) combine multiple assignment and ranges in Golang
ex pseudo code of what I'd like to do
files := [2]*os.File{}
for i, _, fileName := 0, range os.Args[1:3] {
files[i], _ = os.Open(fileName)
}
The idea being I want to have both an iteration counter (i) and the filenames (fileName). I know this can be achieved by using the key from range and some math (key -1), thats not the point of the example.
Edit:
Upon debugging the above example, I learned that i will range 0-1 in that example; Because os.Args[1:2] is a slice and that slice has indexing 0-1 . Therefore I dont need "some math" to properly index the keys.
** EDIT 2: **
This post is also a must read as to why the above [2]*os.File{} is not idiomatic go, instead it should not have a size specified (files := []*os.File{}) so that files is of type slice of *os.File
There are a lot of different issues here. First, range already does what you want. There's no need for even math.
for i, fileName := range os.Args[1:] {
i will range from 0 to 1 here, just like you want. Ranging over a slice always starts at index 0 (it's relative to the start of the slice). (http://play.golang.org/p/qlVM6Y7yPD)
Note that os.Args[1:2] is just one element. You probably meant it to be two.
In any case, this is likely what you really meant:
http://play.golang.org/p/G4yfkKrEe7
files := make([]*os.File, 0)
for _, fileName := range os.Args[1:] {
f, err := os.Open(fileName)
if err != nil {
log.Fatalf("Could not open file: %v", err)
}
files = append(files, f)
}
fmt.Printf("%v\n", files)
Fixed-length arrays are very uncommon in Go. Generally you want a slice, created with make.
For example,
so.go:
package main
import (
"fmt"
"os"
)
func main() {
files := [2]*os.File{}
for i, fileName := range os.Args[1:] {
if i >= len(files) {
break
}
var err error
files[i], err = os.Open(fileName)
if err != nil {
// handle error
}
}
fmt.Println(files)
}
Output:
$ go build so.go && ./so no.go so.go to.go
[<nil> 0xc820030020]
$