Unfortunately golang error documentation in the standard library is next to non-existent. e.g. just opening a file, the return values for the error are not documented except that you can print a string. But this is not always the right way to handle it.
Is there a way to determine what the real error code might be through trial and error rather than just printing out the text? It seems silly to match the whole text for the specific error.
e.g. given I want to ultimately achieve something like this (assuming it's right)
if fd, err := io.Open("filename"); err != nil {
if err != io.ErrFileNotFound {
log.Fatalf("Error opening file: %s", err)
}
}
As far as I can tell anything that implements the error interface will be able to be used as an error. But determining what the error is is what I'm struggling with. The error may be a struct that has other fields in it like a number field to tell me what type of error it is aside from the text itself.
But how would I know what other data the error contains short of looking through many source files and sometimes tens of function calls.
Does this make sense?
As a more practical example. I am using a yaml library to load a config file.
If the config file doesn't exist I want to carry on (it'll use defaults). But if there is a permissions error I want the error to be treated as fatal. The problem is, it's not entirely clear what the error will look like ahead of time.
Use os.IsNotExist to check for file not found errors:
f, err := os.Open("filename")
if os.IsNotExist(err) {
// handle missing file
} else if err != nil {
// handle other errors
}
The functions os.IsExist, os.IsPermission and is.Timeout check for other common types of errors.
The os, io and other packages declare variables for specific errors.
Always check godoc if you are not cleared about a library.
.below is the godoc io URL and read more
https://godoc.org/io
Related
I have a task to parse both eml and msg formatted email files using Go. There's a wonderful package for parsing EML files, however, with MSG, no matter what package I research and attempt to implement, I encounter the same error every single time.
malformed MIME header: missing colon:
It isn't the msg file itself. I have the same service in .NET which reads the msg file perfectly (MsgReader library).
Could someone suggest a package I could use in Go to read msg files? I wonder if it's an encoding issue (this wasn't a problem with eml files).
I've tried using these packages:
github.com/veqryn/go-email
net/mail
https://github.com/go-gomail/gomail
github.com/jordan-wright/email
github.com/emersion/go-message
github.com/jpoehls/gophermail
As an example, here is one function I've tried to read an msg file.
func parse_msg_file() {
var filePath string = "c://messages//kraken.msg"
var reader io.Reader
f, err := os.Open(filePath)
checkerr(err, "file "+filePath+" not found or can not be readed")
defer f.Close()
reader = bufio.NewReader(f)
msg, err := email.ParseMessage(reader)
checkerr(err, "failed to parse raw msg file")
if msg == nil {
checkerr(err, "failed to parse raw msg file")
}
}
and the output when I call the function is:
malformed MIME header: missing colon: "\xd0\xcf\x11\u0871\x1a\xe1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00>\x00\x03\x00\xfe\xff\t\x00\x06\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x10\x00\x00\t\x00\x00\x00\x02\x00\x00\x00\xfe\xff\xff\xff\x00\x00\x00\x00\x03\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xffR\x00o\x00o\x00t\x00 \x00E\x00n\x00t\x00r\x00y\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x16\x00\x05\x00\xff\xff\xff\xff\xff\xff\xff\xff\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xf0\t-0r$\xd9\x01"
exit status 255
Just to add to my comment, I have searched for "msg parsers in go" in Google and it has brought up this repository - https://github.com/oucema001/OutlookMessageParser-Go . I don't know if it actually works - it's pretty old, and no documentation, so unlikely it'll be easy to use, but you can start from there.
Here's the specification for Microsoft's Outlook Item File Format (*.msg).
And here's the specification for Microsoft's Compound File Binary File Format, the basis for the Outlook Item File Format (*.msg).
The Compound File Binary File Format is
a general-purpose file format that provides a file-system-like structure within a file for the storage of arbitrary, application-specific streams of data.
I believe that this stuff all came from Microsoft's old OLE/COM stuff (Object Linking and Embedding/Component Object Model).
FWIW, here's a parser for the Compound File Binary File Format. No idea if it works, or anything else about it, but it might be, at least, a jumping-off point for you.
https://github.com/richardlehane/mscfb
[Edited to note]
Seems that the above package is a dependency of https://github.com/oucema001/OutlookMessageParser-Go, referenced in this answer by #astax.
I'm getting Checkmarx issue:
The resource Parse allocated by FUNCTION_X in the file FILE at line LINE is prone to resource exhaustion when used by FUNCTION_Y in the file FILE at line LINE.
I can see that data from Parse result is processed in a loop, so I added a length check before the loop, sth like this:
if len(query) > 100 {
return nil, fmt.Errorf("too many query params [%d]", len(query))
}
for k := range query {
// ...
}
But Checkmarx is still complaining. Does anyone know how to fix this?
Thank you.
Having looked into it, it’s not about url.parse, it is about passing user inputs to a loop controlled by len().
Your code is fine, and Checkmarx SAST needs to understand that len() is the size/length function for Go.
You should mark this as Not Exploitable and report it as False Positive to Checkmarx.
In my journey with go discovered that there are no stacktraces. so whenever something breaks, all we get an simple string error message without any information where is this is coming from. This is in stark contrast with other languages where I am used to seing detailed stacktraces
For example, below is the error message from apex
$ cat event.json | apex invoke --logs webhook
⨯ error parsing response: json: cannot unmarshal array into Go value of type map[string]interface {}
here its telling me that unmarshal to a map ins't working because event.json is an array. We have unmarshal to interface{} to support both arrays & maps.However, it doesn't tell me which file/line is causing this error.
Questions:
What is way to quickly find which file/line this error coming from?
In General, Are there tips/tricks which gophers use to get to the source of problem quickly from this string error message?
is this how stack traces are for most go projects or there are any best practices that should be followed?
What is way to quickly find which file/line this error coming from?
There are no default stacks printed unless it's an unrecovered panic.
In General, Are there tips/tricks which gophers use to get to the source of problem quickly from this string error message? is this how stack traces are for most go projects or there are any best practices that should be followed?
In General, you need to check error returns from most of the function calls. There are more than one way to do that.
I usually use standard library package log to print out error logs with file and line numbers for easy debugging in simple programs. For example:
package main
import "log"
import "errors"
func init() { log.SetFlags(log.Lshortfile | log.LstdFlags) }
func callFunc() error {
return errors.New("error")
}
func main() {
if err := callFunc(); err != nil {
log.Println(err)
}
}
http://play.golang.org/p/0iytNw7eZ7
output:
2009/11/10 23:00:00 main.go:14: error
Also, there are functions for you to print or retrieve current stacks in standard library runtime/debug, e.g. https://golang.org/pkg/runtime/debug/#PrintStack
There are many community efforts on bringing error handling easier, you can search error in GoDoc: https://godoc.org/?q=error
Your attempted solution: Finding the piece of code that produces the error to fix the code.
Your actual problem: The content of event.json.
This is called the X-Y-Problem
Invoke expects a json object, you are passing a json array. Fix that and your problem is gone!
$ echo -n '{ "value": "Tobi the ferret" }' | apex invoke uppercase
Relevant part of the documentation: Invoking Functions
And that's the piece of code that produces the error: Github
And yes, Go does have stack traces! Read Dave Cheneys blog post on errors and exceptions.
Go does produce stack traces when a panic happens, crashing the program. This will happen if the code calls panic() directly, typically in cases like:
if err != nil {
panic("it broke")
}
or, when a runtime error happens:
a := []int{1, 2, 3}
b := a[12] // index out of range
Here's a minimal example:
package main
func main() {
panic("wtf?!")
}
Output:
panic: wtf?!
goroutine 1 [running]:
panic(0x94e60, 0x1030a040)
/usr/local/go/src/runtime/panic.go:464 +0x700
main.main()
/tmp/sandbox366642315/main.go:4 +0x80
Note the main.go:4 indicating the filename and line number.
In your example, the program did not panic, instead opting to call (I'm guessing) os.Exit(1) or log.Fatal("error message") (which calls os.Exit(1)). Or, the panic was simply recovered from in the calling function. Unfortunately, there's nothing you can do about this if you aren't the author of the code.
I would recommend reading Defer, Panic, and Recover on the Golang blog for more about this.
Setting log.SetFlags(log.LstdFlags | log.Lshortfile) in main() should do it.
For example, log.Printf("Deadline!") would print:
03/11/2020 23:59:59 liberty_test.go:42: Deadline!
I am using go:generate to handle automatically generating some database models and I was hoping to run my go template through gofmt, but it chokes with all the extra {{ ... }} dynamic sections.
Am I missing something obvious? I was hoping that this was a use case the gofmt people had addressed, given both gofmt and go generate are prominent parts of the go toolchain.
Obviously, it works to just run go fmt after go generate but it just feels dirty to have poorly formatted templates that are 99% go code sitting around.
Most generation tools execute the template to a *bytes.Buffer, format the buffer bytes using format.Source and write the result to the output file.
Given template t and output writer w, the code looks something like this:
var buf bytes.Buffer
if err := t.Execute(&buf, data); err != nil {
// handle error
}
p, err := format.Source(buf.Bytes())
if err != nil {
// handle error
}
w.Write(p)
Gofmting the template does not ensure that the output will be gofmted. Given how easy it is to gofmt the output using the go/format package, there's little value in creating a tool to gofmt templates.
I want to read some input from stdin and then display it. At the moment I am doing this like this:
in := bufio.NewReader(os.Stdin);
input, err = in.ReadString('\n');
if err != nil {
fmt.Println("Error: ", err)
os.Exit(1)
}
fmt.Printf("This is your", input)
...but after running this and entering some input it is always displaying my input twice like this:
This is just a test
This is your This is just a test
Is there anyway to remove the first line?
I haven't yet tried anything with the package, but I guess it could be helpful in this case: exp/terminal. Specifically the ReadPasword function documentations is:
ReadPassword reads a line of input from a terminal without local echo.
This is commonly used for inputting passwords and other sensitive data.
The slice returned does not include the \n.
I assume your first line is just your echoed input text? That's actually a function of the process' terminal. Since the go runtime treats Stdin like any other file, you don't have direct access to the terminal attributes. However you might be able to hack something together with CGO and the approach described here.