GoLang tailing UTF16LE windows log file - go

How can I combine these two golang scripts to tail an active log that is UTF16LEBOM?
I am working on a log parser for a game (Eve Online). In the game chat logs can be saved, and I would like to use GoLang to read the running log and flag on keywords "die" in "prepare to die scumsucker!"
I've found two examples that work separately. Combining them is a challenge I've been unable to figure out. First tail equivalent here:
https://medium.com/#arunprabhu.1/tailing-a-file-in-golang-72944204f22b
Second reading the file so that is legible and doesn't have extra characters:
https://github.com/TomOnTime/utfutil/
I have been trying to replace the code from utfutil with the "equivalent" but it seems it is missing several functions.
// EDIT Almost working. It doesn't look like utfutil is defering the close.
package main
import (
"bufio"
"fmt"
"io"
"time"
"github.com/TomOnTime/utfutil"
)
func main() {
logfilefile := "C:/Users/user/Documents/EVE/logs/Chatlogs/chat_20220709_022129_1006197774.txt"
file, err := utfutil.OpenFile(logfilefile, utfutil.WINDOWS)
if err != nil {
return
}
defer file.Close()
reader := bufio.NewReader(file)
for {
line, err := reader.ReadString('\n')
if err != nil {
if err == io.EOF {
// without this sleep you would hogg the CPU
time.Sleep(500 * time.Millisecond)
continue
}
break
}
fmt.Printf(string(line))
}
}
<cut white space>
---------------------------------------------------------------
Channel ID: local
Channel Name: Local
Listener: Steve
Session started: 2022.07.07 16:18:21
---------------------------------------------------------------
[ 2022.07.07 17:11:44 ] Steve > hello world
[ 2022.07.07 17:11:48 ] John > hello world
[ 2022.07.07 17:11:51 ] James > hello world
[ 2022.07.07 19:36:53 ] Bob > hello world

You can simplify your code by removing github.com/TomOnTime/utfutil (this is a very thin wrapper over golang.org/x/text/encoding/unicode). It's also often simpler to chain tasks rather than trying to do everything at once; here I'm borrowing tailReader (with a minor change) from this answer.
Note: I have only tested the below very quickly (and don't have an "Eve Online" logfile to hand).
package main
import (
"bufio"
"fmt"
"io"
"os"
"time"
"golang.org/x/text/encoding/unicode"
)
func main() {
file, err := newTailReader("./Local_20220707_170827_1006197774.txt")
if err != nil {
return
}
defer file.Close()
utf := unicode.UTF16(unicode.LittleEndian, unicode.IgnoreBOM)
reader := bufio.NewReader(utf.NewDecoder().Reader(file))
for {
line, err := reader.ReadString('\n')
if err != nil {
fmt.Println(err)
break
}
fmt.Printf(string(line))
}
}
// Code copied from https://stackoverflow.com/a/31122253/11810946
// and modified to output contents of file before beginning to 'tail'
type tailReader struct {
io.ReadCloser
}
func (t tailReader) Read(b []byte) (int, error) {
for {
n, err := t.ReadCloser.Read(b)
if n > 0 {
return n, nil
} else if err != io.EOF {
return n, err
}
time.Sleep(10 * time.Millisecond)
}
}
func newTailReader(fileName string) (tailReader, error) {
f, err := os.Open(fileName)
if err != nil {
return tailReader{}, err
}
return tailReader{f}, nil
}

Related

Check if the program is running and stop more than one [duplicate]

I need to only allow one instance of my Golang executable at a time. I'm not sure how to use a Global Mutex to make sure no other instances are running.
This would be running on a Windows Machine.
I know this topic is a bit old, but I needed it recently on Windows and I'll post here how I did it in case someone else needs.
Thx to #VonC for pointing me in the right direction.
var (
kernel32 = syscall.NewLazyDLL("kernel32.dll")
procCreateMutex = kernel32.NewProc("CreateMutexW")
)
func CreateMutex(name string) (uintptr, error) {
ret, _, err := procCreateMutex.Call(
0,
0,
uintptr(unsafe.Pointer(syscall.StringToUTF16Ptr(name))),
)
switch int(err.(syscall.Errno)) {
case 0:
return ret, nil
default:
return ret, err
}
}
// mutexName starting with "Global\" will work across all user sessions
_, err := CreateMutex("SomeMutexName")
I created a lib with a more complete example: https://github.com/rodolfoag/gow32
Thx!
There doesn't seem to be a cross-platform solution (beside writing a file, and looking for that file at start time).
On Windows, this thread reports
the recommended approach (and the one that has worked great for me) is to use the CreateSemaphore function.
If the name you specify starts with "Global\", then the semaphore is unique for the entire system and a second attempt to open it will fail.
This is a kernel32 call, which has some wrapper in Go available.
kostix adds in the comments:
look at the Go source code around the pkg\syscall hierarchy -- it contains a good wealth of examples on how to call out to DLLs on Windows using syscalls (and that's how you access anything in Windows API).
That would be syscall/dll_windows.go. (And here is a gist)
The odbc package by brainman is another example of direct API calls on Windows -- possibly easier to digest.
Like api/zapi_windows.go.
You could use sockets, simple to use and will work on everything really.
package main
import (
"fmt"
"net"
"os"
"strings"
)
const (
INSTANCE_PORT = 9292
)
func main() {
listener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", INSTANCE_PORT))
if err != nil {
if strings.Index(err.Error(), "in use") != -1 {
//optionally send command line arguments to the other instance
fmt.Fprintln(os.Stderr, "Already running.")
return
} else {
panic(err)
}
}
for {
conn, err := listener.Accept()
if err != nil {
println("Error accept:", err.Error())
return
}
go do_something_with(conn)
}
}
You could adapt the code from tendo's python library source
what they do is
for windows :
creating a file made of the executable absolute path (well it's a library, so in your case, you can just define an identifier, to prevent you from "i put the executable in 2 places")
For windows: trying first to remove the file if existing, and if not creating the file with os.O_CREAT | os.O_EXCL | os.O_RDWR
For POSIX compatible systems: trying first to remove the file if existing and if not creating the file and acquiring a lock on it using fcntl.LOCK_EX | fcntl.LOCK_NB
any failure mean the program is already running
and then you can use a defer action to remove the lock (on posix system) and delete the file
Go permit you to create both version wit a build comment to tell which file to compile depending on your OS so you have
for unix system
// +build !windows
package main
import (
"os"
"syscall"
)
func create_lock_file(filename string) (*os.File, error) {
file, err := os.OpenFile(filename, os.O_WRONLY, 0666)
if err != nil {
return nil, err
}
err = syscall.Flock(int(file.Fd()), syscall.LOCK_EX|syscall.LOCK_NB)
if err != nil {
return nil, err
}
return file, nil
}
for windows:
// +build !windows
package main
import (
"os"
)
func create_lock_file(filename string) (*os.File, error) {
if _, err := os.Stat(filename); err == nil {
err = os.Remove(filename)
if err != nil {
return nil, err
}
}
return os.OpenFile(filename, os.O_CREATE|os.O_EXCL|os.O_RDWR, 0666)
}
and a test
package main
import (
"fmt"
"time"
)
func main() {
_, err := create_lock_file("plop.lock")
if err != nil {
fmt.Println("error ", err.Error())
}
time.Sleep(10 * time.Second)
fmt.Println("end ")
}
I've started a library out of it that you can find here
Improvements to this answer. (I am unsure if this answer will distort the original meaning, so I have written a new answer.)
Features:
deprecated: StringToUTF16Ptr is deprecated. Use UTF16PtrFromString instead.
Add the CloseHandle so that you can cancel the CreateMutexW.
package _test
import (
"syscall"
"testing"
"unsafe"
)
var (
kernel32 = syscall.NewLazyDLL("kernel32.dll")
procCreateMutexW = kernel32.NewProc("CreateMutexW")
procCloseHandle = kernel32.NewProc("CloseHandle")
)
// https://learn.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-createmutexW#return-value
func CreateMutexW(proc *syscall.LazyProc, name string) (uintptr, error) {
if proc.Name != "CreateMutexW" {
panic("proc.Name != CreateMutexW")
}
lpName, _ := syscall.UTF16PtrFromString(name) // LPCWSTR
if handleID, _, err := proc.Call(
0,
0,
uintptr(unsafe.Pointer(lpName)),
); err.(syscall.Errno) == 0 {
return handleID, nil
} else {
return handleID, err
}
}
// https://learn.microsoft.com/en-us/windows/win32/api/handleapi/nf-handleapi-closehandle?redirectedfrom=MSDN
func CloseHandle(proc *syscall.LazyProc, handle uintptr) error {
if proc.Name != "CloseHandle" {
panic("proc.Name != CloseHandle")
}
val, _, err := proc.Call(handle)
if val == 0 {
return err
}
return nil
}
func TestCreateMutexW(t *testing.T) {
handle, err := CreateMutexW(procCreateMutexW, "hello world")
if err != nil {
t.Fatalf(err.Error())
}
_, err = CreateMutexW(procCreateMutexW, "hello world")
if err == nil || err != syscall.ERROR_ALREADY_EXISTS {
t.Error("should panic")
}
if err = CloseHandle(procCloseHandle, handle); err != nil {
t.Error(err)
}
// We can create again since we have closed.
handle, _ = CreateMutexW(procCreateMutexW, "hello world")
if err = CloseHandle(procCloseHandle, handle); err != nil {
t.Error(err)
}
}

behave like a "cat" linux command using io/ioutil

have a task to creat a go file, which bahaves like cat command. I have some ideas how to do it, but i don't know how to read input if no file names written.
student#ubuntu:~/div-01/cat$ go build cat.go
student#ubuntu:~/div-01/cat$ ./cat
Hello
Hello
^C
student#ubuntu:~/div-01/cat$
Also i can use only ioutil, io, os packages and one of schools packages to print output.
In internet found this, but it just saves all input in data and then, after I press ctrl+C it prints it back.
package main
import (
"fmt"
"io/ioutil"
"os"
)
func main() {
var lenght int
args := os.Args[1:]
for i := range args {
lenght++
i++
}
if lenght == 0 {
data, err := ioutil.ReadAll(os.Stdin)
if err == nil {
fmt.Println(data)
}
}
}
To check the number of arguments, use len:
package main
import (
"fmt"
"io/ioutil"
"os"
)
func main() {
// No arguments give, read from STDIN.
if len(os.Args) == 1 {
data, err := ioutil.ReadAll(os.Stdin)
if err != nil {
os.Exit(1)
}
fmt.Printf("%s", data)
os.Exit(0)
}
// TODO Read the content of all files.
}
well, I found this and it works
if len(os.Args) == 1 {
_, err := io.Copy(os.Stdout, os.Stdin)
if err != nil {
fmt.Println(err)
}
fmt.Println("^C")
}

How to write log into .log files in golang?

I'am new use go language, still learn and need help. How can I write my application logs in go language into ".log" files similiar like PHP phalcon framework ?
I have searching google and get go language tutorials, but there is no simple understanding example. I have try some example but the logs text doesn't write continously. Here's the example that I used to learn for logging log, and I think it's completly wrong.
package main
import (
"bufio"
"fmt"
"io/ioutil"
"os"
"time"
)
func check(e error) {
if e != nil {
panic(e)
}
}
func main() {
now := time.Now()
dt := now.Format("2006-01-02")
dt2 := now.Format("2006-01-02 15:04:05")
// To start, here's how to dump a string (or just
// bytes) into a file.
d1 := []byte("hello\ngo11\n" + dt2)
err := ioutil.WriteFile("/Users/my/Documents/work/src/logs/log-"+dt+".log", d1, 0644)
check(err)
// For more granular writes, open a file for writing.
f, err := os.Create("/Users/my/Documents/work/src/logs/log1.log")
check(err)
// It's idiomatic to defer a `Close` immediately
// after opening a file.
defer f.Close()
// You can `Write` byte slices as you'd expect.
d2 := []byte{115, 111, 109, 101, 10}
n2, err := f.Write(d2)
check(err)
fmt.Printf("wrote %d bytes\n", n2)
// A `WriteString` is also available.
n3, err := f.WriteString("writes\n" + dt)
fmt.Printf("wrote %d bytes\n", n3)
// Issue a `Sync` to flush writes to stable storage.
f.Sync()
// `bufio` provides buffered writers in addition
// to the buffered readers we saw earlier.
w := bufio.NewWriter(f)
n4, err := w.WriteString("buffered\n")
fmt.Printf("wrote %d bytes\n", n4)
// Use `Flush` to ensure all buffered operations have
// been applied to the underlying writer.
w.Flush()
}
You're trying so many different things it's hard to tell what you're aiming for specifically, but if you simply want to write logs to a file, here is an example:
package main
import (
"log"
"os"
)
func main() {
//create your file with desired read/write permissions
f, err := os.OpenFile("filename", os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0644)
if err != nil {
log.Fatal(err)
}
//defer to close when you're done with it, not because you think it's idiomatic!
defer f.Close()
//set output of logs to f
log.SetOutput(f)
//test case
log.Println("check to make sure it works")
}
I recommend taking a look at Lumberjack. It handles rolling logs with several file conditions quite nicely. I'm using it for a streaming network sensor.
I have a small project that uses the log.Logger as following:
package main
import (
"errors"
"log"
"os"
)
const (
logNone = iota
logInfo
logWarning
logError
logVerbose
logDebug
)
type myFileLogger struct {
logger *log.Logger
logFile *os.File
logLevel int
}
func newFileLogger() *myFileLogger {
return &myFileLogger{
logger: nil,
logFile: nil,
logLevel: logNone,
}
}
func (myLogger *myFileLogger) startLog(level int, file string) error {
f, err := os.OpenFile(file, os.O_WRONLY|os.O_CREATE|os.O_APPEND, os.ModePerm)
if err != nil {
return err
}
myLogger.logger = log.New(f, "", 0)
myLogger.logLevel = level
myLogger.logFile = f
return nil
}
func (myLogger *myFileLogger) stopLog() error {
if myLogger.logFile != nil {
return myLogger.logFile.Close()
}
return nil
}
// You can add a log of auxiliary functions here to make the log more easier
func (myLogger *myFileLogger) log(level int, msg string) error {
if myLogger.logger == nil {
return errors.New("myFileLogger is not initialized correctly")
}
if level >= myLogger.logLevel {
myLogger.logger.Print(msg) // maybe you want to include the loglevel here, modify it as you want
}
return nil
}
func main() {
logger := newFileLogger()
if err := logger.startLog(logError, "myLogFile.log"); err != nil {
panic(err.Error())
}
defer func() {
logger.stopLog()
}()
logger.log(logInfo, "Info level log msg\n") // this will be ignored
logger.log(logError, "Error: error message\n") // this should included in the log file
}
import (
"os/exec"
)
func main() {
// check error here...
exec.Command("/bin/sh", "-c", "echo "+err.Error()+" >> log.log").Run()
}
I have my logger do this, here is an example.
Send message sends error to my IRC and sendSMS sends error to my phone.
in my config file I have:
var ErrorFile = "error.log"
type errorLog struct {
}
func (e errorLog) Write(p []byte) (n int, err error) {
fmt.Println("Error: " + string(p))
if config.Verbose == 0 {
file, _ := os.OpenFile(config.ErrorFile, os.O_RDWR|os.O_APPEND|os.O_CREATE, 0666)
file.WriteString(string(p))
sendMSG(string(p))
sendSMS(string(p))
// Close the file when the surrounding function exists
defer file.Close()
}
return n, err
}
// Logger is a helpper method to print out a more useful error message
var Logger = log.New(errorLog{}, "", log.Lmicroseconds|log.Lshortfile)

How to read a text file? [duplicate]

This question already has answers here:
How can I read a whole file into a string variable
(7 answers)
Closed 4 years ago.
I'm trying to read "file.txt" and put the contents into a variable using Golang. Here is what I've tried...
package main
import (
"fmt"
"os"
"log"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
log.Fatal(err)
}
fmt.Print(file)
}
The file gets read successfully and the return from os.Open returns a type of *os.File
It depends on what you are trying to do.
file, err := os.Open("file.txt")
fmt.print(file)
The reason it outputs &{0xc082016240}, is because you are printing the pointer value of a file-descriptor (*os.File), not file-content. To obtain file-content, you may READ from a file-descriptor.
To read all file content(in bytes) to memory, ioutil.ReadAll
package main
import (
"fmt"
"io/ioutil"
"os"
"log"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
log.Fatal(err)
}
defer func() {
if err = file.Close(); err != nil {
log.Fatal(err)
}
}()
b, err := ioutil.ReadAll(file)
fmt.Print(b)
}
But sometimes, if the file size is big, it might be more memory-efficient to just read in chunks: buffer-size, hence you could use the implementation of io.Reader.Read from *os.File
func main() {
file, err := os.Open("file.txt")
if err != nil {
log.Fatal(err)
}
defer func() {
if err = file.Close(); err != nil {
log.Fatal(err)
}
}()
buf := make([]byte, 32*1024) // define your buffer size here.
for {
n, err := file.Read(buf)
if n > 0 {
fmt.Print(buf[:n]) // your read buffer.
}
if err == io.EOF {
break
}
if err != nil {
log.Printf("read %d bytes: %v", n, err)
break
}
}
}
Otherwise, you could also use the standard util package: bufio, try Scanner. A Scanner reads your file in tokens: separator.
By default, scanner advances the token by newline (of course you can customise how scanner should tokenise your file, learn from here the bufio test).
package main
import (
"fmt"
"os"
"log"
"bufio"
)
func main() {
file, err := os.Open("file.txt")
if err != nil {
log.Fatal(err)
}
defer func() {
if err = file.Close(); err != nil {
log.Fatal(err)
}
}()
scanner := bufio.NewScanner(file)
for scanner.Scan() { // internally, it advances token based on sperator
fmt.Println(scanner.Text()) // token in unicode-char
fmt.Println(scanner.Bytes()) // token in bytes
}
}
Lastly, I would also like to reference you to this awesome site: go-lang file cheatsheet. It encompassed pretty much everything related to working with files in go-lang, hope you'll find it useful.

Restricting to Single Instance of Executable with Golang

I need to only allow one instance of my Golang executable at a time. I'm not sure how to use a Global Mutex to make sure no other instances are running.
This would be running on a Windows Machine.
I know this topic is a bit old, but I needed it recently on Windows and I'll post here how I did it in case someone else needs.
Thx to #VonC for pointing me in the right direction.
var (
kernel32 = syscall.NewLazyDLL("kernel32.dll")
procCreateMutex = kernel32.NewProc("CreateMutexW")
)
func CreateMutex(name string) (uintptr, error) {
ret, _, err := procCreateMutex.Call(
0,
0,
uintptr(unsafe.Pointer(syscall.StringToUTF16Ptr(name))),
)
switch int(err.(syscall.Errno)) {
case 0:
return ret, nil
default:
return ret, err
}
}
// mutexName starting with "Global\" will work across all user sessions
_, err := CreateMutex("SomeMutexName")
I created a lib with a more complete example: https://github.com/rodolfoag/gow32
Thx!
There doesn't seem to be a cross-platform solution (beside writing a file, and looking for that file at start time).
On Windows, this thread reports
the recommended approach (and the one that has worked great for me) is to use the CreateSemaphore function.
If the name you specify starts with "Global\", then the semaphore is unique for the entire system and a second attempt to open it will fail.
This is a kernel32 call, which has some wrapper in Go available.
kostix adds in the comments:
look at the Go source code around the pkg\syscall hierarchy -- it contains a good wealth of examples on how to call out to DLLs on Windows using syscalls (and that's how you access anything in Windows API).
That would be syscall/dll_windows.go. (And here is a gist)
The odbc package by brainman is another example of direct API calls on Windows -- possibly easier to digest.
Like api/zapi_windows.go.
You could use sockets, simple to use and will work on everything really.
package main
import (
"fmt"
"net"
"os"
"strings"
)
const (
INSTANCE_PORT = 9292
)
func main() {
listener, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", INSTANCE_PORT))
if err != nil {
if strings.Index(err.Error(), "in use") != -1 {
//optionally send command line arguments to the other instance
fmt.Fprintln(os.Stderr, "Already running.")
return
} else {
panic(err)
}
}
for {
conn, err := listener.Accept()
if err != nil {
println("Error accept:", err.Error())
return
}
go do_something_with(conn)
}
}
You could adapt the code from tendo's python library source
what they do is
for windows :
creating a file made of the executable absolute path (well it's a library, so in your case, you can just define an identifier, to prevent you from "i put the executable in 2 places")
For windows: trying first to remove the file if existing, and if not creating the file with os.O_CREAT | os.O_EXCL | os.O_RDWR
For POSIX compatible systems: trying first to remove the file if existing and if not creating the file and acquiring a lock on it using fcntl.LOCK_EX | fcntl.LOCK_NB
any failure mean the program is already running
and then you can use a defer action to remove the lock (on posix system) and delete the file
Go permit you to create both version wit a build comment to tell which file to compile depending on your OS so you have
for unix system
// +build !windows
package main
import (
"os"
"syscall"
)
func create_lock_file(filename string) (*os.File, error) {
file, err := os.OpenFile(filename, os.O_WRONLY, 0666)
if err != nil {
return nil, err
}
err = syscall.Flock(int(file.Fd()), syscall.LOCK_EX|syscall.LOCK_NB)
if err != nil {
return nil, err
}
return file, nil
}
for windows:
// +build !windows
package main
import (
"os"
)
func create_lock_file(filename string) (*os.File, error) {
if _, err := os.Stat(filename); err == nil {
err = os.Remove(filename)
if err != nil {
return nil, err
}
}
return os.OpenFile(filename, os.O_CREATE|os.O_EXCL|os.O_RDWR, 0666)
}
and a test
package main
import (
"fmt"
"time"
)
func main() {
_, err := create_lock_file("plop.lock")
if err != nil {
fmt.Println("error ", err.Error())
}
time.Sleep(10 * time.Second)
fmt.Println("end ")
}
I've started a library out of it that you can find here
Improvements to this answer. (I am unsure if this answer will distort the original meaning, so I have written a new answer.)
Features:
deprecated: StringToUTF16Ptr is deprecated. Use UTF16PtrFromString instead.
Add the CloseHandle so that you can cancel the CreateMutexW.
package _test
import (
"syscall"
"testing"
"unsafe"
)
var (
kernel32 = syscall.NewLazyDLL("kernel32.dll")
procCreateMutexW = kernel32.NewProc("CreateMutexW")
procCloseHandle = kernel32.NewProc("CloseHandle")
)
// https://learn.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-createmutexW#return-value
func CreateMutexW(proc *syscall.LazyProc, name string) (uintptr, error) {
if proc.Name != "CreateMutexW" {
panic("proc.Name != CreateMutexW")
}
lpName, _ := syscall.UTF16PtrFromString(name) // LPCWSTR
if handleID, _, err := proc.Call(
0,
0,
uintptr(unsafe.Pointer(lpName)),
); err.(syscall.Errno) == 0 {
return handleID, nil
} else {
return handleID, err
}
}
// https://learn.microsoft.com/en-us/windows/win32/api/handleapi/nf-handleapi-closehandle?redirectedfrom=MSDN
func CloseHandle(proc *syscall.LazyProc, handle uintptr) error {
if proc.Name != "CloseHandle" {
panic("proc.Name != CloseHandle")
}
val, _, err := proc.Call(handle)
if val == 0 {
return err
}
return nil
}
func TestCreateMutexW(t *testing.T) {
handle, err := CreateMutexW(procCreateMutexW, "hello world")
if err != nil {
t.Fatalf(err.Error())
}
_, err = CreateMutexW(procCreateMutexW, "hello world")
if err == nil || err != syscall.ERROR_ALREADY_EXISTS {
t.Error("should panic")
}
if err = CloseHandle(procCloseHandle, handle); err != nil {
t.Error(err)
}
// We can create again since we have closed.
handle, _ = CreateMutexW(procCreateMutexW, "hello world")
if err = CloseHandle(procCloseHandle, handle); err != nil {
t.Error(err)
}
}

Resources