I had a problem when I tried run a Go app with cron job. It's seem that every func of os library can not execute in cron job. Here is my code. I've searched for a lot of time but haven't got any solotion yet.
Here is my code.
package main
import (
"fmt"
"os"
"os/exec"
)
func main() {
out, err := exec.Command("ls").Output()
file, _ := os.Create("test.txt")
_, err1 := file.Write([]byte(string(out) + "\n"))
if err == nil && err1 == nil {
fmt.Println("test")
}
fmt.Println(string(out))
}
Here is my cron job
* * * * * go run /root/code/main.go
Please help me fix this problem or any recommend to run a go app with cron job.
By default, cron jobs are run with the root user, and probably there is no go binary in your root user's path.
To check this, you can run.
# crontab -e
* * * * * whoami >> /tmp/debug.txt && where go && echo OK >> /tmp/debug.txt || echo ERROR >> /tmp/debug.txt
Which will show you user info and "Can I find go binary" information.
You can change the user who runs the cronjob
Better Way
As others said, it's not a good idea to run Go code with go run. Each time compiler needs to compile the code and run it.
If you run go build and run the produced executable, it'll simplify your workflow. Also, by default, go binaries are single binaries that contain all dependencies, which simplifies lots of things. You can just ./executable-name anywhere
Related
I'm trying to do a report of all the objects in all the projects we have in Cloud Storage of our Org. I'm using this repo from the Google Professionnal Services as it's doing exactly what we want: https://github.com/GoogleCloudPlatform/professional-services/tree/main/tools/gcs2bq
We want to use containers instead of just the go code on a Cloud Function for portability mainly.
Locally everything is good and the program behave as expected but when I try in Cloud Run things get tricky. From what I understand, the go part needs to listen to a port, which I added at the beginning of the main so the container can be deployed, which it is:
// Determine port for HTTP service
port := os.Getenv("PORT")
if port == "" {
port = "8080"
log.Printf("defaulting to port %s", port)
}
Start HTTP server.
log.Printf("listening on port %s", port)
if err := http.ListenAndServe(":"+port, nil); err != nil {
log.Fatal(err)
}
But as you can see in the repo, the first file called is the run.sh one. Which set environment variables and then call the main.go. It sucessfully complete it's task, which is get all the size of the different files. But after that the run.sh doesnt "resume" and go to the part where it uploads the data in a BigQuery table, which locally work.
Here is the part in the run.sh file where I have a problem. Note : I don't have errors from executing the ./gcs2bq Note 2 : Every environment variable has a correct value
./gcs2bq $GCS2BQ_FLAGS || error "Export failed!" 2 <- doesnt get past this line
gsutil mb -p "${GCS2BQ_PROJECT}" -c standard -l "${GCS2BQ_LOCATION}" -b on "gs://${GCS2BQ_BUCKET}" || echo "Info: Storage bucket already exists: ${GCS2BQ_BUCKET}"
gsutil cp "${GCS2BQ_FILE}" "gs://${GCS2BQ_BUCKET}/${GCS2BQ_FILENAME}" || error "Failed copying ${GCS2BQ_FILE} to gs://${GCS2BQ_BUCKET}/${GCS2BQ_FILENAME}!" 3
bq mk --project_id="${GCS2BQ_PROJECT}" --location="${GCS2BQ_LOCATION}" "${GCS2BQ_DATASET}" || echo "Info: BigQuery dataset already exists: ${GCS2BQ_DATASET}"
bq load --project_id="${GCS2BQ_PROJECT}" --location="${GCS2BQ_LOCATION}" --schema bigquery.schema --source_format=AVRO --use_avro_logical_types --replace=true "${GCS2BQ_DATASET}.${GCS2BQ_TABLE}" "gs://${GCS2BQ_BUCKET}/${GCS2BQ_FIL$
error "Failed to load gs://${GCS2BQ_BUCKET}/${GCS2BQ_FILENAME} to BigQuery table ${GCS2BQ_DATASET}.${GCS2BQ_TABLE}!" 4
gsutil rm "gs://${GCS2BQ_BUCKET}/${GCS2BQ_FILENAME}" || error "Failed deleting gs://${GCS2BQ_BUCKET}/${GCS2BQ_FILENAME}!" 5
rm -f "${GCS2BQ_FILE}"
I'm kinda new to containers and Cloud Run and even after reading projects and documentation, I'm not sure what I'm doing wrong, Is it normal that the .sh is "stuck" when calling the main.go? I can provide more details/explaination if needed.
Okay so for anyone who encounter similar situation this is how I made it work for me.
The container isn't supposed to stop so no exit, it will just go back to the main function.
That means that when I called executable it just looped and never exited and completed the task. So the solution here is to "recode" everything past the call in golang directly into the main.go
Here the run.sh is then useless so I used another .go file that listen for http request and then call the code that gather data and send it to Bigquery.
I have method, as part of zeebe workflow job. And when it fails, I want to restart all job. I found, that it can be done with NewFailJobCommand, but it seems that the job fails on the first try. How I can restart the job if it fails?
err := w.workflowStore.InitScanEventsTTL(ctx, scanID, job.Msg.Tenant)
if err != nil {
return w.client.NewFailJobCommand().JobKey(job.Key).Retries(job.Retries -
1).ErrorMessage(reason).Send(ctx)
}
You need to specify the retry count in the task properties in the process model.
I need to test if a process terminates and all I have is its pid number. To do so I test if the pseudo file "/proc/<pid>" exist.
Writing a test for this function, I noticed that the process doesn’t terminate as expected.
For the test, I run "sleep 10" as sub-process which should run for 10 seconds. After starting this process, I poll for the disappearance of the pseudo file "/proc/<pid>". That pseudo file never disappear and the termination of the sub-process is not detected.
Testing the code of golang playground reproduces the problem: https://play.golang.org/p/fb4CbXkIjh3.
I checked that the process is created, and that the pid is correct. While checking the process is seen that it turns into <defunct>. It isn’t thus removed.
The questions are the following:
why doesn’t the sub-process terminate ?
how can I change the code to make it terminate ?
package main
import (
"fmt"
"log"
"os"
"os/exec"
"strconv"
"time"
)
func main() {
fmt.Println("Hello, playground")
cmd := exec.Command("sleep", "10")
if err := cmd.Start(); err != nil {
log.Fatal("unexpected error:", err)
}
pidStr := strconv.Itoa(cmd.Process.Pid)
log.Println("sleep pid:", pidStr)
for {
if _, err := os.Stat("/proc/" + pidStr); os.IsNotExist(err) {
log.Println("detect termination of /proc/" + pidStr)
return
}
log.Println("pgm /proc/" + pidStr + " is running")
time.Sleep(3 * time.Second)
}
}
At the OS level, in any POSIX-compatible OS (Unix, Linux, Darwin, etc), a process that has completed, but not yet been collected by its superior, is in "defunct" or "zombie" state. It still exists, but cannot be killed: it is already dead. It exists precisely so that its superior—the process that can call the OS level wait system call—can call the OS level wait system call and see that the process is now dead.
Once its superior has waited for it, the process is truly removed: there is no longer a zombie process hogging that process ID. If you have a /proc file system, this is when the process vanishes from /proc.
In Go, calling cmd.Wait() invokes the OS-level wait call, so that is the way to do this. If you would like to collect the result of this cmd.Wait(), a good way to do that is to send it through a channel.
(If you want to spawn a very long running process and not wait for it, you can disown it, so that you are no longer its superior. The details on doing this correctly are full of OS-specific doodads, such as discarding control ttys, setting sessions, using procctl or prctl, and so on.)
How can I get an exclusive read access to a file in go? I have tried documentations from docs but I am still able to open the file in notepad and edit it. I want to deny any other process to have access to read and write while the first process has not closed it explicitly. In .NET I could do something as:
File.Open("a.txt", FileMode.Open, FileAccess.ReadWrite, FileShare.None);
How do I do it in go?
I finally found a go package that can lock a file.
Here is the repo:
https://github.com/juju/fslock
go get -u github.com/juju/fslock
this package does exactly what it says
fslock provides a cross-process mutex based on file locks that works
on windows and *nix platforms. fslock relies on LockFileEx on Windows
and flock on *nix systems. The timeout feature uses overlapped IO on
Windows, but on *nix platforms, timing out requires the use of a
goroutine that will run until the lock is acquired, regardless of
timeout. If you need to avoid this use of goroutines, poll TryLock in
a loop.
To use this package, first, create a new lock for the lockfile
func New(filename string) *Lock
This API will create the lockfile if it already doesn't exist.
Then we can use the lockhandle to lock (or try lock) the file
func (l *Lock) Lock() error
There is also a timeout version of the above function that will try to get the lock of the file until timeout
func (l *Lock) LockWithTimeout(timeout time.Duration) error
Finally, if you are done, release the acquired lock by
func (l *Lock) Unlock() error
Very basic implementation
package main
import (
"time"
"fmt"
"github.com/juju/fslock"
)
func main() {
lock := fslock.New("../lock.txt")
lockErr := lock.TryLock()
if lockErr != nil {
fmt.Println("falied to acquire lock > " + lockErr.Error())
return
}
fmt.Println("got the lock")
time.Sleep(1 * time.Minute)
// release the lock
lock.Unlock()
}
I'm using following code segment to get the XML definition of a virtual machine running on XEN Hypervisor. The code is trying to execute the command virsh dumpxml Ubutnu14 which will give the XML of the VM named Ubuntu14
virshCmd := exec.Command("virsh", "dumpxml", "Ubuntu14")
var virshCmdOutput bytes.Buffer
var stderr bytes.Buffer
virshCmd.Stdout = &virshCmdOutput
virshCmd.Stderr = &stderr
err := virshCmd.Run()
if err != nil {
fmt.Println(err)
fmt.Println(stderr.String())
}
fmt.Println(virshCmdOutput.String())
This code always goes into the error condition for the given domain name and I get the following output.
exit status 1
error: failed to get domain 'Ubuntu14'
error: Domain not found: no domain with matching name 'Ubuntu14'
But if I run the standalone command virsh dumpxml Ubuntu14, I get the correct XML definition.
I would appreciate if someone could give me some hints on what I'm doing wrong. My host machine is Ubuntu-16.04 and golang version is go1.6.2 linux/amd64
I expect you are running virsh as a different user in these two scenarios, and since you don't provide any URI, it is connecting to a different libvirtd instance. If you run virsh as non-root, then it'll usually connect to qemu:///session, but if you run virsh as root, then it'll usually connect to qemu:///system. VMs registered against one URI, will not be visible when connecting to the other URI.
BTW, if you're using go, you'd be much better off using the native Go library bindings for libvirt instead of exec'ing virsh. Your "virsh dumpxml" invokation is pretty much equivalent to this:
import (
"github.com/libvirt/libvirt-go"
)
conn, err := libvirt.NewConnect("qemu:///system")
dom, err := conn.LookupDomainByName("Ubuntu14")
xml, err := dom.GetXMLDesc(0)
(obviously do error handling too)