I'm using golang revel and I need a job to be run every first monday of every month, a quartz cron spec for that would look like this:
0 0 0 ? 1/1 MON#1
But robfig/cron doesn't accept a spec like that, hence neither revel/jobs.
Anyone knows how can I solve that [using revel jobs]?
To me, the easiest solution would be something like this:
func (e SomeStruct) Run() {
t := time.Now().Local()
day_num, _ := t.Day()
if day_num <= 7 {
fmt.Println("Hello, playground")
}
}
func init() {
revel.OnAppStart(func() {
jobs.Schedule("0 0 * * 1", SomeStruct{})
})
Where you simply run the job EVERY monday, but in the job itself, check if it's the FIRST monday before you actually do anything. There may be a better way (not very familiar with Revel), but glancing through how their jobs work this would work and it's not like it will be a performance issue.
To check for the first Monday in the month,
package main
import (
"fmt"
"time"
)
func IsFirstMonday() bool {
t := time.Now().Local()
if d := t.Day(); 1 <= d && d <= 7 {
if wd := t.Weekday(); wd == time.Monday {
return true
}
}
return false
}
func main() {
fmt.Println(IsFirstMonday())
}
Related
I am using github.com/robfig/cron library. I want to run cronjob at epoc time with millisecond and work every second. The cron starts at 000 millisecond. I need it to start at specific times.
For example if I take the following:
c := cron.New()
c.AddFunc("#every 1s", func() {
// Do Something
})
c.Start()
And run it at 1657713890300 epoc timestamp then I want the function to run at:
1657713891300
1657713892300
1657713893300.
Currently, cron running at
1657713891000
1657713892000
1657713893000.
Is this possible?
When you use #every 1s the library creates a ConstantDelaySchedule which "rounds so that the next activation time will be on the second".
If that is not what you want then you can create your own scheduler (playground):
package main
import (
"fmt"
"time"
"github.com/robfig/cron/v3"
)
func main() {
time.Sleep(300 * time.Millisecond) // So we don't start cron too near the second boundary
c := cron.New()
c.Schedule(CustomConstantDelaySchedule{time.Second}, cron.FuncJob(func() {
fmt.Println(time.Now().UnixNano())
}))
c.Start()
time.Sleep(time.Second * 5)
}
// CustomConstantDelaySchedule is a copy of the libraries ConstantDelaySchedule with the rounding removed
type CustomConstantDelaySchedule struct {
Delay time.Duration
}
// Next returns the next time this should be run.
func (schedule CustomConstantDelaySchedule) Next(t time.Time) time.Time {
return t.Add(schedule.Delay)
}
Follow up: The above uses the time.Time passed to Next which is time.Now() so will the time will slowly advance over time.
Addressing this is possible (see below - playground) but doing this introduces some potential issuers (the CustomConstantDelaySchedule must not be reused and if the jobs take too long to run then you will still end up with discrepancies). I'd suggest that you consider moving away from the cron package and just use a time.Ticker.
package main
import (
"fmt"
"time"
"github.com/robfig/cron/v3"
)
func main() {
time.Sleep(300 * time.Millisecond) // So we don't start cron too nead the second boundary
c := cron.New()
c.Schedule(CustomConstantDelaySchedule{Delay: time.Second}, cron.FuncJob(func() {
fmt.Println(time.Now().UnixNano())
}))
c.Start()
time.Sleep(time.Second * 5)
}
// CustomConstantDelaySchedule is a copy of the libraries ConstantDelaySchedule with the rounding removed
// Note that because this stored the last time it cannot be reused!
type CustomConstantDelaySchedule struct {
Delay time.Duration
lastTarget time.Time
}
// Next returns the next time this should be run.
func (schedule CustomConstantDelaySchedule) Next(t time.Time) time.Time {
if schedule.lastTarget.IsZero() {
schedule.lastTarget = t.Add(schedule.Delay)
} else {
schedule.lastTarget = schedule.lastTarget.Add(schedule.Delay)
}
return schedule.lastTarget
}
I've slightly modified a very simple Switch example from "A Tour of Go" website:
package main
import (
"fmt"
"time"
)
func main() {
day := "Wednesday"
fmt.Printf("When's %s?\n", day)
today := time.Now().Weekday()
switch time.Wednesday {
case today + 0:
fmt.Println("Today.")
case today + 1:
fmt.Println("Tomorrow.")
case today + 2:
fmt.Println("In two days.")
default:
fmt.Println("Too far away.")
}
}
Instead of time.Wednesday I would like to use the value of day to fetch the time.Weekday.
In Python/JS it would be time[day].
How does one do this in Go ?
I have Apache Beam code implementation on Go SDK as described below. The pipeline has 3 steps. One is textio.Read, other one is CountLines and the last step is ProcessLines. ProcessLines step takes around 10 seconds time. I just added a Sleep function for the sake of brevity.
I am calling the pipeline with 20 workers. When I run the pipeline, my expectation was 20 workers would run in parallel and textio.Read read 20 lines from the file and ProcessLines would do 20 parallel executions in 10 seconds. However, the pipeline did not work like that. It's currently working in a way that textio.Read reads one line from the file, pushes the data to the next step and waits until ProcessLines step completes its 10 seconds work. There is no parallelism and there is only one line string from the file throughout the pipeline. Could you please clarify me what I'm doing wrong for parallelism? How should I update the code to achieve parallelism as described above?
package main
import (
"context"
"flag"
"time"
"github.com/apache/beam/sdks/go/pkg/beam"
"github.com/apache/beam/sdks/go/pkg/beam/io/textio"
"github.com/apache/beam/sdks/go/pkg/beam/log"
"github.com/apache/beam/sdks/go/pkg/beam/x/beamx"
)
// metrics to be monitored
var (
input = flag.String("input", "", "Input file (required).")
numberOfLines = beam.NewCounter("extract", "numberOfLines")
lineLen = beam.NewDistribution("extract", "lineLenDistro")
)
func countLines(ctx context.Context, line string) string {
lineLen.Update(ctx, int64(len(line)))
numberOfLines.Inc(ctx, 1)
return line
}
func processLines(ctx context.Context, line string) {
time.Sleep(10 * time.Second)
}
func CountLines(s beam.Scope, lines beam.PCollection) beam.PCollection
{
s = s.Scope("Count Lines")
return beam.ParDo(s, countLines, lines)
}
func ProcessLines(s beam.Scope, lines beam.PCollection) {
s = s.Scope("Process Lines")
beam.ParDo0(s, processLines, lines)
}
func main() {
// If beamx or Go flags are used, flags must be parsed first.
flag.Parse()
// beam.Init() is an initialization hook that must be called on startup. On
// distributed runners, it is used to intercept control.
beam.Init()
// Input validation is done as usual. Note that it must be after Init().
if *input == "" {
log.Fatal(context.Background(), "No input file provided")
}
p := beam.NewPipeline()
s := p.Root()
l := textio.Read(s, *input)
lines := CountLines(s, l)
ProcessLines(s, lines)
// Concept #1: The beamx.Run convenience wrapper allows a number of
// pre-defined runners to be used via the --runner flag.
if err := beamx.Run(context.Background(), p); err != nil {
log.Fatalf(context.Background(), "Failed to execute job: %v", err.Error())
}
}
EDIT:
After I got the answer about the problem might be caused by fusion, I changed the related part of the code but it did not work again.
Now the first and second step is working in parallel, however the third step ProcessLines is not working in parallel. I only made the following changes. Can anyone tell me what the problem is?
func AddRandomKey(s beam.Scope, col beam.PCollection) beam.PCollection {
return beam.ParDo(s, addRandomKeyFn, col)
}
func addRandomKeyFn(elm beam.T) (int, beam.T) {
return rand.Int(), elm
}
func countLines(ctx context.Context, _ int, lines func(*string) bool, emit func(string)) {
var line string
for lines(&line) {
lineLen.Update(ctx, int64(len(line)))
numberOfLines.Inc(ctx, 1)
emit(line)
}
}
func processLines(ctx context.Context, _ int, lines func(*string) bool) {
var line string
for lines(&line) {
time.Sleep(10 * time.Second)
numberOfLinesProcess.Inc(ctx, 1)
}
}
func CountLines(s beam.Scope, lines beam.PCollection) beam.PCollection {
s = s.Scope("Count Lines")
keyed := AddRandomKey(s, lines)
grouped := beam.GroupByKey(s, keyed)
return beam.ParDo(s, countLines, grouped)
}
func ProcessLines(s beam.Scope, lines beam.PCollection) {
s = s.Scope("Process Lines")
keyed := AddRandomKey(s, lines)
grouped := beam.GroupByKey(s, keyed)
beam.ParDo0(s, processLines, grouped)
}
Many advanced runners of MapReduce-type pipelines fuse stages that can be run in memory together. Apache Beam and Dataflow are no exception.
What's happening here is that the three steps of your pipeline are fused, and happening in the same machine. Furthermore, the Go SDK does not currently support splitting the Read transform, unfortunately.
To achieve parallelism in the third transform, you can break the fusion between Read and ProcessLines. You can do that adding a random key to your lines, and a GroupByKey transform.
In Python, it would be:
(p | beam.ReadFromText(...)
| CountLines()
| beam.Map(lambda x: (random.randint(0, 1000), x))
| beam.GroupByKey()
| beam.FlatMap(lambda k, v: v) # Discard the key, and return the values
| ProcessLines())
This would allow you to parallelize ProcessLines.
How to add 1 sec to the date in golang ? I have:
t := time.Now().Format("2006/02/01 03:04:05")
and want something like below but so far getting mismatched types string and time.Duration error
t1, t2, t3 = t + 1*time.Second, t+3*time.Second, t+2*time.Second
func (t Time) Add(d Duration) Time
https://golang.org/pkg/time/#Time.Add
You are asigning a string to t (the result of calling Format) instead of a Time (the result of calling Now). Here's an working example:
package main
import (
"fmt"
"time"
)
func main() {
t := time.Now()
fmt.Println(t.Format(time.RFC3339))
t = t.Add(time.Second)
fmt.Println(t.Format(time.RFC3339))
}
// prints
// 2017-01-21T16:51:31-05:00
// 2017-01-21T16:51:32-05:00
in the interests of learning more about Go, I have been playing with goroutines, and have noticed something - but am not sure what exactly I'm seeing, and hope someone out there might be able to explain the following behaviour.
the following code does exactly what you'd expect:
package main
import (
"fmt"
)
type Test struct {
me int
}
type Tests []Test
func (test *Test) show() {
fmt.Println(test.me)
}
func main() {
var tests Tests
for i := 0; i < 10; i++ {
test := Test{
me: i,
}
tests = append(tests, test)
}
for _, test := range tests {
test.show()
}
}
and prints 0 - 9, in order.
now, when the code is changed as shown below, it always returns with the last one first - doesn't matter which numbers I use:
package main
import (
"fmt"
"sync"
)
type Test struct {
me int
}
type Tests []Test
func (test *Test) show(wg *sync.WaitGroup) {
fmt.Println(test.me)
wg.Done()
}
func main() {
var tests Tests
for i := 0; i < 10; i++ {
test := Test{
me: i,
}
tests = append(tests, test)
}
var wg sync.WaitGroup
wg.Add(10)
for _, test := range tests {
go func(t Test) {
t.show(&wg)
}(test)
}
wg.Wait()
}
this will return:
9
0
1
2
3
4
5
6
7
8
the order of iteration of the loop isn't changing, so I guess that it is something to do with the goroutines...
basically, I am trying to understand why it behaves like this...I understand that goroutines can run in a different order than the order in which they're spawned, but, my question is why this always runs like this. as if there's something really obvious I'm missing...
As expected, the ouput is pseudo-random,
package main
import (
"fmt"
"runtime"
"sync"
)
type Test struct {
me int
}
type Tests []Test
func (test *Test) show(wg *sync.WaitGroup) {
fmt.Println(test.me)
wg.Done()
}
func main() {
fmt.Println("GOMAXPROCS", runtime.GOMAXPROCS(0))
var tests Tests
for i := 0; i < 10; i++ {
test := Test{
me: i,
}
tests = append(tests, test)
}
var wg sync.WaitGroup
wg.Add(10)
for _, test := range tests {
go func(t Test) {
t.show(&wg)
}(test)
}
wg.Wait()
}
Output:
$ go version
go version devel +af15bee Fri Jan 29 18:29:10 2016 +0000 linux/amd64
$ go run goroutine.go
GOMAXPROCS 4
9
4
5
6
7
8
1
2
3
0
$ go run goroutine.go
GOMAXPROCS 4
9
3
0
1
2
7
4
8
5
6
$ go run goroutine.go
GOMAXPROCS 4
1
9
6
8
4
3
0
5
7
2
$
Are you running in the Go playground? The Go playground, by design, is deterministic, which makes it easier to cache programs.
Or, are you running with runtime.GOMAXPROCS = 1? This runs one thing at a time, sequentially. This is what the Go playground does.
Go routines are scheduled randomly since Go 1.5. So, even if the order looks consistent, don't rely on it.
See Go 1.5 release note :
In Go 1.5, the order in which goroutines are scheduled has been changed. The properties of the scheduler were never defined by the language, but programs that depend on the scheduling order may be broken by this change. We have seen a few (erroneous) programs affected by this change. If you have programs that implicitly depend on the scheduling order, you will need to update them.
Another potentially breaking change is that the runtime now sets the default number of threads to run simultaneously, defined by GOMAXPROCS, to the number of cores available on the CPU. In prior releases the default was 1. Programs that do not expect to run with multiple cores may break inadvertently. They can be updated by removing the restriction or by setting GOMAXPROCS explicitly. For a more detailed discussion of this change, see the design document.