I have a Julia code which runs for quite sometime. I want to run the code for 3 hours and then terminate it. Does anyone know what is the most efficient way to do this. Would appreciate any advice. Thanks
#async and #sync are really useful for coroutine type process control in Julia. You can start an async process that calls exit at a later point so as to terminate the entire program:
function killafterseconds(s)
#async begin
sleep(s)
println("terminating after $s seconds")
exit(0)
end
end
function countdown(n)
for t in n:-1:0
println(t)
sleep(1)
end
end
killafterseconds(10)
countdown(10000)
Here's a solution using threads. I'm just begginging coding in Julia so it's not high quality code, but it works. You first need to wrap your function as a Task, by using the #task macro, and then call runTask():
function runTask(origtask:: Task, timeoutms:: Int)
startTime = Dates.datetime2epochms(now())
function internal_task(taskFinished)
try
schedule(origtask)
yield()
wait(origtask)
if istaskdone(origtask)
taskFinished[] = true
Base.task_result(origtask)
else
throw(ErrorException("Task is not done even after wait() for it to finish - something is wrong"))
end
catch e
#warn "Error while processing task: $e"
taskFinished[] = true
missing
end
end
taskFinished = Threads.Atomic{Bool}(false)
thread = Threads.#spawn internal_task(taskFinished)
while !taskFinished[] && (Dates.datetime2epochms(now()) - startTime) < timeoutms
sleep(0.1)
end
if taskFinished[]
return fetch(thread)
end
# Task timeouted
origtask.exception = InterruptException()
return missing
end # function
The worst part of this code is active waiting using sleep(0.1). I guess I could get rid of it by spawning one more thread, so there would be 3 in total: one would execute the actual work from the origtask and after doing it would notify a condition, one would wait on the mentioned condition and another one would sleep for the timeout number of seconds and then notify the condition.
I recommend using Distributed to spawn your function as a new process and control it's time (I believe I have been answering similar question but I cannot find the answer).
Here is the code:
using Distributed
function run_with_timeout(timeout::Int,f::Function, wid::Int)
result = RemoteChannel(()->Channel{Tuple}(1));
#spawnat wid put!(result, (f(),myid()))
res = (:timeout, wid)
time_elapsed = 0.0
while time_elapsed < timeout && !isready(result)
sleep(0.5)
time_elapsed += 0.5
end
if !isready(result)
println("Timeout! at $wid")
else
res = take!(result)
end
return res
end
You can use it like this (please note how external packages are imported):
wid = addprocs(1)[1]
#everywhere using DataFrames
#everywhere function ff()
sleep(2)
#code fir making heavy computations
#this is the place to write whatever you need
return DataFrame(x=[1,2],y=[1,3])
end
Now let us run it. Note that the second returned value is the workerid to run computations (you might want to remove it):
julia> run_with_timeout(10,() ->(try;ff();catch ee;dump(ee);end ),wid)
(2×2 DataFrame
│ Row │ x │ y │
│ │ Int64 │ Int64 │
├─────┼───────┼───────┤
│ 1 │ 1 │ 1 │
│ 2 │ 2 │ 3 │, 2)
If we give our code only 1 sec to execute it will fail:
julia> run_with_timeout(1,() ->(try;ff();catch ee;dump(ee);end ),wid)
Timeout! at 2
(:timeout, 2)
Finally, since the timeout occured the process should be killed (otherwise it would continue to execute):
rmprocs(wid)
Related
I'm self study of Python and it's my first code.
I'm working for analyze logs from the servers. Usually I need analyze full day logs. I created script (this is example, simple logic) just for check speed. If I use normal coding the duration of analyzing 20mil rows about 12-13 minutes. I need 200mil rows by 5 min.
What I tried:
Use multiprocessing (met issue with share memory, think that fix it). But as the result - 300K rows = 20 sec and no matter how many processes. (PS: Also need control processors count in advance)
Use threading (I found that it's not give any speed, 300K rows = 2 sec. But normal code same, 300K = 2 sec)
Use asyncio (I think that script is slow because need reads many files). Result same as threading - 300K = 2 sec.
Finally I think that all three my script incorrect and didn't work correctly.
PS: I try to avoid use specific python modules (like pandas) because in this case it will be more difficult to execute on different servers. Better to use common lib.
Please help to check 1st - multiprocessing.
import csv
import os
from multiprocessing import Process, Queue, Value, Manager
file = {"hcs.log", "hcs1.log", "hcs2.log", "hcs3.log"}
def argument(m, a, n):
proc_num = os.getpid()
a_temp_m = a["vod_miss"]
a_temp_h = a["vod_hit"]
with open(os.getcwd() + '/' + m, newline='') as hcs_1:
hcs_2 = csv.reader(hcs_1, delimiter=' ')
for j in hcs_2:
if j[3].find('MISS') != -1:
a_temp_m[n] = a_temp_m[n] + 1
elif j[3].find('HIT') != -1:
a_temp_h[n] = a_temp_h[n] + 1
a["vod_miss"][n] = a_temp_m[n]
a["vod_hit"][n] = a_temp_h[n]
if __name__ == '__main__':
procs = []
manager = Manager()
vod_live_cuts = manager.dict()
i = "vod_hit"
ii = "vod_miss"
cpu = 1
n = 1
vod_live_cuts[i] = manager.list([0] * cpu)
vod_live_cuts[ii] = manager.list([0] * cpu)
for m in file:
proc = Process(target=argument, args=(m, vod_live_cuts, (n-1)))
procs.append(proc)
proc.start()
if n >= cpu:
n = 1
proc.join()
else:
n += 1
[proc.join() for proc in procs]
[proc.close() for proc in procs]
I'm expect, each file by def argument will be processed by independent process and finally all results will be saved in dict vod_live_cuts. For each process I added independent list in dict. I think it will help cross operation for use this parameter. But maybe it's wrong way :(
using IPC is costly, so only use "shared objects" for saving the final result, not for intermediate results while parsing the file.
limiting the number of processes is done by using a multiprocessing.Pool, the following code uses it to reach the max hard-disk speed, you only need to post-process the results.
you can only parse data as fast as your HDD can read it (typically 30-80 MB/s), so if you need to improve the performance further you should use SSD or RAID0 for higher disk speed, you cannot get much faster than this without changing your hardware.
import csv
import os
from multiprocessing import Process, Queue, Value, Manager, Pool
file = {"hcs.log", "hcs1.log", "hcs2.log", "hcs3.log"}
def argument(m, a):
proc_num = os.getpid()
a_temp_m_n = 0 # make it local to process
a_temp_h_n = 0 # as shared lists use IPC
with open(os.getcwd() + '/' + m, newline='') as hcs_1:
hcs_2 = csv.reader(hcs_1, delimiter=' ')
for j in hcs_2:
if j[3].find('MISS') != -1:
a_temp_m_n = a_temp_m_n + 1
elif j[3].find('HIT') != -1:
a_temp_h_n = a_temp_h_n + 1
a["vod_miss"].append(a_temp_m_n)
a["vod_hit"].append(a_temp_h_n)
if __name__ == '__main__':
manager = Manager()
vod_live_cuts = manager.dict()
i = "vod_hit"
ii = "vod_miss"
cpu = 1
vod_live_cuts[i] = manager.list()
vod_live_cuts[ii] = manager.list()
with Pool(cpu) as pool:
tasks = []
for m in file:
task = pool.apply_async(argument, args=(m, vod_live_cuts))
tasks.append(task)
for task in tasks:
task.get()
print(list(vod_live_cuts[i]))
print(list(vod_live_cuts[ii]))
Assertion of condition is well known way to design application in strategic way. You could be completely sure that your code will work correctly a day after release, but also when other dev in your team will change this code.
There are 2 common ways to put assertion in Lua code:
assert(1 > 0, "Assert that math works")
if 1 <= 0 then
error("Assert that math doesn't work")
end
I would expect this thing similar from performance point of view.
Consider it only matter of style. But it happens to be not true.
assert works longer on my machine:
function with_assert()
for i=1,100000 do
assert(1 < 0, 'Assert')
end
end
function with_error()
for i=1,100000 do
if 1 > 0 then
error('Error')
end
end
end
local t = os.clock()
pcall(with_assert)
print(os.clock() - t)
t = os.clock()
pcall(with_error)
print(os.clock() - t)
>> 3.1999999999999e-05
>> 1.5e-05
Why does it happen?
Look at the source code of assert and error: assert does some work and then calls error.
But the loop in your code serves no purpose: throwing an error during the first iteration means that the rest of the iterations don't run. Perhaps you meant to put the loop around the pcalls as below. Then you'll find hardly any difference between the times.
function with_assert()
assert(1 < 0, 'Assert')
end
function with_error()
if 1 > 0 then
error('Error')
end
end
function test(f)
local t = os.clock()
for i=1,100000 do
pcall(f)
end
print(os.clock() - t)
end
test(with_assert)
test(with_error)
My project involves computing in parallel a map using Julia's Distributed's pmap function.
Mapping a given element could take a few seconds, or it could take essentially forever. I want a timeout or time limit for an individual map task/computation to complete.
If a map task finishes in time, great, return the result of the computation. If the task doesn't complete by the time limit, stop computation when the time limit has been reached, and return some value or message indicating a timeout occurred.
A minimal example follows. First are imported modules, and then worker processes are launched:
num_procs = 1
using Distributed
if num_procs > 1
# The main process (no calling addprocs) can be used for `pmap`:
addprocs(num_procs-1)
end
Next, the mapping task is defined for all the worker processes. The mapping task should timeout after 1 second:
#everywhere import Random
#everywhere begin
"""
Compute stuff for `wait_time` seconds, and return `wait_time`.
If `timeout` seconds elapses, stop computation and return something else.
"""
function waitForTimeUnlessTimeout(wait_time, timeout=1)
# < Insert some sort of timeout code? >
# This block of code simulates a long computation.
# (pretend the computation time is unknown)
x = 0
while time()-t0 < wait_time
x += Random.rand() - 0.5
end
# computation completed before time limit. Return wait_time.
round(wait_time, digits=2)
end
end
The function that executes the parallel map (pmap) is defined on the main process. Each map task randomly takes up to 2 seconds to complete, but should time out after 1 second.
function myParallelMapping(num_tasks = 20, max_runtime=2)
# random task runtimes between 0 and max_runtime
runtimes = Random.rand(num_tasks) * max_runtime
# return the parallel computation of the mapping tasks
pmap((runtime)->waitForTimeUnlessTimeout(runtime), runtimes)
end
print(myParallelMapping())
How should this time-limited parallel map be implemented?
You could put something like this inside your pmap body
pmap(runtimes) do runtime
t0 = time()
task = #async waitForTimeUnlessTimeout(runtime)
while !istaskdone(task) && time()-t0 < time_limit
sleep(1)
end
istaskdone(task) && (return fetch(task))
error("time over")
end
Also note that (runtime)->waitForTimeUnlessTimeout(runtime) is the same as just waitForTimeUnlessTimeout .
Following #Fredrik Bagge's very helpful answer, here is the full working example implementation with some extra explanation.
num_procs = 8
using Distributed
if num_procs > 1
addprocs(num_procs-1)
end
#everywhere import Random
#everywhere begin
function waitForTime(wait_time)
# This code block simulates a long computation.
# Pretend the computation time is unknown.
t0 = time()
x = 0
while time()-t0 < wait_time
x += Random.rand() - 0.5
yield() # CRITICAL to release computation to check if task is done.
# If you comment out #yield(), you will see timeout doesn't work!
end
return round(wait_time, digits=2)
end
end
function myParallelMapping(num_tasks = 16, max_runtime=2, time_limit=1)
# random task runtimes between 0 and max_runtime
runtimes = Random.rand(num_tasks) * max_runtime
# parallel compute the mapping tasks. See "do block" in
# the Julia documentation, it's just syntactic sugar.
return pmap(runtimes) do runtime
t0 = time()
task = #async waitForTime(runtime)
while !istaskdone(task) && time()-t0 < time_limit
# releases computation to waitForTime
sleep(0.1)
# nothing past here will run until waitForTime calls yield()
# *and* 0.1 seconds have passed.
end
# equal to if istaskdone(task); return fetch(task); end
istaskdone(task) && (return fetch(task))
return "TimeOut"
# `return error("TimeOut")` halts pmap unless pmap is
# given an error handler argument. See pmap documentation.
end
end
The output is
julia> print(myParallelMapping())
Any["TimeOut", "TimeOut", 0.33, 0.35, 0.56, 0.41, 0.08, 0.14, 0.72,
"TimeOut", "TimeOut", "TimeOut", 0.52, "TimeOut", 0.33, "TimeOut"]
Note that there are two tasks per process in this example. The original task (the "time checker") is checking every 0.1 seconds if the other task has completed computation. The other task (created with #async) is computing something, periodically calling yield() to release control to the time checker; if it doesn't call yield(), time checking cannot occur.
I'm making a smart home system using nodeMCU, and I need to store and retrieve data from the module. I used the following function.
function save_settings(name,value)
file.remove(name)
file.open(name,"w+")
file.writeline(value)
file.close()
end
It works but it's slow and the NodeMCU crashes if I trigger the above function rapidly... Sometimes requiring a FS format to be able to use it again.
So my question is: is there any other way to make variables persistent between restarts?
I'm using the latest firmware, 0.9.6-dev_20150704, the float version (https://github.com/nodemcu/nodemcu-firmware/releases)
This code took 62-63 ms to complete at first, and seems to add a few fractions of a millisecond with each successive run of the code, after a few hundred executions, it was up to almost 100 ms. It never crashed on me.
function save_setting(name, value)
file.open(name, 'w') -- you don't need to do file.remove if you use the 'w' method of writing
file.writeline(value)
file.close()
end
function read_setting(name)
if (file.open(name)~=nil) then
result = string.sub(file.readline(), 1, -2) -- to remove newline character
file.close()
return true, result
else
return false, nil
end
end
startTime = tmr.now()
test1 = 1200
test2 = 15.7
test3 = 75
test4 = 15000001
save_setting('test1', test1)
save_setting('test2', test2)
save_setting('test3', test3)
save_setting('test4', test4)
1exists, test1 = read_setting('test1')
2exists, test2 = read_setting('test2')
3exists, test3 = read_setting('test3')
4exists, test4 = read_setting('test4')
completeTime = (tmr.now()-startTime)/(1000)
print('time to complete (ms):')
print(tostring(completeTime))
If you upgrade to the newer version (based on SDK 1.4.0) you can use the rtcmem memory slots:
local offset = 10
local val = rtcmem.read32(offset, 1)
rtcmem.write32(offset, val + 1)
That memory is documented to persist through a deep sleep cycle; I've found it to persist across both hardware and software resets (but not cycling power.)
I've written a feature for my library Rubikon that displays a throbber (a spinning — as you may have seen in other console apps) as long as some other code is running.
To test this feature I capture the output of the throbber in a StringIO and compare it with the expected value. As the throbber is only displayed as long as the other code is running the content of the IO gets longer when the code runs longer. In my tests I do a simple sleep 1 and should have a constant 1 second delay. This works most of the time, but sometimes (apparently due to external factors like heavy load on the CPU) it fails, because the code doesn't run for 1 second, but for a bit more, so that the throbber prints a few additional characters.
My question is: Is there any possibility to test such time critical features in Ruby?
From your github repository, I found this test for the Throbber class:
should 'work correctly' do
ostream = StringIO.new
thread = Thread.new { sleep 1 }
throbber = Throbber.new(ostream, thread)
thread.join
throbber.join
assert_equal " \b-\b\\\b|\b/\b", ostream.string
end
I'll assume that a throbber iterates over ['-', '\', '|', '/'], backspacing before each write, once per second. Consider the following test:
should 'work correctly' do
ostream = StringIO.new
started_at = Time.now
ended_at = nil
thread = Thread.new { sleep 1; ended_at = Time.now }
throbber = Throbber.new(ostream, thread)
thread.join
throbber.join
duration = ended_at - started_at
iterated_chars = " -\\|/"
expected = ""
if duration >= 1
# After n seconds we should have n copies of " -\\|/", excluding \b for now
expected << iterated_chars * duration.to_i
end
# Next append the characters we'd get from working for fractions of a second:
remainder = duration - duration.to_i
expected << iterated_chars[0..((iterated_chars.length*remainder).to_i)] if remainder > 0.0
expected = expected.split('').join("\b") + "\b"
assert_equal expected, ostream.string
end
The last assignment of expected is a bit unpleasant, but I made the assumption that the throbber would write character/backspace pairs atomically. If this is not true, you should be able to insert the \b escape sequence into the iterated_chars string and remove the last assignment entirely.
This question is similar (I think, altough I'm not completely sure) to this one:
Only real time operating system can
give you such precision. You can
assume Thread.Sleep has a precision of
about 20 ms so you could, in theory
sleep until the desired time - the
actual time is about 20 ms and THEN
spin for 20 ms but you'll have to
waste those 20 ms. And even that
doesn't guarantee that you'll get real
time results, the scheduler might just
take your thread out just when it was
about to execute the RELEVANT part
(just after spinning)
The problem is not rubby (possibly, I'm no expert in ruby), the problem is the real time capabilities of your operating system.