I'm testing with a minimal example how futures work when using ProcessPoolExecutor.
First, I want to know the result of my processing functions, then I would like to add complexity to the scenario.
import time
import string
import random
import traceback
import concurrent.futures
from concurrent.futures import ProcessPoolExecutor
def process(*args):
was_ok = True
try:
name = args[0][0]
tiempo = args[0][1]
print(f"Task {name} - Sleeping {tiempo}s")
time.sleep(tiempo)
except:
was_ok = False
return name, was_ok
def program():
amount = 10
workers = 2
data = [''.join(random.choices(string.ascii_letters, k=5)) for _ in range(amount)]
print(f"Data: {len(data)}")
tiempo = [random.randint(5, 15) for _ in range(amount)]
print(f"Times: {len(tiempo)}")
with ProcessPoolExecutor(max_workers=workers) as pool:
try:
index = 0
futures = [pool.submit(process, zipped) for zipped in zip(data, tiempo)]
for future in concurrent.futures.as_completed(futures):
name, ok = future.result()
print(f"Task {index} with code {name} finished: {ok}")
index += 1
except Exception as e:
print(f'Future failed: {e}')
if __name__ == "__main__":
program()
If I run this program, the output is as expected, obtaining all the future results. However, just at the end I also get a failure:
Data: 10
Times: 10
Task utebu - Sleeping 14s
Task klEVG - Sleeping 10s
Task ZAHIC - Sleeping 8s
Task 0 with code klEVG finished: True
Task RBEgG - Sleeping 9s
Task 1 with code utebu finished: True
Task VYCjw - Sleeping 14s
Task 2 with code ZAHIC finished: True
Task GDZmI - Sleeping 9s
Task 3 with code RBEgG finished: True
Task TPJKM - Sleeping 10s
Task 4 with code GDZmI finished: True
Task CggXZ - Sleeping 7s
Task 5 with code VYCjw finished: True
Task TUGJm - Sleeping 12s
Task 6 with code CggXZ finished: True
Task THlhj - Sleeping 11s
Task 7 with code TPJKM finished: True
Task 8 with code TUGJm finished: True
Task 9 with code THlhj finished: True
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/concurrent/futures/process.py", line 101, in _python_exit
thread_wakeup.wakeup()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/concurrent/futures/process.py", line 89, in wakeup
self._writer.send_bytes(b"")
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 183, in send_bytes
self._check_closed()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 136, in _check_closed
raise OSError("handle is closed")
OSError: handle is closed
AFAIK the code doesn't have an error itself. I've been researching really old questions like the following, without luck to find a fix for it:
this one related to the same error msg but for Python 2,
this issue in GitHub which seems to be exactly the same I'm having (and seems not fixed yet...?),
this other issue where the comment I linked to seems to point to the actual problem, but doesn't find a solution to it,
of course the official docs for Python 3.7,
...
And so forth. However still unlucky to find how to solve this behaviour. Even found a old question here in SO who pointed to avoid using as_completed function and using submit instead (and from there came this actual test, before I just had a map to my process function).
Any idea, fix, explanation or workaround are welcome. Thanks!
Related
I would like to run the pytest for all the items in the for loop. The pytest should fail in the end but it should run all the elements in the for loop.
The code looks like this
#pytest.fixture
def library():
return Library( spec_dir = service_spec_dir)
#pytest.fixture
def services(library):
return list(library.service_map.keys())
def test_properties(service, services):
for service_name in services:
model = library.models[service_name]
proxy = library.get_service(service_name)
if len(model.properties ) != 0 :
for prop in model.properties:
try:
method = getattr(proxy, f'get_{prop.name}')
method()
except exception as ex:
pytest.fail(ex)
The above code fails if one property of one service fails. I am wondering if there is a way to to run the test for all the service and get a list of failed cases for all the services.
I tried parametrize But based on this stackoverflow discussion. The parameter list should be resolved during the collection phase and in our case the library is loaded during the execution phase. Hence I am also not sure if it can be parametrized.
The goal is run all the services and its properties and get the list of failed items in the end.
I moved the variables to the global scope. I can parametrize the test now\
library = Library( spec_dir = service_spec_dir)
service_names = list(library.service_map.keys())
#pytest .mark.paramertize("serivce_name", service_names)
def test_properties(service):
pass
Don't use pytest.fail, but pytest_check.check instead.
fail point, is that you stop test execution on condition, while check made for collect how much cases were failed.
import logging
import pytest
import pytest_check as check
def test_000():
li = [1, 2, 3, 4, 5, 6]
for i in li:
logging.info(f"Test still running. i = {i}")
if i % 2 > 0:
check.is_true(False, msg=f"value of i is odd: {i}")
Output:
tests/main_test.py::test_000
-------------------------------- live log call --------------------------------
11:00:05 INFO Test still running. i = 1
11:00:05 INFO Test still running. i = 2
11:00:05 INFO Test still running. i = 3
11:00:05 INFO Test still running. i = 4
11:00:05 INFO Test still running. i = 5
11:00:05 INFO Test still running. i = 6
FAILED [100%]
================================== FAILURES ===================================
__________________________________ test_000 ___________________________________
FAILURE: value of i is odd: 1
assert False
FAILURE: value of i is odd: 3
assert False
FAILURE: value of i is odd: 5
assert False
This is a code snippet to run KMeans using GPU.
Documentation-link:https://pycave.borchero.com/sites/generated/clustering/kmeans/pycave.clustering.KMeans.html
import torch
from pycave.clustering import KMeans
X = torch.cat([
torch.randn(1000, 6) - 5,
torch.randn(1000, 6),
torch.randn(1000, 6) + 5,
])
estimator = KMeans(num_clusters = 3, trainer_params=dict(gpus=1,
enable_progress_bar=0,
max_epochs=100,))
labels = estimator.fit_predict(X).numpy()
pd.value_counts(labels)
The issue is with how to disable the console output from the estimator.
Current Output:
Running initialization...
{'batch_size': 3000, 'collate_fn': <function collate_tensor at 0x000002BE21221700>}
Fitting K-Means...
{'batch_size': 3000, 'collate_fn': <function collate_tensor at 0x000002BE21221700>}
{'batch_size': 1, 'sampler': None, 'batch_sampler': <pytorch_lightning.overrides.distributed.IndexBatchSamplerWrapper object at 0x000002BE593A55B0>, 'collate_fn': <function collate_tensor at 0x000002BE21221700>, 'shuffle': False, 'drop_last': False}
0 1000
2 1000
1 1000
dtype: int64
Expected Output:
0 1000
2 1000
1 1000
dtype: int64
Info regarding trainer_params parameter
(Optional[Dict[str, Any]]) --
Initialization parameters to use when initializing a PyTorch Lightning trainer. By default, it disables various stdout logs unless PyCave is configured to do verbose logging. Checkpointing and logging are disabled regardless of the log level.
The dictionaries that are printed should never be there, that's a bug in a dependency. Resolved in the latest build.
As far as the PyCave logs are concerned (Running initialization... and Fitting K-Means...), you can turn them off easily by adding the following:
import logging
from pycave import set_logging_level
set_logging_level(logging.WARNING)
Note that set_logging_level(logging.WARNING) also turns off the progress bar and the model summary automatically so you don't have to set these flags explicitly.
While using time based rotating file handler.Getting error
os.rename('logthred.log', dfn)
WindowsError: [Error 32] The process cannot access the file because it
is being used by another process
config :
[loggers]
keys=root
[logger_root]
level=INFO
handlers=timedRotatingFileHandler
[formatters]
keys=timedRotatingFormatter
[formatter_timedRotatingFormatter]
format = %(asctime)s %(levelname)s %(name)s.%(functionname)s:%(lineno)d %
(output)s
datefmt=%y-%m-%d %H:%M:%S
[handlers]
keys=timedRotatingFileHandler
[handler_timedRotatingFileHandler]
class=handlers.TimedRotatingFileHandler
level=INFO
formatter=timedRotatingFormatter
args=('D:\\log.out', 'M', 2, 0, None, False, False)
Want to achieve time based rotating file handler and multiple process can write same log file.In python,I didn't find any thing which can help to resolve this issue.
I have read discussion on this issue (python issues).
Any suggestion which can resolve this issue.
Found the solution : Problem in Python 2.7 whenever we create child Process then file handles of parent process also inherited by child process so this is causing error. We can block this inheritance using this code.
import sys
from ctypes import windll
import msvcrt
import __builtin__
if sys.platform == 'win32':
__builtin__open = __builtin__.open
def __open_inheritance_hack(*args, **kwargs):
result = __builtin__open(*args, **kwargs)
handle = msvcrt.get_osfhandle(result.fileno())
if filename in args: # which filename handle you don't want to give to child process.
windll.kernel32.SetHandleInformation(handle, 1, 0)
return result
__builtin__.open = __open_inheritance_hack
I'm trying to call lapply within a function applied on spark data frame. According to documentation it's possible since Spark 2.0.
wrapper = function(df){
out = df
out$len <- unlist(lapply(df$value, function(y) length(y)))
return(out)
}
# dd is Spark Data Frame with one column (value) of type raw
dapplyCollect(dd, wrapper)
It returns error:
Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...): org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 37.0 failed 1 times, most recent failure: Lost task 0.0 in stage 37.0 (TID 37, localhost): org.apache.spark.SparkException: R computation failed with
Error in (function (..., deparse.level = 1, make.row.names = TRUE) :
incompatible types (from raw to logical) in subassignment type fix
The following works fine:
wrapper(collect(dd))
But we want computation to run on nodes (not on driver).
What could be the problem? There is a related question but it does not help.
Thanks.
You need to add the schema as it can only be defaulted if the columns of the output are the same mode as the input.
I came across the Timeout module in Ruby, and wanted to test it out. I looked at their official source code at http://ruby-doc.org/stdlib-2.1.1/libdoc/timeout/rdoc/Timeout.html
Here is the code I had
require 'timeout'
require 'benchmark'
numbers = [*1..80]
Timeout::timeout(5) { numbers.combination(5).count }
=> 24040016
I did some benchmarking tests, and got the following.
10.828000 0.063000 10.891000 11.001676
According to the documentation, this method is supposed to return an exception if the block is not executed within 5 seconds. If it is executed within the time frame, it will return the result of the code block
For what it's worth, I've tried timeout with 1 second, instead of 5 seconds, and I still get returned the result of the code block.
Here is the official documentation
timeout(sec, klass=nil)
Performs an operation in a block, raising an error if it takes longer than sec seconds to complete.
sec: Number of seconds to wait for the block to terminate. Any number may be used,
including Floats to specify fractional seconds. A value of 0 or nil will execute the
block without any timeout.
klass: Exception Class to raise if the block fails to terminate in sec seconds. Omitting
will use the default, Timeout::Error
I am mystified as to why this doesn't work.
The problem is the way MRI (Matz's Ruby Implementation) thread scheduling works. MRI uses a GIL (Global Interpreter Lock), which in practice means only one thread is truly running at a time.
There are some exception, but for the majority of the time there is only one thread executing Ruby code at any one time.
Normally you do not notice this, even during heavy computations that consume 100% CPU, because the MRI keeps time-slicing the threads at regular intervals so that each thread gets a turn to run.
However there's one exception where time-slicing isn't active and that's when a Ruby thread is executing native C-code instead of Ruby code.
Now it so happens that Array#combination is implemented in pure C:
[1] pry(main)> show-source Array#combination
From: array.c (C Method):
static VALUE
rb_ary_combination(VALUE ary, VALUE num)
{
...
}
When we combine this knowledge with how Timeout.timeout is implemented we can start to get a clue of what is happening:
[7] pry(main)> show-source Timeout#timeout
From: /opt/ruby21/lib/ruby/2.1.0/timeout.rb # line 75:
75: def timeout(sec, klass = nil) #:yield: +sec+
76: return yield(sec) if sec == nil or sec.zero?
77: message = "execution expired"
78: e = Error
79: bl = proc do |exception|
80: begin
81: x = Thread.current
82: y = Thread.start {
83: begin
84: sleep sec
85: rescue => e
86: x.raise e
87: else
88: x.raise exception, message
89: end
90: }
91: return yield(sec)
92: ensure
93: if y
94: y.kill
95: y.join # make sure y is dead.
96: end
97: end
98: end
99: ...
1xx: end
Your code running Array.combination most likely actually starts executing even BEFORE the timeout thread runs sleep sec on line 84. Your code is launched on line 91 through yield(sec).
This means the order of execution actually becomes:
1: [thread 1] numbers.combination(5).count
# ...some time passes while the combinations are calculated ...
2: [thread 2] sleep 5 # <- The timeout thread starts running sleep
3: [thread 1] y.kill # <- The timeout thread is instantly killed
# and never times out.
In order to make sure the timeout thread starts first you can try this, which will most likely trigger the timeout exception this time:
Timeout::timeout(5) { Thread.pass; numbers.combination(5).count }
This is because by running Thread.pass you allow the MRI scheduler to start and run the code on line 82 before the native combination C-code executes. However even in this case the exception won't be triggered until combination exits because of the GIL.
There is no way around this unfortunately. You would have to use something like JRuby instead, which has real concurrent threads. Or you could run the combination calculation in a Process instead of a thread.