so I am trying to delve into multiprocessing with python. I went to the python 3 website, to see some example code, and they have this:
from multiprocessing import Process
def f(name):
print('hello', name)
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
p.join()
I put it in my IDE and ran it, but nothing happens. If I run the debugger, it takes me to the process, and I see that everything happens, but just running it does nothing. Can someone help me?
Same code is working here :http://ideone.com/9kcQru
from multiprocessing import Process
def f(name):
print('hello', name)
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
p.join()
output : hello bob
There is something wrong with your environment
Related
Ive got a snipped of my actual python script written in this post. Basically I want to have a C programm and a Pyserial function executed in parallel (the C programm is for controlling a motor, the pySerial is for communicating with a arduino). My programm will be executed on a RPi3b using Spyder3 and Rasbipian.
What Ive already figured out from the sources below is that if you want to have a terminal program executed in python you should use the subprocess class. If you want to execute something in parallel the Process package from multiprocessing will do the job.
So Ive mixed them together and tried to archive my goals by using the code bleow. Unluckly without any success. The p1 process immediatly starts after the p1 process is called [ p1 = Process(target=run_c_file()) ] and the script stops until the C file has finished. Does anyone out there can help? Thank you very much!
BTW Im using python 3.5...
My sources:
https://docs.python.org/3.5/library/multiprocessing.html , https://docs.python.org/3.5/library/subprocess.html?highlight=subprocess
import serial_comm as ssf #My own function. Tested and working when single calling
import subprocess as sub
from multiprocessing import Process
def run_c_file():
sub.run("./C_File") #Call the C File in the same directory. Immeidatly starts when script is at line 14 -> p1 = Process(target=run_c_file())
def run_pyserial(ser_obj):
ssf.command(ser_obj,"Command") #Tell the arduino to do something fancy (tested and working)
ser_obj = ssf.connect()
p1 = Process(target=run_c_file())
p2 = Process(target=run_pyserial(ser_obj))
try:
p1.start()
p2.start()
p1.join() #Process one should start here (as far as I understood)
p2.join() #Process two should start here (as far as I understood)
'''The following part is still in progress'''
except KeyboardInterrupt:
print("Aborting")
p1.terminate()
p2.terminate()
try
p1 = Process(target=run_c_file)
p2 = Process(target=run_pyserial, args=(ser_obj,))
currently you are calling the function instead of passing it.
So I had a paragraph of code "train.py" that goes like
do something
print('log something...')
do something else
and I used pytorch's multiprocessing toolbox
import torch.multiprocessing as mp
to execute multithreads of "train.py", however, the print function works well under python3, but not python2. Why?
so in python 2 multiprocessing, new process creation is by default fork() method and forking a multithreaded process could be problematic.
Pytorch uses multithreaded process and to do it safely they use the below-mentioned functionality of python 3
import multiprocessing as mp
def foo(q):
q.put('hello')
if __name__ == '__main__':
mp.set_start_method('spawn')
Note that above snippet will only work with python 3. They use method "set_start_method" and ask python interpreter to start a new process by using 'spawn' rather than 'fork'
As I said the above method only works in python 3's multiprocessing module and because python 2's multiprocessing module does not have "set_start_method", your code
import torch.multiprocessing as mp
might not work as expected. It does not give you an error but the results of the computation are not reliable
Although there are more differences, one of the main differences is the use of the with statement. An easy way to understand the difference is to see how to use multiprocessing using with for both Python 2 and Python 3. If you add (in Python 2 or 3):
# For python 2/3 compatibility, define pool context manager
# to support the 'with' statement in Python 2
if sys.version_info[0] == 2:
from contextlib import contextmanager
#contextmanager
def multiprocessing_context(*args, **kwargs):
pool = multiprocessing.Pool(*args, **kwargs)
yield pool
pool.terminate()
else:
multiprocessing_context = multiprocessing.Pool
After that, you can use multiprocessing the regular Python 3 way, regardless of which version of Python you are using. For example:
def _function_to_run_for_each(x):
return x.lower()
with multiprocessing_context(processes=3) as pool:
results = pool.map(_function_to_run_for_each, ['Bob', 'Sue', 'Tim']) print(results)
will work in Python 2 or Python 3.
When I am trying to run my program based off a sample Tkinter GUI, nothing happens. I am still new to Pydev, but I find this rather unusual. I have a Main.py file containing the code, and I try to simply run the module without any success.
I simply copy/pasted from this reference,
# Main.py
import tkinter as tk;
class Application(tk.Frame):
def __init__(self, master=None):
super().__init__(master)
self.pack()
self.create_widgets()
def create_widgets(self):
self.hi_there = tk.Button(self)
self.hi_there["text"] = "Hello World\n(click me)"
self.hi_there["command"] = self.say_hi
self.hi_there.pack(side="top")
self.quit = tk.Button(self, text="QUIT", fg="red",
command=root.destroy)
self.quit.pack(side="bottom")
def say_hi(self):
print("hi there, everyone!")
root = tk.Tk()
app = Application(master=root)
app.mainloop()
The only result of running the module is an empty Liclipse console dedicated to Main.py.
I have also tried other examples from other sites with no luck. Also, I am currently on MacOS if it matters.
Are you sure you have everything properly (working directory, python path...) configured in Liclipse? I have just tried in a fresh install and, after configuring Liclipse to run the current project in Python 3.6 and after selecting the main project file as being this source code, it runs and shows a window with a button, as intended, and it also prints the text to console.
Also, it does not feel very "pythonic" to initialize a button that way. I would rather suggest like this:
# Main.py
import tkinter as tk;
class Application(tk.Frame):
def __init__(self, master=None):
super().__init__(master)
self.pack()
self.create_widgets()
def create_widgets(self):
mytext = "Hello World\n(click me)"
self.hi_there = tk.Button(self, text=mytext, command=self.say_hi)
self.hi_there.pack(side="top")
self.quit = tk.Button(self, text="QUIT", fg="red", command=root.destroy)
self.quit.pack(side="bottom")
def say_hi(self):
print("hi there, everyone!")
root = tk.Tk()
app = Application(master=root)
app.mainloop()
The code looks a little more readable and works in the same way. When you start growing your interface, it will be many lines long and by reducing two lines at each button, you can make it a lot shorter. I have been coding a tkinter app with more that 1500 lines of code and at that point I promised myself that I would try to learn how to keep it more organized and short ;)
import multiprocessing as mul
def f(x):
return x**2
pool = mul.Pool(5)
rel = pool.map(f,[1,2,3,4,5,6,7,8,9,10])
print(rel)
When I run the program above, the application is stuck in a loop and can't stop.
I am using python 3.5 in windows, is there something wrong?
This is what I see on my screen:
I am new to finance data analysis; and I am trying to find out a way to solve the big data problem with parallel computing.
Its not working because you are typing the commands in a shell; try saving the code in a file and running it directly.
Don't forget to copy the code correctly, you were missing a very important if statement (see the documentation).
Save this to a file, for example example.py on the desktop:
import multiprocessing as mul
def f(x):
return x**2
if __name__ == '__main__':
pool = mul.Pool(5)
rel = pool.map(f,[1,2,3,4,5,6,7,8,9,10])
print(rel)
Then, open a command prompt and type:
python %USERPROFILE%\Desktop\example.py
All code written and tested on python 3.4 windows 7.
I was designing a console app and had a need to use stdin from command-line (win os) to issue commands and to change the operating mode of the program. The program depends on multiprocessing to deal with cpu bound loads to spread to multiple processors.
I am using stdout to monitor that status and some basic return information and stdin to issue commands to load different sub-processes based on the returned console information.
This is where I found a problem. I could no get the multiprocessing module to accept stdin inputs but stdout was working just fine. I think found the following help on stack So I tested it and found that with the threading module this all works great, except for the fact that all output to stdout is paused until each time stdin is cycled due to GIL lock with stdin blocking.
I will say I have been successful with a work around implemented with msvcrt.kbhit(). However, I can't help but wonder if there is some sort of bug in the multiprocessing feature that is making stdin not read any data. I tried numerous ways and nothing worked when using multiprocessing. Even attempted to use Queues, but I did not try pools, or any other methods from multiprocessing.
I also did not try this on my linux machine since I was focusing on trying to get it to work.
Here is simplified test code that does not function as intended (reminder this was written in Python 3.4 - win7):
import sys
import time
from multiprocessing import Process
def function1():
while True:
print("Function 1")
time.sleep(1.33)
def function2():
while True:
print("Function 2")
c = sys.stdin.read(1) # Does not appear to be waiting for read before continuing loop.
sys.stdout.write(c) #nothing in 'c'
sys.stdout.write(".") #checking to see if it works at all.
print(str(c)) #trying something else, still nothing in 'c'
time.sleep(1.66)
if __name__ == "__main__":
p1 = Process(target=function1)
p2 = Process(target=function2)
p1.start()
p2.start()
Hopefully someone can shed light on whether this is intended functionality, if I didn't implement it correctly, or some other useful bit of information.
Thanks.
When you take a look at Pythons implementation of multiprocessing.Process._bootstrap() you will see this:
if sys.stdin is not None:
try:
sys.stdin.close()
sys.stdin = open(os.devnull)
except (OSError, ValueError):
pass
You can also confirm this by using:
>>> import sys
>>> import multiprocessing
>>> def func():
... print(sys.stdin)
...
>>> p = multiprocessing.Process(target=func)
>>> p.start()
>>> <_io.TextIOWrapper name='/dev/null' mode='r' encoding='UTF-8'>
And reading from os.devnull immediately returns empty result:
>>> import os
>>> f = open(os.devnull)
>>> f.read(1)
''
You can work this around by using open(0):
file is either a string or bytes object giving the pathname (absolute or relative to the current working directory) of the file to be opened or an integer file descriptor of the file to be wrapped. (If a file descriptor is given, it is closed when the returned I/O object is closed, unless closefd is set to False.)
And "0 file descriptor":
File descriptors are small integers corresponding to a file that has been opened by the current process. For example, standard input is usually file descriptor 0, standard output is 1, and standard error is 2:
>>> def func():
... sys.stdin = open(0)
... print(sys.stdin)
... c = sys.stdin.read(1)
... print('Got', c)
...
>>> multiprocessing.Process(target=func).start()
>>> <_io.TextIOWrapper name=0 mode='r' encoding='UTF-8'>
Got a