I know I can make IPython parallel use dill instead of pickle with a direct view using
c = Client()
dv = c[:]
dv.use_dill()
But how can I enable it with dv.load_balanced_view()?
I think you can call use_dill on a direct view and then use a load-balanced view:
c = Client()
c.direct_view().use_dill()
v = c.load_balanced_view()
...
From the DirectView.use_dill documentation:
This calls IPython.utils.pickleutil.use_dill() here and on each
engine.
Related
I am using an image processing code in python opencv. Since that process is taking a lot of time to process say 30 images. I tried to process these image parallel using Multiprocessing. The multiprocessing part is working good in CPU but I want to use that multiprocessing thing in GPU(cuda).
I use torch.multiprocessing for running task in parallel. So I am using torch.device('cuda') for our class to run whole thing in to this perticular device. When I run the code it's showing device using "cuda" but not using any GPU processing.
import cv2
import numpy as np
import torch
import torch.nn as nn
from torch.multiprocessing import Process, Pool, Manager, set_start_method
import sys
import os
class RoadShoulderWidth(nn.Module):
def __init__(self):
super(RoadShoulderWidth, self).__init__()
pass
// Want to run below method in parallel for 30 images.
#staticmethod
def get_dim(image, road_shoulder_width_list):
..... code
def get_road_shoulder_width(self, _root_dir, _img_path_list):
manager = Manager()
road_shoulder_width_list = manager.list()
processes = []
for img_path in img_path_list[:30]:
img = cv2.imread(_root_dir + '/' + img_path)
img = img[72 * 5:72 * 6, 0:1280]
# Do work
p = Process(target=self.get_dim,args=(img,road_shoulder_width_list))
p.start()
processes.append(p)
for p in processes:
p.join()
return road_shoulder_width_list
Use below set of code to run your class
if __name__ == '__main__':
root_dir = '/home/nikhil_m/r'
img_path_list = os.listdir(root_dir)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('Using device:', device)
dataloader_kwargs = {'pin_memory': True}
set_start_method('fork')
obj = RoadShoulderWidth().to(device)
val = obj.get_road_shoulder_width(str(root_dir), img_path_list)
print(val)
print(torch.cuda.is_available())
Can anybody suggest me how to fix this?
Your class RoadShoulderWidth is a nn.Module subclass which lets you use .to(device). This only means that all other nn.Module objects or nn.Parameters that are members of your RoadShoulderWidth object are moved to the device. As from your example, there are none, so nothing happens.
In general PyTorch does not move code to GPU but data. If all data of a pytorch operation are on the GPU (e.g. a + b, a and b are on GPU) then the operation is executed on the GPU. You can move the data with a.to(device), given a is a torch.Tensor object.
PyTorch can only execute its own operations on GPU. It's not able to execute OpenCV code on GPU.
I am currently trying to implement some functions using sparkR version 1.5.1. I have seen older (version 1.3) examples, where people used the apply function on DataFrames, but it looks like this is no longer directly available. Example:
x = c(1,2)
xDF_R = data.frame(x)
colnames(xDF_R) = c("number")
xDF_S = createDataFrame(sqlContext,xDF_R)
Now, I can use the function sapply on the data.frame object
xDF_R$result = sapply(xDF_R$number, ppois, q=10)
When I use a similar logic on the DataFrame
xDF_S$result = sapply(xDF_S$number, ppois, q=10)
I get the error message "Error in as.list.default(X) :
no method for coercing this S4 class to a vector"
Can I somehow do this?
This is possible with user defined functions in Spark 2.0.
wrapper = function(df){
+ out = df
+ out$result = sapply(df$number, ppois, q=10)
+ return(out)
+ }
> xDF_S2 = dapplyCollect(xDF_S, wrapper)
> identical(xDF_S2, xDF_R)
[1] TRUE
Note you need a wrapper function like this because you can't pass the extra arguments in directly, but that may change in the future.
The native R functions do not support Spark DataFrames. We can use user defined functions in SparkR to execute native R modules. These are executed on the executors and thus the libraries must be available on all the executors.
For example, suppose we have a custom function holt_forecast which takes in a data.table as an argument.
Sample R code
sales_R_df %>%
group_by(product_id) %>%
do(holt_forecast(data.table(.))) %>%
data.table(.) -> dt_holt
For using UDFs, we need to specify the schema of the output data.frame returned by the execution of the native R method. This schema is used by Spark to generate back the Spark DataFrame.
Equivalent SparkR code
Define the schema
structField("product_id", "integer"),
structField("audit_date", "date"),
structField("holt_unit_forecast", "double"),
structField("holt_unit_forecast_std", "double")
)
Execute the method
library(data.table)
library(lubridate)
library(dplyr)
library(forecast)
sales <- data.table(x)
y <- data.frame(key,holt_forecast(sales))
}, dt_holt_schema)
Reference: https://shbhmrzd.medium.com/stl-and-holt-from-r-to-sparkr-1815bacfe1cc
I have the basic PBS Controller and EngineSet Launchers working with my PBS/torque cluster. But now I'd like to pass in additional options through the config file and/or the command line to be rendered in the batch_template file using the {} template formatter syntax.
As an example, I want to run:
ipcluster start -n 10 --pmem=10gb
And then have, in my template:
#PBS -l pmem={pmem}
I guess that I have to subclass the PBSControllerLauncher & PBSEngineSetLauncher classes? This is mentioned in the documentation, but I can't find a clear example of how to do it.
Simply inheriting from these classes and adding pmem = Unicode(...) as a class attribute doesn't make pmem available in the template's context when it calls write_batch_script.
I can access the config options and insert it into the context, as shown below. This code works, but is pretty awkward and doesn't seem like the right way to do it:
from IPython.parallel.apps.launcher import PBSControllerLauncher, PBSEngineSetLauncher, BatchClusterAppMixin, ipcontroller_cmd_argv, ipengine_cmd_argv
from IPython.utils.traitlets import (
Any, Integer, CFloat, List, Unicode, Dict, Instance, HasTraits, CRegExp
)
import pipes
class MyPBSLauncherExtension(object):
def update_context(self):
cfg = self.config
ctx = self.context
cls = type(self).__name__
for c in ('extra_pbs_directives', 'extra_modules'):
if cls in cfg:
if c in cfg[cls]:
ctx[c] = cfg[cls][c]
continue
if c in cfg:
ctx[c] = cfg[c]
else:
ctx[c] = getattr(self, c)
class MyPBSControllerLauncher(PBSControllerLauncher, MyPBSLauncherExtension):
extra_pbs_directives = Unicode(u'', config=True, help='')
extra_modules = Unicode(u'', config=True, help='')
def write_batch_script(self, n):
self.update_context()
super(MyPBSControllerLauncher, self).write_batch_script(n)
class MyPBSEngineSetLauncher(PBSEngineSetLauncher, MyPBSLauncherExtension):
extra_pbs_directives = Unicode(u'', config=True, help='')
extra_modules = Unicode(u'', config=True, help='')
def write_batch_script(self, n):
self.update_context()
super(MyPBSEngineSetLauncher, self).write_batch_script(n)
Would very much appreciate a simple example that adds an additional option to a launcher subclass.
The following code fetches parameter from request and respond from couchbase db as per the value of the parameter.
couchbase = Couchbase("ubuntumartini03:8091", "thebucket", "")
bucket = couchbase["thebucket"]
class MH(tornado.web.RequestHandler):
def get(self):
key = self.get_argument("pub_id", strip=True)
result = json.loads(bucket.get(key)[2])
self.write(result['metaTag'])
if __name__=="__main__":
app = tornado.web.Application(handlers=[(r"/", MH)])
app.listen(8888,"")
tornado.ioloop.IOLoop.instance().start()
Problem: For the given hardware, we can make 10k/sec calls to Couchbase from Tornado machine. But when we are making a call from client to Tornado machine, we are only able to make 350 calls/sec.
Surely the bottleneck here is Tornado. How to optimize it to be able to make atleast 7k calls/sec?
Edit your code like this :
from tornado.ioloop import IOLoop
couchbase = Couchbase("ubuntumartini03:8091", "thebucket", "")
bucket = couchbase["thebucket"]
class MH(tornado.web.RequestHandler):
async def get(self):
key = self.get_argument("pub_id", strip=True)
result = await IOLoop.current().run_in_executor(None,bucket.get,*(key))
self.write(result[2]['metaTag'])
if __name__=="__main__":
app = tornado.web.Application(handlers=[(r"/", MH)])
app.listen(8888,"")
tornado.ioloop.IOLoop.instance().start()
What client do you use, is it a synchronous or an ansynchronous client? If this is a synchronous client, it can't not make full use of the tornado ioloop reactor.
The SWT-Gui looks very nice.
Is there an easy way to use it in Jython ?
Given that you can use all Java classes from within Jython, it is also possible to use SWT.
For the example, adapted from an SWT snippet, make sure you have the SWT jar on your CLASSPATH:
import org.eclipse.swt as swt
import org.eclipse.swt.widgets as widgets
import org.eclipse.swt.layout as layout
result = None
display = widgets.Display()
shell = widgets.Shell(display)
shell.pack()
shell.open()
dialog = widgets.Shell(shell, swt.SWT.DIALOG_TRIM | swt.SWT.APPLICATION_MODAL)
dialog.setLayout(layout.RowLayout())
ok = widgets.Button(dialog, swt.SWT.PUSH)
ok.setText ("OK")
cancel = widgets.Button(dialog, swt.SWT.PUSH);
cancel.setText("Cancel");
class MyListener(widgets.Listener):
def handleEvent(self, event):
global result
result = event.widget == ok
dialog.close()
listener = MyListener()
ok.addListener(swt.SWT.Selection, listener)
cancel.addListener(swt.SWT.Selection, listener)
dialog.pack()
dialog.open()
while not dialog.isDisposed():
if not display.readAndDispatch():
display.sleep ()
print "Result:", result
display.dispose()
Jython has a few other niceties that makes the code cleaner.
Jython automagically translates getters & setters into public properties so that
ok.setText ("OK")
becomes simply
ok.text = 'OK'
You can then supply them as named arguments to the constructor. Jython also handles creating listener objects for your event handlers:
def handleEvent(self, event):
global result
result = event.widget == ok
dialog.close()
ok = widgets.Button(dialog, swt.SWT.PUSH
text='OK',
widgetSelected=handleEvent)
cancel = widgets.Button(dialog, swt.SWT.PUSH
text='Cancel',
widgetSelected=handleEvent)
Actually, there is no need for a special module. This talk by Sean McGrath contains a simple example of a Jython/SWT GUI.
Slide 11 of the talk begins with:
"""
Simple SWT Example
Sean McGrath
"""
from org.eclipse.swt.events import *
from org.eclipse.swt.graphics import *
from org.eclipse.swt.layout import *
from org.eclipse.swt.widgets import *
from org.eclipse.swt.layout.GridData import *
from org.eclipse.swt import *
It shows that SWT is directly usable from Jython.
The full example is right there at Sean's site.