Torch multi-processing on single GPU is extremely slow - multiprocessing

I have a torch code in which I am trying to calculate the output of 10 layers that are independent of each other. It seems like a case where torch multi-processing fits perfectly. Here is the code for that replicates the situation in my case:
from torch.multiprocessing import Pool, set_start_method
import torch
from torch.autograd import Variable
from kernel_2_0 import *
from kernel_2_1 import *
from kernel_2_2 import *
from kernel_2_3 import *
from kernel_2_4 import *
from kernel_2_5 import *
from kernel_2_6 import *
from kernel_2_7 import *
from kernel_2_8 import *
from kernel_2_9 import *
def apply_fn(args):
f = args[0]
x = args[1]
return f(x)
def forward_2(x):
assert isinstance(x.data, torch.cuda.BoolTensor)
try:
set_start_method('spawn', force=True)
except RuntimeError:
print('Could not set spawn method')
with Pool(12) as pool:
mapped_out = pool.map(apply_fn,
[(kernel_2_0, x), (kernel_2_1, x), (kernel_2_2, x), (kernel_2_3, x), (kernel_2_4, x),
(kernel_2_5, x), (kernel_2_6, x), (kernel_2_7, x), (kernel_2_8, x), (kernel_2_9, x)])
out = Variable(torch.cuda.BoolTensor(x.size()[0], 10))
out[:, 0] = mapped_out[0]
out[:, 1] = mapped_out[1]
out[:, 2] = mapped_out[2]
out[:, 3] = mapped_out[3]
out[:, 4] = mapped_out[4]
out[:, 5] = mapped_out[5]
out[:, 6] = mapped_out[6]
out[:, 7] = mapped_out[7]
out[:, 8] = mapped_out[8]
out[:, 9] = mapped_out[9]
return out
However, when the code runs it is almost 100 times slower than the version where I do not do multi processing an instead call all the kernels in a sequential manner. FYI I am using torch==1.4.0 and cuda-10.0. I am not able to understand why this happens. Can someone please help?
Thanks in advance!

Related

How can I plot different types of seaborn plots on different x ticks?

I want to have multiple types of seaborn plots using the same y axis but with different x coordinates (see image below).
I've tried doing this multiple different ways with specifying the X-axis coordinates differently but can't seem to get it to work.
Here is an example of almost working code
x=[1,2,3,3,3,4,4,5,5,6] # first violin
y=[4,4,5,5,5,5,6] # second violin
z=[5,5,6] # swarmplot over second violin
for data,label in [(x,'x'),(y,'y'),(z,'z')]:
for i in data:
c2v['value'].append(i)
c2v['key'].append(label)
data=pd.DataFrame(c2v)
data.head()
print(data.loc[data.key=='z'])
fig,ax=plt.subplots(1,figsize=(5,5),dpi=200)
ax = sns.violinplot(data=data.loc[data.key.isin(['x','y'])], x='key', y='value',palette=['honeydew','lightgreen'])
sns.swarmplot(x=['swarmplot']*len(data), y=data['value'], order=ax.get_xticklabels() + ['swarmplot'], ax=ax) #.loc[data.key=='z',:]
ax.set_xlabel('')
It produces the following image:
However, it is plotting all values associated with x/y/z instead of just z. When I slice the dataframe to only 'z' in the swarmplot as below, I get an error:
sns.swarmplot(x=['swarmplot']*len(data), y=data.loc[data.key=='z',:]['value'], order=ax.get_xticklabels() + ['swarmplot'], ax=ax)
KeyError: 'swarmplot'
Any suggestions?
To draw a second plot onto the same x-axis, you can use order= giving a list of existing tick labels, appending the new labels.
Here is an example:
import seaborn as sns
tips = sns.load_dataset('tips')
ax = sns.swarmplot(data=tips, x='day', y='total_bill')
sns.violinplot(x=['violin']*len(tips), y=tips['total_bill'], order=ax.get_xticklabels() + ['violin'], ax=ax)
ax.set_xlabel('')
The problem with the code in the new question, is that the x= and y= of the swarmplot need the same number of elements. It also seems the swarmplot resets the y limits, so I added some code to readjust those:
from matplotlib import pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
x = [1, 2, 3, 3, 3, 4, 4, 5, 5, 6] # first violin
y = [4, 4, 5, 5, 5, 5, 6] # second violin
z = [5, 5, 6] # swarmplot over second violin
data = pd.DataFrame({'value': np.concatenate([x, y, z]),
'key': ['x'] * len(x) + ['y'] * len(y) + ['z'] * len(z)})
fig, ax = plt.subplots(1, figsize=(5, 5))
ax = sns.violinplot(data=data.loc[data.key.isin(['x', 'y'])], x='key', y='value', palette=['honeydew', 'lightgreen'])
ymin1, ymax1 = ax.get_ylim()
swarm_data = data.loc[data.key == 'z', :]['value']
sns.swarmplot(x=['swarmplot'] * len(swarm_data), y=swarm_data, order=ax.get_xticklabels() + ['swarmplot'], ax=ax)
ymin2, ymax2 = ax.get_ylim()
ax.set_ylim(min(ymin1, ymin2), max(ymax1, ymax2))
ax.set_xlabel('')
ax.set_xticks(np.arange(3))
ax.set_xticklabels(['x', 'y', 'swarmplot'])
plt.show()
You can simplify things by directly using the data without creating a dataframe:
x = [1, 2, 3, 3, 3, 4, 4, 5, 5, 6] # first violin
y = [4, 4, 5, 5, 5, 5, 6] # second violin
z = [5, 5, 6] # swarmplot over second violin
fig, ax = plt.subplots(1, figsize=(5, 5))
ax = sns.violinplot(x=['x']*len(x) + ['y']*len(y), y=x + y, palette=['honeydew', 'lightgreen'])
ymin1, ymax1 = ax.get_ylim()
sns.swarmplot(x=['swarmplot'] * len(z), y=z, order=ax.get_xticklabels() + ['swarmplot'], ax=ax)
ymin2, ymax2 = ax.get_ylim()
ax.set_ylim(min(ymin1, ymin2), max(ymax1, ymax2))
ax.set_xticks(np.arange(3))
ax.set_xticklabels(['x', 'y', 'swarmplot'])
plt.show()

Why does this simple LightGBM binary classifier perform poorly?

I tried to train a LightGBM binary classifier using the Python API the relation -
if feature > 5, then 1 else 0
import pandas as pd
import numpy as np
import lightgbm as lgb
x_train = pd.DataFrame([4, 7, 2, 6, 3, 1, 9])
y_train = pd.DataFrame([0, 1, 0, 1, 0, 0, 1])
x_test = pd.DataFrame([8, 2])
y_test = pd.DataFrame([1, 0])
lgb_train = lgb.Dataset(x_train, y_train)
lgb_eval = lgb.Dataset(x_test, y_test, reference=lgb_train)
params = { 'objective': 'binary', 'metric': {'binary_logloss', 'auc'}}
gbm = lgb.train(params, lgb_train, valid_sets=lgb_eval)
y_pred = gbm.predict(x_test, num_iteration=gbm.best_iteration)
y_pred
array([0.42857143, 0.42857143])
np.where((y_pred > 0.5), 1, 0)
array([0, 0])
Clearly it failed to predict the first test 8. Can anyone see what went wrong?
LightGBM's parameter defaults are set with the expectation of moderate-sized training data, and might not work well on extremely small datasets like the one in this question.
There are two in particular that are impacting your result:
min_data_in_leaf: minimum number of samples that must fall into a leaf node
min_sum_hessian_in_leaf: basically, the minimum contribution to the loss function for one leaf node
Setting these to the lowest possible values can force LightGBM to overfit to such a small dataset.
import pandas as pd
import numpy as np
import lightgbm as lgb
x_train = pd.DataFrame([4, 7, 2, 6, 3, 1, 9])
y_train = pd.DataFrame([0, 1, 0, 1, 0, 0, 1])
x_test = pd.DataFrame([8, 2])
y_test = pd.DataFrame([1, 0])
lgb_train = lgb.Dataset(x_train, y_train)
lgb_eval = lgb.Dataset(x_test, y_test, reference=lgb_train)
params = {
'objective': 'binary',
'metric': {'binary_logloss', 'auc'},
'min_data_in_leaf': 1,
'min_sum_hessian_in_leaf': 0
}
gbm = lgb.train(params, lgb_train, valid_sets=lgb_eval)
y_pred = gbm.predict(x_test, num_iteration=gbm.best_iteration)
y_pred
# array([6.66660313e-01, 1.89048958e-05])
np.where((y_pred > 0.5), 1, 0)
# array([1, 0])
For details on all the parameters and their defaults, see https://lightgbm.readthedocs.io/en/latest/Parameters.html.

Why Indexing error returns in python GEKKO

I want to transform LINGO code to Python GEKKO code. Here is Lingo code, lingo results and gekko codes. I cant write second and third constraints. It returns indexing error but, I dont understand why? Can someone help? (It's a graph coloring problem)
from gekko import GEKKO
import numpy as np
m = GEKKO(remote=False)
# x = m.Array(m.Var,(7,5),lb=0,ub=1,integer=True)
x = m.Array(m.Var,(6,6),lb=0,ub=1,integer=True)
y= np.array([1, 2, 3, 4, 5, 6])
country=6
arcs=np.array([[1,3],
[5,4],
[3,6],
[2,4],
[2,5],
[2,6],
[4,5],
[4,6]])
for i in range(6):
m.Minimize(y)
for i in range(6):
# for j in range(2):
# m.Equation(m.sum(x[i,j])==1)
m.Equation(m.sum(x[i,:])==1)
for k in range (6):
for i in range(8):
m.Equation(x[arcs[i,1],k]+x[arcs[i,2],k]<=1)
# m.Equation(x[arcs[i,1],k]+x[arcs[i,2],k])<=1)
# m.Equation(m.sum(x[arcs[i,1],k],x[arcs[i,2],k]))<=1)
Revised version is=
from gekko import GEKKO
import numpy as np
m = GEKKO(remote=False)
x = m.Array(m.Var,(6,6),lb=0,ub=1,integer=True)
y = m.Array(m.Var,6,lb=0,ub=1,integer=True)
y= np.array([1, 2, 3, 4, 5, 6])
country=6
arcs=np.array([[1,3],
[1,4],
[3,4],
[3,4],
[4,5],
[2,6],
[4,5],
[4,6]])
for i in range(6):
m.Minimize(y[i])
for i in range(6):
m.Equation(m.sum(x[i,:])==1)
for k in range (6):
for i in range(8):
m.Equation(x[arcs[i,0]-1,k-1]+x[arcs[i,1]-1,k-1]<=1)
for i in range(6):
m.Equation(m.sum(x[i,:]<=y[i])
m.options.solver = 1
m.solve()
print('Objective Function: ' + str(m.options.objfcnval))
print(x)
print(y)
now it gives invalid syntax error for m.solve and m options?
A couple things that you need to consider for Python:
Lists and arrays are zero-index so you need to shift them by -1 relative to the LINDO / LINGO language.
The objective function y is a list of constants. Gekko generates warnings that there are no variables in that expression.
Here is a corrected version of your Python script that you probably need to supplement with a correct objective statement and any additional equations that are needed.
from gekko import GEKKO
import numpy as np
m = GEKKO(remote=False)
x = m.Array(m.Var,(6,6),lb=0,ub=1,integer=True)
y= np.array([1, 2, 3, 4, 5, 6])
country=6
arcs=np.array([[1,3],
[1,4],
[3,4],
[2,4],
[2,5],
[2,6],
[4,5],
[4,6]])
for i in range(6):
m.Minimize(y[i])
for i in range(6):
m.Equation(m.sum(x[i,:])==1)
for k in range (6):
for i in range(8):
m.Equation(x[arcs[i,0]-1,k]\
+x[arcs[i,1]-1,k]<=1)
m.solve()
Response to Edit
The revised version is missing a closing parenthesis on the m.sum(). Here is a corrected version.
from gekko import GEKKO
import numpy as np
m = GEKKO(remote=False)
x = m.Array(m.Var,(6,6),lb=0,ub=1,integer=True)
y = m.Array(m.Var,6,lb=0,ub=1,integer=True)
y= np.array([1, 2, 3, 4, 5, 6])
country=6
arcs=np.array([[1,3],[1,4],[3,4],[3,4],[4,5],[2,6],[4,5],[4,6]])
for i in range(6):
m.Minimize(y[i])
for i in range(6):
m.Equation(m.sum(x[i,:])==1)
for k in range (6):
for i in range(8):
m.Equation(x[arcs[i,0]-1,k-1]+x[arcs[i,1]-1,k-1]<=1)
for i in range(6):
m.Equation(m.sum(x[i,:])<=y[i])
m.options.solver = 1
m.solve()
print('Objective Function: ' + str(m.options.objfcnval))
print(x)
print(y)
You can find additional tips on troubleshooting gekko applications with tutorial 18.

How to slice on arbitrary indices with breeze?

In Python's numpy, I can do this:
>>> import numpy as np
>>> m = np.array([[1,2,3,4],[5,6,7,8],[9,10,11,12]])
>>> indices = [1,3]
>>> m[:,indices]
array([[ 2, 4],
[ 6, 8],
[10, 12]])
In other words, I can slice based on an arbitrary (not necessarily contiguous) list of indices. How can I do something similar in Breeze? I'm looking for something efficient and preferably elegant.
More or less identically to numpy:
scala> import breeze.linalg._
import breeze.linalg._
scala> val m = DenseMatrix((1,2,3,4),(5,6,7,8),(9,10,11,12))
m: breeze.linalg.DenseMatrix[Int] =
1 2 3 4
5 6 7 8
9 10 11 12
scala> val indices = IndexedSeq(1,3)
indices: IndexedSeq[Int] = Vector(1, 3)
scala> m(::, indices)
res0: breeze.linalg.SliceMatrix[Int,Int,Int] =
2 4
6 8
10 12

How can I put 2d array to 2d array in numpy?

I have the following code
a_alpha_beta = zeros((2, len( atime ) ))
for i in range( len( atime ) ):
alpha_beta = transf.clarke(signal[0][i], signal[1][i], signal[2][i])
a_alpha_beta[0][i] = alpha_beta[0][0]
a_alpha_beta[1][i] = alpha_beta[1][0]
How can I optimize the code above, for example how can I copy alpha_beta to a_alpha_beta?
I don't exactly know what the function transf.clarke does, but copy operations that you desire can be done as follows:
import numpy as np
# generate a test input
x = np.arange(0, 10).reshape((2, 5))
print x
# simply copy 2d array
y = x.copy()
print y
# some new data (i.e., alpha_beta in the original code)
z = np.array([[10, 11, 12],
[13, 14, 15]])
print z
# replace the iteration by numpy slicing to obtain the same result
x[0, :] = z[0, 0]
x[1, :] = z[1, 0]
print x

Resources