I am trying to use pymc3 to generate some samples from a GMM distribution, here is my code:
w = sp.array([.3, .6, 0.1])
w = sp.array([.3, .6, 0.1])
mu = sp.array([-2, 1, 4])
sd = sp.array([1, 0.5, 0.5])
with pm.Model() as model:
pm.NormalMixture('x', w=w, mu=mu, sd=sd)
step = pm.Metropolis(tune=False, S=sp.array([1]))
trace = pm.sampling.sample(1000, step=step, start={'x':5},
chain=10, cores=1, tune=0)
result = trace['x']
However, no matter what I do with "chain" and "cores", I get the following :
Multiprocess sampling (2 chains in 2 jobs)
Metropolis: [x]
100%|██████████| 1000/1000 [00:00<00:00, 1407.68it/s]
You should use chains, and njobs. Note that setting n_chains with 1000 samples means you will actually get n_chains * 1000 total draws from your model. The njobs argument is passed to joblib, which figures out how to distribute those chains on your machine.
cores will be accepted starting with PyMC 3.4 (or on master as of January, 2018). It is A Bad Thing that sample accepts keyword arguments and silently does nothing with them. That would be a useful contribution, or issue, in the project.
Related
I ran a brms model in R with a complex three-way interaction that I am trying to decompose. See below for code. I have a multinomial dependent variable with three levels (happy/angry/fear) and multiple categorical factors: group (three levels; social anxiety, other anxiety, non-anxious) / emotion (three levels; happy, angry, fear) / percentage (four levels: 10%, 30%, 50%, 70%). I am trying to figure out how I can answer the following question: Do groups differ in their responses to emotional faces (e.g., is the socially anxious group more likely than the other groups to pick angry as their response to happy or fearful faces at any of the percentage levels? The three-way interaction was significant.
My question is, how should I use emmeans to get the output I would like to answer this question? Thanks so much in advance for your response.
model code:
cormod_multi <- brm(Emochoice ~ f_group + f_emotion_r + f_percentage + f_groupf_emotion_rf_percentage + (1 + f_emotion_r*f_percentage | f_ppn), data = data_mult, family = 'categorical', warmup = 1000, iter = 10000, chains = 4, cores = 10, control = list(adapt_delta = 0.95, max_treedepth = 15))
I'm quite new to Gekko. Is it possible to vary the size of a model array as part of an optimization? I am running a simple problem where various numbers of torsional springs engage at different angles, and I would like to allow the model to change the number of engagement angles. Each spring has several component variables, which I am also attempting to define as arrays of variables. However, the size definition of the array theta_engage, below, has not accepted int(n_engage.value). I get the following error:
TypeError: int() argument must be a string, a bytes-like object or a number, not 'GK_Value'
Relevant code:
n_engage = m.Var(2, lb=1, ub=10, integer=True)
theta_engage = m.Array(m.Var, (int(n_engage.value)))
theta_engage[0].value = 0.0
theta_engage[0].lower = 0.0
theta_engage[0].upper = 85.0
theta_engage[1].value = 15.0
theta_engage[1].lower = 0.0
theta_engage[1].upper = 85.0
If I try to define the size of theta_engage only by n_engage.value, I get this error:
TypeError: expected sequence object with len >= 0 or a single integer
I suppose I could define the array at the maximum size I am willing to accept and allow the number of springs to have a lower bound of 0, but I would have to enforce a minimum number of total springs somehow in the constraints. If Gekko is capable of varying the size of the arrays this way it seems to me the more elegant solution.
Any help is much appreciated.
The problem structure can't be changed iteration-to-iteration. However, it is easy to define a binary variable b that either activates or deactivates those parts of the model that should be included or excluded.
from gekko import GEKKO
import numpy as np
m = GEKKO()
# number of springs
n = 10
# number of engaged springs (1-10)
nb = m.Var(2, lb=1, ub=n, integer=True)
# engaged springs (binary, 0-1)
b = m.Array(m.Var,n,lb=0,ub=1,integer=True)
# angle of engaged springs
θ = m.Array(m.Param,n,lb=0,ub=85)
# initialize values
t0 = [0,15,20,25,30,15,30,25,10,50]
for i,ti in enumerate(t0):
θ[i].value = ti
# contributing spring forces
F = [m.Intermediate(b[i]*m.cos((np.pi/180.0)*θ[i])) \
for i in range(10)]
# force constraint
m.Equation(m.sum(F)>=3)
# engaged springs
m.Equation(nb==m.sum(b))
# minimize engaged springs
m.Minimize(nb)
# optimize with APOPT solver
m.options.SOLVER=1
m.solve()
# print solution
print(b)
This gives a solution in 0.079 sec that springs 1, 3, 9, and 10 should be engaged. It selects the minimum number of springs (4) to achieve the required force that is equivalent to 3 springs at 0 angle.
Successful solution
---------------------------------------------------
Solver : APOPT (v1.0)
Solution time : 7.959999999729916E-002 sec
Objective : 4.00000000000000
Successful solution
---------------------------------------------------
[[1.0] [0.0] [1.0] [0.0] [0.0] [0.0] [0.0] [0.0] [1.0] [1.0]]
I'm using the caret::train() function to develop a weighted knn classification model (kknn) with 10-fold cross-validation and a tuneGrid containing 15 values for kmax, one value for distance, and 3 values for kernel.
That’s 450 total iterations if I understand the process correctly (an iteration being the computation of the probability of a given outcome for a given combination of kmax, distance, and kernel). x has about 480,000 data points (6 predictors each having about 80,000 observations), and y has about 80,000 data points.
Understanding that there are innumerable variables affecting performance, how long can I reasonably expect the train function to take if run on a pc with an 8-core 3GHz Intel processor and 32GB of RAM?
It currently takes about 70 minutes per fold, which is about 1.5 minutes per iteration. Is this reasonable, or excessive?
This is a kknn learning exercise. I realize there are other types of algorithms that produce better results more efficiently.
Here is the essential code:
x <- as.matrix(train_set2[, c("n_launch_angle", "n_launch_speed", "n_spray_angle_Kolp", "n_spray_angle_adj", "n_hp_to_1b", "n_if_alignment")])
y <- train_set2$events
set.seed(1)
fitControl <- trainControl(method = "cv", number = 10, p = 0.8, returnData = TRUE,
returnResamp = "all", savePredictions = "all",
summaryFunction = twoClassSummary, classProbs = TRUE,
verboseIter = TRUE)
tuneGrid <- expand.grid(kmax = seq(11, 39, 2),
distance = 2,
kernel = c("triangular", "gaussian", "optimal"))
kknn_train <- train(x, y, method = "kknn",
tuneGrid = tuneGrid, trControl = fitControl)
As we have established in the comments, it is reasonable to expect this type of runtime. There are a few step to reduce this;
Running your code in parallel
Using a more efficient OS; like Linux
Be more efficient in your trainControl(), is it really necessary to have returnResamps=TRUE? There is small gains in controlling these.
Clearly, the first one is a no-brainer. For the second one, I can find as many computer-engineers who swears to linux as those who swears to windows. What convinced me to switch to Linux, was this particular test, which I hope will give you what it gave me.
# Calculate distance matrix
test_data <- function(dim, num, seed = 1903) {
set.seed(seed)
dist(
matrix(
rnorm(dim * num), nrow = num
)
)
}
# Benchmarking
microbenchmark::microbenchmark(test_data(120,4500))
This piece of code simply just runs faster on the exact same machine that runs Linux. At least this was my experience.
I am going through the following example on Problog probabilistic graph
I tried to computed probability of path from 1 to 5. Here are my manual computations
0.6*0.4+0.1*0.3*0.8 = 0.264
However, Problog returns P(path(1,5)) = 0.25824
Am I computing it correctly?
No, you can't just add up all the probabilities for the different paths. To see that, just assume that both paths from 1 to 5 had a probability of 0.7 each. You would get a probability of 1.4 which is clearly wrong (meaning that it is impossible that there is no path).
The way to calculate the probability for either of two events A and B is to get the probability of neither being true and then looking at the inverse of this event.
P(1->2->5) = 0.24
P(1->3->4->5) = 0.024
P(either) = 1 - (1 - 0.24) * (1 - 0.024)
= 1 - 0.74176
= 0.25824
Sorry for probably bad terminology, my statistics knowledge is a bit rusty.
I was persuaded some time ago to drop my comfortable matlab programming and start programming in Julia. I have been working for a long with neural networks and I thought that, now with Julia, I could get things done faster by parallelising the calculation of the gradient.
The gradient need not be calculated on the entire dataset in one go; instead one can split the calculation. For instance, by splitting the dataset in parts, we can calculate a partial gradient on each part. The total gradient is then calculated by adding up the partial gradients.
Though, the principle is simple, when I parallelise with Julia I get a performance degradation, i.e. one process is faster then two processes! I am obviously doing something wrong... I have consulted other questions asked in the forum but I could still not piece together an answer. I think my problem lies in that there is a lot of unnecessary data moving going on, but I can't fix it properly.
In order to avoid posting messy neural network code, I am posting below a simpler example that replicates my problem in the setting of linear regression.
The code-block below creates some data for a linear regression problem. The code explains the constants, but X is the matrix containing the data inputs. We randomly create a weight vector w which when multiplied with X creates some targets Y.
######################################
## CREATE LINEAR REGRESSION PROBLEM ##
######################################
# This code implements a simple linear regression problem
MAXITER = 100 # number of iterations for simple gradient descent
N = 10000 # number of data items
D = 50 # dimension of data items
X = randn(N, D) # create random matrix of data, data items appear row-wise
Wtrue = randn(D,1) # create arbitrary weight matrix to generate targets
Y = X*Wtrue # generate targets
The next code-block below defines functions for measuring the fitness of our regression (i.e. the negative log-likelihood) and the gradient of the weight vector w:
####################################
## DEFINE FUNCTIONS ##
####################################
#everywhere begin
#-------------------------------------------------------------------
function negative_loglikelihood(Y,X,W)
#-------------------------------------------------------------------
# number of data items
N = size(X,1)
# accumulate here log-likelihood
ll = 0
for nn=1:N
ll = ll - 0.5*sum((Y[nn,:] - X[nn,:]*W).^2)
end
return ll
end
#-------------------------------------------------------------------
function negative_loglikelihood_grad(Y,X,W, first_index,last_index)
#-------------------------------------------------------------------
# number of data items
N = size(X,1)
# accumulate here gradient contributions by each data item
grad = zeros(similar(W))
for nn=first_index:last_index
grad = grad + X[nn,:]' * (Y[nn,:] - X[nn,:]*W)
end
return grad
end
end
Note that the above functions are on purpose not vectorised! I choose not to vectorise, as the final code (the neural network case) will also not admit any vectorisation (let us not get into more details regarding this).
Finally, the code-block below shows a very simple gradient descent that tries to recover the parameter weight vector w from the given data Y and X:
####################################
## SOLVE LINEAR REGRESSION ##
####################################
# start from random initial solution
W = randn(D,1)
# learning rate, set here to some arbitrary small constant
eta = 0.000001
# the following for-loop implements simple gradient descent
for iter=1:MAXITER
# get gradient
ref_array = Array(RemoteRef, nworkers())
# let each worker process part of matrix X
for index=1:length(workers())
# first index of subset of X that worker should work on
first_index = (index-1)*int(ceil(N/nworkers())) + 1
# last index of subset of X that worker should work on
last_index = min((index)*(int(ceil(N/nworkers()))), N)
ref_array[index] = #spawn negative_loglikelihood_grad(Y,X,W, first_index,last_index)
end
# gather the gradients calculated on parts of matrix X
grad = zeros(similar(W))
for index=1:length(workers())
grad = grad + fetch(ref_array[index])
end
# now that we have the gradient we can update parameters W
W = W + eta*grad;
# report progress, monitor optimisation
#printf("Iter %d neg_loglikel=%.4f\n",iter, negative_loglikelihood(Y,X,W))
end
As is hopefully visible, I tried to parallelise the calculation of the gradient in the easiest possible way here. My strategy is to break the calculation of the gradient in as many parts as available workers. Each worker is required to work only on part of matrix X, which part is specified by first_index and last_index. Hence, each worker should work with X[first_index:last_index,:]. For instance, for 4 workers and N = 10000, the work should be divided as follows:
worker 1 => first_index = 1, last_index = 2500
worker 2 => first_index = 2501, last_index = 5000
worker 3 => first_index = 5001, last_index = 7500
worker 4 => first_index = 7501, last_index = 10000
Unfortunately, this entire code works faster if I have only one worker. If add more workers via addprocs(), the code runs slower. One can aggravate this issue by create more data items, for instance use instead N=20000.
With more data items, the degradation is even more pronounced.
In my particular computing environment with N=20000 and one core, the code runs in ~9 secs. With N=20000 and 4 cores it takes ~18 secs!
I tried many many different things inspired by the questions and answers in this forum but unfortunately to no avail. I realise that the parallelisation is naive and that data movement must be the problem, but I have no idea how to do it properly. It seems that the documentation is also a bit scarce on this issue (as is the nice book by Ivo Balbaert).
I would appreciate your help as I have been stuck for quite some while with this and I really need it for my work. For anyone wanting to run the code, to save you the trouble of copying-pasting you can get the code here.
Thanks for taking the time to read this very lengthy question! Help me turn this into a model answer that anyone new in Julia can then consult!
I would say that GD is not a good candidate for parallelizing it using any of the proposed methods: either SharedArray or DistributedArray, or own implementation of distribution of chunks of data.
The problem does not lay in Julia, but in the GD algorithm.
Consider the code:
Main process:
for iter = 1:iterations #iterations: "the more the better"
δ = _gradient_descent_shared(X, y, θ)
θ = θ - α * (δ/N)
end
The problem is in the above for-loop which is a must. No matter how good _gradient_descent_shared is, the total number of iterations kills the noble concept of the parallelization.
After reading the question and the above suggestion I've started implementing GD using SharedArray. Please note, I'm not an expert in the field of SharedArrays.
The main process parts (simple implementation without regularization):
run_gradient_descent(X::SharedArray, y::SharedArray, θ::SharedArray, α, iterations) = begin
N = length(y)
for iter = 1:iterations
δ = _gradient_descent_shared(X, y, θ)
θ = θ - α * (δ/N)
end
θ
end
_gradient_descent_shared(X::SharedArray, y::SharedArray, θ::SharedArray, op=(+)) = begin
if size(X,1) <= length(procs(X))
return _gradient_descent_serial(X, y, θ)
else
rrefs = map(p -> (#spawnat p _gradient_descent_serial(X, y, θ)), procs(X))
return mapreduce(r -> fetch(r), op, rrefs)
end
end
The code common to all workers:
#= Returns the range of indices of a chunk for every worker on which it can work.
The function splits data examples (N rows into chunks),
not the parts of the particular example (features dimensionality remains intact).=#
#everywhere function _worker_range(S::SharedArray)
idx = indexpids(S)
if idx == 0
return 1:size(S,1), 1:size(S,2)
end
nchunks = length(procs(S))
splits = [round(Int, s) for s in linspace(0,size(S,1),nchunks+1)]
splits[idx]+1:splits[idx+1], 1:size(S,2)
end
#Computations on the chunk of the all data.
#everywhere _gradient_descent_serial(X::SharedArray, y::SharedArray, θ::SharedArray) = begin
prange = _worker_range(X)
pX = sdata(X[prange[1], prange[2]])
py = sdata(y[prange[1],:])
tempδ = pX' * (pX * sdata(θ) .- py)
end
The data loading and training. Let me assume that we have:
features in X::Array of the size (N,D), where N - number of examples, D-dimensionality of the features
labels in y::Array of the size (N,1)
The main code might look like this:
X=[ones(size(X,1)) X] #adding the artificial coordinate
N, D = size(X)
MAXITER = 500
α = 0.01
initialθ = SharedArray(Float64, (D,1))
sX = convert(SharedArray, X)
sy = convert(SharedArray, y)
X = nothing
y = nothing
gc()
finalθ = run_gradient_descent(sX, sy, initialθ, α, MAXITER);
After implementing this and run (on 8-cores of my Intell Clore i7) I got a very slight acceleration over serial GD (1-core) on my training multiclass (19 classes) training data (715 sec for serial GD / 665 sec for shared GD).
If my implementation is correct (please check this out - I'm counting on that) then parallelization of the GD algorithm is not worth of that. Definitely you might get better acceleration using stochastic GD on 1-core.
If you want to reduce the amount of data movement, you should strongly consider using SharedArrays. You could preallocate just one output vector, and pass it as an argument to each worker. Each worker sets a chunk of it, just as you suggested.