I have a (100,64,22,3,3) shaped pytorch tensor, and I would like to sort along axis=0 by the trace of the (3,3) components. The code I have below works, but it is very slow due to the for loops. Is there a way to vectorize the operation to speed it up?
x=torch.rand(100,64,22,3,3)
x_sorted=torch.zeros((x.shape[0],x.shape[1],x.shape[2],x.shape[3],x.shape[4]))
for i in range(x.shape[0]):
#compute tensorized trace
trace=new=torch.diagonal(x[i], dim1=-2, dim2=-1).sum(-1)
#Sort the trace
trace_values,trace_ind=torch.sort(trace,dim=0,descending=True)
for j in range(x_sorted.shape[1]):
for k in range(x_sorted.shape[2]):
x_sorted[i,j,k]=x[i,trace_ind[j,k],k]
Try:
tensor = torch.tensor(np.random.rand(100,64, 3, 3))
orders = torch.argsort(torch.einsum('ijkk->ijk', tensor).sum(-1), axis=0)
orders.shape
tensor[orders, torch.arange(s.shape[1])[None, :]]
Related
I'm trying to get a single element of an adjugate A_adj of a matrix A, both of which need to be symbolic expressions, where the symbols x_i are binary and the matrix A is symmetric and sparse. Python's sympy works great for small problems:
from sympy import zeros, symbols
size = 4
A = zeros(size,size)
x_i = [x for x in symbols(f'x0:{size}')]
for i in range(size-1):
A[i,i] += 0.5*x_i[i]
A[i+1,i+1] += 0.5*x_i[i]
A[i,i+1] = A[i+1,i] = -0.3*(i+1)*x_i[i]
A_adj_0 = A[1:,1:].det()
A_adj_0
This calculates the first element A_adj_0 of the cofactor matrix (which is the corresponding minor) and correctly gives me 0.125x_0x_1x_2 - 0.28x_2x_2^2 - 0.055x_1^2x_2 - 0.28x_1x_2^2, which is the expression I need, but there are two issues:
This is completely unfeasible for larger matrices (I need this for sizes of ~100).
The x_i are binary variables (i.e. either 0 or 1) and there seems to be no way for sympy to simplify expressions of binary variables, i.e. simplifying polynomials x_i^n = x_i.
The first issue can be partly addressed by instead solving a linear equation system Ay = b, where b is set to the first basis vector [1, 0, 0, 0], such that y is the first column of the inverse of A. The first entry of y is the first element of the inverse of A:
b = zeros(size,1)
b[0] = 1
y = A.LUsolve(b)
s = {x_i[i]: 1 for i in range(size)}
print(y[0].subs(s) * A.subs(s).det())
print(A_adj_0.subs(s))
The problem here is that the expression for the first element of y is extremely complicated, even after using simplify() and so on. It would be a very simple expression with simplification of binary expressions as mentioned in point 2 above. It's a faster method, but still unfeasible for larger matrices.
This boils down to my actual question:
Is there an efficient way to compute a single element of the adjugate of a sparse and symmetric symbolic matrix, where the symbols are binary values?
I'm open to using other software as well.
Addendum 1:
It seems simplifying binary expressions in sympy is possible with a simple custom substitution which I wasn't aware of:
A_subs = A_adj_0
for i in range(size):
A_subs = A_subs.subs(x_i[i]*x_i[i], x_i[i])
A_subs
You should make sure to use Rational rather than floats in sympy so S(1)/2 or Rational(1, 2) rather than 0.5.
There is a new (undocumented and for the moment internal) implementation of matrices in sympy called DomainMatrix. It is likely to be a lot faster for a problem like this and always produces polynomial results in a fully expanded form. I expect that it will be much faster for this kind of problem but it still seems to be fairly slow for this because is is not sparse internally (yet - that will probably change in the next release) and it does not take advantage of the simplification from the symbols being binary-valued. It can be made to work over GF(2) but not with symbols that are assumed to be in GF(2) which is something different.
In case it is helpful though this is how you would use it in sympy 1.7.1:
from sympy import zeros, symbols, Rational
from sympy.polys.domainmatrix import DomainMatrix
size = 10
A = zeros(size,size)
x_i = [x for x in symbols(f'x0:{size}')]
for i in range(size-1):
A[i,i] += Rational(1, 2)*x_i[i]
A[i+1,i+1] += Rational(1, 2)*x_i[i]
A[i,i+1] = A[i+1,i] = -Rational(3, 10)*(i+1)*x_i[i]
# Convert to DomainMatrix:
dM = DomainMatrix.from_list_sympy(size-1, size-1, A[1:, 1:].tolist())
# Compute determinant and convert back to normal sympy expression:
# Could also use dM.det().as_expr() although it might be slower
A_adj_0 = dM.charpoly()[-1].as_expr()
# Reduce powers:
A_adj_0 = A_adj_0.replace(lambda e: e.is_Pow, lambda e: e.args[0])
print(A_adj_0)
I need to perform a double sum over a weighted matrix product like this: mathematical expression of problem
where C is a list containing some different weights (complex) for each term and
X are an 3d-array where the index refers to the i'ths 2d-array along the depth axis. I need to evaluate a sum of such expression thousands of times (different 'm' arrays) so the execution time is very important. in the real problem the arrays are of greater dimensions. An illustrative minimum example of my code is
import numpy as np
from time import clock
X = np.random.random((12,12,31))
m = np.random.random((12,12))
C = list(map(lambda x: x*2+1j*x, range(31)))
def summing(m):
start = clock()
t2 = np.zeros((12,12),dtype = complex)
for i in range(len(C)):
for j in range(len(C)):
t2 += C[i]*(X[:,:,i].dot(m).dot(np.conj(X[:,:,j].T)))
return t2, clock()-start
summing(m)[1]
The execution time for me is around 0.025s, which is way too slow for the number of computations I need to perform. Running in parallel is not an option as this function is part of a larger parallel computation, so parallelising this would create nested parallel computations.
Do any of you see a much cleverer way of performing this computation in a faster and hopefully a more pythonic way?
I have a nX2 matrix A and a 3D matrix K. I would like to take element-wise multiplication specifying 2 indices in 3rd dimension of K designated by each row vector in A and take summation of them.
For instance of a simplified example when n=2,
A=[1 2;3 4];%2X2 matrix
K=unifrnd(0.1,0.1,2,2,4);%just random 3D matrix
L=zeros(2,2);%save result to here
for t=1:2
L=L+prod(K(:,:,A(t,:)),3);
end
Can I get rid of the for loop in this case?
How's this?
B = A.'; %'
L = squeeze(sum(prod(...
reshape(permute(K(:,:,B(:)),[3 1 2]),2,[],size(K,1),size(K,2)),...
1),...
2));
Although your test case is too simple, so I can't be entirely sure that it's correct.
The idea is that we first take all the indices in A, in column-major order, then reshape the elements of K such that the first two dimensions are of size [2, n], and the second two dimensions are the original 2 of K. We then take the product, then the sum along the necessary dimensions, ending up with a matrix that has to be squeezed to get a 2d matrix.
Using a bit more informative test case:
K = rand(2,3,4);
A = randi(4,4,2);
L = zeros(2,3);%save result to here
for t=1:size(A,1)
L = L+prod(K(:,:,A(t,:)),3);
end
B = A.'; %'
L2 = squeeze(sum(prod(reshape(permute(K(:,:,B(:)),[3 1 2]),2,[],size(K,1),size(K,2)),1),2));
Then
>> isequal(L,L2)
ans =
1
With some reshaping magic -
%// Get sizes
[m1,n1,r1] = size(K);
[m2,n2] = size(A);
%// Index into 3rd dim of K; perform reductions and reshape back
Lout = reshape(sum(prod(reshape(K(:,:,A'),[],n2,m2),2),3),m1,n1);
Explanation :
Index into the third dimension of K with a transposed version of A (transposed because we are using rows of A for indexing).
Perform the prod() and sum() operations.
Finally reshape back to a shape same as K but without the third dimension as that was removed in the earlier reduction steps.
I have a scalar function f([x,y],[i,j])= exp(-norm([x,y]-[i,j])^2/sigma^2) which receives two 2-dimensional vectors as input (norm here implements the Euclidean norm). The values of x,i range in 1:w and the values y,j range in 1:h. I want to create a cell array X such that X{x,y} will contain a w x h matrix such that X{x,y}(i,j) = f([x,y],[i,j]). This can obviously be done using 4 nested loops like so:
for x=1:w;
for y=1:h;
X{x,y}=zeros(w,h);
for i=1:w
for j=1:h
X{x,y}(i,j)=f([x,y],[i,j])
end
end
end
end
This is however extremely inefficient. I would very much appreciate an efficient way to create X.
The one way to do this is to remove the 2 innermost loops and replace then with a vectorised version. By the look of your f function this shouldn't be too bad
First we need to construct two matrices containing the 1 to w on every row and 1 to h on every column like so
wMat=repmat(1:w,h,1);
hMat=repmat(1:h,w,1)';
This is going to represent the inner two loops, and the transpose will allow us to get all combinations. Now we can vectorise the calculation (f([x,y],[i,j])= exp(-norm([x,y]-[i,j])^2/sigma^2)):
for x=1:w;
for y=1:h;
temp1=sqrt((x-wMat).^2+(y-hMat).^2);
X{x,y}=exp(temp1/(sigma^2));
end
end
Where we have computed the Euclidean norm for all pairs of nodes in the inner loops at once.
Some discussion and code
The trick here is to perform the norm-calculations with numeric arrays and save the results into a cell array version as late as possible. For performing the norm-calculations you can take help of ndgrid, bsxfun and some permute + reshape to give it the "shape" as needed for the final cell array version. So, here's the vectorized approach to perform these tasks -
%// Create x-y/i-j values to be used for calculation of function values
[xi,yi] = ndgrid(1:w,1:h);
%// Get the norm values
normvals = sqrt(bsxfun(#minus,xi(:),xi(:).').^2 + ...
bsxfun(#minus,yi(:),yi(:).').^2);
%// Get the actual function values
vals = exp(-normvals.^2/sigma^2);
%// Get the values into blocks of a 4D array and then re-arrange to match
%// with the shape of numeric array version of X
blks = reshape(permute(reshape(vals, w*h, h, []), [2 1 3]), h, w, h, w);
arranged_blks = reshape(permute(blks,[2 3 1 4]),w,h,w,h);
%// Finally get the cell array version
X = squeeze(mat2cell(arranged_blks,w,h,ones(1,w),ones(1,h)));
Benchmarking and runtimes
After improving the original loopy code with pre-allocation for X and function-inling f, runtime-benchmarks were performed with it against the proposed vectorized approach with datasizes as w, h = 60 and the runtime results thus obtained were -
----------- With Improved loopy code
Elapsed time is 41.227797 seconds.
----------- With Vectorized code
Elapsed time is 2.116782 seconds.
This suggested a whooping close to 20x speedup with the proposed solution!
For extremely huge datasizes
If you are dealing with huge datasizes, essentially you are not giving enough memory for bsxfun to work with, and bsxfun is known to use up a lot of memory for giving you a performance-efficient vectorized solution. So, for such huge-datasize cases, you can use the following loopy approach to replace normvals calculations that was listed in the earlier bsxfun based solution -
%// Get the norm values
nx = numel(xi);
normvals = zeros(nx,nx);
for ii = 1:nx
normvals(:,ii) = sqrt( (xi(:) - xi(ii)).^2 + (yi(:) - yi(ii)).^2 );
end
It seems to me that when you run through the cycle for x=w, y=h, you are calculating all the values you need at once. So you don't need recalculate them. Once you have this:
for i=1:w
for j=1:h
temp(i,j)=f([x,y],[i,j])
end
end
Then, e.g. X{1,1} is just temp(1,1), X{2,2} is just temp(1:2,1:2), and so on. If you can vectorise the calculation of f (norm here is just the Euclidean norm of that vector?) then it will get even simpler.
I'm trying to take the square root of a matrix. That is find the matrix B so B*B=A. None of the methods I've found around gives a working result.
First I found this formula on Wikipedia:
Set Y_0 = A and Z_0 = I then the iteration:
Y_{k+1} = .5*(Y_k + Z_k^{-1}),
Z_{k+1} = .5*(Z_k + Y_k^{-1}).
Then Y should converge to B.
However implementing the algorithm in python (using numpy for inverse matrices), gave me rubbish results:
>>> def denbev(Y,Z,n):
if n == 0: return Y,Z
return denbev(.5*(Y+Z**-1), .5*(Z+Y**-1), n-1)
>>> denbev(matrix('1,2;3,4'), matrix('1,0;0,1'), 3)[0]**2
matrix([[ 1.31969074, 1.85986159],
[ 2.78979239, 4.10948313]])
>>> denbev(matrix('1,2;3,4'), matrix('1,0;0,1'), 100)[0]**2
matrix([[ 1.44409972, 1.79685675],
[ 2.69528512, 4.13938485]])
As you can see, iterating 100 times, gives worse results than iterating three times, and none of the results get within a 40% error margin.
Then I tried the scipy sqrtm method, but that was even worse:
>>> scipy.linalg.sqrtm(matrix('1,2;3,4'))**2
array([[ 0.09090909+0.51425948j, 0.60606061-0.34283965j],
[ 1.36363636-0.77138922j, 3.09090909+0.51425948j]])
>>> scipy.linalg.sqrtm(matrix('1,2;3,4')**2)
array([[ 1.56669890+0.j, 1.74077656+0.j],
[ 2.61116484+0.j, 4.17786374+0.j]])
I don't know a lot about matrix square rooting, but I figure there must be algorithms that perform better than the above?
(1) the square root of the matrix [1,2;3,4] should give something complex, as the eigenvalues of that matrix are negative. SO your solution can't be correct to begin with.
(2) linalg.sqrtm returns an array, NOT a matrix. Hence, using * to multiply them is not a good idea. In your case, the solutions is thus correct, but you're not seeing it.
edit try the following, you'll see it's correct:
asmatrix(scipy.linalg.sqrtm(matrix('1,2;3,4')))**2
Your matrix [1 2; 3 4] isn't positive so there is no solution to the problem in the domain of real matrices.
What is the purpose of the matrix square root that you're doing? I suspect a practical application the matrix really could be symmetric positive definite (e.g. covariance) so you shouldn't encounter complex numbers.
In that case you can compute a cholesky decomposition, like a scaled LU factorization, see here: http://en.wikipedia.org/wiki/Cholesky_decomposition
Another practical example is if your matrices are rotations, then you can first decompose with matrix log and just divide by 2 in the log space, then go back to rotation with matrix exponent... in any event it sounds strange that you ask for a 'generic matrix square root', you probably want to understand the specific application in more depth.