I am very new in Apache Spark Scala. Can you help me with some operations?
I have two distributed matrix H and Y in Spark Scala.
I want to compute the pseudo-inverse of H and then multiply H and Y.
How can I do this?
Here is an implementation for the inverse.
import org.apache.spark.mllib.linalg.{Vectors,Vector,Matrix,SingularValueDecomposition,DenseMatrix,DenseVector}
import org.apache.spark.mllib.linalg.distributed.RowMatrix
def computeInverse(X: RowMatrix): DenseMatrix = {
val nCoef = X.numCols.toInt
val svd = X.computeSVD(nCoef, computeU = true)
if (svd.s.size < nCoef) {
sys.error(s"RowMatrix.computeInverse called on singular matrix.")
}
// Create the inv diagonal matrix from S
val invS = DenseMatrix.diag(new DenseVector(svd.s.toArray.map(x => math.pow(x,-1))))
// U cannot be a RowMatrix
val U = new DenseMatrix(svd.U.numRows().toInt,svd.U.numCols().toInt,svd.U.rows.collect.flatMap(x => x.toArray))
// If you could make V distributed, then this may be better. However its alreadly local...so maybe this is fine.
val V = svd.V
// inv(X) = V*inv(S)*transpose(U) --- the U is already transposed.
(V.multiply(invS)).multiply(U)
}
To calculate the pseudo-inverse of non-square matrices you need to be able to calculate the transpose (easy) and the matrix inverse (others have supplied that functionality). There are two different calculations, depending on whether M has full column rank or full row rank.
Full column rank means that the columns of the matrix are linearly independent which requires that the the number of columns is less than or equal to the number of rows. (In pathological cases, an mxn matrix with m>=n might still not have full column rank, but we'll ignore that statistical impossibility. If it is a possibility in your case, the matrix inversion step below will fail.) For full column rank, the pseudo-inverse is
M^+ = (M^T M)^{-1} M^T
where M^T is the transpose of M. Matrix multiply M^T by M, then take the inverse, then matrix multiply by M^T again. (I'm assuming M has real number entries; if the entries are complex numbers, you also have to take complex conjugates.)
A quick check to make sure you have calculated the psuedo-inverse correctly is to check M^+ M. It should be the identity matrix (up to floating point error).
On the other hand, if M has full row rank, in other words M is mxn with m<=n, the pseudo-inverse is
M^+ = M^T (M M^T)^{-1}
To check whether you have the correct pseudo-inverse in this case, right multiply with the original matrix: M M^+. That should equal the identity matrix, up to floating point error.
Matrix multiplication is the easier one: there are several Matrix implementations with a multiply method in packages org.apache.spark.mllib.linalg and org.apache.spark.mllib.linalg.distributed. Pick whatever fits your needs most.
I have not seen (pseudo-)inverse anywhere in the Spark API. But RowMatrix is able to compute the singular value decomposition which can be used to calculate the inverse of a matrix. Here is a very naive implementation, inspired by How can we compute Pseudoinverse for any Matrix (warning: dimensions of the 2x2 matrix are hard-coded):
val m = new RowMatrix(sc.parallelize(Seq(Vectors.dense(4, 3), Vectors.dense(3, 2))))
val svd = m.computeSVD(2, true)
val v = svd.V
val sInvArray = svd.s.toArray.toList.map(x => 1.0 / x).toArray
val sInverse = new DenseMatrix(2, 2, Matrices.diag(Vectors.dense(sInvArray)).toArray)
val uArray = svd.U.rows.collect.toList.map(_.toArray.toList).flatten.toArray
val uTranspose = new DenseMatrix(2, 2, uArray) // already transposed because DenseMatrix is column-major
val inverse = v.multiply(sInverse).multiply(uTranspose)
// -1.9999999999998297 2.999999999999767
// 2.9999999999997637 -3.9999999999996767
Unfortunately, a lot of conversion from Matrix to Array and so forth is necessary. If you need a fully distributed implementation, try using DistributedMatrix instead of DenseMatrix. If not, maybe using Breeze is preferable here.
Related
I'm trying to get a single element of an adjugate A_adj of a matrix A, both of which need to be symbolic expressions, where the symbols x_i are binary and the matrix A is symmetric and sparse. Python's sympy works great for small problems:
from sympy import zeros, symbols
size = 4
A = zeros(size,size)
x_i = [x for x in symbols(f'x0:{size}')]
for i in range(size-1):
A[i,i] += 0.5*x_i[i]
A[i+1,i+1] += 0.5*x_i[i]
A[i,i+1] = A[i+1,i] = -0.3*(i+1)*x_i[i]
A_adj_0 = A[1:,1:].det()
A_adj_0
This calculates the first element A_adj_0 of the cofactor matrix (which is the corresponding minor) and correctly gives me 0.125x_0x_1x_2 - 0.28x_2x_2^2 - 0.055x_1^2x_2 - 0.28x_1x_2^2, which is the expression I need, but there are two issues:
This is completely unfeasible for larger matrices (I need this for sizes of ~100).
The x_i are binary variables (i.e. either 0 or 1) and there seems to be no way for sympy to simplify expressions of binary variables, i.e. simplifying polynomials x_i^n = x_i.
The first issue can be partly addressed by instead solving a linear equation system Ay = b, where b is set to the first basis vector [1, 0, 0, 0], such that y is the first column of the inverse of A. The first entry of y is the first element of the inverse of A:
b = zeros(size,1)
b[0] = 1
y = A.LUsolve(b)
s = {x_i[i]: 1 for i in range(size)}
print(y[0].subs(s) * A.subs(s).det())
print(A_adj_0.subs(s))
The problem here is that the expression for the first element of y is extremely complicated, even after using simplify() and so on. It would be a very simple expression with simplification of binary expressions as mentioned in point 2 above. It's a faster method, but still unfeasible for larger matrices.
This boils down to my actual question:
Is there an efficient way to compute a single element of the adjugate of a sparse and symmetric symbolic matrix, where the symbols are binary values?
I'm open to using other software as well.
Addendum 1:
It seems simplifying binary expressions in sympy is possible with a simple custom substitution which I wasn't aware of:
A_subs = A_adj_0
for i in range(size):
A_subs = A_subs.subs(x_i[i]*x_i[i], x_i[i])
A_subs
You should make sure to use Rational rather than floats in sympy so S(1)/2 or Rational(1, 2) rather than 0.5.
There is a new (undocumented and for the moment internal) implementation of matrices in sympy called DomainMatrix. It is likely to be a lot faster for a problem like this and always produces polynomial results in a fully expanded form. I expect that it will be much faster for this kind of problem but it still seems to be fairly slow for this because is is not sparse internally (yet - that will probably change in the next release) and it does not take advantage of the simplification from the symbols being binary-valued. It can be made to work over GF(2) but not with symbols that are assumed to be in GF(2) which is something different.
In case it is helpful though this is how you would use it in sympy 1.7.1:
from sympy import zeros, symbols, Rational
from sympy.polys.domainmatrix import DomainMatrix
size = 10
A = zeros(size,size)
x_i = [x for x in symbols(f'x0:{size}')]
for i in range(size-1):
A[i,i] += Rational(1, 2)*x_i[i]
A[i+1,i+1] += Rational(1, 2)*x_i[i]
A[i,i+1] = A[i+1,i] = -Rational(3, 10)*(i+1)*x_i[i]
# Convert to DomainMatrix:
dM = DomainMatrix.from_list_sympy(size-1, size-1, A[1:, 1:].tolist())
# Compute determinant and convert back to normal sympy expression:
# Could also use dM.det().as_expr() although it might be slower
A_adj_0 = dM.charpoly()[-1].as_expr()
# Reduce powers:
A_adj_0 = A_adj_0.replace(lambda e: e.is_Pow, lambda e: e.args[0])
print(A_adj_0)
To numerically diagonalize a unitary matrix I use the LAPACK routine zgeev.
The problem is: In case of degeneracies the degenerate subspace is not orthonormalized, since the routine is for general matrices.
However, since in my case the matrices are unitary, the basis can be always orthonormalized. Is there a better solution than applying QR-algorithm afterwards to the degenerate subspace?
Short answer: Schur decomposition!
If a square matrix A is complex, then its Schur factorization is A=ZTZ*, where Z is unitary and T is upper triangular.
If A happens to be unitary, T must also be unitary. Since T is both unitary and triangular, it is diagonal (proof here,.or there)
Let's consider the vectors Z.e_i, where e_i are the vectors of the canonical basis. These vectors obviously form an orthonormal basis. Moreover, these vectors are eigenvectors of the matrix A.
Hence, the columns of the unitary matrix Z are eigenvectors of the unitary matrix A and form an orthonormal basis.
As a consequence, computing a Schur decomposition of a unitary matrix is equivalent to finding one of its orthogonal basis of eigenvectors.
ZGEESX computes the eigenvalues, the Schur form, and, optionally, the matrix of Schur vectors for GE matrices
The resulting T can also be tested to check that A is unitary.
Here is a piece of python code testing it, though scipy's scipy.linalg.schur makes use of Lapack's zgees for Schur decomposition. I used hpaulj's code to generate random unitary matrix as shown in How to create random orthonormal matrix in python numpy
import numpy as np
import scipy.linalg
#from hpaulj, https://stackoverflow.com/questions/38426349/how-to-create-random-orthonormal-matrix-in-python-numpy
def rvs(dim=3):
random_state = np.random
H = np.eye(dim)
D = np.ones((dim,))
for n in range(1, dim):
x = random_state.normal(size=(dim-n+1,))
D[n-1] = np.sign(x[0])
x[0] -= D[n-1]*np.sqrt((x*x).sum())
# Householder transformation
Hx = (np.eye(dim-n+1) - 2.*np.outer(x, x)/(x*x).sum())
mat = np.eye(dim)
mat[n-1:, n-1:] = Hx
H = np.dot(H, mat)
# Fix the last sign such that the determinant is 1
D[-1] = (-1)**(1-(dim % 2))*D.prod()
# Equivalent to np.dot(np.diag(D), H) but faster, apparently
H = (D*H.T).T
return H
n=42
A= rvs(n)
A = A.astype(complex)
T,Z=scipy.linalg.schur(A,output='complex',lwork=None,overwrite_a=False,sort=None,check_finite=True)
#print T
normT=np.linalg.norm(T,ord=None) #2-norm
eigenvalues=[]
for i in range(n):
eigenvalues.append(T[i,i])
T[i,i]=0.
normTu=np.linalg.norm(T,ord=None)
print 'must be very low if A is unitary: ',normTu/normT
#print Z
for i in range(n):
v=Z[:,i]
w=A.dot(v)-eigenvalues[i]*v
print i,'must be very low if column i of Z is eigenvector of A: ',np.linalg.norm(w,ord=None)/np.linalg.norm(v,ord=None)
I have a nX2 matrix A and a 3D matrix K. I would like to take element-wise multiplication specifying 2 indices in 3rd dimension of K designated by each row vector in A and take summation of them.
For instance of a simplified example when n=2,
A=[1 2;3 4];%2X2 matrix
K=unifrnd(0.1,0.1,2,2,4);%just random 3D matrix
L=zeros(2,2);%save result to here
for t=1:2
L=L+prod(K(:,:,A(t,:)),3);
end
Can I get rid of the for loop in this case?
How's this?
B = A.'; %'
L = squeeze(sum(prod(...
reshape(permute(K(:,:,B(:)),[3 1 2]),2,[],size(K,1),size(K,2)),...
1),...
2));
Although your test case is too simple, so I can't be entirely sure that it's correct.
The idea is that we first take all the indices in A, in column-major order, then reshape the elements of K such that the first two dimensions are of size [2, n], and the second two dimensions are the original 2 of K. We then take the product, then the sum along the necessary dimensions, ending up with a matrix that has to be squeezed to get a 2d matrix.
Using a bit more informative test case:
K = rand(2,3,4);
A = randi(4,4,2);
L = zeros(2,3);%save result to here
for t=1:size(A,1)
L = L+prod(K(:,:,A(t,:)),3);
end
B = A.'; %'
L2 = squeeze(sum(prod(reshape(permute(K(:,:,B(:)),[3 1 2]),2,[],size(K,1),size(K,2)),1),2));
Then
>> isequal(L,L2)
ans =
1
With some reshaping magic -
%// Get sizes
[m1,n1,r1] = size(K);
[m2,n2] = size(A);
%// Index into 3rd dim of K; perform reductions and reshape back
Lout = reshape(sum(prod(reshape(K(:,:,A'),[],n2,m2),2),3),m1,n1);
Explanation :
Index into the third dimension of K with a transposed version of A (transposed because we are using rows of A for indexing).
Perform the prod() and sum() operations.
Finally reshape back to a shape same as K but without the third dimension as that was removed in the earlier reduction steps.
I have a nxn singular matrix. I want to add k rows (which must be from the standard basis e1, e2, ..., en) to this matrix such that the new (n+k)xn matrix is full column rank. The number of added rows k must be minimum and they can be added in any order (not just e1, e2 ,..., it can be e4, e10, e1, ...) as long as k is minimum.
Does anybody know a simple way to do this? Any help is appreciated.
You can achieve this by doing a QR decomposition with column pivoting, then taking the transpose of the last n-rank(A) columns of the permutation matrix.
In matlab, this is achieved by the qr function(See the matlab documentation here):
r=rank(A);
[Q,R,E]=qr(A);
newA=[A;transpose(E(:,end-r+1:end))];
Each row of transpose(E(:,end-r+1:end)) will be a member of standard basis, rank of newA will be n, and this is also the minimal number of standard basis you will need to do so.
Here is how this works:
QR decomposition with column pivoting is a standard procedure to decompose a matrix A into products:
A*E==Q*R
where Q is an orthogonal matrix if A is real, or an unitary matrix if A is complex; R is upper triangular matrix, and E is a permutation matrix.
In short, the permutations are chosen so that the diagonal elements are larger than the off-diagonals in the same row, and that size of the diagonal elements are non-increasing. More detailed description can be found on the netlib QR factorization page.
Since Q and E are both orthogonal (or unitary) matrices, the rank of R is the same as the rank of A. To bring up the rank of A, we just need to find ways to increase the rank of R; and this is much more straight forward thanks to the structure of R as the result of pivoting and the fact that it is upper-triangular.
Now, with the requirement placed on pivoting procedure, if any diagonal element of R is 0, the entire row has to be 0. The n-rank(A) rows of 0s in the bottom if R is responsible for the nullity. If we replace the lower right corner with an identity matrix, the that new matrix would be full rank. Well, we cannot really do the replacement, but we can append the rows matrix to the bottom of R and form a new matrix that has the same rank:
B==[ 0 I ] => newR=[ R ; B ]
Here the dimensionality of I is the nullity of A and that of R.
It is readily seen that rank(newR)=n. Then we can also define a new unitary Q matrix by expanding its dimensionality in a trivial manner:
newQ=[Q 0 ; 0 I]
With that, our new rank n matrix can be obtained as
newA=newQ*newR.transpose(E)=[Q*R ; B ]*transpose(E) =[A ; B*transpose(E)]
Note that B is [0 I] and E is a permutation matrix, so B*transpose(E) is simply the transpose
of the last n-rank(A) columns of E, and thus a set of rows made of standard basis, and that's just what you wanted!
Is n very large? The simplest solution without using any math would be to try adding e_i and seeing if the rank increases. If it does, keep e_i. proceed until finished.
I like #Xiaolei Zhu's solution because it's elegant, but another way to go (that's even more computationally efficient is):
Determine if any rows, indexed by i, of your matrix A are all zero. If so, then the corresponding e_i must be concatenated.
After that process, you can simply concatenate any subset of the n - rank(A) columns of the identity matrix that you didn't add in step 1.
rows/cols from Identity matrix can be added in any order. it does not need to be added in usual order as e1,e2,... in general situation for making matrix full rank.
I don't have enough memory to simply create a diagonal D-by-D matrix, since D is large. I keep getting an 'out of memory' error.
Instead of performing M x D x D operations in the first multiplication, I do M x D operations, but still my code takes ages to run.
Can anybody find a more effective way to perform the multiplication A'*B*A? Here's what I've attempted so far:
D=20000
M=25
A = floor(rand(D,M)*10);
B = floor(rand(1,D)*10);
for i=1:D
for j=1:M
result(i,j) = A(i,j) * B(1,j);
end
end
manual = result * A';
auto = A*diag(B)*A';
isequal(manual,auto)
One option that should solve your problem is using sparse matrices. Here's an example:
D = 20000;
M = 25;
A = floor(rand(D,M).*10); %# A D-by-M matrix
diagB = rand(1,D).*10; %# Main diagonal of B
B = sparse(1:D,1:D,diagB); %# A sparse D-by-D diagonal matrix
result = (A.'*B)*A; %'# An M-by-M result
Another option would be to replicate the D elements along the main diagonal of B to create an M-by-D matrix using the function REPMAT, then use element-wise multiplication with A.':
B = repmat(diagB,M,1); %# Replicate diagB to create an M-by-D matrix
result = (A.'.*B)*A; %'# An M-by-M result
And yet another option would be to use the function BSXFUN:
result = bsxfun(#times,A.',diagB)*A; %'# An M-by-M result
Maybe I'm having a bit of a brainfart here, but can't you turn your DxD matrix into a DxM matrix (with M copies of the vector you're given) and then .* the last two matrices rather than multiply them (and then, of course, normally multiply the first with the found product quantity)?
You are getting "out of memory" because MATLAB can not find a chunk of memory large enough to accommodate the entire matrix. There are different techniques to avoid this error described in MATLAB documentation.
In MATLAB you obviously do not need programming explicit loops in most cases because you can use operator *. There exists a technique how to speed up matrix multiplication if it is done with explicit loops, here is an example in C#. It has a good idea how (potentially large) matrix can be split into smaller matrices. To contain these smaller matrices in MATLAB you can use cell matrix. It is much more probably that system finds enough RAM to accommodate two smaller sub-matrices then the resulting large matrix.