I have a conceptual question. We can read sparse matrices in Spark and convert them into Compressed Sparse Column (CSC) format using the Matrices.sparse method from the import org.apache.spark.mllib.linalg.{Matrix, Matrices} class. If I convert this into RowMatrix Format and perform SVD, will it be faster and more efficient than directly converting the dense matrix into the RowMatrix format and performing SVD?
Common sense may suggest that sparse matrix would be faster but is it indeed faster?, does the RowMatrix interpretation of the sparse matrix give it a performance boost? If so why?
Related
I know that in scipy, it is good practice to construct your graph in the LIL format then convert said graph in the CRS or CRS format when you want to do operations on this matrix.
however I don't see how it could increase performances when using CRS/CRC format over the LIL format as the time to access data is the same in both cases, or is there something I am missing ?
===
$\sum_{n} F^n(v)$
with
F(v) := $\bigvee_{j \in V} M_{i, j} \wedge v^n_j$
and M the adjacency matrix, v a vector (usually e_i).
third, the sparsity of the F^n(v) vectors would be between 10^(-5) and 10^(-3).
I'm looking for a way to implement block-diagonal matrices in Tensorflow. Specifically, I have block-diagonal matrix A with N blocks of size S x S each. Further, I have a vector v of length N*S. I want to calculate A dot v. Is there any efficient way to do it in Tensorflow?
Also, I would prefer the implementation which supports a batch dimension of v (e.g. its real dimension is batch_size x (N*S)) and which is memory efficient, keeping in memory only block-diagonal parts of A.
Thanks for any help!
You can simply convert your tensor to a sparse tensor since a block-diagonal matrix is just a special case of it. Then, the operations are done in a efficient way. If you already have a dense representation of the tensor you can just cast it using sparse_tensor = tf.contrib.layers.dense_to_sparse(dense_tensor). Otherwise, you can construct it with the tf.SparseTensor(...) function. To get the indices, you might use tf.strided_slice, see this post for more information.
I need to calculate the SVD with a very large matrix of complex numbers.
The matrix dimensions are [260000][570].
By using the EIGEN libs I have defined:
MatrixXcf Lop_tmp
(even if MatrixXcd would be better).
However, I'm able to instantiate Lop_tmp but the SVD crashed (bad alloc) when invoked in this way:
Eigen::BDCSVD<Eigen::MatrixXcf> svd(Lop_tmp.transpose(), Eigen::ComputeThinU | Eigen::ComputeThinV);
Supposing not to partitiong the algoritm, is there a way to compute the Eigen SVD with this Lop_tmp?
Thank you!
I am using armadillo mostly for symmetric and triangular matrices. I wanted to be efficient in terms of memory storage. However, it seems there is no other way than to create a new mat and fill with zeros(for triangular) or with duplicates(for symmetric) the lower/upper part of the matrix.
Is there a more efficient way of using triangular/symmetric matrices using Armadillo?
Thanks,
Antoine
There is no specific support for triangular or banded matrices in Armadillo. However, since version 3.4 support for sparse matrices has gradually been added. Depending on what Armadillo functions you need, and the sparsity of your matrix, you might gain from using SpMat<type> which implements the compressed sparse column (CSC) format. For each nonzero value in your matrix the CSC format stores the row index along with the value so you would likely not save much memory for a triangular matrix. A banded diagonal matrix should however consume significantly less memory.
symmatu()/symmatl() and trimatu()/trimatl()
may be what you are looking for:
http://arma.sourceforge.net/docs.html
Are there any algorithms that allow efficient creation (element filling) of sparse (e.g. CSR or coordinate) matrix in parallel?
If you store your matrix as a coordinate map, any language which has a concurrent dictionary implementation available should do the job for you.
Java's got the ConcurrentHashMap, and .NET 4 has ConcurrentDictionary, both of which allow multi-threaded non-blocking (afaik) element insertion in parallel.
There are no efficient algorithms for creating sparse matrices in data-parallel way. Plausible is coordinate matrix type which requires sorting after content filling, but that type is slow for matrix products etc.
Solution is you don't build sparse matrix - you don't keep it in memory; you do implicit operations in place when you're calculating elements of sparse matrix.