Discr. convolution can be represented as multiplication of input with matrix M.
Where M is presented a special case of Toeplitz matrices - circulant matrices.
The questions is: is 2d convolution can also be represented as matrix multiplication?
p.s. By dicr. convolution I mean dicr. convolution with indexing discrete samples in modulus fashion, i.e. the discrete signal is repeating
....X[n-1]x[0]x[1]...x[N-1]x[0]...
Yes, it can, but it will generally be a rather big matrix. If your data set is on a grid of size NxM, then the convolution is a matrix operating on a vector of length N*M; the convolution matrix has N2M2 elements.
If your convolution kernel is small, then the matrix will typically a band matrix where the width of the band is at least N or M.
Related
If I have to approximate the difference of norm of two matrices X and Y, it can be calculated through their eigen values by Mirsky's inequality .Now, I want to approximate the norm of difference of the eigen vector of the matrices, but I am stucked. Any help would be appreciated.
Can a 2D transpose convolution operation be represented as a matrix multiplication with the Toeplitz matrix, as can be done for a normal convolution?
I want to generalise some ideas from a dense network to a convolutional network. For normal convolutions, this is not a problem, as they can be represented as matrix multiplications with the Toeplitz matrix. But I couldn't find a clear mathematical formulation of transposed convolution, so I am not sure about this case.
I was looking for a mathematical answer, so should have probably asked somewhere else, anyways I think my latex write-up is correct and answers the question:
formula transposed convolution
I have a square matrix that is largely zeros but interspersed with values. Is there a way to 'solve' this matrix so that all the information in it is contained in its lower triangle only, with the upper triangle containing only zeros?
Not in general.
If symmetric and positive definite you can do a Cholesky Decomposition.
If non-symmetric you can do an LU decomposition.
The Q matrix in quadratic forms (x'Qx) can be made symmetric and then lower-triangular. This is sometimes used when setting up a Quadratic Programming (QP) model.
Sparse versions of decomposition approaches are a well-studied area (not trivial though). In large-scale LP solvers sparse LU (simplex) or sparse Cholesky (interior point) are widely used.
How can we find inverse of a matrix using Simplex method? Do we need to have square matrix only or inverse can be found of any matrix? Also specify about the upper bound on the matrix size?
The Matrix Inverse is required during simplex only over the Basis Matrix (Basis Inversion).
Base matrix is a square matrix of dimensions (mxm) where m is the total number of constraints.
This Matrix Inversion is carried out using either the Product form of Inverse or LU Decomposition.
Basically I am solving the diffusion equation in 3D using FFT and one of the ways to parallelise this is to decompose the 3D FFT in 2D FFTs.
As described in this paper: https://cmb.ornl.gov/members/z8g/csproject-report.pdf
The way to decompose a 3d fft would be by doing:
2d fft in xy direction
global transpose
1d fft in z direction
Basically, my problem is that I am not sure how to do this global transpose (as I assume it's transposing a 3d array I suppose). Anyone has came accross this? Thanks a lot.
Think of a 3d cube with nx*ny*nz elements. The 3d FFT of these elements is mathematically 3 stages of 1-d FFTs, one along each axis:
Do ny*nz transforms along the X axis, each transform handles nx elements
nx*nz transforms along the Y axis
nx*ny transforms along the Z axis
More generally, an N-dimensional FFT (N>1) is composed of many (N-1)-dimensional FFTs along that axis.
If the signal is real and you have an FFT that can return the half spectrum, then stage 1 would be about half as expensive (real FFT is cheaper), the remaining stages need to be complex, but they only need to have about half as many transforms. So the cost is roughly half.
If your 1d FFT can read input elements that are strided and pack the output into a contiguous buffer, then you end up doing a transposition at each stage.
This is how kissfft performs multi-dimensional FFTs.
P.S. When I need to get a mental pictures of higher dimensions, I think of:
sheets of paper with matrices of numbers (2d), in folders of numbered papers (3d), in numbered filing cabinets (4d), in numbered rooms (5d), in numbered buildings (6d), and so on ... So I can visualize the "filing cabinet" dimension
The "global transposition" mentioned in the paper is not a mathematical operation, but a rearrangement of data between the distributed memory machines.
The data calculated on one machine in step 1 has to be transferred to all other machines, vice versa, for step to. It has nothing to do with a matrix transposition.