I want to be able to compute the Jacobian matrix using the Eigen C++ library, but I cannot find any documentation on how to do this.
Previously, I have achieved this using the numdifftools package in Python. The function is:
numdifftools.Jacobian(ForwardsFunction)([input 1, input 2, input 3, ....])
Here, ForwardsFunction is a user-defined function which calculates the output state given the input state ([input 1, input 2, input 3, ...]). The numdifftools.Jacobian() method then automatically calculates the Jacobian for these input values, presumable using some automatic differentiation.
Is there an equivalent function in the Eigen library?
There are some tools in Eigen library that perform numerical differentiation.
Take a look at:
https://eigen.tuxfamily.org/dox/unsupported/group__NumericalDiff__Module.html
https://eigen.tuxfamily.org/dox/unsupported/classEigen_1_1AutoDiffScalar.html
You might notice, that those modules are "unsupported" (not a part of a official Eigen library). The reasoning is the following: Eigen is a library for linear algebra, thus manipulating with sparse and dense matrices, and numerical differentiation are a little bit on the edge of its scope. - So the priority of including them inside the library is lower. Those modules, as far as I know, are used in their solvers in a very specific way. I don't have an experience of using those Eigen::Numerical Differentiation classes in my project, though you might give it a try.
https://eigen.tuxfamily.org/dox/unsupported/classEigen_1_1NumericalDiff.html
Related
Purely for my own knowledge and understanding of code and computers, I am trying to create an array/matrix class with multiple matrix functions, which I will then use in any projects I need a matrix or array class for. Most significantly, I would like to make a neural network library using this matrix/array class, and therefore require it to be as fast as possible.
The function I require to be fastest is the matrix product calculation of two matrices, however, I have had little luck trying to make this calculation fast with larger matrices.
My current method for calculating the dot product is:
Note, this code is in python, however, if python is not the optimal language, I can use any other
a = [[1, 2, 3], [4, 5, 6]]
b = [[1], [2], [3]]
def dot(a, b):
c = [[0 for j in range(len(b[i]))] for i in range(len(a))]
for i in range(len(c)):
for j in range(len(c[i])):
t = 0
for k in range(len(b)):
t += a[i][k] * b[k][j]
c[i][j] = t
return c
print(dot(a, b))
# [[14], [32]]
I have looked into the Intel MKL (I have an intel core i7) and other BLAS implementations like OpenBLAS, however I have not been able to get any results that worked, and no amount of googling can make them work, so my question is, what is the fastest way to calculate the dot product of two matrices? (CPU and memory usage do not matter much to me currently, however, being more efficient would be nice)
PS:
I am trying to do all of this using no external libraries (numpy, for example, in python)
***** UPDATE *****
I am using a mac
***** UPDATE 2 *****
Thank you everyone for all of your help, however, I am unsure how to implement these methods of calculating the dot product as my math skills are not yet advanced enough to understand what any of it means (I am yet to start my GCSEs), though I will keep these ideas in mind and will experiment with these ideas further.
Thank you again for everyone's help.
If it possible, you can use CUDA to utilize GPU for very fast calculations.
You can use GPU
as AbdelAziz AbdelLatef suggested in his answer. However this limits the usage of your lib to computers with GPU.
Parallelize the dot products for big matrices
use SIMD instructions
use state of the art algorithms
some operations on big data sets can be done much faster using more advanced techniques which are too slow for small matrices ... usually involving FFT or NTT ... Matrix multiplication is set of dot products and dot product is form of convolution so FFT approach should be applicable but had never done that for matrices/vectors ...
there are also special algorithms solely for matrices like Strassen algorithm
for powers you can use power by squaring, for sqr I think you can simplify even more some multiplications would be the same ...
Python is far from optimal as its slow I would do such thing in C++ or even combine with asm if there is the need for extreme speed (like the SIMD instructions use). IIRC you still can use C++ created libs in python (link as DLL,obj,...)
However if you need fast neural network then use dedicated HW. There are neural network processing ICs out there too.
I am using boost::numeric::odeint rosenbrock4 integrator. The problem is I can not get a analytic Jacobean Matrix, Is there any solution to fix this?
You can use a numerical scheme to compute an approximate jacobian. But this has to be implemented by yourself, or some other library. odeint does not provide this functionality.
Out of curiosity: What exactly is your system where you can't find a Jacobian? If the rhs is not differentiable, you might encounter other problems with the numerics.
I have found automatic differentiation to be extremely useful when writing mathematical software. I now have to work with random variables and functions of the random variables, and it seems to me that an approach similar to automatic differentiation could be used for this, too.
The idea is to start with a basic random vector with given multivariate distribution and then you want to work with the implied probability distributions of functions of components of the random vector. The idea is to define operators that automatically combine two probability distributions appropriately when you add, multiply, divide two random variables and transform the distribution appropriately when you apply scalar functions such as exponentiation. You could then combine these to build any function you need of the original random variables and automatically have the corresponding probability distribution available.
Does this sound feasible? If not, why not? If so and since it's not a particularly original thought, could someone point me to an existing implementation, preferably in C
There has been a lot of work on probabilistic programming. One issue is that as your distribution gets more complicated you start needing more complex techniques to sample from it.
There are a number of ways this is done. Probabilistic graphical models gives one vocabulary for expressing these models, and you can then sample from them using various Metropolis-Hastings-style methods. Here is a crash course.
Another model is Probabilistic Programming, which can be done through an embedded domain specific language, directly. Oleg Kiselyov's HANSEI is an example of this approach. Once they have the program they can inspect the tree of decisions and expand them out by a form of importance sampling to gain the most information possible at each step.
You may also want to read "Nonstandard Interpretations of Probabilistic
Programs for Efficient Inference" by Wingate et al. which describes one way to use extra information about the derivative of your distribution to accelerate Metropolis-Hastings-style sampling techniques. I personally use automatic differentiation to calculate those derivatives and this brings the topic back to automatic-differentiation. ;)
I want to implement some matrix–vector math. There are vector types like: float2, int2, but I cannot find any built-in type matrix in CUDA.
Is there a library that suitable for such operations?
You're right to look for a library for matrix data types. I recommend taking a look at ArrayFire.
Here is the quick reference page with a listing of the supported types. Here are the functions you can run with is, which is organized into the categories of data analysis, linear algebra, image and signal processing, sparse matrices, and a bunch of common place algorithms for data indexing, sorting, reductions, visualization, and faster for loops.
Other libraries include CULA or MAGMA (focused on linear algebra), Thrust (targeted at 1D operations), and a host of niche academic libraries.
Disclaimer: I work on ArrayFire myself.
I am trying to write a function that produces a single solution to an underrepresented system of equations (e.g. the matrix that describes the system is wider than it is tall). In order to do this, I have been looking in the LAPACK documentation for a way to row-reduce a matrix to it's reduced-echelon form, similar to the function rref() in both Mathematica and TI calculators. The closest I came across was http://software.intel.com/en-us/forums/intel-math-kernel-library/topic/53107/ this tiny thread. This thread, however, seems to imply that simply taking the "U" upper triangular matrix (and dividing each row by the diagonal) is the same as the reduced echelon form of a matrix, which I do not believe to be the case. I could code up rref() myself, but I do not believe I could achieve the performance LAPACK is famous for.
1) Is there a better way to simply get any one specific solution to an underrepresented system?
2) If not, is there a way for LAPACK to row-reduce a matrix?
Thanks!
One often used method for this is the least square solution, see lapack's sgelsx