I am working on a little puzzle-game-project. The basic idea is built around projecting multi-dimensonal data down to 2D. My only problem is how to generate the randomized scenario data. Here is the problem:
I got muliple randomized vectors v_i and a target vector t, all 2D. Now I want to randomize scalar values c_i that:
t = sum c_i v_i
Because there are more than two v_i this is a overdetermined system. I also took care that the linear combination of v_i is actual able to reach t.
How can I create (randomized) values for my c_i?
Edit: After finding this Question I can additionally state, that it is possible for me also (slightly) change the v_i.
All values are based on double
Let's say your v_i form a matrix V with 2 rows and n columns, each vector is a column. The coefficients c_i form a column vector c. Then the equation can be written in matrix form as
V×c = t
Now apply a Singular Value Decomposition to matrix V:
V = A×D×B
with A being an orthogonal 2×2 matrix, D is a 2×n matrix and B an orthogonal n×n matrix. The original equation now becomes
A×D×B×c = t
multiply this equation with the inverse of A, the inverse is the same as the transposed matrix AT:
D×B×c = AT×t
Let's introduce new symbols c'=B×c and t'=AT×t:
D×c' = t'
The solution of this equation is simple, because Matrix D looks like this:
u 0 0 0 ... // n columns
0 v 0 0 ...
The solution is
c1' = t1' / u
c2' = t2' / v
And because all the other columns of D are zero, the remaining components c3'...cn' can be chosen freely. This is the place where you can create random numbers for c3'...cn. Having vector c' you can calculate c as
c = BT×c'
with BT being the inverse/transposed of B.
Since the v_i are linearly dependent there are non trivial solutions to 0 = sum l_i v_i.
If you have n vectors you can find n-2 independent such solutions.
If you have now one solution to t = sum c_i v_i you can add any multiple of l_i to c_i and you will still have a solution: c_i' = p l_i + c_i.
For each independent solution of the homogenous problem determine a random p_j and calculate
c_i'' = c_i + sum p_j l_i_j.
Related
This must be surely well known, being a particular linear programming problem. What I want is a specific easy to implement efficient algorithm adapted to this very case, for relatively small sizes (about, say, ten vectors of dimension less than twenty).
I have vectors v(1), ..., v(m) of the same dimension. Want an
algorithm that produces strictly positive numbers c(1), ..., c(m)
such that c(1)v(1) + ... + c(m)v(m) is the zero vector, or tells for
sure that no such numbers exist.
What I found (in some clever code by a colleague) gives an approximate algorithm like this:
start with, say, c(1) = ... = c(m) = 1/m;
at each stage, given current approximation v = c(1)v(1) + ... + c(m)v(m), seek for j such that v - v(j) is longer than v(j).
If no such j exists then output "no solution" (or c(1), ..., c(m) if v is zero).
If such j exists, change v to the new approximation (1 - c)v + cv(j) with some small positive c.
This changes c(j) to (1 - c)c(j) + c and each other c(i) to (1 - c)c(i), so that the new coefficients will remain positive and strictly less than 1 (in fact they will sum to 1 all the time, i. e. we will remain in the convex hull of the v(i)).
Moreover the new v will have strictly smaller length, so eventually the algorithm will either discover that there is no solution or will produce arbitrarily small v.
Clearly this is incomplete and not satisfactory from several points of view. Can one do better?
Update
There are by now two useful answers; however one final step is missing.
They both boil down to the following (unless I miss some essential point).
Take a basis of the nullspace of v(1), ..., v(m).
One obtains a collection of not necessarily strictly positive solutions c(1), ..., c(m), c'(1), ..., c'(m), c''(1), ..., c''(m), ... such that any such solution is their linear combination (in a unique way). So we are reduced to the question whether this new collection of m-dimensional vectors admits a linear combination with strictly positive entries.
Example: take four 2d-vectors (2,1), (3,-1), (-1,2), (-3,-3). Their nullspace has a basis consisting of two solutions c = (12,-3,0,5), c' = (-1,1,1,0). None of these are strictly positive but their combination c + 4c' = (8,1,4,5) is. So the latter is the desired solution. But in general it might be not so easy to find out whether a strictly positive solution exists and if yes, how to find it.
As suggested in the answer by btilly one might use Fourier-Motzkin elimination for that, but again, I would be grateful for more details about it.
This is doable as follows.
First write your vectors as columns. Put them into a matrix. Now create a single column with entries c(1), c(2), ..., c(m_)). If you multiply that matrix times that column, you get your linear combination.
Now consider the elementary row operations. Multiply a row by a constant, swap two rows, add a multiple of one row to another. If you do an elementary row operation to the matrix, your linear combination after the row operation will be 0 if and only if it was before the row operation. Therefore doing elementary row operations DOESN'T CHANGE the coefficients that you're looking for.
Therefore you may simplify life by doing elementary row operations to put the matrix into reduced row echelon form. Once it is in reduced row echelon form, life gets easier. Columns which do not contain a pivot correspond to free coefficients. Columns which do contain a pivot correspond to coefficients that must be a specific linear combination of free coefficients. This reduces your problem being to find positive values for the free coefficients that make the others also positive. So you're now just solving a system of inequalities (and generally in far fewer variables).
Whether a system of linear inequalities has a solution can be answered with the FME method.
Denoting by A the matrix where the ith row is v(i) and by x the vector whose ith index is c(i), your problem can be describes as Ax = b where b=0 is the zero vector. The problem of Ax=b when b is not equal to zero is called the least squares problem (or the inhomogeneous least squares) and has a close form solution in the sense of Minimal Mean Square Error (MMSE). In your case however, b = 0 therefore we are in the homogeneous least squares problem. In Linear Algebra this can be looked as an eigenvalue problem, whose solution is the eigenvector x of the matrix A^TA whose eigenvalue is equal to 0. If no such eigenvalue exists, the MMSE solution will the the eigenvalue x whose matching eigenvalue is the smallest (closest to 0). A nice discussion on this topic is given here.
The solution is, as stated above, will be the eigenvector of A^TA with the lowest matching eigenvalue. This can be done using Singular Value Decomposition (SVD), which will decompose the matrix A into
The column of V matching with the lowest eigenvalue in the diagonal matrix Sigma will be your solution.
Explanation
When we want to minimize the Ax = 0 in the MSE sense, we can compute the vector derivative w.r.t x as follows:
Therefore, the eigenvector of A^TA matching the smallest eigenvalue will solve your problem.
Practical solution example
In python, you can use numpy.linalg.svd to perform the SVD decomposition. numpy orders the matrices U and V^T such that the leftmost column matches the largest eigenvalue and the rightmost column matches the lowest eigenvalue. Thus, you need to compute the SVD and take the rightmost column of the resulting V:
from numpy.linalg import svd
[_, _, vt] = svd(A)
x = vt[-1] # we take the last row since this is a transposed matrix, so the last column of V is the last row of V^T
One zero eigenvalue
In this case there is only one non trivial vector who solves the problem and the only way to satisfy the strictly positive condition will be if the values in the vector are all positive or all negative (multiplying the vector by -1 will not change the result)
Multiple zero eigenvalues
In the case where we have multiple zero eigenvalues, any of their matching eigenvectors is a possible solution and any linear combination of them. In this case one would have to check if there is a linear combination of these eigenvectors which creates a vector where all the values are strictly positive in order to satisfy the strictly positive condition.
How do we find the solution if one exists? once we are left with the basis of eigenvectors matching zero eigenvalue (also known as null-space) what we need to do is to solve a system of linear inequalities. I'll explain by example, since it will be clearer this way. Suppose we have the following matrix:
import numpy as np
A = np.array([[ 2, 3, -1, -3],
[ 1, -1, 2, -3]])
[_, Sigma, Vt] = np.linalg.svd(A) # Sigma has only 2 non-zero values, meaning that the null-space have a dimension of 2
We can extract the eigenvectors as explained above:
C = Vt[len(Sigma):]
# array([[-0.10292809, 0.59058542, 0.75313786, 0.27092073],
# [ 0.89356997, -0.15289589, 0.09399548, 0.4114856 ]])
What we want to find are two real coefficients, noted as x and y such that:
-0.10292809*x + 0.89356997*y > 0
0.59058542*x - 0.15289589*y > 0
0.75313786*x + 0.09399548*y > 0
0.27092073*x + 0.4114856*y > 0
We have a system of 4 inequalities with 2 variables, therefore in this case a solution is not promised. A solution can be found in many ways but I will propose the following. We can start with an initial guess and go over each hyperplane to check if the initial guess satisfies the inequality. if not we can reflect the guess to the other side of the hyperplane. After passing all the hyperplanes we check for a solution. (explanation of hot to reflect a point w.r.t a line can be found here). An example for python implementation will be:
import numpy as np
def get_strictly_positive(A):
[_, Sigma, Vt] = np.linalg.svd(A)
if len(Sigma[np.abs(Sigma) > 1e-5]) == Vt.shape[0]: # No zero eigenvalues, taking MMSE solution if exists
c = Vt[-1]
if np.sum(c > 0) == len(c) or np.sum(c < 0) == len(c):
return c if np.sum(c) == np.sum(abs(c)) else -1 * c
else:
return -1
# This means we have a zero solution
# Building matrix C of all the null-space basis vectors
C = Vt[len(Sigma[np.abs(Sigma) > 1e-5]):]
# 1. What we have here is a set of linear system of inequalities. Each equation inequality is a hyperplane and for
# each equation there is a valid half-space. We want to find the intersection of all the half-spaces, if it exists.
# 2. A vey important observations is that the basis of the null-space that we found using SVD is ORTHOGONAL!
coeffs = np.ones(C.shape[0]) # initial guess
for hyperplane in C.T:
if coeffs.dot(hyperplane) <= 0: # the guess is on the wrong side of the hyperplane
orthogonal_part = coeffs - (coeffs.dot(hyperplane) / hyperplane.dot(hyperplane)) * hyperplane
# reflecting the coefficients to the other side of the hyperplane
coeffs = 2 * orthogonal_part - coeffs
# If this yielded a solution, we return it
c = C.T.dot(coeffs)
if np.sum(c > 0) == len(c) or np.sum(c < 0) == len(c):
return c if np.sum(c) == np.sum(abs(c)) else -1 * c
else:
return -1
The equations are taken from one of my summaries and therefore I do not have a link to the source
Let's say A is a column vecter with shape (m,1), B is a row vector with shape (1,p), C is the matrix product of A and B, i.e. C=AB, so C's shape is (m,p).
Now I want to compute the square of Frobenius norm of C, i.e. sum_i sum_j c_{ij}^2 (sum of all the squares of C's elements)
Note that c_{ij}=a_i*b_j, a_i and b_j are the elements in A and B. So I can rewrite the formula above.
sum_i sum_j c_{ij}^2=sum_i sum_j (a_ib_j)^2=sum_i a_i^2 *sum_j b_j^2
The complexity of formula sum_i sum_j c_{ij}^2 is O(mp).
And the complexity of formula sum_i a_i^2*sum_j b_j^2 is O(m+p), since A and B has lower dimention than C.
However, this trick is off when A and B are both matricies.
Consider this, A is a matrix with shape (m,n), B is a matrix with shape (n,p), C is the matrix product of A and B, i.e. C=AB, so C's shape is also (m,p).
I still want to compute this sum_i sum_j c_{ij}^2
Note that c_{ij}=sum_k a_{ik}*b_{kj}, so
sum_i sum_j c_{ij}^2=sum_i sum_j(sum_k a_{ik}*b_{kj})^2
Therefore, at this time, there is no trick I can use like before (the vector version)
So my question is, I need matrix C and I also need the square of Frobenius norm of C, would it be faster using C directly than using A and B?
square of Frobenius norm of C = trace of CC^T
You want the square of the Frobenius norm of AB. The Frobenius norm is
preserved by rotations, which leads to the following O((m+n+p)
n2)-time algorithm.
Assuming that m ≥ p and n ≥ p (this shouldn’t be critical if we define
the degenerate cases the right way), let A = QR be the reduced
QR-decomposition of A and tB = Q′R′ be the reduced
QR-decomposition of tB. We want the Frobenius norm of AB =
QRt(Q′R′) = QRtR′tQ′. Since Q and Q′
are semi-orthogonal, it suffices to find the Frobenius norm of
RtR′. Since R and R′ are n × n matrices, we can use the
straightforward O(n3)-time algorithm.
We need to transform a set of integers A to another set of integers B such that the sum of squares of the elements of B is equal to a certain given value M.
Since there can be multiple such transformations, we need to find the one in which the sum of the square of the difference between the corresponding elements of A and B is minimum.
Input:
A set of non-negative integers A = {a1, a2, a3 ... an}
A non-negative integer M
Output:
A set of numbers B = {b1, b2, b3 ... bn}, such that:
sumi=1->n[ bi ^ 2 ] = M
sumi=1->n[ (ai-bi) ^ 2 ] = S is minimized.
The minimum sum S.
A bit of math.
Sum (ai - bi)2 = Sum (ai2 - 2 aibi + bi2) = Sum ai2 - 2 Sum aibi + Sum bi2
The first term is constant; the last one is M (also constant), so you are seeking to maximize
Sum aibi
with the restriction Sum bi2 = M.
In other words, you need a hyperplane normal to a vector A = { ai }, tangent to a hypersphere with a radius sqrt(M). Such hyperplane passes through a point where the normal line intersects with the sphere. This point is fA with |fA| = sqrt(M):
f = sqrt(M)/sqrt(Sum ai2)
The solution to your problem is
bi = ai * sqrt(M)/sqrt(Sum ai2)
EDIT: The answers so far, including the one below, map A to a set of real numbers instead of integers. As far as I can tell there is no general fix for this because there are many values of M for which there is no integer vector satisfying the constraint. Ex: M = 2. There is no vector of integers the sum of whose squares is 2. Even if M is a sum of squares, it is a sum of a certain number of squares, so even M = 4 has no solution if A has 3 or more non-zero components. As such, there is no general mapping that satisfies the problem as stated.
Here is the version that allows B to be a vector of reals:
The answer by #user58697 is quite elegant. Here is a restatement that is, perhaps, more intuitive for those of us less used to thinking with hyper geometry:
Treat A and B as vectors. Then start the same way: sum(ai - bi)2 = sum(ai2) - 2sum(aibi) + sum(bi2)
The first term is the magnitude of the vector A squared just as the last term is the magnitude of vector B squared. Both are constant so only the middle term can change. That means we want to maximize sum(aibi) which is exactly the dot product of A and B (https://en.wikipedia.org/wiki/Dot_product). The dot product of two vectors is maximized when the angle between them is 0, which is to say when they are co-directional (that is they point in the same direction).
This means that the unit vector forms of A and B must be the same. That is:
ai/|A| = bi/|B|. Solve this for bi: bi = ai * |B| / |A|
But |B| is just sqrt(M) and A is sqrt(sum(ai2)). So, just like in user58697's version:
bi = ai * sqrt(M) / sqrt(sum(ai2))
I am fitting a plane to a 3D point set with the least square method. I already have algorithm to do that, but I want to modify it to use weighted least square. Meaning I have a weight for each point (the bigger weight, the closer the plane should be to the point).
The current algorithm (without weight) looks like this:
Compute the sum:
for(Point3D p3d : pointCloud) {
pos = p3d.getPosition();
fSumX += pos[0];
fSumY += pos[1];
fSumZ += pos[2];
fSumXX += pos[0]*pos[0];
fSumXY += pos[0]*pos[1];
fSumXZ += pos[0]*pos[2];
fSumYY += pos[1]*pos[1];
fSumYZ += pos[1]*pos[2];
}
than make the matrices:
double[][] A = {
{fSumXX, fSumXY, fSumX},
{fSumXY, fSumYY, fSumY},
{fSumX, fSumY, pointCloud.size()}
};
double[][] B = {
{fSumXZ},
{fSumYZ},
{fSumZ}
};
than solve Ax = B and the 3 components of the solution are the coefficients of the fitted plain...
So, can you please help me how to modify this to use weights? Thanks!
Intuition
A point x on a plane defined by normal n and a point on the plane p obeys: n.(x - p) = 0. If a point y does not lie on the plane, n.(y -p) will not be equal to zero, so a useful way to define a cost is by |n.(y - p)|^2 . This is the squared distance of the point y from the plane.
With equal weights, you want to find an n that minimizes the total squared error when summing over the points:
f(n) = sum_i | n.(x_i - p) |^2
Now this assumes we know some point p that lies on the plane. We can easily compute one as the centroid, which is simply the component-wise mean of the points in the point cloud and will always lie in the least-squares plane.
Solution
Let's define a matrix M where each row is the ith point x_i minus the centroid c, we can re-write:
f(n) = | M n |^2
You should be able to convince yourself that this matrix multiplication version is the same as the sum on the previous equation.
You can then take singular value decomposition of M, and the n you want is then given by the right singular vector of M that corresponds to the smallest singular value.
To incorporate weights you simply need to define a weight w_i for each point. Calculate c as the weighted average of the points, and change sum_i | n.(x_i - c) |^2 to sum_i | w_i * n.(x_i - c) |^2, and the matrix M in a similar way. Then solve as before.
Multiply each term in each sum by the corresponding weight. For example:
fSumZ += weight * pos[2];
fSumXX += weight * pos[0]*pos[0];
Since pointCloude.size() is the sum of 1 for all points, it should be replaced with the sum of all weights.
Start from redefining the least-square error calculation. The formula tries to minimize the sum of squares of errors. Multiply the squared error by a function of two points which decreases with their distance. Then try to minimize the weighted sum of squared errors and derive the coefficients from that.
assume there are three group of high dimension vectors:
{a_1, a_2, ..., a_N},
{b_1, b_2, ... , b_N},
{c_1, c_2, ..., c_N}.
each of my vector can be represented as: x = a_i + b_j + c_k, where 1 <=i, j, k <= N. then the vector is encoded as (i, j, k) wich is then can be decoded as x = a_i + b_j + c_k.
my question is, if there are two vector: x = (i_1, j_1, k_1), y = (i_2, j_2, k_2), is there a method to compute the euclidian distance of these two vector without decode x and y.
Square root of the sum of squares of the differences between components. There's no other way to do it.
You should scale the values to guard against overflow/underflow issues. Search for the max difference and divide all the components by it before squaring, summing, and taking the square root.
Let's assume you have only two groups. You are trying to compute the scalar product
(a_i1 + b_j1, a_i2 + b_j2)
= (a_i1,a_i2) + (b_j1,b_j2) + (a_i1,b_j2) + (a_i2,b_j1) # <- elementary scalar products
So if you know the necessary elementary scalar products between the elements of your vectors a_i, b_j, c_k, then, you do not need to "decode" x and y and can compute the scalar product directly.
Note that this is exactly what happens when you compute an ordinary euclidian distance on a non orthogonal basis.
If you are happy with an approximate result, you could project your high dimension basis vectors using a random projection into a small dimensional space. Johnson-Lindenstrauss lemma says that you can reduce your dimension to O(log N), so that distances remain approximately the same with high probability.