I have to solve multiple least squares problem sequentially - that is one by one. Every least square problem from the previous one changes by only one row. The right hand side is same for all. For eg., Problem 1 : ||Ax-b|| and Problem 2 : ||Cy-b|| where C and A changes by only one row. That is, it is equivalent to deleting a row from A and including a new row in A. When solving problem 2, I also have x. Is there a fast way for solving y of Problem 2?
You can use the Sherman-Morrison formula.
The key piece of the linear regression solution is computing the inverse of A'A.
If b is the old row from A and a is the new row in C, then
C'C=A'A-bb'+aa'=A'A+(a-b)(a+b)'
This expression can be plugged into the Sherman-Morrison formula to compute (C'C)^{-1} given (A'A)^{-1}.
Unfortunately the answer may be NO...
Changing one row of a matrix will lead to completely different spectrum of the matrix. All the eigenvalues and eigenvectors are changed with both magnitude and orientation. As a result, the gradient of problem 1 won't remain in problem 2. You can try to use your x from problem 1 as a initial guess for y in problem 2, but it is not guaranteed to reduce your searching time in optimization.
Yet a linear matrix equation solving is not that hard with the powerful packages. You can use LU decomposition or QR decomposition to improve the computing efficiency very much.
Related
I'm working on a problem, and it feels like it might be analogous to an existing problem in mathematical programming, but I'm having trouble finding any such problem.
The problem goes like this: We have n sets of d dimensional vectors, such that each set contains exactly d+1 vectors. Within each set, all vectors have the same length (furthermore, the angle between any two vectors in a set is the same for any set, but I'm not sure whether this relevant). We then need to choose exactly one vector out of every set, and compute the sum of these vectors. Our objective is to make our choices so that the sum of the vectors is minimized.
It feels like the problem is sort of related to the Shortest Vector Problem, or a variant of job scheduling, where scheduling a job on a machine affects all machines, or a partition problem.
Does this problem ring a bell? Specifically, I'm looking for research into solving this problem, as currently my best bet is using an ILP, but I feel there must be something more clever that can be done.
I think this is an MIQP (Mixed Integer Quadratic Programming) or MISOCP (mixed integer second-order cone) problem:
Let
v(i,j) be i vectors in group j (data)
x(i,j) in {0,1}: binary decision variables
w: sum of selected vectors (decision variable)
Then the problem can be stated as:
min ||w||
sum(i, x(i,j)) = 1 for all j
w = sum((i,j), x(i,j)*v(i,j))
If you want you can substitute out w. Indeed I don't use your angle restriction (this is a restriction on the data and not on the decision variables of the model). The x variables are chosen such that we select exactly one vector from each group.
Minimizing the 2-norm can be replaced by minimizing the sum of the squares of the elements (i.e. minimizing the square of the norm).
Assuming the 2-norm, this is a MISOCP problem or convex MIQP problem for which quite a few solvers are available. For 1-norm and infinity-norms we can formulate a linear MIP problem. MIP solvers are widely available.
I have a set of N (N is very large) linear equations with W variables.
For efficiency sake, I need to find the smallest number of linear equations that are solvable (have a unique solution). It can be assumed that a set of X equations containing Y variables has a unique solution when X == Y.
For example, if I have the following as input:
2a = b - c
a = 0.5b
b = 2 + a
I want to return the equation set:
a = 0.5b
b = 2 + a
Currently, I have an implementation that uses some heuristics. I create a matrix, columns are variables and rows are equations. I search the matrix to find a set of fully connected equations, and then one-by-one try removing equations to see if the remaining set of equations is still solvable, if it is continue, if not, return the set of equations.
Is there a known algorithm for this, and am I trying got reinvent the wheel?
Does anyone have input on how to better approach this?
Thanks.
Short answer is "yes", there are known algorithms. For example, you could add a single equation and then compute the rank of the matrix. Then add the next equation and compute the rank. If it hasn't gone up that new equation isn't helping any and you can get rid of it. Once the rank == the number of variables you have a unique solution and you're done. There are libraries (e.g. Colt, JAMA, la4j, etc.) that will do this for you.
Longer answer is that this is surprisingly difficult to do correctly, especially if your matrix gets big. You end up with lots of numerical stability issues and so on. I'm not a numerical linear algebra expert but I know enough to know there are dragons here if you're not careful. Having said that, if your matrices are small and "well conditioned" (the rows/columns aren't almost parallel) then you should be in good shape. It depends on your application.
Premise
I've a system of linear equations
dot(A,x) = y
whose solutions have many degrees of freedom: indeed the Number of linearly independent Equations (E) is less than the dimension of x, A.K.A. the Number of Variables (N).
The number of degrees of freedom left constrains the solutions to be a hyperplane N-E of the overall space R^N. Given the (unimportant) characteristics of A, I am always able to write the solutions x (a vector N x 1) as
x=dot(B,t)+q
where B is a N x (N-E) matrix, t a (N-E) x 1 vector and q a N x 1 vector. This define the hyperplane of the solutions of my original problem, A x = y in parametric form.
I need to extract a random solution, with uniform probability over any possible point of the hyperplane, such that all x are positive (we will refer to it as a positive solution). Note that, for the specific problem I am dealing with, the space of positive solutions of x exists and it is bounded (that's how the notion of uniform probability is reasonable for the specific case, to clarify as suggested by #Petr comment). In the beginning, once I was able to write x=Bt+q, I thought it extremely simple. Now I am starting to doubt it.
Proposed Solution
By now I do something like this:
For each dimension i in range(N-E) I compute the maximum and minimum value of t[i]: t_min[i] and t_max[i]. Intervals big enough to not exclude any possible positive solution. Those are algebraically computed, always existing and defining a limited space.
I extract N-E uniform random values t[i], each comprised between t_min [i] and t_max[i].
I compute x = dot(B,t)+q
If all x[j] are positives, accept the solution. If some x[j] is negative, go back to point 2.
An example is visible for a two dimensional space N-E in the next figure.
Caption: A problem in N dimension reduced to a N-E=2 space. The yellow diamond is the space of positive solutions of the N-dimensional problem. I randomly sample points in the orange box between (t1(min),t2(min)) and (t1(max),t2(max)) until I find a point in the yellow box.
I think it is a good enough solution, but...
Problem
When N-E is big, the space of the hyperparallelogram bounded inside the hypercube can be small. In general it will be small^(N-E), that can be very small. How small?
While for sure an infinite number of positive solutions to the original problem exist, the space of the solutions can have measure zero in the N-E dimensional space. This can happen if all the positive solutions of the original problem have one dimension of x = 0. The borders of a diamond will make contact, transforming the diamond of solutions to a line. Of course you will never randomly pick EXACTLY a line in 2D, let alone in 5D.
A obvious idea would be to further reduce the dimensionality from N-E to a smaller number, i.e. to extract directly points from the aforementioned line instead of the square. Algebra is not easy, but I'm working on it. I'm not positive I will be able to solve it.
Note that choosing first one dimension (for example t1), computing the new limits of t2 conditional to the value of t1 extracted and then extract a possible value of t2 in this boundary, while much faster, does not give a uniform probability among all the possible solutions.
I know that the problem is very specific, but even some general ideas or thoughts would be gladly received. I am doubtful if there is some computing technique to extract directly the solution in the diamond...
I have the following question on a practice exam:
I need to use MATLAB to solve it. The problem is, I have not seen a problem like this before and I'm struggling to get started.
I have my 1x1 grid, split into 10x10. I know I can calculate the whole bottom row besides the corners using 1/10 * x*2. I also know I can calculate the entire right row using (1/10)(1+t)^2. However, I cannot figure out how to get enough points to be able to fill in the values for the entire grid. I know it must have something to do with the partial derivatives given in the problem, but I'm not quite sure where they come into play (especially the u_x equation). Can someone help me get a start here?
I don't need the whole solution. Once I have enough points I can easily write a matlab program to solve the rest. Really, I think I just need the x=0 axis solved, then I just fill in the middle of the grid.
I have calculated the bottom row, minus the two corners, to be 0.001, 0.004, 0.009, 0.016, 0.025, 0.036, 0.049, 0.064, 0.081. And similarly, the entire right row is trival to calculate using the given boundry condition. I just can't piece together where to go from there.
Edit: the third boundry condition equation was mistyped. it should read:
u_x(0,t) = 1/5t, NOT u(0,t) = 1/5t
First realise that the equation you have to solve is the linear wave equation, and the numerical scheme you are given can be rewritten as
( u^(n+1)_m - 2u^n_m + u^(n-1)_m )/k^2 = ( u^n_(m-1) - 2u^n_m + u^n_(m+1) )/h^2
where k is the time step and h is the delta x in space.
The reformulated numerical scheme makes clear that the left- and right-hand sides are the second order centred finite difference approximations of u_tt and u_xx respectively.
To solve the problem numerically, however, you need to use the form given to you because it is the explicit update formula that you need to implement numerically: it gives you the solution at time n+1 as a function of the previous two times n and n-1. You need to start from the initial condition and march the solution in time.
Observe that the solution is assigned on the boundaries of the domain (x=0 and x=1), so the values of the discretized solution u^(n)_0 and u^(n)_10 are known for any n (t=n*k). At the nth time step your unknown is the vector [u^(n+1)_1, u^(n+1)_2, ..., u^(n+1)_9].
Observe also that to use the update formula to find the solution at the n+1 step, requires the knowledge of the solution at two previous steps. So, how do you start from n=0 if you need information from two previous times? This is where the initial conditions come into play.
You have the solution at n=0 (t=0), but you also have u_t at t=0. These two pieces of information combined can give you both u^0 and u^1 and get you started.
I would use the following start-up scheme:
u^0_m = u(h*m,0) // initial condition on u
(u^2_m - u^0_m)/(2k) = u_t(h*m,0) // initial condition on u_t
that combined with the numerical scheme used with n=1 gives you everything you need to define a linear system for both u^1_m and u^2_m for m=1,...,9.
To summarize:
--use the start-up scheme to find solution at n=1 and n=2 simultaneously.
--from there on march in time using the numerical scheme you are given.
If you are completely lost check out things like: finite difference schemes, finite difference schemes for advection equations, finite difference schemes for hyperbolic equations, time marching.
EDITING:
For the boundary condition on u_x you typically use the ghost cell method:
Introduce a ghost cell at m=-1, i.e. a fictitious (or auxiliary) grid point that is used to deal with boundary condition, but that is not part of the solution.
The first node m=0 is back into your unknown vector, i.e. you are now working with [u_0 u_1 ... u_9].
Use the left side boundary condition to close the system.
Specifically, by writing down the centered approx of the boundary condition
u^n_(1) - u^n_(-1) = 2*h*u_x(0,k*n)
The above equation allows you to express the solution on the ghost node in terms on the solution on an internal, real node. Therefore you can apply the time-marching numerical scheme (the one you are given) to the m=0 node. (The numerical scheme applied to m=0 would contain contributions from the m=-1 ghost node, but now you have that expressed in terms of the m=1 node.)
I am having some trouble to find or implement an algorithm to find a signal source. The objective of my work is to find the sound emitter position.
To accomplish this I am using three microfones. The technique that I am using is multilateration that is based on the time difference of arrival.
The time difference of arrival between each microfones are found using Cross Correlation of the received signals.
I already implemented the algorithm to find the time difference of arrival, but my problem is more on how multilateration works, it's unclear for me based on my reference, and I couldn't find any other good reference for this that are free/open.
If you have some references on how I can implement a multilateration algorithm, or some other trilateration algorithm that I can use based on time difference of arrival it would be a great help.
Thanks in advance.
The point you are looking for is the intersection of three hyperbolas. I am assuming 2D here since you only use 3 receptors. Technically, you can find a unique 3D solution but as you likely have noise, I assume that if you wanted a 3D result, you would have taken 4 microphones (or more).
The wikipedia page makes some computations for you. They do it in 3D, you just have to set z = 0 and solve for system of equations (7).
The system is overdetermined, so you will want to solve it in the least squares sense (this is the point in using 3 receptors actually).
I can help you with multi-lateration in general.
Basically, if you want a solution in 3d - you have to have at least 4 points and 4 distances from them (2-give you the circle in which is the solution - because that is the intersection between 2 spheres, 3 points give you 2 possible solutions (intersection between 3 spheres) - so, in order to have one solution - you need 4 spheres). So, when you have some points (4+) and the distance between them (there is an easy way to transform the TDOA into the set of equations for just having the length type distances /not time/) you need a way to solve the set of equations. First - you need a cost function (or solution error function, as I call it) which would be something like
err(x,y,z) = sum(i=1..n){sqrt[(x-xi)^2 + (y-yi)^2 + (z-zi)^2] - di}
where x, y, z are coordinates of the current point in the numerical solution and xi, yi, zi and di are the coordinates and distance towards the ith reference point. In order to solve this - my advice is NOT to use Newton/Gauss or Newton methods. You need first and second derivative of the aforementioned function - and those have a finite discontinuation in some points in space - hence that is not a smooth function and these methods won't work. What will work is direct search family of algorithms for optimization of functions (finding minimums and maximums. in our case - you need minimum of the error/cost function).
That should help anyone wanting to find a solution for similar problem.