I have 5000 vectors; each one has a length of 10000 (values)
I want to fill a matrix column by column such that each columns represents a vector.
5000 columns and 10000 rows.
it did't work in this way . I have this structure:
vector = Vector()
for i in 1:5000
println(vector[i])
end
for example
julia> vector[502]
10000-element Array{Float64,1}: -3.0 1.0 . . . -2.0
when I do
a = zeros(10000,0)
for v in vector
a = hcat(a,v)
end
it doesn't work.
This is almost identical to the question you asked previously on populating a matrix row by row. The solution likewise is:
A = [rand(10000) for idx in 1:5000];
B = hcat(A...);
a = zeros(10000,0)
vector = #whatever is here
for v in vector
a = hcat(a,v)
end
Note
If you get a
**ERROR: ArgumentError: number of rows of each array must match**
the loop must be
a = hcat(a,v')
Related
I create an optimization model in Julia-JuMP using the symbolic variables and constraints e.g. below
using JuMP
using CPLEX
# model
Mod = Model(CPLEX.Optimizer)
# sets
I = 1:2;
# Variables
x = #variable( Mod , [I] , base_name = "x" )
y = #variable( Mod , [I] , base_name = "y" )
# constraints
Con1 = #constraint( Mod , [i in I] , 2 * x[i] + 3 * y[i] <= 100 )
# objective
ObjFun = #objective( Mod , Max , sum( x[i] + 2 * y[i] for i in I) ) ;
# solve
optimize!(Mod)
I guess JuMP creates the problem in the form minimize c'*x subj to Ax < b before it is passes to the solver CPLEX. I want to extract the matrices A,b,c. In the above example I would expect something like:
A
2×4 Array{Int64,2}:
2 0 3 0
0 2 0 3
b
2-element Array{Int64,1}:
100
100
c
4-element Array{Int64,1}:
1
1
2
2
In MATLAB the function prob2struct can do this https://www.mathworks.com/help/optim/ug/optim.problemdef.optimizationproblem.prob2struct.html
In there a JuMP function that can do this?
This is not easily possible as far as I am aware.
The problem is stored in the underlying MathOptInterface (MOI) specific data structures. For example, constraints are always stored as MOI.AbstractFunction - in - MOI.AbstractSet. The same is true for the MOI.ObjectiveFunction. (see MOI documentation: https://jump.dev/MathOptInterface.jl/dev/apimanual/#Functions-1)
You can however, try to recompute the objective function terms and the constraints in matrix-vector-form.
For example, assuming you still have your JuMP.Model Mod, you can examine the objective function closer by typing:
using MathOptInterface
const MOI = MathOptInterface
# this only works if you have a linear objective function (the model has a ScalarAffineFunction as its objective)
obj = MOI.get(Mod, MOI.ObjectiveFunction{MOI.ScalarAffineFunction{Float64}}())
# take a look at the terms
obj.terms
# from this you could extract your vector c
c = zeros(4)
for term in obj.terms
c[term.variable_index.value] = term.coefficient
end
#show(c)
This gives indeed: c = [1.;1.;2.;2.].
You can do something similar for the underlying MOI.constraints.
# list all the constraints present in the model
cons = MOI.get(Mod, MOI.ListOfConstraints())
#show(cons)
in this case we only have one type of constraint, i.e. (MOI.ScalarAffineFunction{Float64} in MOI.LessThan{Float64})
# get the constraint indices for this combination of F(unction) in S(et)
F = cons[1][1]
S = cons[1][2]
ci = MOI.get(Mod, MOI.ListOfConstraintIndices{F,S}())
You get two constraint indices (stored in the array ci), because there are two constraints for this combination F - in - S.
Let's examine the first one of them closer:
ci1 = ci[1]
# to get the function and set corresponding to this constraint (index):
moi_backend = backend(Mod)
f = MOI.get(moi_backend, MOI.ConstraintFunction(), ci1)
f is again of type MOI.ScalarAffineFunction which corresponds to one row a1 in your A = [a1; ...; am] matrix. The row is given by:
a1 = zeros(4)
for term in f.terms
a1[term.variable_index.value] = term.coefficient
end
#show(a1) # gives [2.0 0 3.0 0] (the first row of your A matrix)
To get the corresponding first entry b1 of your b = [b1; ...; bm] vector, you have to look at the constraint set of that same constraint index ci1:
s = MOI.get(moi_backend, MOI.ConstraintSet(), ci1)
#show(s) # MathOptInterface.LessThan{Float64}(100.0)
b1 = s.upper
I hope this gives you some intuition on how the data is stored in MathOptInterface format.
You would have to do this for all constraints and all constraint types and stack them as rows in your constraint matrix A and vector b.
Use the following lines:
Pkg.add("NLPModelsJuMP")
using NLPModelsJuMP
nlp = MathOptNLPModel(model) # the input "< model >" is the name of the model you created by JuMP before with variables and constraints (and optionally the objective function) attached to it.
x = zeros(nlp.meta.nvar)
b = NLPModelsJuMP.grad(nlp, x)
A = Matrix(NLPModelsJuMP.jac(nlp, x))
I didn't try it myself. But the MathProgBase package seems to be able to provide A, b, and c in matrix form.
I want to perform matrix related operations such as multiplication, transpose and inversion of a matrix. I could find out matrix support in Lua here
I have a table which I want to convert to matrix. The table has following structure-
for i=1,myTableSize[1],1 do
str=''
for j=1,myTableSize[2],1 do
if #str~=0 then
str=str..', '
end
str=str..string.format("%.1e",myTable[(j-1)*myTableSize[1]+i])
end
print(str)
end
I am looking for something like myMatrix=matrix(myTable) or myMatrix=matrix.init(myTable), which is compatible with Lua Matrix.
-
Thanks
Try (not tested)
local function tableToMatrix(table, rows cols)
local myMatrix = matrix:new(rows, cols) -- function returns matrix of size rows x cols
for i=1, rows do
for j=1, cols do
matrix.setelement(myMatrix, i, j, table[(i - 1) * cols + j] )
end
end
return matrix
end
I have 200 vectors; each one has a length of 10000.
I want to fill a matrix such that each line represents a vector.
If your vectors are already stored in an array then you can use vcat( ) here:
A = [rand(10000)' for idx in 1:200]
B = vcat(A...)
Julia stores matrices in column-major order so you are going to have to adapt a bit to that
If you have 200 vectors of length 100000 you should make first an empty vector, a = [], this will be your matrix
Then you have to vcat the first vector to your empty vector, like so
v = your vectors, however they are defined
a = []
a = vcat(a, v[1])
Then you can iterate through vectors 2:200 by
for i in 2:200
a = hcat(a,v[i])
end
And finally transpose a
a = a'
Alternatively, you could do
a = zeros(200,10000)
for i in 1:length(v)
a[i,:] = v[i]
end
but I suppose that wont be as fast, if performance is at all an issue, because as I said, julia stores in column major order so access will be slower
EDIT from reschu's comment
a = zeros(10000,200)
for i in 1:length(v)
a[:,i] = v[i]
end
a = a'
for an input matrix
in = [1 1;
1 2;
1 3;
1 4;
2 5;
2 6;
2 7;
3 8;
3 9;
3 10;
3 11];
i want to get the output matrix
out = [1 5 8;
2 6 9;
3 7 10;
4 0 11];
meaning i want to reshape the second input column into an output matrix, where all values corresponding to one value in the first input column are written into one column of the output matrix.
As there can be different numbers of entries for each value in the first input column (here 4 values for "1" and "3", but only 3 for "2"), the normal reshape function is not applicable. I need to pad all columns to the maximum number of rows.
Do you have an idea how to do this matlab-ish?
The second input column can only contain positive numbers, so the padding values can be 0, -x, NaN, ...
The best i could come up with is this (loop-based):
maxNumElem = 0;
for i=in(1,1):in(end,1)
maxNumElem = max(maxNumElem,numel(find(in(:,1)==i)));
end
out = zeros(maxNumElem,in(end,1)-in(1,1));
for i=in(1,1):in(end,1)
tmp = in(in(:,1)==i,2);
out(1:length(tmp),i) = tmp;
end
Either of the following approaches assumes that column 1 of in is sorted, as in the example. If that's not the case, apply this initially to sort in according to that criterion:
in = sortrows(in,1);
Approach 1 (using accumarray)
Compute the required number of rows, using mode;
Use accumarray to gather the values corresponding to each column, filled with zeros at the end. The result is a cell;
Concatenate horizontally the contents of all cells.
Code:
[~, n] = mode(in(:,1)); %//step 1
out = accumarray(in(:,1), in(:,2), [], #(x){[x; zeros(n-numel(x),1)]}); %//step 2
out = [out{:}]; %//step 3
Alternatively, step 1 could be done with histc
n = max(histc(in(:,1), unique(in(:,1)))); %//step 1
or with accumarray:
n = max(accumarray(in(:,1), in(:,2), [], #(x) numel(x))); %//step 1
Approach 2 (using sparse)
Generate a row-index vector using this answer by #Dan, and then build your matrix with sparse:
a = arrayfun(#(x)(1:x), diff(find([1,diff(in(:,1).'),1])), 'uni', 0); %//'
out = full(sparse([a{:}], in(:,1), in(:,2)));
Introduction to proposed solution and Code
Proposed here is a bsxfun based masking approach that uses the binary operators available as builtins for use with bsxfun and as such I would consider this very appropriate for problems like this. Of course, you must also be aware that bsxfun is a memory hungry tool. So, it could pose a threat if you are dealing with maybe billions of elements depending also on the memory available for MATLAB's usage.
Getting into the details of the proposed approach, we get the counts of each ID from column-1 of the input with histc. Then, the magic happens with bsxfun + #le to create a mask of positions in the output array (initialized by zeros) that are to be filled by the column-2 elements from input. That's all you need to tackle the problem with this approach.
Solution Code
counts = histc(in(:,1),1:max(in(:,1)))'; %//' counts of each ID from column1
max_counts = max(counts); %// Maximum counts for each ID
mask = bsxfun(#le,[1:max_counts]',counts); %//'# mask of locations where
%// column2 elements are to be placed
out = zeros(max_counts,numel(counts)); %// Initialize the output array
out(mask) = in(:,2); %// place the column2 elements in the output array
Benchmarking (for performance)
The benchmarking presented here compares the proposed solution in this post against the various methods presented in Luis's solution. This skips the original loopy approach presented in the problem as it appeared to be very slow for the input generated in the benchmarking code.
Benchmarking Code
num_ids = 5000;
counts_each_id = randi([10 100],num_ids,1);
num_runs = 20; %// number of iterations each approach is run for
%// Generate random input array
in = [];
for k = 1:num_ids
in = [in ; [repmat(k,counts_each_id(k),1) rand(counts_each_id(k),1)]];
end
%// Warm up tic/toc.
for k = 1:50000
tic(); elapsed = toc();
end
disp('------------- With HISTC + BSXFUN Masking approach')
tic
for iter = 1:num_runs
counts = histc(in(:,1),1:max(in(:,1)))';
max_counts = max(counts);
out = zeros(max_counts,numel(counts));
out(bsxfun(#le,[1:max_counts]',counts)) = in(:,2);
end
toc
clear counts max_counts out
disp('------------- With MODE + ACCUMARRAY approach')
tic
for iter = 1:num_runs
[~, n] = mode(in(:,1)); %//step 1
out = accumarray(in(:,1), in(:,2), [], #(x){[x; zeros(n-numel(x),1)]}); %//step 2
out = [out{:}];
end
toc
clear n out
disp('------------- With HISTC + ACCUMARRAY approach')
tic
for iter = 1:num_runs
n = max(histc(in(:,1), unique(in(:,1))));
out = accumarray(in(:,1), in(:,2), [], #(x){[x; zeros(n-numel(x),1)]}); %//step 2
out = [out{:}];
end
toc
clear n out
disp('------------- With ARRAYFUN + Sparse approach')
tic
for iter = 1:num_runs
a = arrayfun(#(x)(1:x), diff(find([1,diff(in(:,1).'),1])), 'uni', 0); %//'
out = full(sparse([a{:}], in(:,1), in(:,2)));
end
toc
clear a out
Results
------------- With HISTC + BSXFUN Masking approach
Elapsed time is 0.598359 seconds.
------------- With MODE + ACCUMARRAY approach
Elapsed time is 2.452778 seconds.
------------- With HISTC + ACCUMARRAY approach
Elapsed time is 2.579482 seconds.
------------- With ARRAYFUN + Sparse approach
Elapsed time is 1.455362 seconds.
slightly better, but still uses a loop :(
out=zeros(4,3);%set to zero matrix
for i = 1:max(in(:,1)); %find max in column 1, and loop for that number
ind = find(in(:,1)==i); %
out(1: size(in(ind,2),1),i)= in(ind,2);
end
don't know if you can avoid the loop...
Suppose I have a matrix A and I sort the rows of this matrix. How do I replicate the same ordering on a matrix B (same size of course)?
E.g.
A = rand(3,4);
[val ind] = sort(A,2);
B = rand(3,4);
%// Reorder the elements of B according to the reordering of A
This is the best I've come up with
m = size(A,1);
B = B(bsxfun(#plus,(ind-1)*m,(1:m)'));
Out of curiosity, any alternatives?
Update: Jonas' excellent solution profiled on 2008a (XP):
n = n
0.048524 1.4632 1.4791 1.195 1.0662 1.108 1.0082 0.96335 0.93155 0.90532 0.88976
n = 2m
0.63202 1.3029 1.1112 1.0501 0.94703 0.92847 0.90411 0.8849 0.8667 0.92098 0.85569
It just goes to show that loops aren't anathema to MATLAB programmers anymore thanks to JITA (perhaps).
A somewhat clearer way to do this is to use a loop
A = rand(3,4);
B = rand(3,4);
[sortedA,ind] = sort(A,2);
for r = 1:size(A,1)
B(r,:) = B(r,ind(r,:));
end
Interestingly, the loop version is faster for small (<12 rows) and large (>~700 rows) square arrays (r2010a, OS X). The more columns there are relative to rows, the better the loop performs.
Here's the code I quickly hacked up for testing:
siz = 10:100:1010;
tt = zeros(100,2,length(siz));
for s = siz
for k = 1:100
A = rand(s,1*s);
B = rand(s,1*s);
[sortedA,ind] = sort(A,2);
tic;
for r = 1:size(A,1)
B(r,:) = B(r,ind(r,:));
end,tt(k,1,s==siz) = toc;
tic;
m = size(A,1);
B = B(bsxfun(#plus,(ind-1)*m,(1:m).'));
tt(k,2,s==siz) = toc;
end
end
m = squeeze(mean(tt,1));
m(1,:)./m(2,:)
For square arrays
ans =
0.7149 2.1508 1.2203 1.4684 1.2339 1.1855 1.0212 1.0201 0.8770 0.8584 0.8405
For twice as many columns as there are rows (same number of rows)
ans =
0.8431 1.2874 1.3550 1.1311 0.9979 0.9921 0.8263 0.7697 0.6856 0.7004 0.7314
Sort() returns the index along the dimension you sorted on. You can explicitly construct indexes for the other dimensions that cause the rows to remain stable, and then use linear indexing to rearrange the whole array.
A = rand(3,4);
B = A; %// Start with same values so we can programmatically check result
[A2 ix2] = sort(A,2);
%// ix2 is the index along dimension 2, and we want dimension 1 to remain unchanged
ix1 = repmat([1:size(A,1)]', [1 size(A,2)]); %//'
%// Convert to linear index equivalent of the reordering of the sort() call
ix = sub2ind(size(A), ix1, ix2)
%// And apply it
B2 = B(ix)
ok = isequal(A2, B2) %// confirm reordering
Can't you just do this?
[val ind]=sort(A);
B=B(ind);
It worked for me, unless I'm understanding your problem wrong.