I am rather confused by how view acts on sparse matrices in Julia:
using LinearAlgebra, SparseArrays, BenchmarkTools
v = SparseVector(1000, [212,554,873], [.3, .4, .3]);
A = sparse(diagm(rand(1000))); # same effect observed for non-diag matrix
B = view(A, :, :);
julia> #btime A*v;
1.536 μs (4 allocations: 23.84 KiB)
julia> #btime B*v;
11.557 ms (5 allocations: 288 bytes)
B*v seems to have much lower memory footprint but is 8000x slower than A*v.
What is going on and what causes these performance differences?
Update June 2021: The missing specialized algorithm for sparse views mentioned below is now implemented, so the performance is much more reasonable these days (Julia 1.6+):
julia> #btime A*v;
2.063 μs (4 allocations: 23.84 KiB)
julia> #btime B*v;
2.836 μs (9 allocations: 25.30 KiB)
And you can see that, yes, indeed, this is using a sparse specialization for *:
julia> #which B*v
*(A::Union{SparseArrays.AbstractSparseMatrixCSC{Tv, Ti}, SubArray{Tv, 2, var"#s832", Tuple{Base.Slice{Base.OneTo{Int64}}, I}, L} where {I<:AbstractUnitRange, var"#s832"<:SparseArrays.AbstractSparseMatrixCSC{Tv, Ti}, L}, LowerTriangular{Tv, var"#s831"} where var"#s831"<:Union{SparseArrays.AbstractSparseMatrixCSC{Tv, Ti}, SubArray{Tv, 2, var"#s832", Tuple{Base.Slice{Base.OneTo{Int64}}, I}, L} where {I<:AbstractUnitRange, var"#s832"<:SparseArrays.AbstractSparseMatrixCSC{Tv, Ti}, L}}, UpperTriangular{Tv, var"#s832"} where var"#s832"<:Union{SparseArrays.AbstractSparseMatrixCSC{Tv, Ti}, SubArray{Tv, 2, var"#s832", Tuple{Base.Slice{Base.OneTo{Int64}}, I}, L} where {I<:AbstractUnitRange, var"#s832"<:SparseArrays.AbstractSparseMatrixCSC{Tv, Ti}, L}}} where {Tv, Ti}, B::AbstractSparseVector{Tv, Ti} where {Tv, Ti}) in SparseArrays at /Users/mbauman/Julia/release-1.6/usr/share/julia/stdlib/v1.6/SparseArrays/src/linalg.jl:163
Previously, that method wasn't implemented, meaning that the below occurred:
Old answer: It's not 8x slower, it's 8000x slower. The reason is because Julia uses multiple dispatch to use specialized algorithms that can exploit the sparse storage of the matrix and vector to completely avoid working on sections of the array it knows will just be zero. You can see which algorithm is getting called with #which:
julia> #which A*v
*(A::SparseArrays.AbstractSparseMatrixCSC, x::AbstractSparseArray{Tv,Ti,1} where Ti where Tv) in SparseArrays at /Users/mbauman/Julia/master/usr/share/julia/stdlib/v1.4/SparseArrays/src/sparsevector.jl:1722
julia> #which B*v
*(A::AbstractArray{T,2}, x::AbstractArray{S,1}) where {T, S} in LinearAlgebra at /Users/mbauman/Julia/master/usr/share/julia/stdlib/v1.4/LinearAlgebra/src/matmul.jl:50
The former is using a highly specialized sparse implementation, whereas the latter is using a slightly more general interface that can also support views. Now, ideally we'd detect the trivial cases like view(A, :, :) and specialize those to effectively be the same, too, but note that in general views may not preserve the sparsity and structure of the matrix:
julia> view(A, ones(Int, 1000), ones(Int, 1000))
1000×1000 view(::SparseMatrixCSC{Float64,Int64}, [1, 1, 1, 1, 1, 1, 1, 1, 1, 1 … 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1 … 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) with eltype Float64:
0.306159 0.306159 0.306159 0.306159 … 0.306159 0.306159 0.306159
0.306159 0.306159 0.306159 0.306159 0.306159 0.306159 0.306159
0.306159 0.306159 0.306159 0.306159 0.306159 0.306159 0.306159
⋮ ⋱
0.306159 0.306159 0.306159 0.306159 0.306159 0.306159 0.306159
0.306159 0.306159 0.306159 0.306159 0.306159 0.306159 0.306159
0.306159 0.306159 0.306159 0.306159 … 0.306159 0.306159 0.306159
Not directly related to the question, but for all who want to take a look/view/examine a matrix/array in sparse format, you can use display(YourSparseArray).
This answer is motivated by the relatively many views of this question, which I assume are not all concerned with the performance of view, but are rather interested in how to see a sparse matrix nicely.
Related
I need something like MatLab's permute(A, order) or Mathematica's Transpose[list, {n,...}] for Clojure vectors (of vectors of...).
Swapping the first and second dimension is easy: (apply mapv vector x) but I'd like an inteface like the functions mentioned above where I can give a new order of dimensions, i.e. (permute x [1 0]) would be that transposition, (permute x [0 1 2]) identity for 3 dimensions etc.
Example for a Mathematica list of lists of lists:
Array[a, {1, 2, 3}]
{{{a[1, 1, 1], a[1, 1, 2], a[1, 1, 3]},
{a[1, 2, 1], a[1, 2, 2], a[1, 2, 3]}}}
Transpose[%, {3, 2, 1}]
{{{a[1, 1, 1]}, {a[1, 2, 1]}},
{{a[1, 1, 2]}, {a[1, 2, 2]}},
{{a[1, 1, 3]}, {a[1, 2, 3]}}}
An existing library would be best of course, but any pointers would be appreciated.
Numeric libraries usually just wrap a one-dimensional array and adjust the offsets, but I want actual lists.
In Wolfram Mathematica，there is a function called Permutations(http://reference.wolfram.com/mathematica/ref/Permutations.html).
It can gives all permutations containing exactly n elements.
For example: Permutations[{1,2,3,4}, {2}] gives
{{1, 2}, {1, 3}, {1, 4}, {2, 1}, {2, 3}, {2, 4}, {3, 1}, {3, 2}, {3,
4}, {4, 1}, {4, 2}, {4, 3}}
I know Matlab have a similar function perms, but it only receive one parameter and gives all possible permutations. Is there other function like Mathematica's Permutations[list,{n}]？
If order doesn't matter, take a look at nchoosek.
If it does (which seems to be the case), there is an inefficient, ugly, but non-toolbox dependent one-liner:
>> unique(builtin('_paren', perms(1:4), :,1:2), 'rows')
ans =
1 2
1 3
1 4
2 1
2 3
2 4
3 1
3 2
3 4
4 1
4 2
4 3
(which is really a hacked two-liner).
I'd suggest you just use combnk from the Statistics toolbox.
You can get all the combinations of your desired size using nchoosek and then permute them using perms. That means for permutations of length k from vector v you can use
A=nchoosek(v,k);
P=reshape(A(:,perms(1:k)), [], k);
Note that the rows of P will not be sorted and you can use sortrows to sort it:
P=sortrows(reshape(A(:,perms(1:k)), [], k));
Using your example
v = 1:4;
k = 2;
A=nchoosek(v,k);
P=sortrows(reshape(A(:,perms(1:k)), [], k))
returns:
P =
1 2
1 3
1 4
2 1
2 3
2 4
3 1
3 2
3 4
4 1
4 2
4 3
I have a 20000 x 185 x 5 tensor, which looks like
{{{a1_1,a2_1,a3_1,a4_1,a5_1},{b1_1,b2_1,b3_1,b4_1,b5_1}...
(continue for 185 times)}
{{a1_2,a2_2,a3_2,a4_2,a5_2},{b1_2,b2_2,b3_2,b4_2,b5_2}...
...
...
...
{{a1_20000,a2_20000,a3_20000,a4_20000,a5_20000},
{b1_20000,b2_20000,b3_20000,b4_20000,b5_20000}... }}
The 20000 represents iteration number, the 185 represents individuals, and each individual has 5 attributes. I need to construct a 185 x 5 matrix that stores the mean value for each individual's 5 attributes, averaged across the 20000 iterations.
Not sure what the best way to do this is. I know Mean[ ] works on matrices, but with a Tensor, the derived values might not be what I need. Also, Mathematica ran out of memory if I tried to do Mean[tensor]. Please provide some help or advice. Thank you.
When in doubt, drop the size of the dimensions. (You can still keep them distinct to easily see where things end up.)
(* In[1]:= *) data = Array[a, {4, 3, 2}]
(* Out[1]= *) {{{a[1, 1, 1], a[1, 1, 2]}, {a[1, 2, 1],
a[1, 2, 2]}, {a[1, 3, 1], a[1, 3, 2]}}, {{a[2, 1, 1],
a[2, 1, 2]}, {a[2, 2, 1], a[2, 2, 2]}, {a[2, 3, 1],
a[2, 3, 2]}}, {{a[3, 1, 1], a[3, 1, 2]}, {a[3, 2, 1],
a[3, 2, 2]}, {a[3, 3, 1], a[3, 3, 2]}}, {{a[4, 1, 1],
a[4, 1, 2]}, {a[4, 2, 1], a[4, 2, 2]}, {a[4, 3, 1], a[4, 3, 2]}}}
(* In[2]:= *) Dimensions[data]
(* Out[2]= *) {4, 3, 2}
(* In[3]:= *) means = Mean[data]
(* Out[3]= *) {
{1/4 (a[1, 1, 1] + a[2, 1, 1] + a[3, 1, 1] + a[4, 1, 1]),
1/4 (a[1, 1, 2] + a[2, 1, 2] + a[3, 1, 2] + a[4, 1, 2])},
{1/4 (a[1, 2, 1] + a[2, 2, 1] + a[3, 2, 1] + a[4, 2, 1]),
1/4 (a[1, 2, 2] + a[2, 2, 2] + a[3, 2, 2] + a[4, 2, 2])},
{1/4 (a[1, 3, 1] + a[2, 3, 1] + a[3, 3, 1] + a[4, 3, 1]),
1/4 (a[1, 3, 2] + a[2, 3, 2] + a[3, 3, 2] + a[4, 3, 2])}
}
(* In[4]:= *) Dimensions[means]
(* Out[4]= *) {3, 2}
Mathematica ran out of memory if I tried to do Mean[tensor]
This is probably because intermediate results are larger than the final result. This is likely if the elements are not type Real or Integer. Example:
a = Tuples[{x, Sqrt[y], z^x, q/2, Mod[r, 1], Sin[s]}, {2, 4}];
{MemoryInUse[], MaxMemoryUsed[]}
b = Mean[a];
{MemoryInUse[], MaxMemoryUsed[]}
{109125576, 124244808}
{269465456, 376960648}
If they are, and are in packed array form, perhaps the elements are such that the array in unpacked during processing.
Here is an example where the tensor is a packed array of small numbers, and unpacking does not occur.
a = RandomReal[99, {20000, 185, 5}];
PackedArrayQ[a]
{MemoryInUse[], MaxMemoryUsed[]}
b = Mean[a];
{MemoryInUse[], MaxMemoryUsed[]}
True
{163012808, 163016952}
{163018944, 163026688}
Here is the same size of tensor with very large numbers.
a = RandomReal[$MaxMachineNumber, {20000, 185, 5}];
Developer`PackedArrayQ[a]
{MemoryInUse[], MaxMemoryUsed[]}
b = Mean[a];
{MemoryInUse[], MaxMemoryUsed[]}
True
{163010680, 458982088}
{163122608, 786958080}
To elaborate a little on the other answers, there is no reason to expect Mathematica functions to operate materially differently on tensors than matrices because Mathemetica considers them both to be nested Lists, that are just of different nesting depth. How functions behave with lists depends on whether they're Listable, which you can check using Attributes[f], where fis the function you are interested in.
Your data list's dimensionality isn't actually that big in the scheme of things. Without seeing your actual data it is hard to be sure, but I suspect the reason you are running out of memory is that some of your data is non-numerical.
I don't know what you're doing incorrectly (your code will help). But Mean[] already works as you want it to.
a = RandomReal[1, {20000, 185, 5}];
b = Mean#a;
Dimensions#b
Out[1]= {185, 5}
You can even check that this is correct:
{Max#b, Min#b}
Out[2]={0.506445, 0.494061}
which is the expected value of the mean given that RandomReal uses a uniform distribution by default.
Assume you have the following data :
a = Table[RandomInteger[100], {i, 20000}, {j, 185}, {k, 5}];
In a straightforward manner You can find a table which stores the means of a[[1,j,k]],a[[2,j,k]],...a[[20000,j,k]]:
c = Table[Sum[a[[i, j, k]], {i, Length[a]}], {j, 185}, {k, 5}]/
Length[a] // N; // Timing
{37.487, Null}
or simply :
d = Total[a]/Length[a] // N; // Timing
{0.702, Null}
The second way is about 50 times faster.
c == d
True
To extend on Brett's answer a bit, when you call Mean on a n-dimensional tensor then it averages over the first index and returns an n-1 dimensional tensor:
a = RandomReal[1, {a1, a2, a3, ... an}];
Dimensions[a] (* This would have n entries in it *)
b = Mean[a];
Dimensions[b] (* Has n-1 entries, where averaging was done over the first index *)
In the more general case where you may wish to average over the i-th argument, you would have to transpose the data around first. For example, say you want to average the 3nd of 5 dimensions. You would need the 3rd element first, followed by the 1st, 2nd, 4th, 5th.
a = RandomReal[1, {5, 10, 2, 40, 10}];
b = Transpose[a, {2, 3, 4, 1, 5}];
c = Mean[b]; (* Now of dimensions {5, 10, 40, 10} *)
In other words, you would make a call to Transpose where you placed the i-th index as the first tensor index and moved everything before it ahead one. Anything that comes after the i-th index stays the same.
This tends to come in handy when your data comes in odd formats where the first index may not always represent different realizations of a data sample. I've had this come up, for example, when I had to do time averaging of large wind data sets where the time series came third (!) in terms of the tensor representation that was available.
You could imagine the generalizedTenorMean would look something like this then:
Clear[generalizedTensorMean];
generalizedTensorMean[A_, i_] :=
Module[{n = Length#Dimensions#A, ordering},
ordering =
Join[Table[x, {x, 2, i}], {1}, Table[x, {x, i + 1, n}]];
Mean#Transpose[A, ordering]]
This reduces to the plain-old-mean when i == 1. Try it out:
A = RandomReal[1, {2, 4, 6, 8, 10, 12, 14}];
Dimensions#A (* {2, 4, 6, 8, 10, 12, 14} *)
Dimensions#generalizedTensorMean[A, 1] (* {4, 6, 8, 10, 12, 14} *)
Dimensions#generalizedTensorMean[A, 7] (* {2, 4, 6, 8, 10, 12} *)
On a side note, I'm surprised that Mathematica doesn't support this by default. You don't always want to average over the first level of a list.
I think Mathematica is biased towards rows not columns.
Given a matrix, to insert a row seems to be easy, just use Insert[]
(a = {{1, 2, 3}, {4, 0, 8}, {7 , 8, 0}}) // MatrixForm
1 2 3
4 0 8
7 8 0
row = {97, 98, 99};
(newa = Insert[a, row, 2]) // MatrixForm
1 2 3
97 98 99
4 0 8
7 8 0
But to insert a column, after some struggle, I found 2 ways, I show below, and would like to ask the experts here if they see a shorter and more direct way (Mathematica has so many commands, and I could have overlooked one that does this sort of thing in much direct way), as I think the methods I have now are still too complex for such a basic operation.
First method
Have to do double transpose:
a = {{1, 2, 3}, {4, 0, 8}, {7 , 8, 0}}
column = {97, 98, 99}
newa = Transpose[Insert[Transpose[a], column, 2]]
1 97 2 3
4 98 0 8
7 99 8 0
Second method
Use SparseArray, but need to watch out for index locations. Kinda awkward for doing this:
(SparseArray[{{i_, j_} :> column[[i]] /; j == 2, {i_, j_} :> a[[i, j]] /; j == 1,
{i_, j_} :> a[[i, j - 1]] /; j > 1}, {3, 4}]) // Normal
1 97 2 3
4 98 0 8
7 99 8 0
The question is: Is there a more functional way, that is little shorter than the above? I could ofcourse use one of the above, and wrap the whole thing with a function, say insertColumn[...] to make it easy to use. But wanted to see if there is an easier way to do this than what I have.
For reference, this is how I do this in Matlab:
EDU>> A=[1 2 3;4 0 8;7 8 0]
A =
1 2 3
4 0 8
7 8 0
EDU>> column=[97 98 99]';
EDU>> B=[A(:,1) column A(:,2:end)]
B =
1 97 2 3
4 98 0 8
7 99 8 0
Your double Transpose method seems fine. For very large matrices, this will be 2-3 times faster:
MapThread[Insert, {a, column, Table[2, {Length[column]}]}]
If you want to mimic your Matlab way, the closest is probably this:
ArrayFlatten[{{a[[All, ;; 1]], Transpose[{column}], a[[All, 2 ;;]]}}]
Keep in mind that insertions require making an entire copy of the matrix. So, if you plan to build a matrix this way, it is more efficient to preallocate the matrix (if you know its size) and do in-place modifications through Part instead.
You can use Join with a level specification of 2 along with Partition in subsets of size 1:
a = {{1, 2, 3}, {4, 0, 8}, {7 , 8, 0}}
column = {97, 98, 99}
newa = Join[a,Partition[column,1],2]
I think I'd do it the same way, but here are some other ways of doing it:
-With MapIndexed
newa = MapIndexed[Insert[#1, column[[#2[[1]]]], 2] &, a]
-With Sequence:
newa = a;
newa[[All, 1]] = Transpose[{newa[[All, 1]], column}];
newa = Replace[a, List -> Sequence, {3}, Heads -> True]
Interestingly, this would seem to be a method that works 'in place', i.e. it wouldn't really require a matrix copy as stated in Leonid's answer and if you print the resulting matrix it apparently works as a charm.
However, there's a big catch. See the problems with Sequence in the mathgroup discussion "part assigned sequence behavior puzzling".
I usually just do like this:
In: m0 = ConstantArray[0, {3, 4}];
m0[[All, {1, 3, 4}]] = {{1, 2, 3}, {4, 0, 8}, {7, 8, 0}};
m0[[All, 2]] = {97, 98, 99}; m0
Out:
{{1, 97, 2, 3}, {4, 98, 0, 8}, {7, 99, 8, 0}}
I don't know how it compare in terms of efficiency.
I originally posted this as a comment (now deleted)
Based on a method given by user656058 in this question (Mathematica 'Append To' Function Problem) and the reply of Mr Wizard, the following alternative method of adding a column to a matrix, using Table and Insert, may be gleaned:
(a = {{1, 2, 3}, {4, 0, 8}, {7, 8, 0}});
column = {97, 98, 99};
Table[Insert[a[[i]], column[[i]], 2], {i, 3}] // MatrixForm
giving
Similarly, to add a column of zeros (say):
Table[Insert[#[[i]], 0, 2], {i, Dimensions[#][[1]]}] & # a
As noted in the comments above, Janus has drawn attention to the 'trick' of adding a column of zeros by the ArrayFlatten method (see here)
ArrayFlatten[{{Take[#, All, 1], 0, Take[#, All, -2]}}] & #
a // MatrixForm
Edit
Perhaps simpler, at least for smaller matrices
(Insert[a[[#]], column[[#]], 2] & /# Range[3]) // MatrixForm
or, to insert a column of zeros
Insert[a[[#]], 0, 2] & /# Range[3]
Or, a little more generally:
Flatten#Insert[a[[#]], {0, 0}, 2] & /# Range[3] // MatrixForm
May also easily be adapted to work with Append and Prepend, of course.
I would like to apply a function to a specific column of a table. Say to the i-th column of a (m x n) table. Actually I just want to multiply all elements in that column with a scalar, but the application of a general function would be fine as well.
It probably just needs some Map or MapAt command, maybe combined with a Transpose in order to apply to rows instead of columns - but I can't figure out the correct syntax for addressing an entire column (or row)..
Any hints would be highly appreciated.
Here's a 3x3 table:
In[1]:= table = {{1,2,3}, {4,5,6}, {7,8,9}}
Out[1]= {{1, 2, 3}, {4, 5, 6}, {7, 8, 9}}
In[2]:= table//TableForm
Out[2]//TableForm= 1 2 3
4 5 6
7 8 9
Column 2 is table[[All, 2]]:
In[3]:= table[[All, 2]]
Out[3]= {2, 5, 8}
So it's a simple matter to modify only that column:
In[4]:= table[[All, 2]] = 10 * table[[All, 2]]
Out[4]= {20, 50, 80}
In[5]:= table//TableForm
Out[5]//TableForm= 1 20 3
4 50 6
7 80 9
For example,
ranfunc=Function[{f,mat, n},Transpose[MapAt[f /# # &, Transpose[mat], n]]]
will apply f to each element of the nth column of mat. So, for instance,
ranfunc[Sin[Cos[#]] &, {{1, 2, 3}, {a, b, c}, {\[Alpha], \[Beta], \[Gamma]}}, 2]
will apply Sin[Cos[#]]& to each element of the second column, while
ranfunc[s*# &, {{1, 2, 3}, {a, b, c}, {\[Alpha], \[Beta], \[Gamma]}},2]
will multiply each element on the second column by s
One versatile approach is to use ReplacePart
For example, to apply f to column 3 of mat:
(mat = Array[Subscript[a, ##] &, {4, 4}]) // MatrixForm
(newmat = ReplacePart[#, 3 -> f ##[[3]] ] & /# mat) // MatrixForm
The following multiplies each entry by 10:
(newmat2 = ReplacePart[#, 3 -> 10 #[[3]] ] & /# mat) // MatrixForm
However, a 'quick' way to do this it as follows:
mat[[All, 3 ]] *= 10
(Unlike the first method, all entries in column 3 of mat are now modified. It is not clear whether you want to modify the existing table, or to create a new table with modifications, leaving the original intact)
MapAt function accepts the following Part specification:
MapAt[f, mat, {All, 3}]
to apply 'f' to column 3 of your matrix.
Another compact solution I found is using Map and MapAt:
Here is an example Matrix:
mat={{3,4,5},{4,7,5},{2,6,7},{3,6,9}}
Now apply the function f to the second column:
n=2;
Map[MapAt[f,#,n]&,mat]
The result is then:
{{3,f[4],5},{4,f[7],5},{2,f[6],7},{3,f[6],9}}