Let's suppose we have a function, that gives us following:
julia> ExampleFunction(Number1, Number2)
5-element Vector{Tuple{Int64, Int64}}:
(2, 2)
(2, 3)
(3, 3)
(3, 2)
(4, 2)
I would like to convert Vector{Tuple{Int64, Int64}}
into a matrix, or in my case I would like to convert it into a 5x2 Matrix.
It's not entirely clear from your question how the 5-element vector of tuples of length 2, which has 10 elements in total, should be converted to a 3x2 matrix which holds 6 elements, but assuming that you meant 5x2 here's one way of doing it:
julia> x = [(1,2), (2, 3), (3, 4)]
3-element Vector{Tuple{Int64, Int64}}:
(1, 2)
(2, 3)
(3, 4)
julia> hcat(first.(x), last.(x))
3×2 Matrix{Int64}:
1 2
2 3
3 4
EDIT: As Phips mentioned an alternative below, here's a quick benchmark on Julia 1.7beta3, Windows 10 - I've thrown in a loop version as well, as it always makes sense to try a straightforward loop in Julia:
julia> convert_to_tuple1(x) = hcat(first.(x), last.(x))
convert_to_tuple1 (generic function with 1 method)
julia> convert_to_tuple2(x) = PermutedDimsArray(reshape(foldl(append!, x, init = Int[]), 2, :), (2, 1))
convert_to_tuple2 (generic function with 1 method)
julia> function convert_to_tuple3(x)
out = Matrix{eltype(x[1])}(undef, length(x), length(x[1]))
for i ∈ 1:length(x)
for j ∈ 1:length(x[1])
out[i, j] = x[i][j]
end
end
out
end
convert_to_tuple3 (generic function with 1 method)
julia> xs = [(rand(1:10), rand(1:10)) for _ ∈ 1:1_000_000];
julia> using BenchmarkTools
julia> #btime convert_to_tuple1($xs);
15.789 ms (6 allocations: 30.52 MiB)
julia> #btime convert_to_tuple2($xs);
22.067 ms (21 allocations: 18.91 MiB)
julia> #btime convert_to_tuple3($xs);
7.286 ms (2 allocations: 15.26 MiB)
(further edited to add $ for interpolation of xs into the benchmark)
The fastest code would be a non-copying one. In example from the other post this would reduce times from milliseconds to nanoseconds:
julia> xs
1000000-element Vector{Tuple{Int64, Int64}}:
(5, 1)
(4, 3)
⋮
(1, 4)
(9, 2)
julia> #btime reshape(reinterpret(Int, $xs), (2,:))'
10.611 ns (0 allocations: 0 bytes)
1000000×2 adjoint(reshape(reinterpret(Int64, ::Vector{Tuple{Int64, Int64}}), 2, 1000000)) with eltype Int64:
5 1
4 3
⋮
1 4
9 2
And for the copying code the fastest is going to be:
function convert_to_tuple4(x)
out = Matrix{eltype(x[1])}(undef, length(x), length(x[1]))
for i ∈ 1:length(x)
#inbounds #simd for j ∈ 1:length(x[1])
out[i, j] = x[i][j]
end
end
out
end
Benchmarks:
julia> #btime convert_to_tuple3($xs);
3.488 ms (2 allocations: 15.26 MiB)
julia> #btime convert_to_tuple4($xs);
2.932 ms (2 allocations: 15.26 MiB)
Related
I am using Julia 1.6.1.
B is a matrix. For example,
B =
[ 2 4 4 4 5 ;
1 2 2 3 5 ;
1 2 3 3 3 ;
1 2 2 5 6 ;
1 3 4 4 4 ; ]
I wanted to sort it forcsing on each row.
sortedB = sortslices( B, dims=1, rev=true)
Then, we get sorted B
sortedB =
[ 2 4 4 4 5 ; # 1st row of the original matrix B
1 3 4 4 4 ; # 5th row of the original matrix B
1 2 3 3 3 ; # 3rd row of the original matrix B
1 2 2 5 6 ; # 4th row of the original matrix B
1 2 2 3 5 ;] # 2nd row of the original matrix B
I would like to get the array [1 5 3 4 2].
How can I do that ?
It seems that sortperm does not work.
sortperm( sortslices( B, dims=1, rev=true) )
# ERROR: MethodError; no method matching sortperm(::Matrix{Int64})
If performance is an issue use a non-allocating version.
julia> sortperm(view.(Ref(B), 1:size(B,1), :), rev=true)
5-element Vector{Int64}:
1
5
3
4
2
Here are some benchmarks using BenchmarkTools:
julia> #btime sortperm(view.(Ref($B), 1:size($B,1), :),rev=true);
376.471 ns (3 allocations: 432 bytes)
julia> #btime sortperm(collect(eachslice($B,dims=1)),rev=true)
642.683 ns (6 allocations: 496 bytes);
you can use eachrow or eachslice:
julia> C = collect(eachslice(B,dims=1))
5-element Vector{SubArray{Int64, 1, Matrix{Int64}, Tuple{Int64, Base.Slice{Base.OneTo{Int64}}}, true}}:
[2, 4, 4, 4, 5]
[1, 2, 2, 3, 5]
[1, 2, 3, 3, 3]
[1, 2, 2, 5, 6]
[1, 3, 4, 4, 4]
julia> sortperm(C,rev=true)
5-element Vector{Int64}:
1
5
3
4
2
although this will allocate more than necessary (collect is needed apparently)
Apparently I must have a fundamental misunderstanding about DistributedArrays.jl.
I have set up a MWE of something similar to what I have to do:
using Distributed
using DistributedArrays
addprocs()
#everywhere using Distributed, DistributedArrays
a = distribute(zeros(5))
#sync #distributed for i in 1:5
a_l = localpart(a)
a_l[i] = 100 * i
end
And then I run into the following Error:
ERROR: TaskFailedException:
On worker 2:
BoundsError: attempt to access 1-element Array{Float64,1} at index [2]
setindex! at ./array.jl:847
macro expansion at /home/user/test.jl:36 [inlined]
[inlined]
#17 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/macros.jl:301
#160 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/macros.jl:87
#103 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:290
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:79
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:88
#96 at ./task.jl:356
...and 3 more exception(s).
Stacktrace:
[1] sync_end(::Channel{Any}) at ./task.jl:314
[2] (::Distributed.var"#159#161"{var"#17#18",UnitRange{Int64}})() at ./task.jl:333
Stacktrace:
sync_end(::Channel{Any}) at ./task.jl
top-level scope at task.jl
Using a = dzeros((5,1), workers()) also gives the same Error. Any help is appreciated!
There are two problems:
localpart is indexed starting from 1
the number of your workers is greater than loop size. Since workers are randomly assigned your loop elements get empty localparts.
Let us consider this code:
a = distribute(zeros(5));
#sync #distributed for i in 1:5
for j in keys(a[:L])
a[:L][j] = 100 * i+myid()
end
end
While it solves the first issue the second is still there:
julia> a
5-element DArray{Float64, 1, Vector{Float64}}:
402.0
503.0
0.0
0.0
0.0
Why it does not work as expected? because addprocs is adding all processes so I have now 8 workers and the size of loop is 5.
Perhaps the simplest solution is to replace the range from 1:5 to 1:max(5,nworkers()). This makes sure that each localpart is going to get processed.
julia> #sync #distributed for i in 1:max(5,nworkers())
#show i, myid(), length(a[:L])
for j in keys(a[:L])
a[:L][j] = 100 * i+myid()
end
end
From worker 9: (i, myid(), length(a[:L])) = (6, 9, 0)
From worker 7: (i, myid(), length(a[:L])) = (4, 7, 0)
From worker 2: (i, myid(), length(a[:L])) = (7, 2, 1)
From worker 8: (i, myid(), length(a[:L])) = (5, 8, 0)
From worker 3: (i, myid(), length(a[:L])) = (8, 3, 1)
From worker 4: (i, myid(), length(a[:L])) = (1, 4, 1)
From worker 5: (i, myid(), length(a[:L])) = (2, 5, 1)
From worker 6: (i, myid(), length(a[:L])) = (3, 6, 1)
Task (done) #0x0000000073e09f50
This code run shows clearly what is happening when you loop over 5 elements and use 8 workers.
The result is now as expected (with regard that tasks are randomly allocated around workers):
julia> a
5-element DArray{Float64, 1, Vector{Float64}}:
702.0
803.0
104.0
205.0
306.0
I'm attempting to create an array of matrices by multiplying a t = range(0,stop=2*pi,length=101) by a matrix [1, 0] as follows
A = t .* [1 ,0]
but this produces the error ERROR: LoadError: DimensionMismatch("arrays could not be broadcast to a common size"). I would like each scalar, or element of t to be multiplied elementwise (in terms of t) with the elements of the vector [1 , 0], essentially performing an elementwise scalar--matrix product.
The reason I'm doing this is because I would later like to be able to multiply another constant matrix M with each column vector found in A. How can this be done in Julia v1.1?
You have to wrap the element you do not want to be broadcasted over in a container. Here is a standard way to do it (I decreased length kwarg to 3 to make the example more clear):
julia> t = range(0,stop=2*pi,length=3)
0.0:3.141592653589793:6.283185307179586
julia> A = t .* Ref([1 ,0])
3-element Array{Array{Float64,1},1}:
[0.0, 0.0]
[3.141592653589793, 0.0]
[6.283185307179586, 0.0]
julia> Ref([1 2; 3 4]) .* A
3-element Array{Array{Float64,1},1}:
[0.0, 0.0]
[3.141592653589793, 9.42477796076938]
[6.283185307179586, 18.84955592153876]
Instead of Ref container you can also use a 1-element tuple or 1-element vector as wrappers:
julia> t .* ([1 ,0],)
3-element Array{Array{Float64,1},1}:
[0.0, 0.0]
[3.141592653589793, 0.0]
[6.283185307179586, 0.0]
julia> t .* [[1 ,0]]
3-element Array{Array{Float64,1},1}:
[0.0, 0.0]
[3.141592653589793, 0.0]
[6.283185307179586, 0.0]
The reason why Ref should be preferred is that it is 0-dimensional, so that it is the most neutral of these three methods (i.e. influences the output in the least way - retaining the broadcast style of the other argument). Here are some examples:
julia> f1(x) = x .* (2, )
f1 (generic function with 1 method)
julia> f2(x) = x .* [2]
f2 (generic function with 1 method)
julia> f3(x) = x .* Ref(2)
f3 (generic function with 1 method)
julia> f1(1)
(2,)
julia> f2(1)
1-element Array{Int64,1}:
2
julia> f3(1)
2
julia> f1((1,2))
(2, 4)
julia> f2((1,2))
2-element Array{Int64,1}:
2
4
julia> f3((1,2))
(2, 4)
Consider these two functions:
Function 1:
function testVec(t,v,k,n)
for i=1:n
t .= v .* 3 .+ k .* 4;
end
end
Function 2:
module params
r = 3;
end
function testVec2(t,v,k,n)
for i=1:n
t .= v .* params.r .+ k .* 4;
end
end
They have drastically different performance:
#time testVec([1 2 3 4], [2 3 4 5], [3 4 5 6], 1000)
0.000036 seconds (7 allocations: 496 bytes)
#time testVec2([1 2 3 4], [2 3 4 5], [3 4 5 6], 1000)
0.003180 seconds (4.01 k allocations: 141.109 KiB)
Why does including the parameter r in a module make the function perform worse?
If I export the module params and include r in testVec2 without using the prefix params, its performance immediately improves (same as testVec). Why?
r in params module is a non-const global, which makes its type unstable (because some function can assign r to something else, of a different type).
Replace r = 3 with const r = 3 and the timings will be the same. Also see first section of Performance Tips.
Given a function/expression that yields a single column, how to build a matrix from those columns in Julia? I tried the following (simplified example):
column(j) = [j, j, j] # for example
my_matrix = Float64[column(j)[i] for i in 1:3, j in 1:4]
==> 3x4 Array{Float64,2}:
1.0 2.0 3.0 4.0
1.0 2.0 3.0 4.0
1.0 2.0 3.0 4.0
The result is correct, but this is not what I want because the column-vector-expression is evaluated unnecessarily (once for every row).
I also tried this alternative approach:
[column(j) for j in 1:4]
==> 4-element Array{Array{Float64,1},1}:
[1.0,1.0,1.0]
[2.0,2.0,2.0]
[3.0,3.0,3.0]
[4.0,4.0,4.0]
But I found no way to convert or reshape this into the form I want (matrix with two dimensions instead of array of arrays).
How to achieve this in Julia?
Have you tried hcat?:
julia> column(j) = [j, j ,j]
column (generic function with 1 method)
julia> my_matrix = hcat([column(j) for j=1:4]...)
3x4 Array{Int64,2}:
1 2 3 4
1 2 3 4
1 2 3 4
See hcat in Julia docs
Looking at answer #Benoit it is actually faster to use reduce(hcat, c):
julia> c = [[j,j,j] for j in 1:4]
4-element Vector{Vector{Int64}}:
[1, 1, 1]
[2, 2, 2]
[3, 3, 3]
[4, 4, 4]
julia> #btime reshape(collect(Iterators.flatten($c)), 3, 4)
232.791 ns (6 allocations: 448 bytes)
3×4 Matrix{Int64}:
1 2 3 4
1 2 3 4
1 2 3 4
julia> #btime reduce(hcat, $c)
70.825 ns (1 allocation: 176 bytes)
3×4 Matrix{Int64}:
1 2 3 4
1 2 3 4
1 2 3 4
But I found no way to convert or reshape this into the form I want (matrix with two dimensions instead of array of arrays).
You can obtain the matrix by reshaping the flatten'ed version of the vector of columns c:
julia> column(j) = [j, j, j]
column (generic function with 1 method)
julia> c = [column(j) for j in 1:4]
4-element Vector{Vector{Int64}}:
[1, 1, 1]
[2, 2, 2]
[3, 3, 3]
[4, 4, 4]
julia> v = collect(Iterators.flatten(c))
12-element Vector{Int64}:
1
1
1
2
2
2
3
3
3
4
4
4
julia> m = reshape(v, 3, 4)
3×4 Matrix{Int64}:
1 2 3 4
1 2 3 4
1 2 3 4
The alternative hcat(c...) suggested by the accepted answer works as well but as pointed out by a comment, this will not work for a large number of columns.
Indeed, if c is a vector of n columns then hcat(c...) will compile a method with n arguments. This first means that if your code does this for different number of columns then you will need to compile a new method every time. Secondly, compiling a method with too many arguments will make the stack memory overflow.
For this reason, when you don't need a one-line solution (e.g. like the one combining flatten and reshape above), it seems to me that most Julia users do:
m = Matrix{Int}(undef, 3, 4)
for j in 1:4
m[:, j] = column(j)
end
EDIT #Przemyslaw suggested a better solution: https://stackoverflow.com/a/68976092/1189815
stack, which will be in Julia 1.9, and can be used in the meanwhile with Compat, will construct matrix from row/column vectors (returned from function/expression).
column(j) = [j, j, j]
using Compat #for stack which will be in Julia 1.9
stack(column(j) for j=1:4)
#3×4 Matrix{Int64}:
# 1 2 3 4
# 1 2 3 4
# 1 2 3 4
What is fast and memory efficient.
using BenchmarkTools, Compat
column(j) = [j, j, j]
#btime stack(column(j) for j=1:4)
# 193.315 ns (5 allocations: 480 bytes)
#3×4 Matrix{Int64}:
# 1 2 3 4
# 1 2 3 4
# 1 2 3 4
#btime begin ##Benoît Legat
m = Matrix{Int}(undef, 3, 4)
for j in 1:4
m[:, j] = column(j)
end
m
end
# 201.942 ns (5 allocations: 480 bytes)
#3×4 Matrix{Int64}:
# 1 2 3 4
# 1 2 3 4
# 1 2 3 4
#btime reduce(hcat, [column(j) for j=1:4]) ##Przemyslaw Szufel
# 249.676 ns (6 allocations: 560 bytes)
#3×4 Matrix{Int64}:
# 1 2 3 4
# 1 2 3 4
# 1 2 3 4
#btime reshape(collect(Iterators.flatten([column(j) for j in 1:4])), 3, 4) ##Benoît Legat
# 392.325 ns (10 allocations: 976 bytes)
#3×4 Matrix{Int64}:
# 1 2 3 4
# 1 2 3 4
# 1 2 3 4
#btime Int[column(j)[i] for i in 1:3, j in 1:4] ##radioflash
# 440.211 ns (13 allocations: 1.09 KiB)
#3×4 Matrix{Int64}:
# 1 2 3 4
# 1 2 3 4
# 1 2 3 4
#btime hcat([column(j) for j=1:4]...) ##spencerlyon2
# 644.676 ns (10 allocations: 784 bytes)
#3×4 Matrix{Int64}:
# 1 2 3 4
# 1 2 3 4
# 1 2 3 4
In case you have already a vector.
c = [[j,j,j] for j in 1:4]
#4-element Vector{Vector{Int64}}:
# [1, 1, 1]
# [2, 2, 2]
# [3, 3, 3]
# [4, 4, 4]
using Compat
stack(c)
#3×4 Matrix{Int64}:
# 1 2 3 4
# 1 2 3 4
# 1 2 3 4
using BenchmarkTools
#btime stack($c)
# 63.204 ns (1 allocation: 160 bytes)
#3×4 Matrix{Int64}:
# 1 2 3 4
# 1 2 3 4
# 1 2 3 4
#btime reduce(hcat, $c) ##Przemyslaw Szufel
# 68.758 ns (1 allocation: 160 bytes)
#3×4 Matrix{Int64}:
# 1 2 3 4
# 1 2 3 4
# 1 2 3 4
#btime begin ##Benoît Legat
m = Matrix{Int}(undef, 3, 4)
for j in 1:4
m[:, j] = $c[j]
end
m
end
# 75.586 ns (1 allocation: 160 bytes)
#3×4 Matrix{Int64}:
# 1 2 3 4
# 1 2 3 4
# 1 2 3 4
#btime reshape(collect(Iterators.flatten($c)), 3, 4) ##Benoît Legat
# 210.752 ns (5 allocations: 576 bytes)
#3×4 Matrix{Int64}:
# 1 2 3 4
# 1 2 3 4
# 1 2 3 4
#btime hcat($c...) ##spencerlyon2
# 453.213 ns (5 allocations: 384 bytes)
#3×4 Matrix{Int64}:
# 1 2 3 4
# 1 2 3 4
# 1 2 3 4