I'm trying to implement a convert for struct containing NTuple:
import Base: convert
abstract type AbstractMyType{N, T} end
struct MyType1{N, T} <: AbstractMyType{N, T}
data::NTuple{T, N}
end
struct MyType2{N, T} <: AbstractMyType{N, T}
data::NTuple{T, N}
end
foo(::Type{MyType2}, x::AbstractMyType{N, T}) where {N, T} = x
convert(::Type{MyType2}, x::AbstractMyType{N, T}) where {N, T} = MyType2{T}(x.data)
println(foo(MyType2, MyType1((1,2,3)))) # MyType1{Int64,3}((1, 2, 3))
println(convert(MyType2, MyType1((1,2,3)))) # MethodError
Defined functions foo and convert have the same signature. For some reason function foo returns normally while convert throws MethodError. Why Julia cannot find my convert method?
julia version 1.4.1
Julia is finding your convert method:
julia> println(convert(MyType2, MyType1((1,2,3)))) # MethodError
ERROR: MethodError: no method matching MyType2{3,T} where T(::Tuple{Int64,Int64,Int64})
Stacktrace:
[1] convert(::Type{MyType2}, ::MyType1{Int64,3}) at ./REPL[16]:1
[2] top-level scope at REPL[18]:1
That stack trace is saying that it's inside your convert function (in my case, I defined it on the first line of the 16th REPL prompt). The problem is that it cannot find a MyType2{T}(::Tuple) constructor.
Julia automatically creates a number of constructors for you when you don't use an inner constructor; in this case you can either call MyType(()) or MyType{T, N}(()), but Julia doesn't know what to do with only one type parameter passed (by default):
julia> MyType2((1,2,3))
MyType2{Int64,3}((1, 2, 3))
julia> MyType2{Int, 3}((1,2,3))
MyType2{Int64,3}((1, 2, 3))
julia> MyType2{Int}((1,2,3))
ERROR: MethodError: no method matching MyType2{Int64,T} where T(::Tuple{Int64,Int64,Int64})
Stacktrace:
[1] top-level scope at REPL[7]:1
[2] eval(::Module, ::Any) at ./boot.jl:331
[3] eval_user_input(::Any, ::REPL.REPLBackend) at /Users/mbauman/Julia/release-1.4/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:86
[4] run_backend(::REPL.REPLBackend) at /Users/mbauman/.julia/packages/Revise/AMRie/src/Revise.jl:1023
[5] top-level scope at none:0
So the fix is either to define that method yourself, or change the body of your convert method to call MyType{T,N} explicitly.
Just define the method
convert(::Type{MyType2}, x::AbstractMyType{N, T}) where {N, T} = MyType2(x.data)
Testing:
julia> convert(MyType2, MyType1((1,2,3)))
MyType2{Int64,3}((1, 2, 3))
Related
I am trying to permute the order of elements in a CharacterVector. In R I would simply use:
sample(charvec)
I am trying the same thing in Rcpp using the sample sugar function, but it keeps throwing 'error: no matching function for call to 'sample(Rcpp::CharacterVector&)'. Other sugar functions I have tried, like intersect or sort_unique work fine with CharacterVector, but sample refuses to work. This is the minimal example I have been experimenting with:
cppFunction('CharacterVector samplefunc() {
CharacterVector v = {"Cat", "Dog", "Fox", "Fish", "Lion"} ;
CharacterVector v2 = sample(v) ;
return v2 ;
}')
What I am doing wrong when trying to use the sample sugar function?
You are just missing the size parameter, which is mandatory for Rcpp::sample:
set.seed(42)
Rcpp::cppFunction('CharacterVector samplefunc() {
CharacterVector v = {"Cat", "Dog", "Fox", "Fish", "Lion"} ;
CharacterVector v2 = sample(v, v.size()) ;
return v2 ;
}')
samplefunc()
#> [1] "Lion" "Fish" "Cat" "Dog" "Fox"
UPDATE (about debugging this kind of errors): Admittedly, the error you see when you do not provide the size argument is kind of obscure (at least with gcc), but you can see:
file1294a34f4734f.cpp: In function ‘Rcpp::CharacterVector samplefunc()’:
file1294a34f4734f.cpp:8:30: error: no matching function for call to ‘sample(Rcpp::CharacterVector&)’
8 | CharacterVector v2 = sample(v) ;
| ~~~~~~^~~
This is the error: no matching function. And then,
In file included from /***/Rcpp/include/Rcpp/sugar/functions/functions.h:89,
from /***/Rcpp/include/Rcpp/sugar/sugar.h:31,
from /***/Rcpp/include/Rcpp.h:78,
from file1294a34f4734f.cpp:1:
/***/Rcpp/include/Rcpp/sugar/functions/sample.h:437:1: note: candidate: ‘template<int RTYPE> Rcpp::Vector<RTYPE, Rcpp::PreserveStorage> Rcpp::sample(const Rcpp::Vector<RTYPE, Rcpp::PreserveStorage>&, int, bool, Rcpp::sugar::probs_t)’
437 | sample(const Vector<RTYPE>& x, int size, bool replace = false, sugar::probs_t probs = R_NilValue)
| ^~~~~~
where gcc is showing you a candidate, and you can see that this function accepts a constant Vector of any RTYPE (numeric, character...), and then it needs a size argument, because there is no default. The others (replace, probs) do have a default. R functions may have missing arguments, C++ functions cannot.
I have an Array{UInt16,2} in Julia of size 5328×3040. I want to save it to a png image.
I tried the following:
save("gray.png", colorview(Gray, img))
But got the following error:
ERROR: TypeError: in Gray, in T, expected T<:Union{Bool, AbstractFloat, FixedPoint}, got Type{UInt16}
Stacktrace:
[1] ccolor_number at C:\Users\ankushar\.julia\packages\ImageCore\KbJyT\src\convert_reinterpret.jl:60 [inlined]
[2] ccolor_number at C:\Users\ankushar\.julia\packages\ImageCore\KbJyT\src\convert_reinterpret.jl:57 [inlined]
[3] colorview(::Type{Gray}, ::Array{UInt16,2}) at C:\Users\ankushar\.julia\packages\ImageCore\KbJyT\src\colorchannels.jl:104
[4] top-level scope at REPL[16]:1
caused by [exception 3]
IOError: symlink: operation not permitted (EPERM)
I am using Julia 1.4.2
Can you suggest a good way to store these arrays as images in Julia?
TIA!
You can normalize the pixel values before saving.
using Images
img = rand(UInt16, 10, 20)
img[1:3]
# => 3-element Array{UInt16,1}:
0x7fc2
0x057e
0xae79
gimg = colorview(Gray, img ./ typemax(UInt16))
gimg[1:3] |> channelview
# => 3-element reinterpret(Float64, ::Array{Gray{Float64},1}):
0.4990615701533532
0.02145418478675517
0.6815442130159457
save("gray.png", gimg)
A faster and more accurate solution is to reinterpret your array as an array of N0f16 which is a type from FixedPointNumbers which is basically just a Uint16 scaled between 0 and 1. This will both avoid rounding errors, but also prevent the need for making a copy.
using FixedPointNumbers
img = rand(UInt16, 10, 20)
colorview(Gray, reinterpret(N0f16, img)))
I'm trying to write a simple program to do parallel processing with code in separate modules. I have the following code in two separate files:
main.jl
push!(LOAD_PATH, ".")
#using other # This doesn't work either
importall other
np = 4
b = [Bounds(k, k+8) for k in 1:np]
fut = Array{Future}(np)
for k = 1:np
fut[k] = #spawn center(b[k])
end
for k = 1:np
xc = fetch(fut[k])
println("Center for k ", k, " is ", xc)
end
other.jl
#everywhere module other
export Bounds
export center
#everywhere type Bounds
x1::Int
x2::Int
end
#everywhere function center(bound::Bounds)
return (bound.x1 + bound.x2) / 2
end
end
When i run with a single process with "julia main.jl" it runs with no errors, but if I try to add processes with "julia -p4 main.jl" I get the error below. It looks like maybe the additional processes can't see the code in other.jl, but I have the #everywhere macro at all the right places it seems. What is the problem?
ERROR: LoadError: LoadError: On worker 2:
UndefVarError: ##5#7 not defined
in deserialize_datatype at ./serialize.jl:823
in handle_deserialize at ./serialize.jl:571
in deserialize_msg at ./multi.jl:120
in message_handler_loop at ./multi.jl:1317
in process_tcp_streams at ./multi.jl:1276
in #618 at ./event.jl:68
in #remotecall_fetch#606(::Array{Any,1}, ::Function, ::Function, ::Base.Worker) at ./multi.jl:1070
in remotecall_fetch(::Function, ::Base.Worker) at ./multi.jl:1062
in #remotecall_fetch#609(::Array{Any,1}, ::Function, ::Function, ::Int64) at ./multi.jl:1080
in remotecall_fetch(::Function, ::Int64) at ./multi.jl:1080
in (::other.##6#8)() at ./multi.jl:1959
...and 3 other exceptions.
in sync_end() at ./task.jl:311
in macro expansion; at ./multi.jl:1968 [inlined]
in anonymous at ./<missing>:?
in eval(::Module, ::Any) at ./boot.jl:234
in (::##1#3)() at ./multi.jl:1957
in sync_end() at ./task.jl:311
in macro expansion; at ./multi.jl:1968 [inlined]
in anonymous at ./<missing>:?
in include_from_node1(::String) at ./loading.jl:488
in eval(::Module, ::Any) at ./boot.jl:234
in require(::Symbol) at ./loading.jl:409
in include_from_node1(::String) at ./loading.jl:488
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318
while loading /mnt/mint320/home/bmaier/BillHome/Programs/Julia/parallel/modules/other.jl, in expression starting on line 1
while loading /mnt/mint320/home/bmaier/BillHome/Programs/Julia/parallel/modules/main.jl, in expression starting on line 5
Try #everywhere importall other in main.jl
This will load other on all of the current workers. There is no requirement for other to be anything other than a normal script or module, and so you don't need the #everywhere's in other.jl.
Julia has strftime as a built-in but not gmtime.
julia> strftime
strftime (generic function with 3 methods)
julia> gmtime
ERROR: gmtime not defined
What is the preferred Julia way to do the equivalent of gmtime? The idea is to turn seconds since the epoch into a time structure in the Z (+00:00) time zone. Here in Los Angeles, I see:
julia> strftime("%H:%M:%S", 0)
"16:00:00"
I would like to see "00:00:00". I can do it in Python:
>>> from time import strftime, gmtime
>>> strftime("%H:%M:%S", gmtime(0))
'00:00:00'
I tried to use ccall in Julia but it did not work
julia> ccall( (:gmtime, "libc"), TmStruct, (Int64,), 0)
TmStruct(1590498096,32767,16041550,1,-1924564896,32744,1,0,0,0,0,0,1590498144,32767)
julia> strftime("%H:%M:%S", ans)
"16041550:32767:1590498096"
What went wrong with my ccall? And better, is there just a nicer all-Julia way to get the effect of gmtime?
Looks like gmtime is on Julia's TODO list. Until it gets included, will something like this work for you?
julia> function gmtime(t::Real)
t = floor(t)
tm = TmStruct()
ccall(:gmtime_r, Ptr{TmStruct}, (Ptr{Int}, Ptr{TmStruct}), &t, &tm)
return tm
end
gmtime (generic function with 1 method)
julia> strftime("%H:%M:%S", gmtime(0))
"00:00:00"
How do you move data from one processor to another in julia?
Say I have an array
a = [1:10]
Or some other data structure. What is the proper way to put it on all other available processors so that it will be available on those processors as the same variable name?
I didn't know how to do this at first, so I spent some time figuring it out.
Here are some functions I wrote to pass objects:
sendto
Send an arbitrary number of variables to specified processes.
New variables are created in the Main module on specified processes. The
name will be the key of the keyword argument and the value will be the
associated value.
function sendto(p::Int; args...)
for (nm, val) in args
#spawnat(p, eval(Main, Expr(:(=), nm, val)))
end
end
function sendto(ps::Vector{Int}; args...)
for p in ps
sendto(p; args...)
end
end
Examples
# creates an integer x and Matrix y on processes 1 and 2
sendto([1, 2], x=100, y=rand(2, 3))
# create a variable here, then send it everywhere else
z = randn(10, 10); sendto(workers(), z=z)
getfrom
Retrieve an object defined in an arbitrary module on an arbitrary
process. Defaults to the Main module.
The name of the object to be retrieved should be a symbol.
getfrom(p::Int, nm::Symbol; mod=Main) = fetch(#spawnat(p, getfield(mod, nm)))
Examples
# get an object from named x from Main module on process 2. Name it x
x = getfrom(2, :x)
passobj
Pass an arbitrary number of objects from one process to arbitrary
processes. The variable must be defined in the from_mod module of the
src process and will be copied under the same name to the to_mod
module on each target process.
function passobj(src::Int, target::Vector{Int}, nm::Symbol;
from_mod=Main, to_mod=Main)
r = RemoteRef(src)
#spawnat(src, put!(r, getfield(from_mod, nm)))
for to in target
#spawnat(to, eval(to_mod, Expr(:(=), nm, fetch(r))))
end
nothing
end
function passobj(src::Int, target::Int, nm::Symbol; from_mod=Main, to_mod=Main)
passobj(src, [target], nm; from_mod=from_mod, to_mod=to_mod)
end
function passobj(src::Int, target, nms::Vector{Symbol};
from_mod=Main, to_mod=Main)
for nm in nms
passobj(src, target, nm; from_mod=from_mod, to_mod=to_mod)
end
end
Examples
# pass variable named x from process 2 to all other processes
passobj(2, filter(x->x!=2, procs()), :x)
# pass variables t, u, v from process 3 to process 1
passobj(3, 1, [:t, :u, :v])
# Pass a variable from the `Foo` module on process 1 to Main on workers
passobj(1, workers(), [:foo]; from_mod=Foo)
use #eval #everywhere... and escape the local variable. like this:
julia> a=collect(1:3)
3-element Array{Int64,1}:
1
2
3
julia> addprocs(1)
1-element Array{Int64,1}:
2
julia> #eval #everywhere a=$a
julia> #fetchfrom 2 a
3-element Array{Int64,1}:
1
2
3
Just so everyone here knows, I put these ideas together into a package ParallelDataTransfer.jl for this. So you just need to do
using ParallelDataTransfer
(after installing) in order to use the functions mentioned in the answers here. Why? These functions are pretty useful! I added some testing, some new macros, and updated them a bit (they pass on v0.5, fail on v0.4.x). Feel free to put in pull requests to edit these and add more.
To supplement #spencerlyon2 's answer here are some macros:
function sendtosimple(p::Int, nm, val)
ref = #spawnat(p, eval(Main, Expr(:(=), nm, val)))
end
macro sendto(p, nm, val)
return :( sendtosimple($p, $nm, $val) )
end
macro broadcast(nm, val)
quote
#sync for p in workers()
#async sendtosimple(p, $nm, $val)
end
end
end
The #spawnat macro binds a value to a symbol on a particular process
julia> #sendto 2 :bip pi/3
RemoteRef{Channel{Any}}(9,1,5340)
julia> #fetchfrom 2 bip
1.0471975511965976
The #broadcast macro binds a value to a symbol in all processes except 1 (as I found doing so made future expressions using the name copy the version from process 1)
julia> #broadcast :bozo 5
julia> #fetchfrom 2 bozo
5
julia> bozo
ERROR: UndefVarError: bozo not defined
julia> bozo = 3 #these three lines are why I exclude pid 1
3
julia> #fetchfrom 7 bozo
3
julia> #fetchfrom 7 Main.bozo
5