JULIA: Can pmap modify arguments in place? - parallel-processing
I am applying the same function to multiple independent objects and I'd like to do it in parallel. The problem is the function modifies one of its arguments. This is fine for map but not pmap. Here is a minimal reproducible example:
#everywhere function testmod!(a,μ)
for i=1:length(a)
a[i]=i*μ
end
b=copy(a)
return b
end
myarrays=[zeros(Float64,10) for i=1:10]
pmap((a1,a2)->testmod!(a1,a2),myarrays,[i for i=1:10])
This toy function modifies the elements of the input array a. I'll compare the results of map and pmap:
map
julia> myarrays=[zeros(Float64,10) for i=1:10]
10-element Array{Array{Float64,1},1}:
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
julia> map((a1,a2)->testmod!(a1,a2),myarrays,[i for i=1:10])
10-element Array{Array{Float64,1},1}:
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]
[3.0,6.0,9.0,12.0,15.0,18.0,21.0,24.0,27.0,30.0]
[4.0,8.0,12.0,16.0,20.0,24.0,28.0,32.0,36.0,40.0]
[5.0,10.0,15.0,20.0,25.0,30.0,35.0,40.0,45.0,50.0]
[6.0,12.0,18.0,24.0,30.0,36.0,42.0,48.0,54.0,60.0]
[7.0,14.0,21.0,28.0,35.0,42.0,49.0,56.0,63.0,70.0]
[8.0,16.0,24.0,32.0,40.0,48.0,56.0,64.0,72.0,80.0]
[9.0,18.0,27.0,36.0,45.0,54.0,63.0,72.0,81.0,90.0]
[10.0,20.0,30.0,40.0,50.0,60.0,70.0,80.0,90.0,100.0]
julia> myarrays
10-element Array{Array{Float64,1},1}:
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]
[3.0,6.0,9.0,12.0,15.0,18.0,21.0,24.0,27.0,30.0]
[4.0,8.0,12.0,16.0,20.0,24.0,28.0,32.0,36.0,40.0]
[5.0,10.0,15.0,20.0,25.0,30.0,35.0,40.0,45.0,50.0]
[6.0,12.0,18.0,24.0,30.0,36.0,42.0,48.0,54.0,60.0]
[7.0,14.0,21.0,28.0,35.0,42.0,49.0,56.0,63.0,70.0]
[8.0,16.0,24.0,32.0,40.0,48.0,56.0,64.0,72.0,80.0]
[9.0,18.0,27.0,36.0,45.0,54.0,63.0,72.0,81.0,90.0]
[10.0,20.0,30.0,40.0,50.0,60.0,70.0,80.0,90.0,100.0]
This works as desired. In contrast pmap:
pmap
julia> myarrays=[zeros(Float64,10) for i=1:10]
10-element Array{Array{Float64,1},1}:
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
julia> pmap((a1,a2)->testmod!(a1,a2),myarrays,[i for i=1:10])
10-element Array{Any,1}:
[1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0]
[2.0,4.0,6.0,8.0,10.0,12.0,14.0,16.0,18.0,20.0]
[3.0,6.0,9.0,12.0,15.0,18.0,21.0,24.0,27.0,30.0]
[4.0,8.0,12.0,16.0,20.0,24.0,28.0,32.0,36.0,40.0]
[5.0,10.0,15.0,20.0,25.0,30.0,35.0,40.0,45.0,50.0]
[6.0,12.0,18.0,24.0,30.0,36.0,42.0,48.0,54.0,60.0]
[7.0,14.0,21.0,28.0,35.0,42.0,49.0,56.0,63.0,70.0]
[8.0,16.0,24.0,32.0,40.0,48.0,56.0,64.0,72.0,80.0]
[9.0,18.0,27.0,36.0,45.0,54.0,63.0,72.0,81.0,90.0]
[10.0,20.0,30.0,40.0,50.0,60.0,70.0,80.0,90.0,100.0]
julia> myarrays
10-element Array{Array{Float64,1},1}:
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
Clearly myarray has not been modified. Is there any way to achieve this with pmap or can you only return things with pmap.
First it is unnecessary to do (a1,a2)->testmod!(a1,a2), you can just pass testmod!.
ie. pmap(testmod!,myarrays,[i for i=1:10])
Second, the return value is not used in either case. You are just modifying the array in place in the first (map) example. In the pmap example, the array has to be copied to each worker process, so it is only being modified on the worker process itself.
What does matter for map or pmap (and the intended way of using them) is the value that is returned by each function (which is the same).
You may notice that myarray is changed when using pmap if you don't have any worker processes. This is because the array was not copied anywhere and all modifications were done locally.
Perhaps you could make use of a SharedArray to achieve what you want, or restructure your example so your ultimate goal is more clear.
tl;dr -- pmap can modify its arguments, but the result probably isn't what you expect.
Related
Julia iterates over `Base.Iterators.Reverse`
for i in Iterators.reverse(Iterators.drop([1,2], 1))
println(i)
end
MethodError: no method matching iterate(::Base.Iterators.Reverse{Base.Iterators.Drop{Array{Int64,1}}})
Closest candidates are:
iterate(!Matched::LibGit2.GitRevWalker) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LibGit2/src/walker.jl:29
iterate(!Matched::LibGit2.GitRevWalker, !Matched::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/LibGit2/src/walker.jl:29
iterate(!Matched::Base.EnvDict) at env.jl:119
...
Stacktrace:
[1] top-level scope at ./In[1]:1
[2] include_string(::Function, ::Module, ::String, ::String) at ./loading.jl:1091
Below version runs correctly
for i in Iterators.reverse(collect(Iterators.drop([1,2], 1)))
println(i)
end
Why is collect required?
My julia version is 1.5.3, running on WSL2.
The answer is the the error message: collect is required because there is no method matching iterate(::Base.Iterators.Reverse{Base.Iterators.Drop{Array{Int64,1}}}). From the documentation of reverse:
Not all iterator types T support reverse-order iteration. If T doesn't, then iterating over Iterators.reverse(itr::T) will
throw a MethodError because of the missing iterate methods for Iterators.Reverse{T}. (To implement these methods, the
original iterator itr::T can be obtained from r = Iterators.reverse(itr) by r.itr.)
Using addprocs() and pmap() inside a function in Julia
In Julia, I want to use addprocs and pmap inside a function that is defined inside a module. Here's a silly example:
module test
using Distributions
export g, f
function g(a, b)
a + rand(Normal(0, b))
end
function f(A, b)
close = false
if length(procs()) == 1 # If there are already extra workers,
addprocs() # use them, otherwise, create your own.
close = true
end
W = pmap(x -> g(x, b), A)
if close == true
rmprocs(workers()) # Remove the workers you created.
end
return W
end
end
test.f(randn(5), 1)
This returns a very long error
WARNING: Module test not defined on process 4
WARNING: Module test not defined on process 3
fatal error on fatal error on WARNING: Module test not defined on process 2
43: : WARNING: Module test not defined on process 5
fatal error on fatal error on 5: 2: ERROR: UndefVarError: test not defined
in deserialize at serialize.jl:504
in handle_deserialize at serialize.jl:477
in deserialize at serialize.jl:696
...
in message_handler_loop at multi.jl:878
in process_tcp_streams at multi.jl:867
in anonymous at task.jl:63
Worker 3 terminated.
Worker 2 terminated.ERROR (unhandled task failure): EOFError: read end of file
WARNING: rmprocs: process 1 not removed
Worker 5 terminated.ERROR (unhandled task failure): EOFError: read end of file
4-element Array{Any,1}:Worker 4 terminated.ERROR (unhandled task failure): EOFError: read end of file
ERROR (unhandled task failure): EOFError: read end of file
ProcessExitedException()
ProcessExitedException()
ProcessExitedException()
ProcessExitedException()
What I'm trying to do is write a package that contains functions that perform operations that can be optionally parallelized at the user's discretion. So a function like f might take an argument par::Bool that does something like I've shown above if the user calls f with par = true and loops otherwise. So from within the definition of f (and within the definition of the module test), I want to create workers and broadcast the Distributions package and the function g to them.
What's wrong with using #everywhere in your function? The following, for example, works fine on my computer. function f(A, b) close = false if length(procs()) == 1 # If there are already extra workers, addprocs() # use them, otherwise, create your own. #everywhere begin using Distributions function g(a, b) a + rand(Normal(0, b)) end end close = true end W = pmap(x -> g(x, b), A) if close == true rmprocs(workers()) # Remove the workers you created. end return W end f(randn(5), 1) Note: when I first ran this, I needed to recompile the Distributions package since it had been updated since I had last used it. When I first tried the above script right after the recompiling, it failed. But, I then quit Julia and reopened it and it worked fine. Perhaps that was what is causing your error?
How to handle cljs-ajax responses?
I am making a request from a clojurescript frontend with cljs-ajax to an API that responds with JSON but it seems like I need to do something to it before I can use it in cljs.
(defn all-pieces []
(GET "/art/pieces" {:handler ajax-success-handler}))
When I initialize my app-state I assign the key :all-pieces (all-pieces)
When I go to iterate over :all-pieces in a component I get the error Uncaught Error: [object Object] is not ISeqable.
(defn pieces-component []
[:ul (for [piece (:all-pieces #app-state)]
[:li (art-piece piece)])])
Edited re Pratley:
The code below now results in the state of all-pieces being {}, see anything wrong?
;; -------------------------
;; Remote Data
(defn all-pieces [handler]
(GET "/art/pieces" {:handler handler}))
;; -------------------------
;; State Management
(def app-state (atom
{:doc {}
:saved? false
:page-state {}
:all-pieces {}}))
(defn set-pieces-fresh []
(all-pieces (fn [pcs] swap! app-state assoc :all-pieces pcs)))
Don't set :all-peices to the result of (all-pieces).
The function ajax-success-handler should set :all-peices instead.
The result of (all-pieces) is the result of starting the asynchronous call, not the response. The handler is what gets called when the response arrives.
(fn [pcs] swap! app-state assoc :all-pieces pcs)
Does not do any swapping, as swap! needs to be in parens... it is just a function that returns pcs. Consider promoting it to a named function so you can test it separately:
(def app-state
(atom {:all-pieces {}}))
(defn pieces-handler [pcs]
(swap! app-state assoc :all-pieces pcs))
(defn fetch-pieces []
(GET "/art/pieces" {:handler pieces-handler}))
(fetch-pieces)
How to reference an earlier output in pry
In python's ipython you can reference the output in the following way: In [15]: s = "abcd" In [16]: s Out[16]: 'abcd' In [17]: Out[16] Out[17]: 'abcd' What is the equivalent for ruby's pry?
Using the _out_ and _in_ variables, as described in the docs. E.g.: _out_[16] or _out_[-1] for the last result.
Main class in clojure, leiningan (Conway's game of life)
Hi guys! I'm running Conway's game of life - but I think I had to modify the original version here: https://github.com/sebastianbenz/clojure-game-of-life to reference field.clj (rather than run.clj) in the project.clj file. After doing so, I can start the game by running
repl> (run-game)
However, it crashes on this method, which has no comments regarding input arguments.
Thus, my question is: what does this form (appear to do) from a Clojure syntax perspective?
(defn run-game
([engine seed]
(run-game engine seed
{:columns 50 :rows 50 :speed 500 :cellsize 10}))
([engine seed options]
(let [panel (field-panel engine seed options)
frame (field-frame panel)
timer (Timer. (options :speed) panel)]
(.start timer))))
UPDATE _
Why is "[engine-seed]" nested in a parentheses?
Why is this function recursive?
any other syntax level insights to how this function is designed?
I'm not 100% sure what you're asking, but it basically defines a function that either takes engine and seed arguments or engine, seed and options arguments.
If the option argument isn't specified, the function creates a default map {:columns 50 :rows 50 :speed 500 :cellsize 10} and calls the second form.
You'll have to look at the code to determine what engine and seed should be set to.
Then you can call it as:
(run-game engine seed)
or
(run-game engine seed {:columns 75 :rows 75 :speed 750 :cellsize 15})
Does that help?