MMap and SharedArrays - parallel-processing
I am confronting a very weird problem when parallelizing some function. I understand that I am supposed to post a MWE but I could not recreate the issue in a simple problem.
#everywhere function simulSample(Profits,ν,J,TerminalT,RelevantT,N,S,params,thresh;RelevantPer=RelevantPeriod)
Dₑ = rand([0],J,1) # Final Condition
Dᵢ = rand([0],J,1) # Initial Condition
tempData=SharedArray{Int8}(J,RelevantT,S*N)
#inbounds #sync #distributed for n=1:S*N
#for n=1:N
#println(n)
tempData[:,:,n]=solver(Dᵢ,Dₑ,Profits[:,:,n],continent,cont_lang,language,J,TerminalT,RelevantT,RelevantPeriod,params,thresh,ν[:,:,n])
end
return tempData
end
This function is being called by another function in an iterative procedure. In the first iteration it works, but in the second iteration I get the following error
SystemError: mmap: The operation completed successfully.
#windowserror#45(::Nothing, ::typeof(Base.windowserror), ::Symbol, ::Bool) at error.jl:148
windowserror at error.jl:148 [inlined]
#mmap#1(::Bool, ::Bool, ::typeof(Mmap.mmap), ::Mmap.Anonymous, ::Type{Array{Int8,3}}, ::Tuple{Int64,Int64,Int64}, ::Int64) at Mmap.jl:221
mmap(::Mmap.Anonymous, ::Type{Array{Int8,3}}, ::Tuple{Int64,Int64,Int64}, ::Int64) at Mmap.jl:186
_shm_mmap_array(::Type, ::Tuple{Int64,Int64,Int64}, ::String, ::UInt16) at SharedArrays.jl:670
shm_mmap_array(::Type, ::Tuple{Int64,Int64,Int64}, ::String, ::UInt16) at SharedArrays.jl:649
#call#3(::Bool, ::Array{Int64,1}, ::Type{SharedArray{Int8,3}}, ::Tuple{Int64,Int64,Int64}) at SharedArrays.jl:118
Type at SharedArrays.jl:105 [inlined]
#call#10 at SharedArrays.jl:161 [inlined]
Type at SharedArrays.jl:161 [inlined]
#call#15 at SharedArrays.jl:171 [inlined]
SharedArray{Int8,N} where N(::Int64, ::Int64, ::Int64) at SharedArrays.jl:171
#simulatedMoments#67(::Float64, ::Int64, ::typeof(simulatedMoments), ::Array{Float64,4}, ::Array{Float64,4}, ::Array{Float64,4}, ::Array{Float64,4}, ::Int64, ::Int64, ::Int64, ::Int64, ::Array{Any,3}, ::Array{Float64,1}, ::Int64, ::Int64, ::Int64, ::Float64) at 13-Estimation_Stable.jl:1255
simulatedMoments at 13-Estimation_Stable.jl:1232 [inlined]
#gmm_fun#70(::Float64, ::Float64, ::Int64, ::typeof(gmm_fun), ::SharedArray{Int64,3}, ::Array{Any,3}, ::Array{Float64,1}, ::Array{Float64,1}, ::Int64, ::Int64, ::Int64, ::Array{Float64,4}, ::Array{Float64,4}, ::Array{Float64,4}, ::Array{Float64,4}, ::Array{Array,1}, ::Array{Array,1}, ::Array{Array,1}, ::Array{Array,1}, ::Array{Float64,2}) at 13-Estimation_Stable.jl:1297
gmm_fun(::SharedArray{Int64,3}, ::Array{Any,3}, ::Array{Float64,1}, ::Array{Float64,1}, ::Int64, ::Int64, ::Int64, ::Array{Float64,4}, ::Array{Float64,4}, ::Array{Float64,4}, ::Array{Float64,4}, ::Array{Array,1}, ::Array{Array,1}, ::Array{Array,1}, ::Array{Array,1}, ::Array{Float64,2}) at 13-Estimation_Stable.jl:1284
obj_function_final(::Array{Float64,1}, ::Array{Any,1}) at 13-Estimation_Stable.jl:1370
top-level scope at util.jl:156
I am adding a MWE with the error that I get hoping it can make things more transparent:
using Distributed, SharedArrays
rmprocs()
addprocs()
big_array = rand(100,11,20000)
function donothing(a)
shared_array = convert(SharedArray,a)
end
for i=1:1000
donothing(big_array)
end
With the error:
SystemError: mmap: The operation completed successfully.
#windowserror#45(::Nothing, ::typeof(Base.windowserror), ::Symbol, ::Bool) at error.jl:148
windowserror at error.jl:148 [inlined]
#mmap#1(::Bool, ::Bool, ::typeof(Mmap.mmap), ::Mmap.Anonymous, ::Type{Array{Float64,3}}, ::Tuple{Int64,Int64,Int64}, ::Int64) at Mmap.jl:218
mmap(::Mmap.Anonymous, ::Type{Array{Float64,3}}, ::Tuple{Int64,Int64,Int64}, ::Int64) at Mmap.jl:186
_shm_mmap_array(::Type, ::Tuple{Int64,Int64,Int64}, ::String, ::UInt16) at SharedArrays.jl:670
shm_mmap_array(::Type, ::Tuple{Int64,Int64,Int64}, ::String, ::UInt16) at SharedArrays.jl:649
#call#3(::Bool, ::Array{Int64,1}, ::Type{SharedArray{Float64,3}}, ::Tuple{Int64,Int64,Int64}) at SharedArrays.jl:118
Type at SharedArrays.jl:105 [inlined]
Type at SharedArrays.jl:357 [inlined]
convert at SharedArrays.jl:369 [inlined]
donothing(::Array{Float64,3}) at mwproblem.jl:8
top-level scope at mwproblem.jl:12
Here is a MWE pattern for a scenario that should be similar to yours:
using Distributed
addprocs(4)
using SharedArrays
#everywhere function compute()
rand() + myid()*100
end
function dothejob()
s = SharedArray(zeros(10000000))
#sync #distributed for i in 1:10000000
s[i] = compute()
end
s
end
myres = dothejob();
Related
getting too many arguments error in prolog
customer(chuchi, goyal, 800.00).
customer(rishu, jain, 900.00).
get_cust_bal(X, Y, Z):-
customer(chuchi, Y, 800.00),
customer(X, Y, Z),
format('owes me Bal ~w', [X, Y, Z]).
the error is
Format error: too many arguments
ERROR: In:
ERROR: [11] format('owes me Bal ~w',[_4304,goyal|...])
ERROR: [9] toplevel_call(user:user: ...) at c:/program files/swipl/boot/toplevel.pl:1162
ERROR:
ERROR: Note: some frames are missing due to last-call optimization.
ERROR: Re-run your program in debug mode (:- debug.) to get more detail.
Type issue with prolog
I wanted to create a "function" that "returns" the even elements of a list :
elements_pairs([], []).
elements_pairs([H|T], R) :- H mod 2 =:= 0, elements_pairs(T,Rn), R is [H|Rn].
elements_pairs([H|T], R) :- not(H mod 2 =:= 0), elements_pairs(T,Rn), R is Rn.
This elements_pairs([1,3,4,6], R) produces this error:
ERROR: Type error: [] expected, found [4|6] (a compound) ("x" must hold one character)
ERROR: In: ERROR: [11] _3884 is [4|6] ERROR: [10]
elements_pairs([4,6],_3918) at path/tp.pl:21 ERROR: [9]
elements_pairs([3,4|...],_3956) at path/tp.pl:21 ERROR: [8]
elements_pairs([1,3|...],_3994) at path/tp.pl:21 ERROR: [7]
What's wrong ?
Thank you
WARMR algorithm in ALEPH (SWI-Prolog)
i am trying to use WARMR to find frequent relational patterns in my data; for this i am using ALEPH in SWI-Prolog. however, i am struggling to figure out how to do this and why my previous attempts did not work. i want to make a toy example work before i move on to my full data. for this i took the toy "train" data from the aleph pack page: http://www.swi-prolog.org/pack/list?p=aleph the Aleph manual states about the ar search: ar Implements a simplified form of the type of association rule search conducted by the WARMR system (see L. Dehaspe, 1998, PhD Thesis, Katholieke Universitaet Leuven). Here, Aleph simply finds all rules that cover at least a pre-specified fraction of the positive examples. This fraction is specified by the parameter pos_fraction. accordingly i have inserted :- set(search,ar). :- set(pos_fraction,0.01). into the background file (and deleted :- set(i,2).)) and erased the .n file of negative examples. i have also commented out all the determinations and the modeh declaration logic being that we are searching for frequent patterns, not rules (i.e. in a supervised context head would be an "output" variable and clauses in the body -- "inputs" trying to explain the output), i.e. it is an unsupervised task. now, the original trains dataset is trying to construct rules for "eastbound" trains. this is done by having predicates like car, shape, has_car(train, car) etc. originally all the background knowledge relating to these is located in the .b file and the five positive examples (e.g eastbound(east1).) in the .f file (+ five negative examples, e.g. eastbound(west1)., in the .n file). leaving files unchanged (save for the changes described above) and running induce. does not produce a sensible result (it would return ground terms like train(east1) as a "rule", for example). i have tried moving some of the background knowledge to the .f file but that did not produce anything sensible either. how do i go about constructing the .f and .b files? what should to into the positive examples file if we are not really looking to explain any positive examples (which would surely constitute a supervised problem) but instead to find frequent patterns in the data (unsupervised problem)? am i missing something? any help would be greatly appreciated.
First of all if you can use the original WARMR I think it is better. But I think you need to be an academic for free use. You can try asking for a license. https://dtai.cs.kuleuven.be/ACE/ To get association rules, I put all the examples I want in the f file. The n file can have examples in it or I think be empty. The only thing I change is to put : :- set(search,ar). :- set(pos_fraction,0.01). In the .b file. Keep the determinations and mode declarations. The set(i,2) limits the length of the query to having two additional literals (I think) so you might want this to be larger. ?-read_all(train). induce. You will then get an out of 'good clauses' which I think are the frequent queries. [good clauses] eastbound(A). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), long(B). [pos cover = 2 neg cover = 0] [pos-neg] [2] eastbound(A) :- has_car(A,B), open_car(B). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), shape(B,rectangle). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), wheels(B,2). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), load(B,rectangle,3). [pos cover = 1 neg cover = 0] [pos-neg] [1] eastbound(A) :- has_car(A,B), has_car(A,C). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), has_car(A,C). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), has_car(A,C). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), short(B). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), closed(B). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), shape(B,rectangle). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), wheels(B,2). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), load(B,triangle,1). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), has_car(A,C). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), has_car(A,C). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), long(B). [pos cover = 2 neg cover = 0] [pos-neg] [2] eastbound(A) :- has_car(A,B), open_car(B). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), shape(B,rectangle). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), wheels(B,3). [pos cover = 3 neg cover = 0] [pos-neg] [3] eastbound(A) :- has_car(A,B), load(B,hexagon,1). [pos cover = 1 neg cover = 0] [pos-neg] [1] eastbound(A) :- has_car(A,B), has_car(A,C). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), short(B). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), open_car(B). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), shape(B,rectangle). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), wheels(B,2). [pos cover = 5 neg cover = 0] [pos-neg] [5] eastbound(A) :- has_car(A,B), load(B,circle,1). [pos cover = 3 neg cover = 0] [pos-neg] [3] eastbound(A) :- has_car(A,B), open_car(B), shape(B,rectangle). [pos cover = 4 neg cover = 0] [pos-neg] [4] etc etc The rules are of the form eastbound(A):-blah blah. But it is only counting the eastbound examples. So think of this as example_covered(A):-blah blah
Creating Arrays of permutations
I have an array that looks like this
mainArray =>
[a] =>
[A]
[B]
[C]
[b] =>
[D]
[E]
[F]
[c] =>
[G]
[H]
[I]
I want to put these into an array like this:
secondArray =>
[0] => { [A], [D], [G] }
[1] => { [A], [E], [G] }
.
.
.
[n] => { [C], [F], [I] }
I am having trouble figuring out how to get the right number of elements in $secondArray that start with a certain element in { [A],[B], .. }. For example, how many start with [A].
This is what I think I have to do:
secondArray =>
[0] =>
[A]
secondArray =>
[0] =>
[A]
[D]
secondArray =>
[0] =>
[A]
[D]
[G]
secondArray =>
[0] =>
[A]
[D]
[G]
[1] =>
[A]
[D]
[H]
secondArray =>
[0] =>
[A]
[D]
[G]
[1] =>
[A]
[D]
[H]
[2] =>
[A]
[D]
[I]
secondArray =>
[0] =>
[A]
[D]
[G]
[1] =>
[A]
[D]
[H]
[2] =>
[A]
[D]
[I]
[3] =>
[A]
[E]
[G]
[4] =>
[A]
[E]
[H]
[5] =>
[A]
[E]
[I]
secondArray =>
[0] =>
[A]
[D]
[G]
[1] =>
[A]
[D]
[H]
[2] =>
[A]
[D]
[I]
[3] =>
[A]
[E]
[G]
[4] =>
[A]
[E]
[H]
[5] =>
[A]
[E]
[I]
[6] =>
[A]
[F]
[G]
[7] =>
[A]
[F]
[H]
[8] =>
[A]
[F]
[I]
and so on, but I can't really think of how to implement it...
Any help would be appreciated
You can generate the elements sequentially. To implement the iterator: -remember the array of input arrays, if there exists a next element. and an array of indexes of the same length. -if any input array is empty, the result set is empty and there is no next (first) element. -initialise the array of indexes to all zeroes. To get the next element: -If you remember there is no next element, fail. -compute the result: --start with an empty result --For each input array and its corresponding index ---Append input[index] to the result -compute the next set of indexes: --Iterate the indexes in reverse order. For each index ---Increment the index ---If the index now points past its corresponding input array ----Reset the index to zero ---Else ----Return the result, exiting the function. -Remember there is no next element -Return the result (last element). If you do want all combinations at once, the code simplifies slightly: -if any input array is empty, the result set is empty. -initialise the array of indexes to all zeroes. -compute the result and store in the result set. -while not done generating results: --Iterate the indexes in reverse order. For each index ---Increment the index ---If the index now points past its corresponding input array ----Reset the index to zero ---Else ----Compute the result from the current indexes ----Add the result to the result set ----Continue generating the results --(all indexes were reset to zero) finish generating the results. -return the result set.
In Ruby: ['A', 'B', 'C'].product(['D', 'E', 'F']) # => [['A', 'D'], ['A', 'E']... Array#product can accept multiple arrays.
Polymorphism in Clojure
Suppose I have a bunch of Clojure data structures, all of the same type - for example an object type defined by defrecord. What is the best way to get polymorphic behaviour across these structures? Would it be good practice to embed a function within the structure so that I can do something like: ((:my-method my-object) param1 param2) Or is there a better way?
You can use multimethods for dispatch,
(derive ::subtype ::type)
then you can dispatch based on type
(defmulti my-method
(fn [type] ((meta type) :type)))
(defmethod my-method ::subtype []
;;stuff...
)
You need to add :type to the objects metadata when you create it,
(with-meta [1 2 3] {:type ::subtype})
or I haven't tried this but since protocols and records creates java classes behind the scenes you dispatch based on that name.
If I understand your question, protocols are what you want to use. user=> (defprotocol P (foo [p x])) P user=> (defrecord R [a b] P (foo [_ x] [a b x])) user.R user=> (defrecord S [a b] P (foo [_ x] [x b a])) user.S user=> (def r (R. 1 2)) #'user/r user=> (def s (S. 1 2)) #'user/s user=> (foo r 3) [1 2 3] user=> (foo s 3) [3 2 1] In the above both R and S implements the foo function defined by protocol P, but they each do so differently. When the foo function is called, it dispatches on type of the first arg.