I'm confused as to how to declare which parts of a program are accessible or not from the different workers. On a relatively low level in a program of mine I have a for loop I want to parallelize.
module module_name
[...]
addprocs(3)
totalsum = #parallel (+) for i in 1:large_number
tmp_sum = 0
for j in 1:num
... # calls f1 f2
end
tmp_sum # not sure how to 'return' the result, the examples have a conveniantly placed calculation at the end
end
rmprocs([2 3 4])
[...]
end
As I understand I'd have to put the #everywhere decorator infront of f1 and f2. But the program fails far before with the additional workers complaining that UndefVarError: module_name not defined, and I have no clue how to fix that.
I feel like I’ve missed something needed for setting up parallel processing. As I understood it other than writing the actual #parallel part, one needs to addprocs and then add the #everywhere decorator to those functions used inside the loop. Is that really it?
I know pmap is better suited for what I'm doing here but I wanted to get the simpler option to work first (I'd need to pass several arguments the pmap function).
Related
I am trying to find out how to proper way to use RemoteChannel inside of a macro. In a function or the REPL, the following code works:
addprocs(3)
r = RemoteChannel(3)
#spawnat(3,put!(r,10))
fetch(r) # Gives 10
However, if I put that same stuff in a macro:
macro rrtest(src,val)
quote
r = RemoteChannel($(esc(src)))
#spawnat($(esc(src)), put!(r, $(esc(val))))
println(fetch(r))
end
end
and then call it with the same arguments
#rrtest(3,10)
then the REPL just stalls. Is there something wrong with using RemoteChannels like this?
macro rrtest(src,val)
quote
r = RemoteChannel($(esc(src))) #Using a `Future` here maybe be better
remotecall_wait(r_i->put!(r_i, $(esc(val))), $(esc(src)), r)
wait(r);
println(fetch(r))
end
end
The wait(r) should not be required -- fetch is supposed call wait when used on a Future or RemoteChannel.
But it does seem to be, sometimes.
Changing the #spawnat to a remotecall means you can pass in the r, without that, it gets. I think there are off things with how macro-hygine nests with closures created themself with macros. (#spawnat) creates closures inside another macro. It is awkaward to reasonabout.
In general I find #spawnat harder to reason about, than remote_call.
The reason it needs to be remotecall_wait is because otherwise, there is no garentee when its contents will run. Which means what is happening to r is itself unclear. I feel like it should be safe, but it does not seem to be.
I think because the waiting for r, rather than waiting for the remotecall that sets r is never certain to allow that remotecall to run.
In Conclusion:
prefer remotecall to #spawnat, particularly in macros, for sake of being easier to reason about.
Sometimes you have to wait for things before fetching them. That is probably a bug
sometimes waiting for X, when X is to be set by a remotecall (or #spawnat) returning future Y, requires waiting for Y first. Also probably a bug
I have a code that iterates through two ranges. Please see the example below:
(0..6).each do |wday|
(0..23).each do |hour|
p [wday,hour]
end
end
Although this seems very concise and readable, sometimes 5 lines can be too much. One might want to write a more vertically compact code.
(0..6).to_a.product((0..23).to_a).each do |wday,hour|
p [wday, hour]
end
Above was my try, but the code looks very artificial to me. Am I missing something? Does ruby have a preferred way for this type of loop collapsing? If not, are there other alternatives to this workaround?
The following is slightly cleaner version of your loop:
[*0..6].product([*0..23]).each do |wday,hour|
p [wday, hour]
end
This approach does have the disadvantage of expanding the ranges into memory.
I think my preferred way of "collapsing" loops though, especially if the specific loop structure occurs in multiple places, is to turn the nested loops into a method that takes a block and yields to it. E.g.
def for_each_hour_in_week
(0..6).each do |wday|
(0..23).each do |hour|
yield wday,hour
end
end
end
for_each_hour_in_week do |wday,hour|
p [wday,hour]
end
This keeps the deep nesting out of the way of your logic, and makes your intent clear.
I what to optimize my code. I have 3 option don't know which is better for memory in Lua:
1)
local Test = {}
Test.var1 = function ()
-- Code
end
Test.var2 = function ()
-- Code
end
2) Or
function var1()
-- Code
end
function var2()
-- Code
end
3) Or maybe
local var1 = function ()
-- Code
end
local var2 = function ()
-- Code
end
Quoting from Lua Programming Gem, the two maxims of program optimization:
Rule #1: Don’t do it.
Rule #2: Don’t do it yet. (for experts only)
Back to your examples, the second piece of code is a little bit worse as the access to global ones is slower. But the performance difference is hardly noticeable.
It depends on your needs, the first one uses an extra table than the third one, but the namespace is cleaner.
None will really affect memory, barring the use of a table in #1 (so some 40 bytes + some per entry).
If its performance you want, then option #3 is far better, assuming you can access said functions at the local scope.
If it's about memory usage more than processing and you're using object-oriented programming where you're instantiating multiple instances of Test as you showed above, you have a fourth option with metatables.
TestMt = {}
TestMt.func1 = function(self, ...)
...
end
TestMt.func2 = function(self, ...)
...
end
TestMt.func3 = function(self, ...)
...
end
function new_test()
local t = {}
t.data = ...
setmetatable(t, {__index = TestMt})
return t
end
foo = new_test()
foo:func1()
foo:func2()
foo:func3()
If you're doing object-oriented kind of programming, metatables can lead to a massive savings in memory (I accidentally used over 1 gigabyte once for numerous mathematical vectors this way, only to reduce it down to 40 megabytes by using the metatable).
If it's not about objects and tables that get instantiated many times, and just about organizing your globally-accessible functions, worrying about memory here is ridiculous. It's like putting the entirety of your lua code into one file in order to reduce file system overhead. You're talking about such negligible savings that you should really need an extraordinary use case backed by meticulous measurements to even concern yourself with that.
If it's about processing, then you can get some small improvements by keeping your global functions out of nested tables, and by favoring locals when possible.
It has been a while since I've used Mathematica, and I looked all throughout the help menu. I think one problem I'm having is that I do not know what exactly to look up. I have a block of code, with things like appending lists and doing basic math, that I want to define as a single variable.
My goal is to loop through a sequence and when needed I wanted to call a block of code that I will be using several times throughout the loop. I am guessing I should just put it all in a loop anyway, but I would like to be able to define it all as one function.
It seems like this should be an easy and straightforward procedure. Am I missing something simple?
This is the basic format for a function definition in Mathematica.
myFunc[par1_,par2_]:=Module[{localVar1,localVar2},
statement1; statement2; returnStatement ]
Your question is not entirely clear, but I interpret that you want something like this:
facRand[] :=
({b, x} = Last#FactorInteger[RandomInteger[1*^12]]; Print[b])
Now every time facRand[] is called a new random integer is factored, global variables b and x are assigned, and the value of b is printed. This could also be done with Function:
Clear[facRand]
facRand =
({b, x} = Last#FactorInteger[RandomInteger[1*^12]]; Print[b]) &
This is also called with facRand[]. This form is standard, and allows addressing or passing the symbol facRand without triggering evaluation.
I am working on a code base that has many modules nested 4 or 5 deep. Right now, this results in our code being heavily indented from the beginning.
Is there an acceptable way of putting multiple module declarations on the same line?
For example,
module A
module B
#do stuff
end
end
Is there a way to make it something like this?
module A::B
#do stuff
end
Though the previous block doesn't work, I was able to get this next one to work, however I am not sure if this is considered acceptable code construction.
module A module B
#do stuff
end end
You can use ; instead of \n safely in Ruby source files. Newlines before end are not important.
module A ; module B
#do stuff
end end
Or for example:
def sqr x ; x*x end
etc.
I think you've answered it yourself - your third segment looks pretty bad to my eye.
But, more to the point, if you did write module A::B and module A had never been defined before, you would be implicitly defining an (empty) module A, which doesn't seem very useful. And, once you've defined module A once, you're welcome to write module A::B to define module B. So it seems actively good to me that you can't use your second example.
Do this, though it's a bit naughty:
class Module
def const_missing(m)
const_set(m, Module.new)
end
end
module A::B
def self.hello_my_friend
:hello_my_friend
end
end
A::B.hello_my_friend #=> :hello_my_friend
Ruby doesn't use significant whitespace in the same way as Python does (though it can provide you with warnings if the indentation seems wonky), so you can simply not indent the code if you don't want to do so. It'd just make the code harder to read.
Incidentally, you may want to check whether four or five levels of modules indicates a code smell.