I can't get pool.map or pool.starmap to work the way I think it should - parallel-processing

So for the life of me I can't figure out which combination of pool.map() or pool.starmap() should be used to make this work as I want. The code below executes correctly, firing off (sequentially!) two instances of runSim and appending the results.
results = []
argsToRun = []
for n in range(2):
random.seed()
env = simpy.Environment() # Create the SimPy environment
env.clockRate = 1e9
argsToRun.append([env, adist, sdist, tdist, 108, intSpeed, runUntil])
for arg_list in argsToRun:
line = runSim(*arg_list)
results.append(line)
I was under the impression that something like the code below would be a simple way to run these instances in parallel (the results are not order-dependent)... but neither of them seem to work. Surely I'm missing something obvious?
with multiprocessing.Pool() as pool:
results = pool.map(runSim, argsToRun)
pool.close()
Nor....
with multiprocessing.Pool() as pool:
results = pool.starmap(runSim, argsToRun)
pool.close()

After lots of frustration and searching, it turns out that some of my parameters were not pickleable. This, combined with a bug in VSCode which prevented the debugger from running made it much harder to diagnose than it should have been.
Switching to the pathos versions of Pool() solved the problem.
for n in range(15):
random.seed()
env = simpy.Environment() # Create the SimPy environment
argsToRun.append((env, adist, sdist, tdist, 108, intSpeed, runUntil))
with pathos.helpers.mp.Pool as pool: ## tada!
results = pool.starmap(runSim, argsToRun)

Related

Reading serial output from Pyserial doesnt work reliably

I am connecting to device using code blow on MacOS and out of 100 times this code would make connection only 1 or two times and dones't respond(since there is no timeout) rest of the times.
ser = serial.Serial(port="/dev/xyz",timeout = None, baudrate=115200, parity = serial.PARITY_NONE, bytesize = serial.EIGHTBITS, stopbits = serial.STOPBITS_ONE)
def exitSer(ser):
print("Closing")
ser.close()
atexit.register(exitSer, ser)
if ser.is_open:
time.sleep(2)
while(1):
print(ser.readline().decode("utf-8"))
Could you please tell me how to use programs like fcntl etc to find if this port is completely free and available for use and how to set tty port's flags to free after making port free forcibly.
Once this works, I have to run this multithreaded where each thread is running different devices expecting output in lines. Any suggestions for that just in case this works.
def startSerial(tty_id):
ser = serial.Serial(port = tty_id, timeout = None)
ser.close()
ser.open()
if ser.isOpen():
print(ser.portstr, ":connection successful.")
return ser
else:
return False
Calling ser.close() before .open() fixed it. I tested it about 200 times and i haven't been dissappointed so far. I am testing it now in multithreaded and hopefully that works too.
Thank you everybody.

Does Matlab execute .m files differently from automation than from within the GUI?

I have a .m script that I've been running using Windows Task Scheduler, generally successfully, every 15 minutes for about a year (options: -automation -minimize -r remotedata -logfile logfile.txt;quit).
When I run the code manually in Matlab, everything behaves as expected.
However, when it is run as an automated script, it has two issues I can't resolve, that seem to indicate the code is not being executed the same way.
First, I have the following conditional:
~isempty(remoteData.Time(setdiff(1:end,ni))) which is terrible syntax, I know, but works just fine when I run the script manually. However, when it runs automated, it gives the error:
Error using setdiff (line 80) Not enough input arguments.
I corrected it to ~isempty(remoteData.Time(setdiff(1:height(remoteData),ni)))
but it made me curious.
Second, I have a webread function with a number of queries (see below) that executes normally when I have it open and hit "run", however, when running as an automation the dateutc query is ignored. This one is a bit more puzzling. Can anyone suggest a reason it might be failing to register, or how I might fix it? Debugging is difficult since it works as expected when I run it manually.
WUurl = 'http://weatherstation.wunderground.com/weatherstation/updateweatherstation.php';
WUID = '***';
WUpwd = '***';
WUdateutc = datestr(datenum(webData.Time(WDNewest-newTimes+i))+7/24,'yyyy-mm-dd HH:MM:SS');
WUwindspeedmph = num2str(webData.WndSpd(WDNewest-newTimes+i)*0.62);
WUwinddir = num2str(webData.WndDir(WDNewest-newTimes+i));
WUtempf = num2str(webData.AirTmp(WDNewest-newTimes+i)*1.8+32);
WUrainin = num2str(webData.Rain(WDNewest-newTimes+i)/25.4*4);
WUdailyrainin = num2str(sum(webData.Rain(WDMidnight:WDNewest-newTimes+i))/25.4);
WUbaromin = num2str(webData.BarPress(WDNewest-newTimes+i)*.0295);
WUhumidity = num2str(webData.RelHum(WDNewest-newTimes+i));
gamma = log(webData.RelHum(WDNewest-newTimes+i)/100)+ ...
(17.67*webData.AirTmp(WDNewest-newTimes+i))/ ...
(243.5+webData.AirTmp(WDNewest-newTimes+i));
WUdewptf = num2str((243.5*gamma)/(17.67-gamma)*1.8+32); % Magnus formula estimation
WUsolarradiation = num2str(webData.NetRad_Wm2(WDNewest-newTimes+i));
WUsoiltempf = num2str(nanmean(webData{WDNewest,20:3:77})*1.8+32);
WUsoilmoisture = num2str(nanmean(webData{WDNewest,18:3:75}));
options = weboptions('Timeout',newTimes);
WU_debugging = webread(WUurl,...
'ID',WUID,...
'PASSWORD',WUpwd,...
'dateutc',WUdateutc,...
'windspeedmph',WUwindspeedmph,...
'winddir',WUwinddir,...
'tempf',WUtempf,...
'rainin',WUrainin,...
'dailyrainin',WUdailyrainin,...
'baromin',WUbaromin,...
'humidity',WUhumidity,...
'dewptf',WUdewptf,...
'solarradiation',WUsolarradiation,...
'soiltempf',WUsoiltempf,...
'soilmoisture',WUsoilmoisture,...
'action','updateraw',...
options);

metriks log to file not working

I wrote a basic program to test the ruby metriks gem
require 'metriks'
require 'metriks/reporter/logger'
#registry = Metriks::Registry.new
#logger = Logger.new('/tmp/metrics.log')
#reporter = Metriks::Reporter::Logger.new(:logger => #logger)
#reporter.start
#registry.meter('tasks').mark
print "Hello"
#registry.meter('tasks').mark
#reporter.stop
After i execute the program, there is nothing in the log other than it got created.
$ cat /tmp/metrics.log
# Logfile created on 2015-06-15 14:23:40 -0700 by logger.rb/44203
You should either pass in your own registry while instantiating Metriks::Reporter::Logger or use the deafult registry (Metrics::Resgitry.default) if you are using a logger to log metrics.
Also the default log write interval is 60 seconds, your code completes before that so even if everything is setup okay it won't get recorded. So, since you want to use your own registry, this should work for you (I'm adding a little sleep since I'm gonna use an interval of 1 second) :
require 'metriks'
require 'metriks/reporter/logger'
#registry = Metriks::Registry.new
#logger = Logger.new('/tmp/metrics.log')
#reporter = Metriks::Reporter::Logger.new(:logger => #logger,
:registry => #registry
:interval => 1)
#reporter.start
#registry.meter('tasks').mark
print "Hello"
#registry.meter('tasks').mark
# Just giving it a little time so the metrics will be recorded.
sleep 2
#reporter.stop
But I don't really think short intervals are good.
UPDATE : Also I think #reporter.write will help you write down the logs instantly regardless of the time interval. So you don't have to use sleep (better).

Garry's Mod Custom Cvar not changing

I'm making a script for Garry's Mod and it's almost complete but for some reason when ever I type in the new Cvar i made it won't change host_framerate. if anyone know what the problem is your a life saver.
local speedCvar = CreateClientConVar( "speedhack_enabled", 0, true, false )
local speedHackCvar = CreateClientConVar( "speedhack_enabled", "0", true, false )
local speedHack = SpeedHackCvar:GetString()
local speed = function()
if (speedCvar:GetInt() == 1) then
speedHack = SpeedHackCvar:GetString()
RunConsoleCommand("host_framerate", speedHack)
else
speedHack = SpeedHackCvar:GetString()
RunConsoleCommand("host_framerate " , speedHack)
end
end
Remove the local from first of each line (creation of your objects) to make your objects go globally across the lua. Your problem is that your functions are localized, which means they can only be accessed from the file/code block or chunk they're created in.
Oh and also you had a mistake on your code, which Is fixed (From reading comments on your post).

Using parfor and labSend/labRecieve

I want to run two matlab scripts in parallel for a project and communicate between them. The purpose of this is to have one script do image analysis and sending the results to the other which will use it for more calculations (time consuming, but not related to the task of finding stuff in the images). Since both tasks are time consuming, and should preferably be done in real time, I believe that parallelization is necessary.
To get a feel for how this should be done I created a test script to find out how to communicate between the two scripts.
The first script takes a user input using the built in function input, and then using labSend sends it to the other, which recieves it, and prints it.
function [blarg] = inputStuff(blarg)
mpiInit(); %added because of error message, but do not work...
for i=1:2
labBarrier; % added because of error message
inp = input('Enter a number to write');
labSend(inp);
if (inp == 0)
break;
else
i = 1;
end
end
end
function [ blarg ] = testWrite( blarg )
mpiInit(); % added because of error message, but does not help
par = 0;
if ( blarg == 0)
par = 1;
end
for i = 1:10
if (par == 1)
labBarrier
delta = labReceive();
i = 1;
else
delta = input('Enter number to write');
end
if (delta == 0)
break;
end
s = strcat('This lab no', num2str(labindex), '. Delta is = ')
delta
end
end
%%This is the file test_parfor.m
funlist = {#inputStuff, #testWrite};
matlabpool(2);
mpiInit(); % added because of error message, but does not help
parfor i=1:2
funlist{i}(0);
end
matlabpool close;
Then, when the code is run, the following error message appears:
Starting matlabpool using the 'local' profile ... connected to 2 labs.
Error using parallel_function (line 589)
The MPI implementation has not yet been loaded. Please
call mpiInit.
Error stack:
testWrite.m at 11
Error in test_parfor (line 8)
parfor i=1:2
Calling the method mpiInit does not help... (Called as shown in the code above.)
And nowhere in the examples that mathworks have in the documentation, or on their website, show this error or what to do with it.
Any help is appreciated!
You would typically use constructs such as labSend, labRecieve and labBarrier within an spmd block, rather than a parfor block.
parfor is intended for implementing embarrassingly parallel algorithms, in other words algorithms that consist of multiple independent tasks that can be run in parallel, and do not require communication between tasks.
I'm stretching my knowledge here (perhaps someone more expert can correct me), but as I understand things, it does not set up an MPI ring for communication between workers, which is probably the explanation for the (rather uninformative) error message you're getting.
An spmd block enables communication between workers using labSend, labRecieve and labBarrier. There are quite a few examples of using them all in the documentation.
Sam is right that the MPI functionality is not enabled during parfor, only during spmd. You need to do something more like this:
spmd
funlist{labindex}(0);
end
(Sam is also quite right that the error message you saw is pretty unhelpful)

Resources