To do some exercise to be more familiar with MPI, i installed MS-MPI on my windows 10 machine, and then mpi4py (python MPI). I tried a hello_world code:
from mpi4py import MPI
def main ():
comm = MPI. COMM_WORLD
rank = comm . Get_rank ()
size = comm . Get_size ()
print " hello from " + str( rank ) + " in " + str( size )
if __name__ == " __main__ ":
main ()
Then, with a windows command as admin i executed the following command:
mpiexec -n 8 python MPI_Test.py
I get:
User credentials needed to launch processes: account (domain\user)
[DESKTOP-3CFSBJ8\Hazem]:
I did a registration, as mpiexec - register from username/pwd, then execute again that command, and i get the following error:
Credentials for user rejected connecting to host.
THE PROBLEM COMES WHEN EXECUTING THE COMMAND mpiexec.
I got the same issue, the solution is:
Type ”mpiexec -n 3 cpi.exe” to run the sample program. You will get a response like this:
”user credentials needed to launch process”
Type your Windows username and Windows password, the sample program will run.
In order not to enter credentials every time you run mpiexec, you can register your username
and password by command ”mpiexec -register”
source: https://www.cmpe.boun.edu.tr/sites/default/files/mpi_install_tutorial.pdf
Related
My topology is: Laptop -> 1st Jumphost (my company) -> 2nd jumphost (my clients company) -> Various network devices (my clients network devices).
The network devices are only accessible from the 2nd jumphost, and the 2nd jumphost is only accessible from the 1st jumphost. So I'm using netmiko in Python to achieve this. My code is below.
The 1st block of code SSH's to the 1st jumphost, and then from there SSH's to the 2nd jumphost. This works correctly.
The 2nd block of code then opens a text file containing the hostnames or IP's of the individual network devices that need to be queried. For each host in that file, it SSH's to it, issues the "show version" command and then disconnects from the device (using "exit") so that the session is returned to the 2nd jumphost, ready for the next device in the file.
This works correctly for the very first device, but crashes upon the "output = device.send_command('exit')" line. Netmiko claims that the pattern is not detected. I think I understand why, because netmiko is using the name in the hostname prompt, when this changes back to the 2nd jumphost hostname upon the disconnect it gets confused and throws an error. If this is the case I have 2 questions:
How come it copes OK when moving from the 1st jumphost to the 2nd jumphost AND from the 2nd jumphost to the network device. In both of these cases the hostname prompt also changes...
What's the solution? How can I safely move between the 2nd jumphost and network devices in order to achieve the loop?
from netmiko import ConnectHandler
import time
jump1 = "x.x.x.x"
jump2 = "y.y.y.y"
jump1_username = "myusername"
jump1_password = "mypassword"
jump2_username = "myusername"
jump2_password = "mypassword"
jump_type = "linux"
cmd_jump = "ssh " + jump2_username + "#" + jump2 + "\n"
device = ConnectHandler(device_type=jump_type, ip=jump1, username=jump1_username, password=jump1_password) # ssh to 1st jumphost
output = device.send_command('cat /proc/sys/kernel/hostname') #just shows me that login worked
print(output, flush=True) # just shows me that login worked
device.write_channel(cmd_jump) # enters ssh command for 2nd jumphost
time.sleep(1)
device.write_channel(jump2_password + "\n") # enters password for 2nd jumphost
time.sleep(1)
output = device.send_command('cat /proc/sys/kernel/hostname') #just shows me that second login worked
print(output, flush=True) # just shows me that second login worked
host_list = open(r'C:\device_list.txt','r') # a simple list of IP addresses you want to connect to each one on a new line
for host in host_list: # loop through network devices
host = host.strip()
cmd_device = "ssh " + host
device.write_channel(cmd_device + "\n") # ssh to each device
time.sleep(1)
device.write_channel(jump2_password + "\n") # enter ssh password (credientals are the same as the 2nd jumphost)
time.sleep(1)
output = device.send_command('sh ver') # run show version command
print(output, flush=True)
output = device.send_command('exit') ' disconnect from network device to return to the 2nd jumphost
time.sleep(1)
print(output, flush=True)
Ignore me, I've solved my own problem. It's because I wasn't using the "write_channel" to enter the 'exit' command. Doh!
NOTE 1- All files are running using cmd in my profile and fetching
correct results.But not with the windows task scheduler.
**> NOTE 2- I finally got a lead that glob.glob and os.listdir is not
working in the windows task scheduler in my python script in which I
am making a connection to a remote server, but it is working in my
local using cmd and pycharm.**
**
print("before for loop::", os.path.join(file_path, '*'))
print(glob.glob( os.path.join(file_path, '*') ))
for filename in glob.glob( os.path.join(file_path, '*') ):
print("after for loop")
**
While running above .py script I got: before for loop:: c:\users\path\dir\*
While executing print(glob.glob( os.path.join(file_path, '*') )) giving "[]" and not able to find why?
I followed this StackOverflow link for setting up Windows Scheduler for python by referring to MagTun comment:Scheduling a .py file on Task Scheduler in Windows 10
Currently, I am having scheduler.py which is calling the other 4 more .py files.
When I try to run Scheduler.py from Windows Task SCHEDULER,
It runs Scheduler.py and then after 1 minute it runs all other 4 .py files and exit within a seconds. Not giving any output in elastic search.
I used this for cmd script:
#echo off
cmd /k "cd /d D:\folder\env\Scripts\ & activate & cd /d D:\folder\files & python scheduler.py" >> open.log
timeout /t 15
In this above cmd command, It is not saving anything in open.log when running with windows task scheduler.
Script with multiple .py subprocess schedulers is like this:
from apscheduler.schedulers.blocking import BlockingScheduler
import datetime
from subprocess import call
from datetime import datetime
import os
from apscheduler.schedulers.blocking import BlockingScheduler
def a():
call(['python', r'C:\Users\a.py'])
def b():
call(['python', r'C:\Users\b.py'])
def c():
call(['python', r'C:\Users\c.py'])
def d():
call(['python', r'C:\Users\d.py'])
if __name__ == '__main__':
scheduler = BlockingScheduler()
scheduler.add_job(a, 'interval', minutes=1)
scheduler.add_job(b, 'interval', minutes=2)
scheduler.add_job(c, 'interval', minutes=1)
scheduler.add_job(d, 'interval', minutes=2)
print('Press Ctrl+{0} to exit'.format('Break' if os.name == 'nt' else 'C'))
try:
scheduler.start()
print("$$$$$$$$$$$$$$$$$$")
except (KeyboardInterrupt, SystemExit):
print("****#####")
pass
Having the same bizar issue. Works like a charm when running as user. With a windows task the glob query returns no results.
Edit: was using a network share by its mapping name. Only works when using the full UNC path (including the server name).
I am using a cluster (similar to slurm but using condor) and I wanted to run my code using VS code (its debugger specially) and it's remote sync extension.
I tried running it using my debugger in VS code but it didn't quite work as expected.
First I logged in to the cluster using VS code and remote sync as usual and that works just fine. Then I go ahead an get an interactive job with the command:
condor_submit -i request_cpus=4 request_gpus=1
then that successfully gives a node/gpu to use.
Once I have that I try to run the debugger but somehow it logs me out from the remote session (and it looks like it goes to the head node from the print statements). That's NOT what I want. I want to run my job in the interactive session in the node/gpu I was allocated. Why is VS code running it in the wrong place? How can I run it in the right place?
Some of the output from the integrated terminal:
source /home/miranda9/miniconda3/envs/automl-meta-learning/bin/activate
/home/miranda9/miniconda3/envs/automl-meta-learning/bin/python /home/miranda9/.vscode-server/extensions/ms-python.python-2020.2.60897-dev/pythonFiles/lib/python/new_ptvsd/wheels/ptvsd/launcher /home/miranda9/automl-meta-learning/automl/automl/meta_optimizers/differentiable_SGD.py
conda activate base
(automl-meta-learning) miranda9~/automl-meta-learning $ source /home/miranda9/miniconda3/envs/automl-meta-learning/bin/activate
(automl-meta-learning) miranda9~/automl-meta-learning $ /home/miranda9/miniconda3/envs/automl-meta-learning/bin/python /home/miranda9/.vscode-server/extensions/ms-python.python-2020.2.60897-dev/pythonFiles/lib/python/new_ptvsd/wheels/ptvsd/launcher /home/miranda9/automl-meta-learning/automl/automl/meta_optimizers/differentiable_SGD.py
--> main in differentiable SGD
hello world torch_utils!
vision-sched.cs.illinois.edu
Files already downloaded and verified
Files already downloaded and verified
Files already downloaded and verified
-> initialization of DiMO done!
---> i = 0, iteration/it 1 about to start
lp_norms(mdl) = 18.43514633178711
lp_norms(meta_optimized mdl) = 18.43514633178711
[e=0,it=1], train_loss: 2.304989814758301, train error: -1, test loss: -1, test error: -1
---> i = 1, iteration/it 2 about to start
lp_norms(mdl) = 18.470401763916016
lp_norms(meta_optimized mdl) = 18.470401763916016
[e=0,it=2], train_loss: 2.3068909645080566, train error: -1, test loss: -1, test error: -1
---> i = 2, iteration/it 3 about to start
lp_norms(mdl) = 18.548133850097656
lp_norms(meta_optimized mdl) = 18.548133850097656
[e=0,it=3], train_loss: 2.3019633293151855, train error: -1, test loss: -1, test error: -1
---> i = 0, iteration/it 1 about to start
lp_norms(mdl) = 18.65604019165039
lp_norms(meta_optimized mdl) = 18.65604019165039
[e=1,it=1], train_loss: 2.308889150619507, train error: -1, test loss: -1, test error: -1
---> i = 1, iteration/it 2 about to start
lp_norms(mdl) = 18.441967010498047
lp_norms(meta_optimized mdl) = 18.441967010498047
[e=1,it=2], train_loss: 2.300947666168213, train error: -1, test loss: -1, test error: -1
---> i = 2, iteration/it 3 about to start
lp_norms(mdl) = 18.545459747314453
lp_norms(meta_optimized mdl) = 18.545459747314453
[e=1,it=3], train_loss: 2.30662202835083, train error: -1, test loss: -1, test error: -1
-> DiMO done training!
--> Done with Main
(automl-meta-learning) miranda9~/automl-meta-learning $ conda activate base
(automl-meta-learning) miranda9~/automl-meta-learning $ hostname vision-sched.cs.illinois.edu
Doesn't even run without debugging mode
The problem is more serious than I thought. I can't run the debugger in the interactive session but I can't even "Run Without Debugging" without it switching to the Python Debug Console on it's own. So that means I have to run things manually with python main.py but that won't allow me to use the variable pane...which is a big loss!
What I am doing is switching my terminal to the conoder_ssh_to_job and then clicking the button Run Without Debugging (or ^F5 or Control + fn + f5) and although I made sure to be on the interactive session at the bottom in my integrated window it goes by itself to the Python Debugger window/pane which is not connected to the interactive session I requested from my cluster...
related:
gitissue: https://github.com/microsoft/vscode-remote-release/issues/1722
quora: https://qr.ae/TqCiu8
reddit: https://www.reddit.com/r/vscode/comments/f1giwi/how_to_run_code_in_a_debugging_session_from_vs/
You can try reversing the order of operations; first submitting the job, obtaining the name of the compute node allocated to you, then instructing VSCode to connect to the compute node rather than the login node.
So first would be
condor_submit -i request_cpus=4 request_gpus=1
and noting the name of the compute node. Assuming node001 in the following.
Then, open VSCode on your laptop, click on the Remote Development extension icon and choose "Remote SSH: Connect to Host...". Choose "+ Add new SSH host...". In the "Enter SSH command" box, add the following:
ssh -J vision-sched.cs.illinois.edu miranda9#node001
The VSCode will ask you which SSH configuration file it should update. Make sure to review that configuration: specify the SSH keys if needed, the user name, etc. Also make sure you have the vision-sched.cs.illinois.edu correctly configured in that file.
Then you can choose that host to connect to. VSCode will then execute on the compute node, and will be disconnected when the allocation finishes.
I stumbled upon a related issue recently (I wanted to use VsCode interactive Python capabilities on a compute node) and the above weren't working but this solved it:
ssh to the remote cluster ssh cluster
inside the remote cluster, add my public key to the authorized keys, so typically append the content of ~/.ssh/id_rsa.pub (local machine) to .ssh/authorized_keys (remote cluster)
allocate some resources inside the cluster (this particular cluster uses slurm and not condor so in this case I use something like srun --pty bash)
get the name of the compute node, typically visible in the command line as username#nodename). For argument's sake, let's imagine I get a generic name like node001
for simplicity on my local machine, modify the ~/.ssh/config file and edit it as:
Host cluster
# stuff written
Host node*
HostName %h
ProxyJump cluster
User $USERNAME
Now I'm able to ssh to it from my local machine (as long as the compute node is running) with ssh node001.
In VsCode this boils down to
CTRL+P > Remote-SSH: Connect to Host...
type in the name of the node, here node001
you get connected to the node, now every interactive python you run (including jupyter and jupytext) will have access to your allocated resources
I don't know how generic this solution is, I hope it'll help at least somebody !
Here is a simpler workaround:
on the remote server create a file named bash somewhere for example /home/myuser/pathto/bash
make it executable using chmod +x bash
write salloc [your desired options for the interactive job] in the bash file
In vscode Settings search for Automation Shell: Linux and click on the "Edit in settings.js"
change the line to "terminal.integrated.automationShell.linux": "/home/myuser/pathto/bash" and save it (use the absolute path. for example ~/pathto/bash didn't work for me)
Done :)
now every time you run the debugger it will first ask for the interactive job and the debugger will run on it. but take in to consider that this is also applied to tasks you run in tasks.json.
also you can use srun instead of salloc. for example srun --pty -t 2:00:00 --mem=8G bash
I need a help in wsadmin.sh scripting along with jython.
I am creating a script to provide console access to the users via wsadmin.sh
I am able to make it work if I hardcode the user name details inside the consoleacces.py file but My requirement is to give the user name details at run-time so that i can use the script for multiple times for different user.
Working :
wsadmin.sh -lang jython -f /tmp/consoleaccess.py
content of consoleaccess.py
AdminTask.mapUsersToAdminRole('[-accessids [user:defaultWIMFileBasedRealm/employeenumber=123,ou=people,ou=country,o=office] -roleName administrator -userids user1]')
AdminConfig.save()
agBean=AdminControl.queryNames('type=AuthorizationGroupManager,process=dmgr,*');null=AdminControl.invoke(agBean, 'refreshAll')
Not working
wsadmin.sh -lang jython -f /tmp/consoleaccess.py 123 user1 administrator
content of consoleaccess.py
import sys
AdminTask.mapUsersToAdminRole('[-accessids [user:defaultWIMFileBasedRealm/employeenumber=sys.argv[1],ou=people,ou=americas,o=SIAM_ED] -roleName sys.argv[3] -userids sys.argv[2]]')
AdminConfig.save()
agBean=AdminControl.queryNames('type=AuthorizationGroupManager,process=dmgr,*');null=AdminControl.invoke(agBean, 'refreshAll')
Error :
WASX7209I: Connected to process "dmgr" on node host1 using SOAP connector; The type of process is: DeploymentManager
WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: "[123, user1, administrator]"
WASX7017E: Exception received while running file "/tmp/consoleaccess.py"; exception information: com.ibm.ws.scripting.ScriptingException: WASX8009E: Invalid parameter: [-accessids [user:defaultWIMFileBasedRealm/employeenumber=sys.argv[1],ou=people,ou=americas,o=SIAM_ED] -roleName sys.argv[3] -userids sys.argv[2]]
Update the script as below.
Note : In wsadmin Jython, the name of the program, or script, is not part of sys.argv. So your first argument is sys.argv[0] and not sys.argv[1]
import sys
userEmpNo=sys.argv[0]
userName=sys.argv[1]
userRole=sys.argv[2]
AdminTask.mapUsersToAdminRole('[-accessids [user:defaultWIMFileBasedRealm/employeenumber=' +userEmpNo+ ',ou=people,ou=americas,o=SIAM_ED] -roleName ' +userRole+ ' -userids ' +userName+ ']')
AdminConfig.save()
agBean=AdminControl.queryNames('type=AuthorizationGroupManager,process=dmgr,*');null=AdminControl.invoke(agBean, 'refreshAll')
and run the script as
wsadmin.sh -lang jython -f /tmp/consoleaccess.py 123 user1 administrator
How about -- to separate the flags from the arguments?
wsadmin.sh -lang jython -f /tmp/consoleaccess.py -- 123 user1 administrator
and moving the sys.argv out of the string:
'...employeenumber=' + sys.argv[1] + ',ou=people,ou=americas,o=SIAM_ED] -roleName ' + sys.argv[3] + ' -userids ' + sys.argv[2]])
I Have a Huawei E220 HSDPA Modem on linux xubuntu
I wanted to recieve sms and reply automatically to the sender.
I Use gammu and Gammu-smsd to do this.
To automatically send sms back I added runOnRecieve = /path/to/bash/file into the /etc/gammu-smsdrc configuration-file.
Here is the script:
#!/bin/bash
str=$SMS_1_TEXT //string containing text from sender
tlf=$SMS_1_NUMBER //containing number from sender
tlf=${tlf:3}
if test "$str" = "today"; then
echo "[Weather for today in Norway]
Sol, noe overskyet
[Vind fra sørøst]
Ha en fin dag!" | gammu-smsd-inject TEXT $tlf -unicode -autolen 200
else
echo "fail" >> /home/mattis/sms.txt
fi
This is how I start the daemon
$ sudo gammu-smsd
This works if I run the bash script from terminal using test-input, but when the program gammu-smsd calls the script I get.
gammu-smsd[3183]: Process failed with exit status 2
Now i can remove "gammu-smsd-inject" from the code and replace with "gammu sendsms" , but that would just give me gibberish letters instead of "æøå and [ ]" when received back to the mobile.
Hoping for positive answers.
--//--Working code--//--
The thing is: Gammu sms inject acctually sends data to mysql database called smsd.
Creating this database:
This should be created as specified in wammu sql database. Storing the SQL script for creating tables in MySQL database will able you to import it with phpmyadmin(gui) or any other way to interface mysql.
Run on recieve
Add to the end of /etc/gammu-smsdrc --configuration file for gammu
runOnRecieve = /path/to/bash/file
Open the /path/to/bash/file
#!/bin/bash
str=$SMS_1_TEXT //codeword "weather"
tlf=$SMS_1_NUMBER //+47 41412424
tlf=${tlf:3} //remove +47
if test "$tlf" = "41412424"; then
toSend = "[Weather for today in Norway]"
else
toSend = "[you are not part of this group]"
echo "Someone outside the group send to this number" > /home/user/activity.txt
fi
mysql --host=localhost --user=username --password=pw smsd << EOF
INSERT INTO outbox (
DestinationNumber,
TextDecoded,
CreatorID,
RelativeValidity
) VALUES (
'$tlf',
'$toSend',
'Program',
'255'
);
EOF
Start the daemon
$ sudo gammu-smsd
That should be it!
Extra tips:
Use $ gammu-detect to find out where the device is connected. Look for name = Phone on USB serial port HUAWEI_Technology HUAWEI_Mobile or something similar. Put this info in the configfile.
Be shure to add right permission to the bashfile. Make it readable to user running gammu-smsd.
Make the bashfile executable using chmod u+x /path/to/bash/file.
The gammu-smsd-monitor can be used to check how good signal you have. Be shure not to stop running the gammu-smsd when trying to run this.
You can test the bashfile by running it with dummy tlf and dummytext as input. $ ./bashfile.sh.