I would like to know how to invoke InfiniBand hardware on CentOS 6x cluster with OpenMPI (static libs.) for running my C++ code. This is how I compile and run:
/usr/local/open-mpi/1.10.7/bin/mpic++ -L/usr/local/open-mpi/1.10.7/lib -Bstatic main.cpp -o DoWork
usr/local/open-mpi/1.10.7/bin/mpiexec -mca btl tcp,self --hostfile hostfile5 -host node01,node02,node03,node04,node05 -n 200 DoWork
Here, "-mca btl tcp,self" reveals that TCP is used, and the cluster has InfiniBand.
What should be changed in compiling and running commands for InfiniBand to be invoked? If I just replace "-mca btl tcp,self" with "-mca btl openib,self" then I get plenty of errors with relevant one saying:
At least one pair of MPI processes are unable to reach each other for
MPI communications. This means that no Open MPI device has indicated
that it can be used to communicate between these processes. This is
an error; Open MPI requires that all MPI processes be able to reach
each other. This error can sometimes be the result of forgetting to
specify the "self" BTL.
Thanks very much!!!
Related
I got vmlinux.uimg and vmlinux from my friend. The vmlinux is build with Linux version 4.14.10. I have two machines:
target (arm 32 bit architecture) and
host (x86 architecture)
Both systems are connected with wire, port numbers are /dev/ttySAC3 and /dev/ttyUSB0 respectively. Target boot up by reading image file vmlinux.uimg from sd card. I want to debug this vmlinux from my host machine.
The steps I am following are:
Start minicom in host.
Type
echo ttySAC3,115200 > /sys/module/kgdboc/parameters/kgdboc in minicom.
I am getting "KGDB: Registered I/O driver kgdboc".
Type
echo g >/proc/sysrq-trigger in minicom
I am getting "sysrq DEBUG and Entering KGDB".
Close minicom.
Go to vmlinux folder through terminal in host machine.
Type
gdb vmlinux
[Note: this gdb I am using is 7.10 version for arm]
In GDB type
target remote /dev/ttyUSB0
It give me a message
Remote debugging using /dev/ttyUSB0 0xc0d14f5 in sysrq_dbg_op()
In GDB type
b do_mmap_pgoff
I am getting a message
breakpoint 1 at xxxx: do_map_pgoff (4 location)
In GDB type
cont
This gives me a message
Continuing.
and then does nothing..
The problems:
GDB continue is not working as expected. I am expecting last step(9) to continue with target restart and hit the break point instead it is getting crash.
GDB step and next are not working as expected. Single stepping until exit from sysrq_dbg_op() which has no line number information. "Warning: invalid remote reply".
GDB 'run' is also not working. It display a message "The remote target does not support run"
How can I call a module on my target on which I have put breakpoint as I can not use minicom and gdb concurrently.
What all steps I should follow after I freeze the kernel.
What is wrong in this process?
I've looked through many examples and I'm still confused. I've compiled a simple latency check program from here, and it runs perfectly on one host, but when I try to run it on two hosts it hangs. However, running something like hostname runs fine:
[hamiltont#4 latency]$ mpirun --report-bindings --hostfile hostfile --rankfile rankfile -np 2 hostname
[4:16622] [[5908,0],0] odls:default:fork binding child [[5908,1],0] to slot_list 0
4
[5:12661] [[5908,0],1] odls:default:fork binding child [[5908,1],1] to slot_list 0
5
But here is the compiled latency program:
[hamiltont#4 latency]$ mpirun --report-bindings --hostfile hostfile --rankfile rankfile -np 2 latency
[4:16543] [[5989,0],0] odls:default:fork binding child [[5989,1],0] to slot_list 0
[5:12582] [[5989,0],1] odls:default:fork binding child [[5989,1],1] to slot_list 0
[4][[5989,1],0][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 10.0.2.5 failed: Connection timed out (110)
My current guess is that there is something wrong with my firewall rules (e.g. hostname does not communicate between hosts, but the latency program does).
[hamiltont#4 latency]$ cat rankfile
rank 0=10.0.2.4 slot=0
rank 1=10.0.2.5 slot=0
[hamiltont#4 latency]$ cat hostfile
10.0.2.4 slots=2
10.0.2.5 slots=2
There are two kinds of communication involved in running an Open MPI job. First the job has to be launched. Open MPI uses a special framework to support many kinds of launches and you are probably using the rsh remote login launch mechanism over SSH. Obviously your firewall is correctly set up to allow SSH connections.
When an Open MPI job is launched and the processes are true MPI programs, they connect back to the mpirun process that spawned the job and learn all about the other processes in the job, most importantly the available network endpoints at each process. This message:
[4][[5989,1],0][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to 10.0.2.5 failed: Connection timed out (110)
indicates that the process which runs on host 4 is unable to open a TCP connection to the process which runs on host 5. The most common reason for that is the presence of a firewall, which limits the inbound connections. So checking your firewall is the first thing to do.
Another common reason is if on both nodes there are additional network interfaces configured and up, with compatible network addresses, but without the possibility to establish connection between them. This often happens on newer Linux setups where various virtual and/or tunnelling interfaces are being brought up by default. One can instruct Open MPI to skip those interfaces by listing them (either as interface names or as CIDR network addresses) in the btl_tcp_if_exclude MCA parameter, e.g.:
$ mpirun --mca btl_tcp_if_exclude "127.0.0.1/8,tun0" ...
(one always have to add the loopback interface if setting btl_tcp_if_exclude)
or one can explicitly specify which interfaces to be used for communication by listing them in the btl_tcp_if_include MCA parameter:
$ mpirun --mca btl_tcp_if_include eth0 ...
Since the IP address in the error message matches the address of your second host in the hostfile, then the problem must come from an active firewall rule.
I'm trying to execute parallel MPI program on several Windows machines (actually I have one Win7 and one WinXP machines, but people with other configurations (WinXP on all machines, for example) have the same problem). I use MPICH2 (1.4.1p1). I can execute program if I use wmpiexec (MPIEXEC wrapper - GUI for MPI) but if I try to execute it from console I catch next error:
Command: mpiexec -hosts 2 locahost 2 192.168.0.102 2 <path to the program>
Error: mpiexec running on <name of my machine> is unable to connect to msmpi service on locahost:8677
I have created rule for the program in Windows firewall.
So where can be a problem?
Thanks in advance.
It's a wierd solution, but when mpiexec.exe and executable MPI-program is in the same directory everything is fine.
I am trying to run a remote mathematica kernel between two macs.
Under Kernel Configuration Options
For kernel program I have:
/Applications/Mathematica.app/Contents/MacOS/MathKernel
The arguments of MLOpen:
-LinkMode Listen
-LinkProtocol TCPIP
-LinkOptions MLDontInteract
The launch command is:
java -jar mathssh username#xxxxxx.dynamic.uiowa.edu /usr/local/bin/math -mathlink -LinkMode
Connect -LinkProtocol TCPIP -LinkName "linkname" -LinkHost ipaddress
When i use this remote kernel (for instance 2+2 does not give a result) I get the error message:
"The kernel Thomas Machine failed to connect to the front end. (Error = MLECONNECT). You should try running the kernel connection outside the front end."
It seems that Mathematica is not even opening on the remote machine since I used "top" and do not see it running after I start the remote kernel.
Any help would be greatly appreciated.
I just tried this with 8.0.1 -- here's my config (with bogus machine/user names):
In particular the /usr/local/bin/math looks suspicious. You generally shouldn't need to use the advanced settings.
Drop to a command line and try:
ssh username#xxxxxx.dynamic.uiowa.edu /usr/local/bin/math
and see if you get a Mathematica prompt and can evaluate 1+1 there.
I am trying to set up a MPI Cluster. But I have the problem that the number of CPUs added to the mpd.conf file is not correctly used.
I have three Ubuntu servers.
opteron with 48 Cores
calc1 with 8 Cores
calc2 with 8 Cores.
My mpd.hosts looks like:
opteron:46
calc1:6
calc2:6
After booting (mpdboot -n 3 -f mpd.hosts) the System is running.
mpdtrace -> all three of them are listed.
But running a Programm like "mpiexec -n 58 raxmlHPC-MPI ..." causes that calc1 and calc2 get to many jobs and opteron gets to few at the same time.
What am I doing wrong?
Regards
Bjoern
I found a workaround.
I used the additional parameter "-machinefile /path/to/mpd.hosts" for the mpiexec command. And now, all nodes are running correctly.
One problem I got was that I got following error message:
... MPIU_SHMW_Seg_create_attach_templ(671): open failed - No such file or directory ...
To fix it, I had to set the environment variable MPICH_NO_LOCAL=1
As you figured out, you must pass the machinefile to both mpdboot and mpiexec in order to use per-host process counts. The "open failed" issue is a known bug in MPD, the process manager you are using. Note that the MPICH_NO_LOCAL=1 workaround will work, but will probably result in a big performance penalty for intranode communication.
You are clearly using MPICH2 (or an MPICH2 derivative), but it's not clear what version you are using. If you can, I would strongly recommend upgrading to either MPICH2 1.2.1p1 or (better yet) 1.3.1. Both of these releases include a newer process manager called hydra that is much faster and more robust. In 1.3.1, hydra is the default process manager. It doesn't require an mpdboot phase, and it supports a $HYDRA_HOST_FILE environment variable so that you don't have to specify the machine file on every mpiexec.