I’m trying to use use openmpi over InfiniBand Pkey networks.
I can’t seem to find any documentation on how to do that.
I’m using openmpi 4.1.2 and just a simple ring_c.

mpiexec -n 8 -host cn0001-pkey,cn0002-pkey /home/ring_c/ring_c

Networks:

cn0001
ib1 (default pkey) 10.1.0.101
ib1.00a0 (pkey 0xa0) 10.2.0.101

cn0002
ib1 (default pkey) 10.1.0.102
ib1.00a0 (pkey 0xa0) 10.2.0.102

I have my /etc/hosts file set so that

10.2.0.101  cn0001-pkey
10.2.0.102  cn0002-pkey

This seems to execute on cn0001, but hangs on cn0002. I don’t think it’s actually going over the pkey network. What am I doing wrong?

mpiexec -n 8 -host cn0001-pkey,cn0002-pkey -mca plm_base_verbose 99 /home/ring_c/ring_c

[cn0001:05516] mca: base: components_register: registering framework plm components

[cn0001:05516] mca: base: components_register: found loaded component isolated

[cn0001:05516] mca: base: components_register: component isolated has no register or open function

[cn0001:05516] mca: base: components_register: found loaded component rsh

[cn0001:05516] mca: base: components_register: component rsh register function successful

[cn0001:05516] mca: base: components_register: found loaded component slurm

[cn0001:05516] mca: base: components_register: component slurm register function successful

[cn0001:05516] mca: base: components_open: opening plm components

[cn0001:05516] mca: base: components_open: found loaded component isolated

[cn0001:05516] mca: base: components_open: component isolated open function successful

[cn0001:05516] mca: base: components_open: found loaded component rsh

[cn0001:05516] mca: base: components_open: component rsh open function successful

[cn0001:05516] mca: base: components_open: found loaded component slurm

[cn0001:05516] mca: base: components_open: component slurm open function successful

[cn0001:05516] mca:base:select: Auto-selecting plm components

[cn0001:05516] mca:base:select:(  plm) Querying component [isolated]

[cn0001:05516] mca:base:select:(  plm) Query of component [isolated] set priority to 0

[cn0001:05516] mca:base:select:(  plm) Querying component [rsh]

[cn0001:05516] mca:base:select:(  plm) Query of component [rsh] set priority to 10

[cn0001:05516] mca:base:select:(  plm) Querying component [slurm]

[cn0001:05516] mca:base:select:(  plm) Selected component [rsh]

[cn0001:05516] mca: base: close: component isolated closed

[cn0001:05516] mca: base: close: unloading component isolated

[cn0001:05516] mca: base: close: component slurm closed

[cn0001:05516] mca: base: close: unloading component slurm

[cn0001:05516] [[33746,0],0] plm:rsh: final template argv:

        /usr/bin/ssh <template> ( test ! -r ./.profile || . ./.profile;           PATH=/opt/modules/openmpi/gcc/4.1.2/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/opt/modules/openmpi/gcc/4.1.2/lib:${LD_LIBRARY_PATH:-} ; export LD_LIBRARY_PATH ; DYLD_LIBRARY_PATH=/opt/modules/openmpi/gcc/4.1.2/lib:${DYLD_LIBRARY_PATH:-} ; export DYLD_LIBRARY_PATH ;   /opt/modules/openmpi/gcc/4.1.2/bin/orted -mca ess "env" -mca ess_base_jobid "2211577856" -mca ess_base_vpid "<template>" -mca ess_base_num_procs "2" -mca orte_node_regex "cn[4:1],cn[4:2]-pkey@0(2)" -mca orte_hnp_uri "2211577856.0;tcp://16.1.15.2,10.1.0.201,10.2.0.101:38393" -mca plm_base_verbose "99" -mca plm "rsh" --tree-spawn -mca routed "radix" -mca orte_parent_uri "2211577856.0;tcp://16.1.15.2,10.1.0.201,10.2.0.101:38393" -mca pmix "^s1,s2,cray,isolated" )

[cn0002:05935] mca: base: components_register: registering framework plm components

[cn0002:05935] mca: base: components_register: found loaded component rsh

[cn0002:05935] mca: base: components_register: component rsh register function successful

[cn0002:05935] mca: base: components_open: opening plm components

[cn0002:05935] mca: base: components_open: found loaded component rsh

[cn0002:05935] mca: base: components_open: component rsh open function successful

[cn0002:05935] mca:base:select: Auto-selecting plm components

[cn0002:05935] mca:base:select:(  plm) Querying component [rsh]

[cn0002:05935] mca:base:select:(  plm) Query of component [rsh] set priority to 10

[cn0002:05935] mca:base:select:(  plm) Selected component [rsh]