Testing shared memory communications with Linux on Z

Mainframes allow for shared memory communications between LPARs on the same box through ISM - internal shared memory. More information about the Linux device driver can be found here.

IBM provides open-source tools including smc_run which easily converts an application's usage of TCP/IP sockets to SMC (shared memory connection) sockets.

Let's suppose you want to check two of your LPARs can communicate with each other.

First you'd want to check if ISM is available. As ISM are made available as virtual pci devices, you can simply run lspci on each LPAR.

# lspci
00:00.0 Non-VGA unclassified device: IBM Internal Shared Memory (ISM) virtual PCI device

If your admin tells you have been provided the ISM device but you don't see it you might have to power it on.

# echo 1 > /sys/bus/pci/slots/0000032/power

Now, smc_run will convert any TCP/IP socket usage to an SMC socket. So let's suppose you have an echo server using the AF_NET protocol that you can start via commandline; you'd simply run the same command.

[host1]# smc_run python3 echo_server.py --host my_host_name.example.org --port 12345

You can then simply send data from a client in the same way.

[host2]# smc_run python3 send_data.py --host my_host_name.example.org --port 12345
some data

Looks like it's working right? But does it really?

If you power off one of the ISM devices, the above scenario still works. How can that be?

Reading through the manpages, we'll find in the af_smc manpage:

SMC socket capabilities are negotiated at connection setup. If one peer is not SMC capable, further socket processing falls back to TCP usage automatically.

So, how can we make sure that our LPARs really communicate through the ISM?

The s390-tools luckily deliver another tool smcss. It shows details for AF_SMC socket connections. The Mode column shows how data is exchanged:

SMCD     The SMC socket uses SMC-D for data exchange.

SMCR     The SMC socket uses SMC-R for data exchange.

TCP        The SMC socket uses the TCP protocol for data exchange, because an SMC connection could not be established.

And really, the difference can be confirmed while the connection is open depending on the availability of ISM on both LPARs.

[host1]# smcss
State   UID   Inode   Local Address       Peer Address        Intf Mode
ACTIVE  00000 22045079 192.168.0.10:12223  192.168.0.12:37060  0000 SMCD

vs.

[host1]# smcss
State   UID   Inode   Local Address       Peer Address        Intf Mode
ACTIVE  00000 22049662 192.168.0.10:12223 192.168.0.12:37058  0000 TCP 0x05000000/0x03030000

Finally, the s390-tools since version 1.5 also offers another tool that helps to check the ISM live-connectivity without a TCP application, smc_chk. You can shortly run:

[host1]# smc_chk -S
Server started on port 37374

[host2]# smc_chk -C 192.168.0.12 -p 37374
Test with target IP 192.168.0.12 and port 37374
  Live test (SMC-D and SMC-R, EXPERIMENTAL)
     Success, using SMC-D