Set up Crypto Card passthrough with KVM on IBM Z (vfio-ap)

Crypto Cards on IBM Z systems provide secure key encryption.

This security feature can be passed through to KVM guests.

The passthrough is available through the vfio_ap kernel module (paired with the homonymous driver). It uses another passthrough interface, namely, the VFIO mediated device framework (represented by kernel module vfio_mdev).

More details can be found kernel doc. Here, the focus is on setting up a single passthrough using libvirt.

What we need

1. A System Z host with a crypto card, KVM guest
2. lszcrypt command (from s390tools, often comes preinstalled with distro, package name can be s390utils, too)

What we do

1. Identify the device
2. Mark device queues as not usable by host
3. Create mediated device
4. Assign crypto device to mediated device
5. Attach mediated device to guest
6. Verify setup

Identify the device

# lszcrypt -V

CARD.DOMAIN TYPE  MODE        STATUS  REQUESTS  PENDING HWTYPE QDEPTH FUNCTIONS  DRIVER     

--------------------------------------------------------------------------------------------

01          CEX5C CCA-Coproc  online         1        0     11     08 S--D--N--  cex4card   

01.0011     CEX5C CCA-Coproc  online         1        0     11     08 S--D--N--  cex4queue  

We need two pieces of information:

• HWTYPE: passthrough is only supported if this number is >= 10
• CARD.DOMAIN: 0x01 (adapter id), 0x0011 (domain id)

Mark device queues as not usable by host

• Mark adapter not usable by host:
• echo -0x01 > /sys/bus/ap/apmask
• Mark device queues not usable by host:
• echo -0x0011 > /sys/bus/ap/aqmask

lszcrypt now should only list the card, not the queue.

# lszcrypt

CARD.DOMAIN TYPE  MODE        STATUS  REQUESTS

----------------------------------------------

01          CEX5C CCA-Coproc  online         4

Create mediated device

cd /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough

uuidgen > create

Assign crypto device to mediated device

Below the device dir $uuid that we just created below vfio_ap-passthrough/devices:

cd devices/$uuidgen

echo 0x01 > assign_adapter

echo 0x0011 > assign_domain

We can confirm assignment:

# lszcrypt -V

CARD.DOMAIN TYPE  MODE        STATUS  REQUESTS  PENDING HWTYPE QDEPTH FUNCTIONS  DRIVER     

--------------------------------------------------------------------------------------------

01          CEX5C CCA-Coproc  online         4        0     11     08 S--D--N--  cex4card   

01.0011     CEX5C CCA-Coproc  -              4        0     11     08 S--D--N--  vfio_ap    

Attach the mediated device to guest

Use $uuid for the mediated device and modify guest domain xml (e.g. virsh edit)

<hostdev mode='subsystem' type='mdev' model='vfio-ap'>
    <source>
      <address uuid='$uuid'/>
    </source>
</hostdev>

Verify setup

After starting the guest:

root@guest # lszcrypt -V

CARD.DOMAIN TYPE  MODE        STATUS  REQUESTS  PENDING HWTYPE QDEPTH FUNCTIONS  DRIVER     

--------------------------------------------------------------------------------------------

01          CEX5C CCA-Coproc  online         1        0     11     08 S--D--N--  cex4card   

01.0011     CEX5C CCA-Coproc  online         1        0     11     08 S--D--N--  cex4queue