Among the new and exciting media and IO enhancement in vSphere 6.7 is support for Storage Class Memory. Also called Persistent Memory or Non-Volatile DIMMs. Not to be confused with Non-Volatile Memory, PEM is actually RAM with some added mechanism for making it persistent. Like a battery and or backing by NAND Flash. There are basic types and each manufacturer can have their own proprietary features. NVDIMMs are not drive media and are not accessed as disk devices although a virtual device can be abstracted on top of them.
More information about PMEM can be found here: https://en.wikipedia.org/wiki/NVDIMM
Need Persistent Memory or want to do a PoC? I can do that for you, please contact me.
vSphere 6.7 offers two different mechanisms for granting NVDIMM access to virtual machines. I will briefly describe them here and go into more detail below as I demonstrate each one. The terms can get really overlapped here because PMEM is used in a couple of different contexts. PMEM is an access method for Persistent Memory and PMEM is also an abbreviation for Persistent Memory. I apologize, I am not responsible for this ambiguity 🙂
NVDIMM – Capacity out of your Persistent Memory Pool can be allocated directly to your Virtual Machine. With this method, your VM has a Virtual NVDIMM installed. This method is night and day faster than the PMEM method described below. With this method NVDIMM aware applications can make use of the NVDIMM natively or you can use kernel space tools to create a file system construct on top of the memory addresses allocated to the NVDIMMs.
PMEM – Capacity out of your Persistent Memory Pool is allocated to your Virtual Machine as a VMDK and attached to a Virtual Disk Controller. The VMDK resides in the PMEM datastore which is a different construct with different rules than a regular datastore. PMEM backed virtual disks can be storage migrated out and back into PMEM. This allows PMEM space to be dynamically allocated depending on need. The PMEM shows up in the guest as a disk (or NVMe namespace) and can be used normally. Although there are some operational advantages to the PMEM method, it is not nearly as fast as the direct NVDIMM method.
Persistent Memory in the web interface. First we need to verify that there is PMEM in the ESXi host. You can do this by navigating to the Hardware Summary section or the Memory Section of the ESXi Host Configuration context.
The PMEM Datastore
Each ESXi host with PMEM installed with have a single PMEM Datastore. This Datastore shows up in a variety of places but it doesn’t show up in any of the main Datastore context menus. It is mostly hidden.
The PMEM Datastore has a long randomly generated name that can’t be changed (as of 6.7u1).
The primary method for moving or creating a virtual disk in the PMEM Datastore is to assign the PMEM Storage Profile to the Virtual Disk. This will move it to (or create it in) the PMEM Datastore which is backed by Persistent Memory.
The VM Must be Powered Down to install a PMEM disk or an NVDIMM Device.
The PMEM device will perform optimally as an NVMe Namespace. See my blog on NVMe and vSphere 6.7 here
Adding a new PMEM virtual disk as an NVMe Namespace
Once the device is added, power on the VM and configure as an NVMe device.
See my blog on NVMe and vSphere 6.7 here
Migrating a VMDK off of a PMEM Device
The storage migration menu will have new PMem and Hybrid Options.
PMem – Moves all VMDK files to PMEM storage, leaving only the VM config files on a normal datastore.
Hybrid – Allows migrating non-PMem disks, leaving PMem where it is
Standard – Moves all VMDKs to normal Datastores
Choosing “Configure per disk” will allow specific decisions to be made per disk. While creating a PMEM disk requires selecting a VM Storage Policy and not a Datastore Selection, this menu is the opposite. Choose “Browse” and select the PMem radial to move a non-PMem disk to PMem.
You can storage motion a PMEM VMDK file off of PMEM while the VM is running
You can storage motion a VMDK file on to PMEM while the VM is running
You cannot manually migrate the VMDK to PMEM by changing the VMDK Storage Profile while the VM is running
Manual or DRS vMotion works for VMs with PMEM. The VM is not pinned to a host when using PMEM. A storage migration is not required. As long as the target host has a PMEM pool with sufficient capacity, the PMEM disk will automatically migrate to the target host.
The VM must be powered off to add a virtual NVDIMM. NVDIMM namespaces are configured at boot time and so hot-add is not likely to work.
The VM must be VM Hardware Version 14 (ESXi 6.7) or higher
The Guest Operating System must have native support for NVDIMM
I am using Ubuntu 18.04 here, directions for a different OS may vary.
Install LIBNVDIMM Support Command Line Utility ndctl
Check if pmem device has been created
We are lucky with Ubuntu 18, the pmem device was created by default and we did not have to construct it.
View the properties of the device
The mode of this device matters and you must set it properly for your intended use.
fsdax – Allows direct memory mappings bypassing the page cache. Best for filesystems that are implemented properly for DAX kernel access. (XFS and EXt4)
devdax – Similar to fsdax, use if you will be passing this device through to a virtual machine (would be nested in this case)
sector – Use this mode for legacy file systems without full DAX support that don’t support checksums. Mostly used for small boot file systems.
raw – Just a raw device. Can be used for file systems but without DAX support.
I will be using XFS for performance reasons and so I will change the mode from raw to fsdax.
Now we can format the pmem device and mount it normally.
For more information about managing NVDIMM Namespaces.
vMotioning VM’s with NVDIMM devices
As long as the target ESXi host has a PMEM pool with enough capacity available, a standard vMotion will be successful.