Cloud Field Notes

By Phil Balfanz · · 9 min read

Memory Tiering in VCF: Doubling Memory with DRAM and NVMe

VMware Cloud FoundationMemory TieringvSphereNVMeSecurity

My latest X thread covered Memory Tiering in VMware Cloud Foundation (VCF). A thread doesn't leave much room, so here is the longer version: what the feature is, how it works, what the published performance testing shows, where it fits (and where it doesn't), and how to get started.

What Memory Tiering is

Memory is often the biggest component of a server's hardware cost. Production environments also tend not to over-commit memory, because ballooning, compression, and swapping can cause unpredictable performance. So admins provision the full amount each VM asks for, and expensive DRAM sits underused.

Memory Tiering, introduced in VCF 9.0, takes a different approach. It lets an ESX host use a locally installed NVMe device as a second tier of memory alongside DRAM:

The VM sees one contiguous, larger memory space. It doesn't know or care which tier a given page lives on, and there are no guest OS or application changes required.

Diagram: a VM sees 2 TB of memory, backed by 1 TB of DRAM (Tier 0, hot pages) and 1 TB of NVMe (Tier 1, cold pages), with ESX moving pages between them
With the default 1:1 ratio, 1 TB of DRAM plus 1 TB of NVMe gives VMs about 2 TB of memory. The $/GB figures are the pricing assumptions VCF Operations uses in its savings simulation.

How it works

Under the hood, ESX watches VM memory activity and classifies pages as "hot" (actively used) or "cold" (inactive). It works to keep hot pages in DRAM and moves cold pages out to NVMe. If a cold page becomes active again, it's fetched back from the NVMe tier. Memory pages from NVMe are only used for VM memory allocations.

A few details worth knowing:

What the performance testing showed

Broadcom's performance team published a study (Todd Muirhead and Qasim Ali, July 2025) testing Memory Tiering on both Intel and AMD servers with the default 1:1 ratio. The headline: double the VM density for a single-digit to low-double-digit performance cost.

Bar chart: performance loss at 2x VM density. VDI (Login Enterprise) 0-8%, enterprise apps (VMmark) about 5%, Oracle (DVD Store) under 5%, SQL Server and MySQL (HammerDB) 5-10%
Performance loss at 2x VM density with the default 1:1 DRAM:NVMe ratio. Source: Memory Tiering Performance in VMware Cloud Foundation 9.0.
BenchmarkWorkloadResult
Login EnterpriseVDI2x VM density, 0-8% performance loss
VMmarkEnterprise apps2x VM density, 5% performance loss
DVD StoreOracle Database2x VM density, less than 5% performance loss
HammerDBSQL Server, MySQL2x VM density, 5-10% performance loss

Some specifics from the results:

The study sums it up as less than 10% performance loss across these workloads, doubled VM density, and up to a 40% TCO saving. One important caveat: all of those numbers are at 2x VM density. The paper notes that at lower density the impact will be lower or negligible.

When to use it

Memory Tiering is a good fit when your hosts are memory-bound but not CPU-bound, and much of the allocated memory is idle. The documentation describes the sweet spot as environments where memory consumption is high (more than 50% allocated across all VMs) but active memory stays low (less than 50% of total DRAM). VCF Operations highlights clusters with active memory below 50% and free CPU capacity above 50% as strong candidates.

Two numbers to watch once it's running:

When not to use it

Memory Tiering isn't for everything. The VCF Operations documentation says not to enable it on clusters running latency-sensitive workloads or workloads with large active memory, such as telco workloads, in-memory Java applications, or high-frequency trading.

Some VM types can't be powered on at all on a host with Memory Tiering active:

Others can run on a tiered host but are automatically kept entirely in DRAM: VMs using 1 GB pages, VMs with preallocated memory, PCI passthrough VMs (including VMXNET3 with UPT), and vQAT VMs. Plan enough DRAM-only capacity for those.

There are also some host-level limits: suspend-to-memory isn't supported (suspend-to-disk and snapshots are), the NVMe device can't be hot-plugged or removed while in use, and the host can't have Intel Optane Persistent Memory or NVDIMM-N installed.

Security considerations

DRAM loses its contents when the power goes off. NVMe doesn't. With Memory Tiering, cold VM memory pages are written to a non-volatile device. In a security-conscious or regulated environment, I'd treat the tier like any other data at rest.

New in vCenter 9.1.1: DRS and HA support

The vCenter 9.1.1.0 release notes (GA September 3, 2026) list DRS and HA support for Memory Tiering as new. In the release notes' words: "With VCF 9.1.1, Memory Tiering supports DRS and HA. In clusters with Memory Tiering activated, vSphere HA admission control tracks DRAM as an independent resource alongside total memory to make sure that the cluster retains sufficient DRAM capacity for virtual machines restart after a host failure."

How to get started

1. Simulate first. In VCF Operations, the Resource Efficiency tab (under Capacity > Optimize) shows which clusters are good candidates, how many more VMs you could run, and estimated savings. The estimate uses assumed pricing of $10 per GB for DRAM and about $1 per GB for NVMe, so treat it as a starting point, not a quote.

2. Check requirements. You need vCenter 9.0 and ESX 9.0 or later, plus a locally installed NVMe device (not over fabric or Ethernet). It must be in a similar class to a vSAN cache device: check the Broadcom Compatibility Guide's vSAN SSD list for Endurance Class D (7300 TBW or more) and Performance Class F or G. The device can't already be part of a datastore such as vSAN, and the tier partition can't be larger than 4 TB. If you want resiliency against a single device failure, NVMe behind a supported RAID controller is supported.

3. Create the tier partition. With the host in maintenance mode and all existing partitions removed from the NVMe device:

esxcli system maintenanceMode set --enable true
esxcli storage core adapter device list
esxcli system tierdevice create -d /vmfs/devices/disks/<device>
esxcli system tierdevice list

4. Turn it on and reboot. At the host level, set the advanced setting VMkernel.Boot.memoryTiering to true. At the cluster level, use vSphere Configuration Profiles and enable the memory_tiering boot option. Then reboot. To exclude a specific VM, power it off and set sched.mem.enableTiering to false.

5. Monitor. Watch active memory versus DRAM and NVMe read bandwidth and latency in vCenter's Advanced Memory Performance charts, and use the Memory Tiering dashboard in VCF Operations to track Tier 0 and Tier 1 usage.

If your hosts run out of memory long before CPU, this is worth a look.

Sources

← All posts