Cloud Field Notes

By Phil Balfanz · · 14 min read

vSAN Global Deduplication in VCF: More Capacity from the Same Hosts

VMware Cloud FoundationvSANStorageDeduplicationEncryptionSecurity

My latest X thread covered vSAN Global Deduplication in VMware Cloud Foundation (VCF). A thread doesn't leave much room, so here is the longer version: what the feature is, how it works under the hood, what it requires, how it plays with encryption, and how to judge the savings without fooling yourself.

What Global Deduplication is

Storage is often a big piece of an infrastructure refresh budget. Deduplication cuts that down by finding blocks of data that are identical, keeping one copy, and replacing the duplicates with small metadata pointers. The freed space shows up as usable capacity.

vSAN has had deduplication for a long time, but the original architecture (OSA) had a real limit: its dedup domain was a single disk group in a single host. If the same block lived on two disk groups, it was stored twice.

Global Deduplication in the vSAN Express Storage Architecture (ESA) changes the domain to the entire cluster. Any identical 4KB block anywhere in the cluster can be deduplicated, and the domain grows automatically as you add hosts. Broadcom's example: if an 8-host cluster grows to 16 hosts, capacity doubles but it's still one dedup domain.

Diagram: in vSAN OSA, each disk group dedupes on its own, so block A is stored once per disk group. In vSAN ESA, the whole cluster is one dedup domain, so block A is stored once and other hosts hold pointers
Conceptual: OSA deduped per disk group. ESA dedupes across the whole cluster, and the domain grows as you add hosts.

Two things the bigger domain does not change:

Availability: limited in 9.0, GA in 9.1

Broadcom doesn't label it a tech preview in either release, so I won't either.

How it works

The big design choice is that ESA dedup is post-process, not inline. The write path is the same whether dedup is on or off: the VM writes, vSAN compresses the data high in the stack, encrypts it if Data-at-Rest Encryption is on, and persists it. Dedup happens later, against data that has "cooled down."

That matters for two reasons. First, dedup work stays out of the hot write path, so VM write latency isn't paying for it. Second, it doesn't waste effort on freshly written data that's often overwritten or deleted soon after.

When you turn it on, vSAN creates one or more special objects that are largely invisible to you:

Then, roughly in this order, vSAN:

  1. Reads a 4KB block (colder data first) and generates a cryptographic hash. In 9.1, that work is offloaded lower in the stack, and if the data is encrypted, the block is decrypted in memory first.
  2. Looks for a matching hash in the metadata object.
  3. If a match already lives in the dedup data object, it replaces the block with a pointer and reclaims the space.
  4. If a match exists but isn't in the dedup data object yet, it moves the data there (using a back-pointer to find the original), points both blocks at it, and reclaims the space.
  5. If there's no match, it leaves the data alone and records the hash with a back-pointer, so a future duplicate can be caught.

A few operational details worth knowing:

Requirements and limits (VCF 9.1)

Turning it on is a single toggle: in the vSphere Client, select the cluster, go to Configure > vSAN > Services, edit Data Reduction and Encryption, enable deduplication, and apply.

Encryption and security

This is where 9.1 got interesting for anyone in a security-conscious or regulated environment. Encryption and dedup usually fight each other. Encrypted blocks look random, so identical plaintext doesn't produce identical stored blocks, and dedup ratios collapse. In 9.1, Broadcom says vSAN ESA dedup works with vSAN Data-at-Rest Encryption without reducing dedup ratios.

Diagram: write path (VM write, compress, encrypt with Data-at-Rest, persist to disk), then later post-processing (read cold block, decrypt in host memory, hash and match, pointer and reclaim space)
Conceptual: data is compressed and encrypted on write. Dedup comes later, decrypting blocks in host memory to hash and match them.

How dedup handles encrypted data

Use vSAN encryption, not VM Encryption

This is the most important practical point in the whole post. Broadcom's guidance is blunt: if you need encryption, always use vSAN encryption services (Data-at-Rest, Data-in-Transit, or both) rather than vSphere VM Encryption. VM Encryption encrypts data before it reaches vSAN, which negates the dedup savings. Broadcom also says it lowers performance and adds complexity, and recommends against combining VM Encryption with vSAN encryption.

Key providers: Native Key Provider vs. external KMS

Data-at-Rest Encryption needs a key provider. You have two options:

A few Broadcom recommendations worth following either way:

Encryption in transit

How to judge the savings

Broadcom does not promise a dedup ratio, and neither will I. Savings depend on how alike your data is, what kind of data it is, and how much data is in the cluster. A few things to keep in mind:

Turning a ratio into capacity

Broadcom's space efficiency paper gives a worked example. A 10-host single-site cluster with about 40TB per host has about 394TB raw. The Effective Capacity view, which accounts for resilience overhead, shows about 203TB. Apply a data reduction ratio and you get the effective usable capacity:

Data reduction ratioApproximate usable capacityVersus raw capacity
2:1~400TBAbout equal to raw
4:1~800TBAbout 2x raw
6:1~1.2PBAbout 3x raw
8:1~1.6PBAbout 4x raw

These are illustrations of the math, not predictions for your data.

The licensing angle

Broadcom's August 2025 blog points out that each VCF core license includes 1 TiB of raw vSAN capacity. Dedup doesn't change the entitlement, but it lets you store more data within it. Broadcom's own example: a 6-host cluster with 32 cores per host is entitled to 192 TiB, and if that cluster achieved 6:1, it could store nearly 1.2 PiB.

About the TCO numbers

You may see figures like "up to 34% lower TCO," "29% lower cost per TB," or "dedup alone can cut VCF cost by as much as 4%." Those come from the same Broadcom blog and are Broadcom's own model estimates: a 10,000-core environment with assumed CPU utilization and licensing costs, comparing VCF with vSAN against VCF with a competing array. The 34% and 29% figures are about vSAN overall (compression plus dedup versus an external array), not dedup alone; Broadcom puts dedup's own contribution in that model at up to about 4% of total VCF cost. They're vendor modeling, not measured customer savings, so treat them as a reason to run your own numbers, not as the answer.

How to get started

  1. Check fit. ESA, single site, 3-64 hosts, not stretched or 2-node, and no plan to stretch it later.
  2. Pick your encryption model first. If you need encryption, plan on vSAN Data-at-Rest (and Data-in-Transit if your requirements call for it), not VM Encryption. Choose NKP or an external KMIP KMS, put TPMs in the hosts, and back up NKP keys.
  3. Review policies. Find thick-provisioned VMs and HFTT-3 policies; they won't benefit.
  4. Turn it on under Configure > vSAN > Services > Data Reduction and Encryption.
  5. Wait, then measure. Give it a few days and track the overall Data Reduction Ratio in the Effective Capacity view.

If storage is what's driving your next hardware purchase, this is worth a look before you sign the PO.

Sources

← All posts