Benchmarking Content-Defined Chunking Algorithms for Data Deduplication: what it means for business leaders

A controlled benchmark of eight content-defined chunking algorithms separates the cut condition from the layers built on it, and finds that a copied AE parameter error has been inflating published deduplication ratios for years.

1. What the method is

The paper is a controlled benchmark of content-defined chunking, the step in a deduplication system that decides where a byte stream is cut so that identical pieces are stored once. The authors observe that most chunking methods are modular: a base algorithm sets the byte-level condition for a cut, and separate techniques such as size normalization, minimum and maximum chunk thresholds and caching are layered on top. They strip the layers away and evaluate eight state-of-the-art base algorithms on their own, across four realistic workloads and four metrics: throughput, deduplication ratio, mean chunk size and chunk-size variance. They also derive formulas tying each algorithm's parameters to its expected chunk size for the AE, RAM and MII algorithms, and release the implementations, datasets and analysis scripts as open source. We have the abstract only, so the paper's full tables and dataset descriptions are not in front of us.

2. Why the method was developed

Deduplication ratio and throughput are the two numbers that decide whether a backup, archive or object store pays for itself, and chunking drives both. Published comparisons of chunking algorithms have mixed the base cut condition with whatever normalization, thresholds or caching a given implementation happened to add, so a reported win could come from the layer rather than the algorithm, and results across papers did not line up. The authors argue this conflation has hidden which design choices actually matter. Worse, they report that the standard way of parameterizing AE, copied across several papers and evaluation tools, is wrong: it produces chunks 13 to 39 percent smaller than intended. Smaller chunks mechanically raise the deduplication ratio while adding metadata and index cost, so part of AE's published advantage was an artifact of a misconfigured comparison rather than a property of the method.

3. Who should care

This touches anyone in our organisation who stores or ships repeated data at scale. Storage and platform engineering, who pick or tune chunkers in backup, snapshot, artifact and dataset pipelines, get a cleaner basis for the choice and a warning that existing AE settings may be producing chunks a third smaller than planned. Teams building our ML data infrastructure should care because training corpora, checkpoints and embedding stores are heavily redundant and are exactly the storage-oriented workloads studied here. Procurement and finance should know that vendor deduplication claims are sensitive to chunk-size settings, and that a quoted ratio without a stated mean chunk size is not comparable to another vendor's. Our own deduplication and provenance products depend on chunk boundaries being stable and well sized, so the paper directly informs the defaults we expose to customers and how we describe them.

4. How the method works

Content-defined chunking scans the bytes and declares a cut when a local condition on the data is satisfied, so an insertion or deletion shifts boundaries only nearby rather than across the whole file. The algorithms differ in that condition: hash-based methods test a rolling fingerprint against a mask, while AE, RAM, MII and PCI use local extrema or monotonic runs of byte values, trading hashing cost for simpler arithmetic. The authors implement the eight base conditions without the usual wrappers, so a minimum-size skip or a maximum-size forced cut cannot mask a weak condition. For AE, RAM and MII they derive closed-form relations between the window or threshold parameter and the expected chunk size, which lets every algorithm be set to the same target size before comparison. Checking those formulas against the published AE recipe is what exposed the parameterization error that earlier tools reproduced.

5. How it was evaluated

According to the abstract, eight base algorithms were run over four workloads chosen to be realistic for storage systems, and measured on throughput, deduplication ratio, mean chunk size and chunk-size variance. Including mean size and variance alongside the ratio is the key methodological choice, because it stops an algorithm from winning on deduplication simply by cutting smaller. The derived expected-size formulas are used to align parameters across algorithms so the comparison is at matched target size. One workload is described as low-entropy data, which is where RAM separates from AE. The abstract does not name the datasets, the hardware, the exact parameter grid or any baselines beyond the eight algorithms themselves, so we cannot say how large the corpora were or how throughput was measured; those details sit in the full paper and in the released code, which we can run ourselves.

6. How it performed

The abstract reports that AE is the most balanced base algorithm across all four metrics, that RAM behaves like AE on ordinary data but not on low-entropy data, and that MII and PCI offer limited advantages over simpler alternatives in storage-oriented workloads. The headline correction is that the common AE parameterization yields chunks 13 to 39 percent smaller than intended, which inflated AE's reported deduplication in earlier work; the abstract gives no corrected ratios or throughput figures, so we cannot quote how much the rankings moved. The authors confine their claims to base algorithms without normalization or thresholds, so a production chunker with those layers may rank differently, and the workloads are storage-oriented rather than network or ML-specific. Our call: pilot. Re-derive our AE and RAM parameters against the released formulas and re-run our own deduplication ratios at matched mean chunk size before changing any default. (Source: arXiv 2610.13704, 2026)