VL-Dedup: Graded Multimodal Entity Resolution for Image-Text Data Deduplication: what it means for business leaders

A deduplication pipeline for image-caption data that caps cluster size and spends vision-language model calls only inside clusters, so it keeps captions that add facts and flags mismatched ones rather than deleting by similarity alone.

1. What the method is

VL-Dedup treats the deduplication of image-caption training data as what the authors call graded multimodal entity resolution. For every pair of records it makes three separate decisions: whether the two images show the same visual entity, whether both captions should be kept because each states facts the other lacks, and whether a caption fails to describe its image and should go to review. Records are grouped into visual-entity clusters under a hard cap on cluster size. A vision-language model is then called only on records and pairs inside those clusters, and an export step writes the retained set, recording the actions behind every deletion. The authors also release VL-DedupBench, 50,000 records from COCO and Stanford Online Products with 66,788 labeled pairs, scored at three levels: labeled pairs, clusters over all record pairs, and whether the facts of deleted records survive in what is kept.

2. Why the method was developed

Today's deduplication tools, such as perceptual hashing, SemDeDup, SemHash and Data-Juicer, define redundancy by surface or embedding similarity and are judged by how much they remove or by downstream accuracy. Entity resolution matchers decide identity only and are scored on labeled pairs. Neither says which captions are worth keeping. The authors show how badly pair-level scoring misleads. A gradient boosting classifier selects no false edge among 9,494 labeled test pairs and reaches co-membership F1 of 0.982 on them, yet its clusters join 7,711,852 pairs of different entities across the full test set, for a pairwise F1 of 0.0023, with one cluster of 33,362 records. Wrong edges chain: one bad link merges two scenes, and every later edge into either side makes the merge bigger. Even when clusters are correct, keeping one record per cluster throws away facts that only the deleted captions stated.

3. Who should care

The first audience is our data and ML platform teams who build or curate multimodal corpora for pretraining, fine-tuning or retrieval. The method gives them a way to deduplicate without silently dropping descriptive captions and to send likely caption errors to review. Teams that match products across catalogs should note the multi-photo product results, which show this problem remains hard. Evaluation leads should take the broader lesson to heart: any entity resolution we run, whether on customer records, catalog items or documents, can look nearly perfect on labeled pairs while its clusters are badly wrong, so we should measure clusters over all record pairs. Anyone buying data-curation services should ask vendors for cluster-level and information-loss metrics, not removal rates. Finance and infrastructure owners should know that the size cap is what keeps GPU spend on model calls linear in corpus size.

4. How the method works

Each record gets a perceptual hash, an SSCD copy-detection embedding, CLIP image and text embeddings, and a CLIPScore for image-caption agreement. Candidate pairs come from the union of four channels: near hashes, image neighbors, caption neighbors, and a capped set of records with the same normalized caption. A logistic regression over 11 pixel- and caption-derived features scores each pair, isotonic regression calibrates it, and the threshold maximizes validation recall at a target precision. Edges are added to a union-find structure from highest to lowest probability, and any merge that would push a cluster past the cap is skipped. The cap is chosen on validation data and was five on the main benchmark. Qwen2.5-VL-7B then judges each clustered record's caption. Its yes-probability is combined with CLIPScore into a mismatch score. Remaining in-cluster pairs receive a structured verdict that sets caption actions but never changes the clusters.

5. How it was evaluated

VL-DedupBench combines 36,000 COCO records with human captions and 14,000 product photos whose captions were generated by InternVL3-2B, split about 70/15/15 so that no image, product superclass or labeled pair crosses partitions. Additional test surfaces cover edited copies (mild, strong and mixed crops, rotations, flips, color shifts and watermarks), 8,550 extra photos of the same products from other viewpoints, and entities enlarged to 10, 20 and 40 records. The baselines are Data-Juicer pHash, SemDeDup, SemHash, caption MinHash, an SSCD threshold, a fusion logistic regression and gradient boosting, and the size cap is compared with five clustering algorithms from entity resolution. Caption actions were scored on 400 in-cluster pairs labeled by two AI auditors, with a human settling disagreements. Information loss was audited by AI auditors on 50 sampled clusters of 237 records. Our copy of the text stops partway through the results on full corpora of up to 580,767 records.

6. How it performed

The cap cut the gradient boosting classifier's false joins from 7.7 million to 385 and raised its F1 from 0.0023 to 0.978. VL-Dedup held between 0.994 and 0.995 on the edited surfaces, stayed within 0.005 of the best of the five clustering algorithms across eight settings, and ran in 0.07 seconds against 9.8 for MCL. An image-only classifier scored 0.9997 on originals and zero under strong edits. On multi-photo product pairs VL-Dedup reached F1 0.29, ahead of the 0.17 baseline but still weak. Per-record mismatch scoring reached F1 0.943, against 0.655 for pair verdicts. Caption-keep decisions reached 0.75 precision and 0.61 recall. Keep-one lost facts in 70 to 141 of 187 deletions, VL-Dedup in 5 to 15 of 28. Model calls took about 8.7 GPU-hours per 50,000 records on one RTX 4090. A cap that does not fit the corpus's entity sizes collapses F1, and the audits are small and largely AI-judged. Our call: pilot it on one multimodal corpus. (Source: arXiv 2611.03080, 2026)