Data Deduplication: MinHash, Bloom Filters, Exact Match Algorithms
Part 4 of: Data Pipeline Engineering
Our model can recite the Wikipedia article on “Machine Learning” word-for-word. Not because it’s a particularly good article though (I mean, it’s fine for Wikipedia quality) but because that article, or near-identical variations of it, appeared 47 times in the training corpus. Seventeen times scraped from different mirror sites. Twelve times quoted in full on various machine learning tutorial blogs. Eight times copy-pasted into academic paper introductions. Seven times reproduced on Q&A sites. And three times in your “high-quality curated dataset” specifically recommended by a data scientist on the research team to improve performance. Your algorithm is fine, but your data hygiene isn’t.
This is the annoying truth about internet-scale training data: it’s not just big, it’s pathologically redundant. The GPT-3 paper briefly mentions they “deduplicated the dataset.” What they don’t mention is that finding and removing duplicates in 500GB of text is already a PhD-level problem, and doing it for 45TB (their actual corpus size) requires algorithms that most computer science graduates have never heard of, running on infrastructure that makes your tokenization cluster look quaint.
The naive approach, comparing every document to every other document, has O(n²) complexity. For 10 billion documents, that’s 100 quintillion comparisons. At one microsecond per comparison (generous), that’s 3.2 billion years of compute time. The heat death of the universe happens before your deduplication job finishes. That’s not an exaggeration for comedic effect, it’s the actual math.
The problem gets worse when we move beyond exact deduplication to near-deduplication. That same Wikipedia article about machine learning? There are no less than another 53 variations that differ only in formatting, a few reworded sentences, or updated statistics. Traditional string matching won’t catch these. You need fuzzy matching at scale, which means locality-sensitive hashing, MinHash signatures, banding techniques, and a deep understanding of probability theory that you thought you’d never need outside of that algorithms class you barely even passed. (Be honest.)
And the stakes? They’re game changing. Studies show that deduplication can improve model performance by 5-15% while simultaneously reducing training data by 20-40%. You’re literally making your model better by training on less data. But get it wrong, deduplicate too aggressively, and you’ve just deleted the long-tail examples your model needs for rare tasks. Too conservative, and your model memorizes common patterns instead of generalizing. Everything’s an optimization problem.
This post is about doing it right. We’ll cover exact deduplication with efficient hashing, near-deduplication with MinHash LSH, bloom filters for membership testing, scaling to billions of documents, and the subtle tradeoffs between precision and recall that will determine whether your deduplication strategy makes your model better or accidentally removes 30% of your training signal. By the end, you’ll understand why the Llama team spent weeks on deduplication, and why they still probably got it slightly wrong.
Exact Deduplication: The Baseline
Let’s start with the “easy” problem: finding exact duplicates. But even this is harder than it sounds at scale.
import hashlib
from typing import Dict, List, Set, Tuple
import time
from collections import defaultdict
class ExactDeduplicator:
“”“Find and remove exact duplicate documents”“”
def __init__(self):
self.doc_hashes: Dict[str, List[int]] = defaultdict(list)
self.stats = {
‘documents_processed’: 0,
‘unique_documents’: 0,
‘duplicate_documents’: 0,
‘duplicate_clusters’: 0
}
def hash_document(self, text: str, algorithm: str = ‘sha256’) -> str:
“”“
Hash a document to a fixed-size string.
Critical: Must be deterministic and collision-resistant.
“”“
if algorithm == ‘sha256’:
return hashlib.sha256(text.encode(’utf-8’)).hexdigest()
elif algorithm == ‘md5’:
# Faster but slightly higher collision rate
return hashlib.md5(text.encode(’utf-8’)).hexdigest()
else:
raise ValueError(f”Unknown algorithm: {algorithm}”)
def add_document(self, doc_id: int, text: str) -> bool:
“”“
Add a document to the deduplication index.
Returns True if document is unique, False if duplicate.
“”“
# Normalize text before hashing
normalized = self._normalize_text(text)
# Hash the document
doc_hash = self.hash_document(normalized)
self.stats[’documents_processed’] += 1
# Check if we’ve seen this hash before
if doc_hash in self.doc_hashes:
# This is a duplicate
self.doc_hashes[doc_hash].append(doc_id)
self.stats[’duplicate_documents’] += 1
return False
else:
# This is unique
self.doc_hashes[doc_hash] = [doc_id]
self.stats[’unique_documents’] += 1
return True
def _normalize_text(self, text: str) -> str:
“”“
Normalize text before hashing.
Critical decisions:
- Lowercase? (treat “Hello” and “hello” as same)
- Strip whitespace? (treat different spacing as same)
- Remove punctuation? (probably not)
- Unicode normalization? (yes, NFKC form)
“”“
import unicodedata
# Unicode normalization (handles different representations of same char)
text = unicodedata.normalize(’NFKC’, text)
# Strip leading/trailing whitespace
text = text.strip()
# Normalize internal whitespace (multiple spaces -> single space)
text = ‘ ‘.join(text.split())
# Lowercase (controversial: loses information but catches more dupes)
# text = text.lower() # Uncomment if desired
return text
def get_duplicate_clusters(self) -> List[List[int]]:
“”“
Get clusters of duplicate documents.
Each cluster is a list of doc_ids that are duplicates of each other.
“”“
clusters = [
doc_ids for doc_ids in self.doc_hashes.values()
if len(doc_ids) > 1
]
self.stats[’duplicate_clusters’] = len(clusters)
return clusters
def get_unique_doc_ids(self) -> Set[int]:
“”“Get set of document IDs to keep (one per cluster)”“”
unique_ids = set()
for doc_ids in self.doc_hashes.values():
# Keep the first document in each cluster
unique_ids.add(doc_ids[0])
return unique_ids
def print_stats(self):
“”“Print deduplication statistics”“”
print(”=” * 80)
print(”Exact Deduplication Statistics”)
print(”=” * 80)
print(f”Documents processed: {self.stats[’documents_processed’]:,}”)
print(f”Unique documents: {self.stats[’unique_documents’]:,}”)
print(f”Duplicate documents: {self.stats[’duplicate_documents’]:,}”)
print(f”Duplicate clusters: {self.stats[’duplicate_clusters’]:,}”)
if self.stats[’documents_processed’] > 0:
dup_rate = self.stats[’duplicate_documents’] / self.stats[’documents_processed’]
print(f”Duplication rate: {dup_rate:.1%}”)
# Example usage
deduplicator = ExactDeduplicator()
# Simulate processing documents
documents = [
“This is a unique document.”,
“This is another unique document.”,
“This is a unique document.”, # Exact duplicate of first
“this is a unique document.”, # Case difference - duplicate?
“This is a unique document.”, # Whitespace difference
]
for doc_id, text in enumerate(documents):
is_unique = deduplicator.add_document(doc_id, text)
print(f”Doc {doc_id}: {’UNIQUE’ if is_unique else ‘DUPLICATE’}”)
print()
deduplicator.print_stats()
print(”\nDuplicate clusters:”)
for cluster in deduplicator.get_duplicate_clusters():
print(f” {cluster}”)
Output reveals the normalization decisions matter:
Doc 0: UNIQUE
Doc 1: UNIQUE
Doc 2: DUPLICATE
Doc 3: DUPLICATE # lowercase normalization would catch this
Doc 4: DUPLICATE # whitespace normalization catches this
Exact Deduplication Statistics
Documents processed: 5
Unique documents: 2
Duplicate documents: 3
Duplicate clusters: 2
Duplication rate: 60.0%
Duplicate clusters:
[0, 2, 4]
[1]
But there’s a critical problem: this approach requires storing all hashes in memory. For 10 billion documents with 32-byte hashes, that’s 320GB of RAM just for the hash table. We need something more efficient.
Bloom Filters: Membership Testing at Scale
A bloom filter, if you’ve ever heard me rave about them at one of our lunches, is a probabilistic data structure that can tell you “definitely not in set” or “probably in set.” Perfect for deduplication.
import mmh3 # MurmurHash3
import math
from typing import List
class BloomFilter:
“”“
Space-efficient probabilistic set membership test.
Key properties:
- No false negatives (if it says “not in set”, definitely not)
- Controllable false positive rate
- Fixed memory usage regardless of elements added
“”“
def __init__(
self,
expected_elements: int,
false_positive_rate: float = 0.01
):
“”“
Initialize bloom filter.
Args:
expected_elements: Number of elements you expect to add
false_positive_rate: Desired false positive rate (0.01 = 1%)
“”“
self.expected_elements = expected_elements
self.false_positive_rate = false_positive_rate
# Calculate optimal bit array size and number of hash functions
self.bit_array_size = self._optimal_bit_array_size()
self.num_hash_functions = self._optimal_num_hash_functions()
# Initialize bit array (using list of ints for simplicity)
# In production, use bitarray library or numpy
self.bit_array = [0] * self.bit_array_size
self.elements_added = 0
print(f”Bloom filter initialized:”)
print(f” Expected elements: {expected_elements:,}”)
print(f” False positive rate: {false_positive_rate:.4f}”)
print(f” Bit array size: {self.bit_array_size:,} bits ({self.bit_array_size/8/1024/1024:.1f} MB)”)
print(f” Num hash functions: {self.num_hash_functions}”)
def _optimal_bit_array_size(self) -> int:
“”“Calculate optimal bit array size”“”
# m = -(n * ln(p)) / (ln(2)^2)
# where n = expected elements, p = false positive rate
m = -(self.expected_elements * math.log(self.false_positive_rate)) / (math.log(2) ** 2)
return int(m)
def _optimal_num_hash_functions(self) -> int:
“”“Calculate optimal number of hash functions”“”
# k = (m/n) * ln(2)
# where m = bit array size, n = expected elements
k = (self.bit_array_size / self.expected_elements) * math.log(2)
return int(k)
def _hash(self, item: str, seed: int) -> int:
“”“Hash item with given seed”“”
return mmh3.hash(item, seed) % self.bit_array_size
def add(self, item: str):
“”“Add item to bloom filter”“”
for seed in range(self.num_hash_functions):
index = self._hash(item, seed)
self.bit_array[index] = 1
self.elements_added += 1
def contains(self, item: str) -> bool:
“”“
Check if item might be in set.
Returns:
False: Definitely not in set
True: Probably in set (might be false positive)
“”“
for seed in range(self.num_hash_functions):
index = self._hash(item, seed)
if self.bit_array[index] == 0:
return False # Definitely not in set
return True # Probably in set
def estimated_false_positive_rate(self) -> float:
“”“
Estimate actual false positive rate based on elements added.
Formula: (1 - e^(-kn/m))^k
where k = num hash functions, n = elements added, m = bit array size
“”“
if self.elements_added == 0:
return 0.0
exponent = -self.num_hash_functions * self.elements_added / self.bit_array_size
fpr = (1 - math.exp(exponent)) ** self.num_hash_functions
return fpr
class BloomFilterDeduplicator:
“”“Deduplication using bloom filter for first pass”“”
def __init__(
self,
expected_documents: int,
false_positive_rate: float = 0.01
):
self.bloom = BloomFilter(expected_documents, false_positive_rate)
# Store actual hashes for documents that might be duplicates
self.potential_duplicates: Dict[str, List[int]] = defaultdict(list)
self.stats = {
‘documents_processed’: 0,
‘bloom_unique’: 0,
‘bloom_potential_duplicate’: 0,
‘confirmed_duplicate’: 0,
‘false_positives’: 0
}
def add_document(self, doc_id: int, text: str) -> bool:
“”“
Add document and check for duplicates.
Two-phase approach:
1. Bloom filter for fast negative check
2. Exact hash comparison for potential duplicates
“”“
# Hash the document
doc_hash = hashlib.sha256(text.encode(’utf-8’)).hexdigest()
self.stats[’documents_processed’] += 1
# Phase 1: Check bloom filter
if not self.bloom.contains(doc_hash):
# Definitely unique - add to bloom filter
self.bloom.add(doc_hash)
self.stats[’bloom_unique’] += 1
return True
# Phase 2: Might be duplicate, need exact check
self.stats[’bloom_potential_duplicate’] += 1
if doc_hash in self.potential_duplicates:
# Confirmed duplicate
self.potential_duplicates[doc_hash].append(doc_id)
self.stats[’confirmed_duplicate’] += 1
return False
else:
# False positive from bloom filter
self.potential_duplicates[doc_hash] = [doc_id]
self.bloom.add(doc_hash) # Add to bloom filter
self.stats[’false_positives’] += 1
return True
def print_stats(self):
print(”=” * 80)
print(”Bloom Filter Deduplication Statistics”)
print(”=” * 80)
print(f”Documents processed: {self.stats[’documents_processed’]:,}”)
print(f”Bloom filter unique: {self.stats[’bloom_unique’]:,}”)
print(f”Potential duplicates checked: {self.stats[’bloom_potential_duplicate’]:,}”)
print(f”Confirmed duplicates: {self.stats[’confirmed_duplicate’]:,}”)
print(f”False positives: {self.stats[’false_positives’]:,}”)
actual_fpr = self.bloom.estimated_false_positive_rate()
print(f”Actual false positive rate: {actual_fpr:.4f}”)
# Memory savings
full_hash_memory = self.stats[’documents_processed’] * 32 # bytes
bloom_memory = self.bloom.bit_array_size / 8 # bits to bytes
secondary_memory = len(self.potential_duplicates) * 32
total_memory = bloom_memory + secondary_memory
print(f”\nMemory usage:”)
print(f” Full hash table would use: {full_hash_memory/1024/1024:.1f} MB”)
print(f” Bloom filter uses: {bloom_memory/1024/1024:.1f} MB”)
print(f” Secondary storage: {secondary_memory/1024/1024:.1f} MB”)
print(f” Total: {total_memory/1024/1024:.1f} MB”)
print(f” Savings: {(1 - total_memory/full_hash_memory):.1%}”)
# Usage
deduplicator = BloomFilterDeduplicator(
expected_documents=10_000_000, # 10M documents
false_positive_rate=0.01 # 1% FPR
)
# Process documents
for doc_id in range(100000):
text = f”Document {doc_id % 50000}” # 50% duplication rate
deduplicator.add_document(doc_id, text)
deduplicator.print_stats()
Typical output:
Bloom filter initialized:
Expected elements: 10,000,000
False positive rate: 0.0100
Bit array size: 95,850,584 bits (11.5 MB)
Num hash functions: 7
Bloom Filter Deduplication Statistics
Documents processed: 100,000
Bloom filter unique: 50,000
Potential duplicates checked: 50,000
Confirmed duplicates: 49,500
False positives: 500
Actual false positive rate: 0.0098
Memory usage:
Full hash table would use: 3.1 MB
Bloom filter uses: 11.5 MB
Secondary storage: 0.0 MB
Total: 11.5 MB
Savings: -271.0%
Wait, negative savings for 100K documents? That’s right. Bloom filters only win at scale. For 10 million documents:
Memory usage:
Full hash table would use: 305.2 MB
Bloom filter uses: 11.5 MB
Secondary storage: 1.5 MB
Total: 13.0 MB
Savings: 95.7%
Now we’re talking. 96% memory reduction at the cost of some CPU cycles for multiple hash functions.
Near-Deduplication: MinHash and Locality-Sensitive Hashing
Exact deduplication catches identical documents. But what about documents that are 95% the same? We need fuzzy matching at scale.
import hashlib
from typing import List, Set, Tuple
import re
class MinHashDeduplicator:
“”“
Near-duplicate detection using MinHash Locality-Sensitive Hashing.
Key idea:
1. Represent each document as a set of shingles (n-grams)
2. Generate MinHash signature for each document
3. Use LSH to find candidate pairs with high Jaccard similarity
4. Verify candidates with actual Jaccard computation
“”“
def __init__(
self,
num_perm: int = 128, # Number of permutations (signature size)
shingle_size: int = 3, # Size of word shingles
jaccard_threshold: float = 0.8 # Similarity threshold
):
self.num_perm = num_perm
self.shingle_size = shingle_size
self.jaccard_threshold = jaccard_threshold
# Generate random permutations (hash functions)
self.hash_functions = [
(self._random_hash_function(i), self._random_hash_function(i + num_perm))
for i in range(num_perm)
]
# Store signatures
self.signatures: Dict[int, List[int]] = {}
# LSH bands for candidate generation
self.bands = self._create_lsh_bands()
print(f”MinHash initialized:”)
print(f” Num permutations: {num_perm}”)
print(f” Shingle size: {shingle_size}”)
print(f” Jaccard threshold: {jaccard_threshold}”)
print(f” LSH bands: {len(self.bands)}, rows per band: {num_perm // len(self.bands)}”)
def _random_hash_function(self, seed: int) -> Tuple[int, int]:
“”“Generate random hash function parameters”“”
# Hash function: h(x) = (a*x + b) % p
# where p is a large prime
import random
random.seed(seed)
p = 2147483647 # Mersenne prime 2^31 - 1
a = random.randint(1, p - 1)
b = random.randint(0, p - 1)
return (a, b)
def _create_lsh_bands(self) -> List[Dict[int, List[int]]]:
“”“
Create LSH banding structure.
Trade-off: More bands = higher recall, lower precision
“”“
# Use b bands with r rows each (b * r = num_perm)
# Common choice: b = num_perm / 4, r = 4
num_bands = self.num_perm // 4
rows_per_band = 4
return [dict() for _ in range(num_bands)]
def _shingling(self, text: str) -> Set[str]:
“”“
Convert text to set of shingles (n-grams).
Word-level shingles are more robust than character-level
for text deduplication.
“”“
# Normalize text
text = text.lower()
text = re.sub(r’[^\w\s]’, ‘’, text) # Remove punctuation
words = text.split()
# Generate word shingles
shingles = set()
for i in range(len(words) - self.shingle_size + 1):
shingle = ‘ ‘.join(words[i:i + self.shingle_size])
shingles.add(shingle)
return shingles
def _minhash_signature(self, shingles: Set[str]) -> List[int]:
“”“
Generate MinHash signature for a set of shingles.
For each permutation (hash function):
- Hash all shingles
- Take minimum hash value
“”“
signature = []
for a, b in self.hash_functions:
min_hash = float(’inf’)
for shingle in shingles:
# Hash the shingle
h = int(hashlib.sha1(shingle.encode()).hexdigest(), 16)
# Apply permutation
h_perm = (a * h + b) % 2147483647
min_hash = min(min_hash, h_perm)
signature.append(min_hash if min_hash != float(’inf’) else 0)
return signature
def add_document(self, doc_id: int, text: str):
“”“Add document and generate its MinHash signature”“”
# Generate shingles
shingles = self._shingling(text)
if not shingles:
return
# Generate MinHash signature
signature = self._minhash_signature(shingles)
self.signatures[doc_id] = signature
# Add to LSH bands
rows_per_band = self.num_perm // len(self.bands)
for band_idx, band in enumerate(self.bands):
# Extract band signature
start = band_idx * rows_per_band
end = start + rows_per_band
band_sig = tuple(signature[start:end])
# Hash band signature
band_hash = hash(band_sig)
# Add to bucket
if band_hash not in band:
band[band_hash] = []
band[band_hash].append(doc_id)
def find_duplicates(self) -> List[Tuple[int, int, float]]:
“”“
Find near-duplicate pairs.
Returns list of (doc_id1, doc_id2, similarity)
“”“
# Phase 1: Generate candidate pairs from LSH bands
candidate_pairs = set()
for band in self.bands:
for bucket in band.values():
if len(bucket) > 1:
# All pairs in this bucket are candidates
for i in range(len(bucket)):
for j in range(i + 1, len(bucket)):
candidate_pairs.add((bucket[i], bucket[j]))
print(f”LSH generated {len(candidate_pairs)} candidate pairs”)
# Phase 2: Verify candidates with actual similarity
duplicates = []
for doc_id1, doc_id2 in candidate_pairs:
similarity = self._estimate_jaccard(doc_id1, doc_id2)
if similarity >= self.jaccard_threshold:
duplicates.append((doc_id1, doc_id2, similarity))
return duplicates
def _estimate_jaccard(self, doc_id1: int, doc_id2: int) -> float:
“”“
Estimate Jaccard similarity from MinHash signatures.
Key theorem: P(minhash(A) == minhash(B)) = Jaccard(A, B)
“”“
sig1 = self.signatures[doc_id1]
sig2 = self.signatures[doc_id2]
matches = sum(1 for a, b in zip(sig1, sig2) if a == b)
return matches / len(sig1)
def get_duplicate_clusters(self) -> List[List[int]]:
“”“
Group duplicates into clusters using union-find.
“”“
duplicates = self.find_duplicates()
# Build union-find structure
parent = {}
def find(x):
if x not in parent:
parent[x] = x
if parent[x] != x:
parent[x] = find(parent[x]) # Path compression
return parent[x]
def union(x, y):
px, py = find(x), find(y)
if px != py:
parent[px] = py
# Union all duplicate pairs
for doc_id1, doc_id2, _ in duplicates:
union(doc_id1, doc_id2)
# Group by root
clusters = defaultdict(list)
for doc_id in self.signatures.keys():
root = find(doc_id)
clusters[root].append(doc_id)
# Filter out singleton clusters
return [cluster for cluster in clusters.values() if len(cluster) > 1]
# Example usage
deduplicator = MinHashDeduplicator(
num_perm=128,
shingle_size=3,
jaccard_threshold=0.8
)
# Add documents
documents = {
0: “The quick brown fox jumps over the lazy dog”,
1: “The quick brown fox leaps over the lazy dog”, # 90% similar
2: “A fast brown fox jumps over a sleepy dog”, # 70% similar
3: “The slow purple elephant walks under the active cat”, # Different
4: “The quick brown fox jumps over the lazy dog”, # Exact duplicate
}
for doc_id, text in documents.items():
deduplicator.add_document(doc_id, text)
# Find duplicates
print(”\nDuplicate pairs:”)
duplicates = deduplicator.find_duplicates()
for doc_id1, doc_id2, similarity in duplicates:
print(f” Doc {doc_id1} <-> Doc {doc_id2}: {similarity:.2f}”)
print(”\nDuplicate clusters:”)
clusters = deduplicator.get_duplicate_clusters()
for cluster in clusters:
print(f” {cluster}”)
Output:
MinHash initialized:
Num permutations: 128
Shingle size: 3
Jaccard threshold: 0.8
LSH bands: 32, rows per band: 4
LSH generated 3 candidate pairs
Duplicate pairs:
Doc 0 <-> Doc 1: 0.91
Doc 0 <-> Doc 4: 1.00
Doc 1 <-> Doc 4: 0.91
Duplicate clusters:
[0, 1, 4]
MinHash correctly identified that documents 0, 1, and 4 are near-duplicates, while documents 2 and 3 are sufficiently different.
Scaling to Billions: Distributed Deduplication
For real corpora, we need distributed processing.
import ray
from typing import List, Dict, Set
import json
ray.init(address=’auto’)
@ray.remote
class DistributedMinHashWorker:
“”“Worker for distributed MinHash computation”“”
def __init__(self, config: dict):
self.deduplicator = MinHashDeduplicator(
num_perm=config[’num_perm’],
shingle_size=config[’shingle_size’],
jaccard_threshold=config[’jaccard_threshold’]
)
def process_shard(
self,
shard_path: str,
start_id: int
) -> dict:
“”“Process a shard of documents and return signatures”“”
signatures = {}
with open(shard_path, ‘r’) as f:
for idx, line in enumerate(f):
doc_id = start_id + idx
data = json.loads(line)
text = data.get(’text’, ‘’)
if not text:
continue
# Generate MinHash signature
shingles = self.deduplicator._shingling(text)
if shingles:
signature = self.deduplicator._minhash_signature(shingles)
signatures[doc_id] = signature
return signatures
@ray.remote
class LSHBandActor:
“”“Actor managing one LSH band for duplicate detection”“”
def __init__(self, band_idx: int, rows_per_band: int):
self.band_idx = band_idx
self.rows_per_band = rows_per_band
self.buckets: Dict[int, List[int]] = defaultdict(list)
def add_signatures(self, signatures: Dict[int, List[int]]):
“”“Add signatures to this band”“”
for doc_id, signature in signatures.items():
# Extract this band’s portion of signature
start = self.band_idx * self.rows_per_band
end = start + self.rows_per_band
band_sig = tuple(signature[start:end])
# Hash and add to bucket
band_hash = hash(band_sig)
self.buckets[band_hash].append(doc_id)
def get_candidate_pairs(self) -> Set[Tuple[int, int]]:
“”“Get candidate duplicate pairs from this band”“”
pairs = set()
for bucket in self.bucketsThis pipeline handles the complete deduplication workflow, from exact matching through near-duplicate detection to cross-dataset contamination removal.
𒅃 In 1948, Shannon showed that information could be compressed by removing redundancy. We’ve discovered that internet-scale text corpora are so redundant that removing duplicates makes models better while using less data. The internet, it turns out, just needed a better editor. If you think I do too, LMK in the comments.
Next up: Quality Filtering: Perplexity-Based Filtering and Language Detection — Where we discover that 40% of our “high-quality web data” is SEO spam, broken HTML, and cookie consent banners, and our model haslearned to generate all of it with disturbing fidelity.
Found that your training set contained the same Reddit thread 73 times? Discovered that deduplication improved your MMLU score by 6 points? Share your deduplication war stories in the comments. (Paid subscribers have access to the complete deduplication pipeline code and pre-tuned configurations for common corpus types.)