The Contamination Paradox: Why Test Set Leakage Can Be Both Potent and Negligible

Rylan Schaeffer, Ken Liu, Brando Miranda, Ahmed M Ahmed, Niloofar Mireshghallah, Sanmi Koyejo

NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle Accepted

December 2025

Language Models Data Contamination Evaluation Benchmarks

Summary

Explaining the paradox of when test set contamination matters and when it doesn't.