Benchmarking flaws
Published in 04 13, 2022
This doc is about the benchmarking crimes.
When reviewing systems papers (and sometimes even when reading published papers) I frequently come across highly misleading use of benchmarks. I’m not saying that the authors intend to mislead the reader, it’s just as likely incompetence
Benchmarking crimes include:
- Selective benchmarking
- Improper handling of benchmark results
- Using the wrong benchmarks
- Improper comparison of benchmark results
- Missing information
- …
Reference: van der Kouwe, Erik, et al. “Benchmarking crimes: an emerging threat in systems security.” arXiv preprint arXiv:1801.02381 (2018).