Event-related potentials (ERPs) are increasingly used in cognitive science. With their high temporal resolution, they offer a unique window into cognitive processes and their time course. In this paper, we focus on ERP experiments whose designs involve selecting participants and stimuli amongst many. Recently, Westfall et al. (2017) highlighted the drastic consequences of not considering stimuli as a random variable in fMRI studies with such designs. Most ERP studies in cognitive psychology suffer from the same drawback. We advocate the use of the Quasi-F or Mixed-effects models instead of the classical ANOVA/by-participant F1 statistic to analyze ERP datasets in which the dependent variable is reduced to one measure per trial (e.g., mean amplitude). We combine Quasi-F statistic and cluster mass tests to analyze datasets with multiple measures per trial. Doing so allows us to treat stimulus as a random variable while correcting for multiple comparisons. Simulations show that the use of Quasi-F statistics with cluster mass tests allows maintaining the family wise error rates close to the nominal alpha level of 0.05. Simulations reveal that the classical ANOVA/F1 approach has an alarming FWER, demonstrating the superiority of models that treat both participant and stimulus as random variables, like the Quasi-F approach. Our simulations question the validity of studies in which stimulus is not treated as a random variable. Failure to change the current standards feeds the replicability crisis.