Robust Estimation

OLS is not robust to outliers. It is computed by minimizing the sum of squares of the residuals and each outlying observation has a large residual and consequently a large effect on this sum of squares. On the other hand, The M-estimators used in robust statistics (Heritier et al. 2009; Huber 1964; Maronna, Martin, and Yohai 2006) are not influenced by outlying data. Huber (1964) proposed to minimize functions which are less influenced by outliers rather than the sum of squares. This functions are a central to the theory of robust statistics and are called \(\rho\)-function. Their derivative \(\psi(r) = \frac{\partial}{\partial r}\rho(r)\) are also useful as minimizing \(\rho(r)\) is equivalent to solving \(\psi(r)=0\). Moreover, we can infer robust properties of an estimator from its \(\psi\)-function. \(\psi\)-functions can be un-bounded (like OLS), bounded (like Huber M-estimator) or bounded and redescending (like the bi-square redescending in Koller and Stahel (2011)).

The OLS has a \(\psi\)-function increasing which results in large effect of the outliers on the estimation. The Huber \(\psi\)-function is bounded and outliers have finite effects even if the outlying point “goes” to infinity. Finally, the bi-square redescending \(\psi\)-function consider the influence of large outliers as null.

Using R in regression, the rlm() function from the MASS package (Venables and Ripley 2013) computes the Huber estimator and the lmrob() function from the robustbase (Maechler et al. 2016) package uses the bi-square redescending \(\psi\)-function. Here are an animation (Pedersen, Robinson, and RStudio 2019) showing this effect on the estimation on the intercept and slopes: