Robust Estimation

OLS is not robust to outliers. It is computed by minimizing the sum of squares of the residuals and each outlying observation has a large residual and consequently a large effect on this sum of squares. On the other hand, The M-estimators used in robust statistics (Heritier et al. 2009; Huber 1964; Maronna, Martin, and Yohai 2006) are not influenced by outlying data. Huber (1964) proposed to minimize functions which are less influenced by outliers rather than the sum of squares. This functions are a central to the theory of robust statistics and are called $\rho$-function. Their derivative $\psi (r)=\frac{\partial }{\partial r}\rho (r)$ are also useful as minimizing $\rho (r)$ is equivalent to solving $\psi (r)=0$. Moreover, we can infer robust properties of an estimator from its $\psi$-function. $\psi$-functions can be un-bounded (like OLS), bounded (like Huber M-estimator) or bounded and redescending (like the bi-square redescending in Koller and Stahel (2011)).

The OLS has a $\psi$-function increasing which results in large effect of the outliers on the estimation. The Huber $\psi$-function is bounded and outliers have finite effects even if the outlying point “goes” to infinity. Finally, the bi-square redescending $\psi$-function consider the influence of large outliers as null.

Using R in regression, the rlm() function from the MASS package (Venables and Ripley 2013) computes the Huber estimator and the lmrob() function from the robustbase (Maechler et al. 2016) package uses the bi-square redescending $\psi$-function. Here are an animation (Pedersen, Robinson, and RStudio 2019) showing this effect on the estimation on the intercept and slopes: