If you find the following studies useful and worth of publication in your journal or for other dissemination, let me know. Preprints are available from me upon e-mail request. These studies are to be part of my online freely available textbooks on Bayesian statistical data analysis and its application to natural and other sciences. Coming soon =)
On scoring rules and frequency predictive measures (submitted and rejected at four statistical journals: Bayesian Analysis, Statistics and Probability letters, JRSS-A, The American Statistician)
Abstract. Scoring rules of statistical models have become a common tool for practitioners to use for model comparison and model selection. The objective of this study is to provide pedagogical examples with beautiful geometric illustrations and interpretations which show that scoring rules have very limited ability in ruling out which models among considered are better than others.
The consequences of (mis)understanding of the limited ability of scoring rules to guide a model choice are of great importance to society and researchers who work in areas of contemporary application of statistics, including finance, biology, weather and climate. We show that decisions based on scoring rules are to a great extent subjective and recommend to, instead, employ three customary independent measures of model predictive performance: (1) empirical frequency coverage, (2) an estimate of predictive bias and (3) an estimate of uncertainty and variability in predictions. These measures have much more transparent interpretations than other scores. Manuscript, Supplementary materials, Examples in R.