Interesting paper: "Direction Augmentation in the Evaluation of Armed Conflict Predictions" by
@johannes (Johannes Bracher), Lotta Rüter, Fabian Krüger,
@sebastianlerch and Melanie Schienle
#Prediction #Forecasting #modelevaluation #conflictpredictionhttps://arxiv.org/abs/2304.12108
Direction Augmentation in the Evaluation of Armed Conflict Predictions
In many forecasting settings, there is a specific interest in predicting the
sign of an outcome variable correctly in addition to its magnitude. For
instance, when forecasting armed conflicts, positive and negative log-changes
in monthly fatalities represent escalation and de-escalation, respectively, and
have very different implications. In the ViEWS forecasting challenge, a
prediction competition on state-based violence, a novel evaluation score called
targeted absolute deviation with direction augmentation (TADDA) has therefore
been suggested, which accounts for both for the sign and magnitude of
log-changes. While it has a straightforward intuitive motivation, the empirical
results of the challenge show that a no-change model always predicting a
log-change of zero outperforms all submitted forecasting models under the TADDA
score. We provide a statistical explanation for this phenomenon. Analyzing the
properties of TADDA, we find that in order to achieve good scores, forecasters
often have an incentive to predict no or only modest log-changes. In
particular, there is often an incentive to report conservative point
predictions considerably closer to zero than the forecaster's actual predictive
median or mean. In an empirical application, we demonstrate that a no-change
model can be improved upon by tailoring predictions to the particularities of
the TADDA score. We conclude by outlining some alternative scoring concepts.
#openaccess article "From academia to policy makers: a methodology for real-time
#forecasting of infrequent events" by Alfred Krzywicki, David Muchlinski (Twitter: @DMuchlinski), Benjamin E. Goldsmith (Twitter: @goldsmithbe), Arcot Sowmya in the Journal of Computational Social Science (JCSS)
#conflictprediction #conflictforecasting #machinelearning #masskillings @genocides
👇
https://link.springer.com/article/10.1007/s42001-022-00176-6 
From academia to policy makers: a methodology for real-time forecasting of infrequent events - Journal of Computational Social Science
The field of conflict forecasting has matured greatly over the last decade. Advances in machine learning have allowed researchers to forecast rare political and social events in near real time. Yet the maturity of the field has led to a proliferation of diverse platforms for forecasting, divergent results across forecasts, and an explosion of forecasting methodologies. While the field has done much to establish some baseline results, true, consensual benchmarks against which future forecasts may be evaluated remain elusive, and thus, agreed upon empirical results are still rare. The aim of this work is to address these concerns and provide the field of conflict forecasting with a standardized analysis pipeline to evaluate future forecasts of political violence. We aim to open the black box of the conflict forecasting pipeline and provide empirical evidence on how modeling decisions along all steps of the pipeline affect end results. In this way, we empirically demonstrate best practices that conflict forecasting researchers may utilize in future endeavors. We employ forecasts of targeted mass killings and genocides to support our methodological claims.
Gregor Reisch