Using sequential hypothesis testing techniques to check the modelling assumptions of Bayesian mixture estimators is a promising way of getting value out of combining the Bayesian and frequentist approaches to probability. Here’s a paper to show how that can be done for Context Tree Weighting and related methods.
Paper Abstract: Universal Bayesian sequence predictors like Context Tree Weighting (CTW) offer strong asymptotic convergence guarantees but remain vulnerable to structural misspecification and finite-sample miscalibration issues when deployed in complex, non-stationary environments. This paper introduces an online monitoring framework for Bayesian sequence prediction in the form of a suite of test martingales designed to continuously audit CTW and related methods. The martingales are constructed using betting functions on prequential and conformal p-values, including a universal betting function based on the Transformer architecture. We show that this approach can effectively isolate diverse failure modes in CTW, including static marginal errors and hidden temporal dependencies. Further, we provide an active recalibration mechanism for CTW that can dynamically respond to martingale alerts.
Here’s the full paper.
Mental Models 4 Life 