Bayesian posterior predictive intervals are a central tool for uncertainty quantification in regression. However, in highly overparameterized models—including random feature regression and wide neural networks—recent work has shown that posterior predictive variances can be badly misaligned with the frequentist generalization error of the corresponding point estimator, especially near interpolation thresholds and in low-noise regimes. This misalignment contributes to the “cold posterior effect”, where artificially concentrating the posterior improves empirical performance. In this paper, we propose a principled framework for calibrated Bayesian random feature regression based on posterior tempering. We consider a family of generalized posteriors indexed by a temperature parameter T > 0 that smoothly interpolates between over-dispersed and under-dispersed uncertainty estimates by rescaling the effective regularization in Bayesian random feature regression. Using high-dimensional random matrix asymptotics for random feature models, we derive deterministic limits for the test risk and posterior predictive variance as functions of the feature and sample aspect ratios, signal-to-noise ratio, and temperature. We then define calibration-optimal and risk-optimal temperatures that, respectively, align posterior predictive variance with frequentist prediction error and minimize test risk. Our analysis shows that in overparameterized, low-noise regimes, both optimal temperatures are strictly smaller than one, thus providing a theoretical explanation for the cold posterior effect in random feature models. We complement our asymptotic results with finite-sample simulations that demonstrate substantial improvements in uncertainty calibration and competitive or superior test risk across a wide range of regimes. Finally, we propose practical, data-driven procedures to estimate the temperature from a single dataset, making calibrated Bayesian random feature regression a viable tool for uncertainty-aware prediction in overparameterized systems.
Bayesian posterior predictive intervals are a central tool for uncertainty quantification in regression. However, in highly overparameterized models—including random feature regression and wide neural networks—recent work has shown that posterior predictive variances can be badly misaligned with the frequentist generalization error of the corresponding point estimator, especially near interpolation thresholds and in low-noise regimes. This misalignment contributes to the “cold posterior effect”, where artificially concentrating the posterior improves empirical performance. In this paper, we propose a principled framework for calibrated Bayesian random feature regression based on posterior tempering. We consider a family of generalized posteriors indexed by a temperature parameter T > 0 that smoothly interpolates between over-dispersed and under-dispersed uncertainty estimates by rescaling the effective regularization in Bayesian random feature regression. Using high-dimensional random matrix asymptotics for random feature models, we derive deterministic limits for the test risk and posterior predictive variance as functions of the feature and sample aspect ratios, signal-to-noise ratio, and temperature. We then define calibration-optimal and risk-optimal temperatures that, respectively, align posterior predictive variance with frequentist prediction error and minimize test risk. Our analysis shows that in overparameterized, low-noise regimes, both optimal temperatures are strictly smaller than one, thus providing a theoretical explanation for the cold posterior effect in random feature models. We complement our asymptotic results with finite-sample simulations that demonstrate substantial improvements in uncertainty calibration and competitive or superior test risk across a wide range of regimes. Finally, we propose practical, data-driven procedures to estimate the temperature from a single dataset, making calibrated Bayesian random feature regression a viable tool for uncertainty-aware prediction in overparameterized systems.
