Bayesian Deep Learning

Check our latest paper:

• AISTATS 2025: From deep additive kernel learning to last-layer Bayesian neural networks via induced prior approximation

Deep Additive Kernel Learning

With the strengths of both deep learning and kernel methods like Gaussian Processes (GPs), Deep Kernel Learning (DKL) has gained considerable attention in recent years. From the computational perspective, however, DKL becomes challenging when the input dimension of the last-layer GP is high. To address this challenge, we propose the Deep Additive Kernel (DAK) model, which incorporates i) an additive structure for the last-layer GP; and ii) induced prior approximation for each GP component. This naturally leads to a last-layer Bayesian neural network (BNN) architecture. The proposed method enjoys the interpretability of DKL as well as the computational advantages of BNN. Empirical results show that the proposed approach outperforms state-of-the-art DKL methods in both regression and classification tasks.

Deep Markov Gaussian Processes

Deep Gaussian Processes (DGPs) are a powerful class of probabilistic models that extend traditional GPs by stacking multiple GP layers. We develop software based on a class of DGPs with sparse structure, referred to as Deep Markov GPs (DMGPs). More specifically, we implement DMGPs as sparsely activated Bayesian Neural Networks (BNNs) with learnable weights and biases, which can be used in a wide range of deep learning applications. The depth of DGPs enables them to model intricate dependencies and variations, making them suitable for tasks such as regression, classification, and time-series analysis. The DMGP software can provide not only uncertainty quantification for predictions, but also interpretability by incorporating the neural additive structure. Our evaluation across various tabular datasets and visual object tasks shows that DMGPs not only match but often surpasses the performance of traditional DGP methods, all while significantly reducing the number of parameters. Due to the strong connection to deep neural networks, DMGPs can be easily extended to other state-of-art deep learning architectures and applications.

• DMGP package is available at https://github.com/warrenzha/dmgp.
• Documentation, examples, tutorials on how to construct all sorts of DMGP models.