Probabilistic programming provides a language to describe and fit probability distributions so that we can design, encode, and automatically estimate and evaluate complex models. It aims to abstract away some of the computational and analytical complexity to allow us to focus on the conceptually more straightforward and intuitive aspects of Bayesian reasoning and inference.

The field has become quite dynamic since new languages emerged. Uber open sourced Pyro (based on PyTorch) and Google recently added a probability module to TensorFlow (see the resources linked on GitHub).

As a result, the practical relevance and use of Bayesian methods in machine learning will likely increase to generate insights into uncertainty and for use cases that require transparent rather than black-box models in particular.

In this section, we will introduce the popular PyMC3 library, which implements advanced MCMC sampling and Variational Inference for machine learning models using Python. Together with Stan, named after Stanislaw Ulam, who invented the Monte Carlo method, and developed by Andrew Gelman at Columbia University since 2012, it is the most popular probabilistic programming language.