The Dual Roots of Belief Propagation and Causal Inference
Belief propagation and causal inference are two intertwined concepts that have emerged from distinct fields of study: econometrics and artificial intelligence. Since the “connectionist revolution” of the mid-1980s, researchers have been trying to understand how causal beliefs propagate through real or imagined networks. These ideas have revolutionized both computer science and economics, initially separately, but increasingly in close if not always harmonious contact, with statistics as mediating language.
In this little historical primer, I will focus on the contributions of four influential figures: Turingian Judea Pearl (UCLA) and Michael I. Jordan (UC Berkeley) on the computer science side, vs. Nobelist Guido Imbens and Susan Athey (both Stanford) on the econometrics side.
I’ll offer a front porch view of this sometimes contentious, but ultimately fruitful and always engaging dual development. It is no coincidence that the setting for this development is California.
In the mid-1980, the introduction of a hidden layer in the architecture of neural networks spurred the “connectionist revolution” centered around David Rumelhart and Jay McClelland’s PDP-Group at UC San Diego. An almost immediate question that arose out of that invention was if the parametrizations of those hidden layers had any meaning, or if they were just simply inscrutable computational devices that helped neural networks move beyond the XOR problem that had lead to its first Perceptrons-induced “winter”.
The stock response at the time was that introducing a hidden layer of nodes was a trade-off that increased predictability at the cost of interpretability, and the initial outcome was that those second-generation networks were almost inevitably used as heuristic inference machines for near-intractable problems where introspection didn’t matter all that much, as long as the machine provided a somewhat useful answer.
But even at that stage, the first researchers started asking themselves the question if we could imbue these hidden layers with meaning, or in reverse: could we devise more complex causal structures where observable states allowed us to draw inferences about a hidden internal structure?
Belief propagation, rooted in graph theory and Bayesian inference, offered a means to efficiently update the network’s weights and propagate information through multiple layers. This approach borrowed ideas from message passing in graphical models, enabling neural networks to iteratively refine their internal representations.
But belief propagation not only facilitated training increasingly complex networks, it also offered a theoretical framework to reason about the learning dynamics in deep, partially hidden causal structures, “as if” they were neural networks.
With statistics as the common language, two groups of scientists tried to unravel the enigma from opposite ends.
The Computer Science Perspective: Judea Pearl and Michael I. Jordan
Judea Pearl and Michael I. Jordan have been instrumental in advancing the notions of belief propagation and causal inference from a computer science and artificial intelligence perspective. Their contributions have brought forward a formal and graphical language to express causal relationships and propagate beliefs in complex systems.
Judea Pearl is widely regarded as the pioneer of graphical models and causal inference in AI. His introduction of Bayesian networks and causal Bayesian networks provides a structured approach to representing and reasoning about uncertain knowledge and causality.
Pearl’s work laid the foundation for belief propagation algorithms, which allow for efficient computation of probabilities in graphical models. His development of the “do-calculus” formalism revolutionized the study of causality, enabling researchers to reason about interventions and counterfactuals using graphical models.
Michael I. Jordan’s research extends the AI perspective on belief propagation and causal inference. He has made significant contributions to probabilistic modeling, machine learning, and information theory, including the introduction of latent dirichlet allocation, which allows us to structure large bodies of knowledge into topics.
Jordan’s work emphasizes the importance of combining statistical reasoning with computational efficiency. His research on variational methods and approximate inference techniques enhances the scalability of belief propagation algorithms in large-scale applications.
The Econometric View: Guido Imbens and Susan Athey
Causal inference, a cornerstone of econometrics, seeks to establish causal relationships between variables using observational or experimental data. Economic “power couple” Guido Imbens and Susan Athey are key figures in this realm, known for their contributions to addressing challenges in estimating causal effects.
Guido Imbens has played a pivotal role in developing methodologies for causal inference. His work on Donald Rubin’s potential outcomes framework provides a systematic way to think about causality in observational studies and randomized experiments.
Imbens’ contributions emphasize the importance of considering counterfactuals: what would have happened had a different treatment been applied? His work has been instrumental in bridging the gap between econometrics and statistics, influencing research not only in economics but also in various other fields.
Susan Athey’s work further enriches the econometric perspective on causal inference. Her contributions lie at the intersection of economics, machine learning, and statistics. Athey’s research addresses questions related to policy evaluation, online platforms, and machine learning in economics.
Her collaboration with er husband Guido Imbens on “Machine Learning Methods for Estimating Heterogeneous Causal Effects” demonstrates the power of combining machine learning techniques with traditional econometric methods to enhance causal inference accuracy and applicability.
Comparing and Contrasting Perspectives
While both econometrics and computer science/artificial intelligence share a common interest in causal inference and belief propagation, they approach these concepts from distinct angles.
The econometric perspective often focuses on estimating causal effects using real-world data, addressing challenges related to selection bias, endogeneity, and unobserved confounding.
On the other hand, the AI perspective leans toward formal representation and manipulation of causal relationships, utilizing graphical models and efficient algorithms for probabilistic reasoning.
The creative tension between these two perspectives has led to a rich exchange of ideas. Techniques from computer science, such as belief propagation algorithms, have found applications in econometric settings to address complex causal inference problems.
Conversely, econometric methodologies have inspired the development of more interpretable and applicable causal reasoning tools in AI.
With all remaining disagreements, it has been a noticeable step in the right direction that the fields are finally talking to each other.
Belief propagation and causal inference stand as fundamental concepts that have found homes in both econometrics and computer science/artificial intelligence.
Without ignoring the importance of other pioneers, the work of Judea Pearl, Michael I. Jordan, Guido Imbens, and Susan Athey has shaped the evolution of these concepts and their applications across diverse domains.
By understanding the origins and nuances of these perspectives, researchers can draw upon a rich interdisciplinary heritage to tackle complex problems related to causality and belief propagation.
Oliver Beige (Ph.D. UC Berkeley, MSIE Uni Karlsruhe) is an industrial engineer and economist working at the intersection of game theory, graph theory, machine learning, and belief propagation to develop a fundamental model of economic organization.
