#### DMCA

## Constructing Free Energy Approximations and Generalized Belief Propagation Algorithms (2005)

### Cached

### Download Links

- [www3.cs.pitt.edu]
- [people.cs.pitt.edu]
- [people.cs.pitt.edu]
- [people.cs.pitt.edu]
- [people.cs.pitt.edu]
- [www.cs.utsa.edu]
- [www.merl.com]
- [www.cs.utsa.edu]
- [cba.mit.edu]
- [www.mit.edu]
- [www.ics.uci.edu]
- [www.cs.duke.edu]
- [www.cs.duke.edu]
- [www.cs.huji.ac.il]
- [www.merl.com]
- [www.merl.com]
- [www.merl.com]
- [www.merl.com]
- [www.merl.com]
- [www.merl.com]
- [people.csail.mit.edu]
- [www.merl.com]
- [carlit.toulouse.inra.fr]
- [www.merl.com]
- [www.cs.princeton.edu]
- [www.gatsby.ucl.ac.uk]
- [www.gatsby.ucl.ac.uk]
- [www.stanford.edu]
- [www.stanford.edu]
- [www.cs.princeton.edu]

Venue: | IEEE Transactions on Information Theory |

Citations: | 585 - 13 self |

### Citations

12415 |
Elements of Information Theory
- Cover, Thomas
- 1991
(Show Context)
Citation Context ... ��� ���©��� ������� �������� � (17) � ����� � ����� (18) �������� � ����� ��� � � ��=-=��� � � � is the ����� � Kullback-Leibler ����� � divergence between and . Since there exists a � �������� � theorem [30] that is always nonnegative and is zero if ����� ��� ����� � and only if , we see th-=-at � ��� ������� ��� , with equality precisely when ����� ��� ����� � . � ��� �����sMinimizing the Gibbs free ����� ... |

8904 |
Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference
- Pearl
- 1988
(Show Context)
Citation Context ...5], Gallager’s sum-product algorithm for decoding lowdensity parity check codes [6], the “turbo-decoding” algorithm [7], [8], Pearl’s “belief propagation” algorithm for inference on Bayesi=-=an networks [9], the “K-=-alman filter” for signal processing [10], [11], and the “transfer matrix” approach in statistical mechanics [12]. MERL Cambridge Research Lab, 201 Broadway, 8th Floor, Cambridge £ MA 02139. ¤ ... |

5889 | A tutorial on hidden Markov models and selected applications in speech recognition
- Rabiner
(Show Context)
Citation Context ...quivalent or very closely-related message-passing algorithms have now been independently invented many times. They are well-known by names like the forward-backward algorithm for Hidden Markov Models =-=[3], the Viterb-=-i algorithm [4], [5], Gallager’s sum-product algorithm for decoding lowdensity parity check codes [6], the “turbo-decoding” algorithm [7], [8], Pearl’s “belief propagation” algorithm for i... |

3854 |
A new approach to linear filter and prediction problems
- Kalman
- 1960
(Show Context)
Citation Context ...ng lowdensity parity check codes [6], the “turbo-decoding” algorithm [7], [8], Pearl’s “belief propagation” algorithm for inference on Bayesian networks [9], the “Kalman filter” for sign=-=al processing [10], [11]-=-, and the “transfer matrix” approach in statistical mechanics [12]. MERL Cambridge Research Lab, 201 Broadway, 8th Floor, Cambridge £ MA 02139. ¤ yedidia@merl.com Electrical Engineering and Comp... |

2111 |
Exactly solved models in Statistical Mechanics
- Baxter
- 1982
(Show Context)
Citation Context ..., [8], Pearl’s “belief propagation” algorithm for inference on Bayesian networks [9], the “Kalman filter” for signal processing [10], [11], and the “transfer matrix” approach in statisti=-=cal mechanics [12]. -=-MERL Cambridge Research Lab, 201 Broadway, 8th Floor, Cambridge £ MA 02139. ¤ yedidia@merl.com Electrical Engineering and Computer Science, MIT Artificial Intelligence Laboratory, NE43a, Cambridge M... |

1790 | Factor graphs and the sum-product algorithm
- Kschischang, Frey, et al.
- 2001
(Show Context)
Citation Context ...ressed for other graphical models without difficulty. Using factor graphs has certain practical advantages–in particular we can refer the neophyte reader to the excellent review by Kschischang et.al=-=. [20]-=-. That review explains the equivalence to factor graphs of other graphical models such as Bayesian networks, Tanner graphs for error-correcting codes, or pair-wise Let be a set of discrete-valued rand... |

1776 | Near shannon limit errorcorrecting coding and decoding: Turbo-codes,”
- Berrou, Glavieux, et al.
- 1993
(Show Context)
Citation Context ...orward-backward algorithm for Hidden Markov Models [3], the Viterbi algorithm [4], [5], Gallager’s sum-product algorithm for decoding lowdensity parity check codes [6], the “turbo-decoding” algo=-=rithm [7], [8], Pearl��-=-�s “belief propagation” algorithm for inference on Bayesian networks [9], the “Kalman filter” for signal processing [10], [11], and the “transfer matrix” approach in statistical mechanics ... |

1655 |
Error bounds for convolution codes and an asymptotically optimum decoding algorithm”,
- Viterbi
- 2006
(Show Context)
Citation Context ...elated message-passing algorithms have now been independently invented many times. They are well-known by names like the forward-backward algorithm for Hidden Markov Models [3], the Viterbi algorithm =-=[4], [5], Galla-=-ger’s sum-product algorithm for decoding lowdensity parity check codes [6], the “turbo-decoding” algorithm [7], [8], Pearl’s “belief propagation” algorithm for inference on Bayesian networ... |

1223 |
Applied Optimal Estimation,
- Gelb
- 1974
(Show Context)
Citation Context ...il). Communicated by A. Kavčić, Associate Editor for Detection and Estimation. Digital Object Identifier 10.1109/TIT.2005.850085 Bayesian networks [9], the “Kalman filter” for signal processing [10], =-=[11]-=-, and the “transfer matrix” approach in statistical mechanics [12]. In this list of “standard” belief propagation (BP) algorithms, we have blurred a distinction between two different objectives that o... |

1204 |
Enumerative Combinatorics
- Stanley
- 1997
(Show Context)
Citation Context ...ationship, a region graph must be a directed acyclic graph, in the sense that the arrows cannot loop around. A region graph is closely related to the Hasse diagram for a partially ordered set,orposet =-=[53]-=-, if we consider our regions to be organized into a poset, with the ordering relationship between the regions to be given by the ancestor–descendant relationship [30], [31]. There are, however, some d... |

1180 |
Neuro-dynamic programming
- Bertsekas, Tsitsiklis
- 1997
(Show Context)
Citation Context .... A. Review of Lagrangian Formalism We first briefly review some necessary background about the Lagrangian formalism for constrained optimization. An excellent textbook containing more information is =-=[44]-=-. Consider a function of variables , where the variables may be subject to equality constraint(s) (written as ) and inequality constraint(s) (written as ). We will assume throughout that the equality ... |

1128 | introduction to variational methods for graphical models
- Jordan, Jaakkola, et al.
(Show Context)
Citation Context ... . Instead of a factorized form, one might consider other more � complicated forms for which still lead to tractable approx����� imations. This is the idea behind the “structured mean-fi=-=eld” approach [31]. We will no-=-t follow that path, and will instead describe a quite ����� � different approach to approximating in the next section; one which underlies the BP algorithm. IV. REGION-BASED FREE ENERGY AP... |

994 | The Viterbi algorithm
- Forney
- 1973
(Show Context)
Citation Context ...d message-passing algorithms have now been independently invented many times. They are well-known by names like the forward-backward algorithm for Hidden Markov Models [3], the Viterbi algorithm [4], =-=[5], Gallager��-=-�s sum-product algorithm for decoding lowdensity parity check codes [6], the “turbo-decoding” algorithm [7], [8], Pearl’s “belief propagation” algorithm for inference on Bayesian networks [9... |

819 | Graphical Models, Exponential Families, and Variational Inference.
- Wainwright, Jordan
- 2008
(Show Context)
Citation Context ...osed new variational inference techniques, closely related to our region-based approximations, but differentiated by a requirement that the set of beliefs used must be marginals of some global belief =-=[43]. Th-=-ey call the set of beliefs realizable from a global belief the “marginal polytope.” B. Negative Entropies Because some of the terms in the Bethe entropy have a sign that is flipped from the normal... |

806 | Graphical models
- Jordan
- 2004
(Show Context)
Citation Context ...obabilistic inference using graphical models are important in a wide variety of disciplines, including statistical physics, signal processing, artificial intelligence, and digital communications [1], =-=[2]-=-. Message-passing algorithms are a practical and powerful way to solve such problems. The centrality of such problems and the utility of messagepassing algorithms for solving them is an explanation fo... |

789 |
Statistical Mechanics
- Huang
- 1987
(Show Context)
Citation Context ...ere ��� � � � �sis the temperature. Many systems, for example Ising fer� ¢ romagnets, will have different numbers of solutions above or below a critical � � temperature within the=-= Bethe approximation [34]. � ��-=-� Above , the constrained free energy is convex and has a unique stationary point, while � � below , there are multiple stationary points. Using this equivalence it is easy to define small factor ... |

676 | Loopy belief propagation for approximate inference: an empirical study”,
- Murphy, Weiss, et al.
- 1999
(Show Context)
Citation Context ...phical models with cycles. Nevertheless, in such cases there are no guarantees, and sometimes the results are quite poor, or the algorithm fails to give any result at all because it does not converge =-=[14]-=-. Two major goals of this paper are to explain why the standard BP algorithm often works so well even for graphical models with cycles, and to use that understanding to develop improved algorithms for... |

474 | Generalized belief propagation
- Yedidia, Freeman, et al.
- 2001
(Show Context)
Citation Context ...r. We did this in an attempt der to ensure that the resulting approximations are accurate. to help the reader grasp the fundamental concepts behind our In our original work introducing GBP algorithms =-=[17]-=-, we fo- work and not lose sight of the forest because of all the trees. cused on a sub-class of GBP algorithms that were equivalent The appendices describe a variety of other methods to generate to f... |

448 |
Expectation propagation for approximate bayesian inference
- Minka
- 2001
(Show Context)
Citation Context ...ormulations of the standard BP algorithm provide different insights, and we refer the interested reader to a number of important recent papers that exploit alternative views of the BP algorithm [21], =-=[22]-=-, [23], [24], [25], [26]. After our original work which introduced region-based free energies and GBP algorithms based on the cluster variation method, Aji and McEliece introduced a class of free ener... |

404 | Turbo decoding as an instance of Pearl’s “belief propagation” algorithm.”
- McEliece, MacKay, et al.
- 1998
(Show Context)
Citation Context ...d-backward algorithm for Hidden Markov Models [3], the Viterbi algorithm [4], [5], Gallager’s sum-product algorithm for decoding lowdensity parity check codes [6], the “turbo-decoding” algorithm=-= [7], [8], Pearl’s ��-=-�belief propagation” algorithm for inference on Bayesian networks [9], the “Kalman filter” for signal processing [10], [11], and the “transfer matrix” approach in statistical mechanics [12].... |

395 |
Understanding belief propagation and its generalizations
- Yedidia, Freeman, et al.
- 2001
(Show Context)
Citation Context ...Eliece and Yildirim in [30]. We have also previously released a number of technical reports [33], [34], [35] that are largely superseded by this paper, as well as a somewhat more popular introduction =-=[36]-=-. The outline for the rest of the paper is as follows. In section II, we review and introduce our notation for factor graphs and the standard BP algorithm. In sections III and IV, we introduce and exp... |

366 | A family of algorithms for approximate Bayesian inference.
- Minka
- 2001
(Show Context)
Citation Context ...ave been a number of other recent papers that have tried to explain, reformulate, or generalize the standard belief propagation algorithm in a variety of ways. We point the interested reader to [22], =-=[23]-=-, [24], [25], [26], [27], [28]. After our original work which introduced region-based free energies and GBP algorithms based on the cluster variation method, other works appeared which explored parall... |

359 | The Generalized Distributive Law,”
- Aji, McEliece
- 2000
(Show Context)
Citation Context ...of the standard BP algorithm provide different insights, and we refer the interested reader to a number of important recent papers that exploit alternative views of the BP algorithm [21], [22], [23], =-=[24]-=-, [25], [26]. After our original work which introduced region-based free energies and GBP algorithms based on the cluster variation method, Aji and McEliece introduced a class of free energy approxima... |

320 |
Graphical Models for Machine Learning and Digital Communications.
- Frey
- 1998
(Show Context)
Citation Context ...ng probabilistic inference using graphical models are important in a wide variety of disciplines, including statistical physics, signal processing, artificial intelligence, and digital communications =-=[1]-=-, [2]. Message-passing algorithms are a practical and powerful way to solve such problems. The centrality of such problems and the utility of messagepassing algorithms for solving them is an explanati... |

231 | Correctness of local probability propagation in graphical models with loops,”
- Weiss
- 2000
(Show Context)
Citation Context ...the Bethe approximation. In graphs with no more than a single cycle, it was � known � that� if all (� ��� ����� factors are � strictly � positive for all and ), then there =-=was a unique BP fixed point.[33]-=- For general graphs, we can use the equivalence established above to answer a question about the uniqueness of stationary points for the Bethe free energy. The issue of the number of stationary points... |

225 | A New Class of Upper Bounds on the Log Partition Function”,
- Wainwright, Jaakkola, et al.
- 2005
(Show Context)
Citation Context ...numbers to differ from those given in the Bethe approximation, is one way of deriving the “fractional belief propagation algorithm” [40] and the essentially equivalent “convexified Bethe free en=-=ergy” [41]-=- approximation. In this paper, we will always assume just one set of counting numbers. In fact, not all region-based approximations to the variational free energy are equally good. At this point, we i... |

196 |
The Viterbi algorithm.
- Jr
- 1973
(Show Context)
Citation Context ...d message-passing algorithms have now been independently invented many times. They are well known by names like the forward–backward algorithm for hidden Markov models [3], the Viterbi algorithm [4], =-=[5]-=-, Gallager’s sum–product algorithm for decoding lowdensity parity check codes [6], the “turbo-decoding” algorithm [7], [8], Pearl’s “belief propagation” algorithm for inference on Manuscript received ... |

178 |
Codes on graphs: Normal realizations
- Forney
- 2001
(Show Context)
Citation Context ...ther formulations of the standard BP algorithm provide different insights, and we refer the interested reader to a number of important recent papers that exploit alternative views of the BP algorithm =-=[21]-=-, [22], [23], [24], [25], [26]. After our original work which introduced region-based free energies and GBP algorithms based on the cluster variation method, Aji and McEliece introduced a class of fre... |

158 |
Low-Density Parity Check Codes,
- Gallager
- 1963
(Show Context)
Citation Context ...y are well-known by names like the forward-backward algorithm for Hidden Markov Models [3], the Viterbi algorithm [4], [5], Gallager’s sum-product algorithm for decoding lowdensity parity check code=-=s [6], the “turbo-d-=-ecoding” algorithm [7], [8], Pearl’s “belief propagation” algorithm for inference on Bayesian networks [9], the “Kalman filter” for signal processing [10], [11], and the “transfer matrix... |

137 | CCCP algorithms to minimize the Bethe and Kikuchi free energies: Convergent alternatives to belief propagation.
- Raymond, Yuille
- 2002
(Show Context)
Citation Context ...t the beliefs are in a feasible set. Based on the equivalence, first noted in our earlier work [17], others have recently devised algorithms that directly minimize the free energy on the feasible set =-=[35]-=-, [36], [37]. Such free energy minimizations are somewhat slower than the BP algorithm, but they are guaranteed to converge. VI. THE REGION GRAPH METHOD We now introduce region graphs, which are centr... |

122 | Tree-based reparameterization framework for analysis of sum-product and related algorithms,”
- Wainwright, Jaakkola, et al.
- 2003
(Show Context)
Citation Context ...en a number of other recent papers that have tried to explain, reformulate, or generalize the standard belief propagation algorithm in a variety of ways. We point the interested reader to [22], [23], =-=[24]-=-, [25], [26], [27], [28]. After our original work which introduced region-based free energies and GBP algorithms based on the cluster variation method, other works appeared which explored parallel ide... |

103 | Loopy belief propagation and Gibbs measures
- Tatikonda, Jordan
- 2002
(Show Context)
Citation Context ...fs, but at one of the fixed points, the beliefs are biased towards the first binary state, while at the other fixed point, the beliefs are biased towards the second binary state. Tatikonda and Jordan =-=[47]-=- have explored the question of uniqueness of BP fixed points in detail. They used the connection to the Bethe free energy to obtain a set of sufficient conditions on the strength of the factors fa(xa)... |

98 |
The theory of cooperative phenomena.
- Kikuchi
- 1951
(Show Context)
Citation Context ...re actually a variety of ways to define GBP alenergies were introduced long ago in the physics literature by gorithms for any given region graph, all of which have identical by Bethe [15] and Kikuchi =-=[16]-=-. For the important special case fixed points. We focus on one particular type of GBP algorithm, of the standard BP algorithm, we show that its fixed points are which we call the parent-to-child algor... |

96 |
Random K-satisfiability problem: From an analytic solution to an efficient algorithm,
- Mezard, Zecchina
- 2002
(Show Context)
Citation Context ...ard BP algorithm provide different insights, and we refer the interested reader to a number of important recent papers that exploit alternative views of the BP algorithm [21], [22], [23], [24], [25], =-=[26]-=-. After our original work which introduced region-based free energies and GBP algorithms based on the cluster variation method, Aji and McEliece introduced a class of free energy approximations and GB... |

95 | Bethe free energy, kikuchi approximations, and belief propagation algorithms,” Advances in neural information processing systems,
- Yedidia, Freeman, et al.
- 2001
(Show Context)
Citation Context ...raphs [29]. We also recommend the elegant exposition of generalized belief propagation presented by McEliece and Yildirim in [30]. We have also previously released a number of technical reports [33], =-=[34]-=-, [35] that are largely superseded by this paper, as well as a somewhat more popular introduction [36]. The outline for the rest of the paper is as follows. In section II, we review and introduce our ... |

84 |
Infinite-ranged models of spin-glasses,” Phys
- Sherrington, Kirckpatrick
- 1978
(Show Context)
Citation Context ...riable nodes, where every pair of nodes is connected by a factor. (A version of this factor graph with random factors is known in the physics literature as the Sherrington–Kirpatrick Ising spin glass =-=[54]-=-.) Now take, as the regions to include in the region graph, every triplet of nodes (and all three factors that connect them), every pair of nodes (and the factor that connects them), and every single ... |

78 |
Statistical theory of superlattices
- Bethe
- 1935
(Show Context)
Citation Context ...to another free energy that can be justified by a rigorous vari1sational principle. The first specialized examples of such free energies were introduced long ago in the physics literature by by Bethe =-=[15]-=- and Kikuchi [16]. For the important special case of the standard BP algorithm, we show that its fixed points are the same as the stationary points of the Bethe free energy, thus establishing an impor... |

71 | Stable fixed points of loopy belief propagation are minima of the Bethe free energy. In
- Heskes
- 2003
(Show Context)
Citation Context ...ld be BP fixed points. To complete the general picture of the relation between BP fixed points and the stationary points of the constrained Bethe free energy, we refer the reader to a paper by Heskes =-=[52]-=-, which argues that stable BP fixed points must be local minima of the constrained Bethe free energy, but gives a counter-example that shows that the converse is not true. VII. THE REGION GRAPH METHOD... |

63 | Belief optimization for binary networks: A stable alternative to loopy belief propagation.
- Welling, Teh
- 2001
(Show Context)
Citation Context ... fixed points ane Bethe free energy stationary points, first noted in our earlier work [17], others have devised algorithms that directly minimize the free energy on the feasible set of beliefs [49], =-=[50]-=-, [51]. Such free energy minimizations are somewhat slower than the BP algorithm, but they are guaranteed to converge. D. Factor Graphs Containing Hard Constraints We now return to consider the more g... |

54 |
Introduction to inference for Bayesian Networks.
- Cowell
- 1998
(Show Context)
Citation Context ... of cycles. Thus, a common approach for dealing with graphical models that do have cycles is to try to convert them to equivalent cyclefree graphical models, and then to use the standard BP algorithm =-=[13]-=-. In some cases, this is possible, but for many other cases of practical interest, such an approach is intractable, and one must settle for approximate methods. Fortunately, the standard BP algorithms... |

54 |
Tree-structured approximations by expectation propagation.
- Minka, Qi
- 2004
(Show Context)
Citation Context ...will disfavor the (correct) uniform distribution. It is therefore no surprise that other researchers have noticed that this approximation gives poor results for the Sherrington-Kirpatrick model [51], =-=[55]-=-. B. Example of an Approximation that is Maxent-Normal Fortunately, it is not too hard to find examples of region graph approximations that are maxent-normal, besides those based on the Bethe approxim... |

53 | Tree-based reparameterization for approximate estimation on loopy graphs. In
- Wainwright, Jaakkola, et al.
- 2002
(Show Context)
Citation Context ...tions of the standard BP algorithm provide different insights, and we refer the interested reader to a number of important recent papers that exploit alternative views of the BP algorithm [21], [22], =-=[23]-=-, [24], [25], [26]. After our original work which introduced region-based free energies and GBP algorithms based on the cluster variation method, Aji and McEliece introduced a class of free energy app... |

46 | Fractional belief propagation,”
- Wiegerinck, Heskes
- 2003
(Show Context)
Citation Context ...sed in the Bethe approximation, but modifying the entropic counting numbers to differ from those given in the Bethe approximation, is one way of deriving the “fractional belief propagation algorithm” =-=[40]-=- and the essentially equivalent “convexified Bethe free energy” [41] approximation. In this paper, we will always assume just one set of counting numbers. In fact, not all region-based approximations ... |

45 |
The generalized distributive law and free energy minimization.
- Aji, McEliece
- 2001
(Show Context)
Citation Context ...uced region-based free energies and GBP algorithms based on the cluster variation method, Aji and McEliece introduced a class of free energy approximations and GBP algorithms based on junction graphs =-=[27]-=-. One of the goals of this paper is to unify our previous approach with the one that Aji and McEliece presented. McEliece and Yildirim have independently developed a unified approach Here is an index ... |

41 |
Sherrington D, Infinite-range models of spin-glasses
- Kirkpatrick
- 1978
(Show Context)
Citation Context ...riable nodes, where every pair of nodes is connected by a factor. (A version of this factor graph with random factors is known in the physics literature as the Sherrington-Kirpatrick Ising spin glass =-=[54]-=-.) Now take, as the regions to include in the region graph, every triplet of nodes (and all three factors that connect them), every pair of nodes (and the factor that connects them), and every single ... |

39 |
Theory of the frustration effect in spin glasses:
- Toulouse
- 1977
(Show Context)
Citation Context ... 2 and 3 prefers them to be in different states. Not all of these factors can be satisfied simultaneously; this is thus a very simple example of what statistical physicists call a “frustrated” sys=-=tem [42]. The be-=-liefs ba(xa) and bi(xi) that minimize the constrained Bethe free energy for this model are � � 0.4 0.1 bA(x1, x2) = bB(x1, x3) = , (50) 0.1 0.4 � � 0.1 0.4 bC(x1, x2) = , (51) 0.4 0.1 9 and b1... |

29 | Belief propagation and statistical physics.
- Pakzad, Anantharam
- 2002
(Show Context)
Citation Context ...ropagation which is largely equivalent to our region graph approach, and we recommend their elegant exposition [28]. Pakzad and Anantharam have also recently presented parallel ideas in a brief paper =-=[29]-=-. The outline for the rest of the paper is as follows. In section II, we review and introduce our notation for factor graphs and the standard BP algorithm. In sections III and IV, we introduce and exp... |

28 |
Codes on graphs: normal realizations
- Jr
- 2001
(Show Context)
Citation Context ...here have been a number of other recent papers that have tried to explain, reformulate, or generalize the standard belief propagation algorithm in a variety of ways. We point the interested reader to =-=[22]-=-–[28]. After our original work which introduced region-based free energies and GBP algorithms based on the cluster variation method, other works appeared which explored parallel ideas [29]–[32]. In fa... |

27 |
On the choice of regions for generalized belief propagation
- Welling
(Show Context)
Citation Context ...and automatically construct maxent-normal region graph approximations that also satisfied our heuristics. We do not know of any such method, but we refer the reader to an interesting paper by Welling =-=[56], wh-=-o developed a “bottom-up” approach to generating region graph approximations starting from the Bethe approximation. IX. GENERALIZED BELIEF PROPAGATION ALGORITHMS We have already seen that the stat... |

26 | Belief propagation on partially ordered sets. In
- McEliece, Yildirim
- 2003
(Show Context)
Citation Context ...beling functions , where the function has arguments that are some subset of . to belief propagation which is largely equivalent to our region graph approach, and we recommend their elegant exposition =-=[28]-=-. Pakzad and Anantharam have also recently presented parallel ideas in a brief paper [29]. The outline for the rest of the paper is as follows. In section II, we review and introduce our notation for ... |

22 | Novel iteration schemes for the cluster variation method
- Kappen, Wiegerinck
- 2001
(Show Context)
Citation Context ...s are in a feasible set. Based on the equivalence, first noted in our earlier work [17], others have recently devised algorithms that directly minimize the free energy on the feasible set [35], [36], =-=[37]-=-. Such free energy minimizations are somewhat slower than the BP algorithm, but they are guaranteed to converge. VI. THE REGION GRAPH METHOD We now introduce region graphs, which are central to the re... |

17 |
variation method for non-uniform Ising and Heisenberg models and spin-pair correlation functions
- Morita, Cluster
- 1991
(Show Context)
Citation Context ...pendices describe a variety of other methods to generate to free energy approximations based on Kikuchi’s cluster vari- region graphs and GBP algorithms which could easily prove to ation method [16]=-=, [18]-=-, [19]. We shall show that this method be as important in practice as the methods described in the main is only one of a variety of methods to generate region graphs and their corresponding free energ... |

17 | Estimation and marginalization using Kikuchi approximation methods.
- Pakzad, Anantharam
- 2005
(Show Context)
Citation Context ...28]. After our original work which introduced region-based free energies and GBP algorithms based on the cluster variation method, other works appeared which explored parallel ideas [29], [30], [31], =-=[32]-=-. In fact, one of the goals of this paper is to unify our previous approach with the one that Aji and McEliece presented based on junction graphs [29]. We also recommend the elegant exposition of gene... |

14 |
Foundations and applications of cluster variation method and path probability method
- Morita, Suzuki, et al.
- 1994
(Show Context)
Citation Context ...es describe a variety of other methods to generate to free energy approximations based on Kikuchi’s cluster vari- region graphs and GBP algorithms which could easily prove to ation method [16], [18]=-=, [19]-=-. We shall show that this method be as important in practice as the methods described in the main is only one of a variety of methods to generate region graphs and their corresponding free energies an... |

10 | A conversation about the bethe free energy and sum-product
- Mackay, Yedidia, et al.
- 2001
(Show Context)
Citation Context ...not normally explicitly construct an overall “trial” belief vector ����� � that is consistent with the multi-node beliefs ��������� � , and therefore one does not n=-=ormally obtain any upper bound on � [32]-=-. On the other hand, one can make approximations that are much more accurate than the factorized meanfield approximation, and there is a great deal of flexibility in the exact choice of approximation.... |

6 | On structured-summary propagation, LFSR synchronization, and low-complexity trellis decoding
- Dauwels, Loeliger, et al.
(Show Context)
Citation Context ...ent papers that have tried to explain, reformulate, or generalize the standard belief propagation algorithm in a variety of ways. We point the interested reader to [22], [23], [24], [25], [26], [27], =-=[28]-=-. After our original work which introduced region-based free energies and GBP algorithms based on the cluster variation method, other works appeared which explored parallel ideas [29], [30], [31], [32... |

5 | An algebraic description of iterative decoding schemes
- Offer, Soljanin
- 2000
(Show Context)
Citation Context ... standard BP algorithm provide different insights, and we refer the interested reader to a number of important recent papers that exploit alternative views of the BP algorithm [21], [22], [23], [24], =-=[25]-=-, [26]. After our original work which introduced region-based free energies and GBP algorithms based on the cluster variation method, Aji and McEliece introduced a class of free energy approximations ... |

5 |
Survey propagation: an algorithm for satisfiability. http://fr.arxiv.org/abs/cs.CC/0212002
- Braunstein, Mezard, et al.
- 2003
(Show Context)
Citation Context ...er recent papers that have tried to explain, reformulate, or generalize the standard belief propagation algorithm in a variety of ways. We point the interested reader to [22], [23], [24], [25], [26], =-=[27]-=-, [28]. After our original work which introduced region-based free energies and GBP algorithms based on the cluster variation method, other works appeared which explored parallel ideas [29], [30], [31... |

3 |
An approximation method for order-disorder problems
- Hijmans, Boer
- 1955
(Show Context)
Citation Context ... In our original work introducing GBP algorithms [17], we focused on a sub-class of GBP algorithms that were equivalent to free energy approximations based on Kikuchi’s cluster variation method [16]=-=, [18]-=-, [19], [20]. We shall show that this method is only one of a variety of methods to generate region graphs and their corresponding free energies and message-passing algorithms. In our original work, w... |

3 |
On the uniqueness of belief propagation fixed points
- Heskes
- 2004
(Show Context)
Citation Context ...tion to the Bethe free energy to obtain a set of sufficient conditions on the strength of the factors fa(xa) to ensure unique BP fixed points for arbitrary Markov random fields. More recently, Heskes =-=[48]-=- has analyzed the same question using the connection to the Bethe free energy, but his sufficient conditions for uniqueness also take into consideration the topology of the factor graph. While we have... |

3 | Survey propagation: an algorithm for satisfiability,Random Structures and Algorithms,Volu.27 - Braunstein, Mézard, et al. |

2 |
Belief optimization: a stable alternative to belief propagation
- Welling, Teh
- 2001
(Show Context)
Citation Context ...beliefs are in a feasible set. Based on the equivalence, first noted in our earlier work [17], others have recently devised algorithms that directly minimize the free energy on the feasible set [35], =-=[36]-=-, [37]. Such free energy minimizations are somewhat slower than the BP algorithm, but they are guaranteed to converge. VI. THE REGION GRAPH METHOD We now introduce region graphs, which are central to ... |

2 |
Characterizing belief propagation and its generalizations. Available at: www.merl.com/reports/TR2002-35
- Yedidia, Freeman, et al.
- 2002
(Show Context)
Citation Context ...es in the region graph. We have previously reported promising numerical results obtained using GBP algorithms for inference on random Markov Random Fields [17] and for decoding error-correcting codes =-=[39]-=-. ACKNOWLEDGEMENTS We thank Dave Forney and Robert McEliece for helpful and encouraging discussions, and David MacKay for his comments on a draft of this paper. APPENDIX A: THE JUNCTION GRAPH METHOD A... |

2 |
Poset belief propagation: Experimental results. Caltech senior thesis, 2003
- Harel
(Show Context)
Citation Context ... that do not have any symmetries. Besides the results that we now discuss, which were first reported in [17], the interested reader can find similar empirical results for GBP algorithms in references =-=[58]-=-, [59], [56], [33], [49]. Readers who are more interested in rigorous bounds on the accuracy of marginals will want to consult the work of Wainwright et.al. [24]. We studied factor graphs known in the... |

1 |
Minimal graphical representation of Kikuchi regions
- Pakzad, Anantharam
- 2002
(Show Context)
Citation Context ...27], [28]. After our original work which introduced region-based free energies and GBP algorithms based on the cluster variation method, other works appeared which explored parallel ideas [29], [30], =-=[31]-=-, [32]. In fact, one of the goals of this paper is to unify our previous approach with the one that Aji and McEliece presented based on junction graphs [29]. We also recommend the elegant exposition o... |

1 |
Characterizing belief propagation and its generalizations. http://www.merl.com/reports/TR200115/index.html
- Yedidia, Freeman, et al.
- 2001
(Show Context)
Citation Context ...tion graphs [29]. We also recommend the elegant exposition of generalized belief propagation presented by McEliece and Yildirim in [30]. We have also previously released a number of technical reports =-=[33]-=-, [34], [35] that are largely superseded by this paper, as well as a somewhat more popular introduction [36]. The outline for the rest of the paper is as follows. In section II, we review and introduc... |

1 |
Frustration effect on the d-dimensional ising spin glass: Ii. existence of the spin glass phase
- Fujiki, Katsura
- 1980
(Show Context)
Citation Context ...e Tc is approximately 2.2692J, compared to the mean field theory prediction of 4.0J, the Bethe approximation prediction of 2.8854J, and the Kikuchi prediction (using 2 by 2 clusters) of 2.4257J [16], =-=[57]-=-. Qualitatively similar results are available in the physics literature for a wide variety of models of magnetic systems with translationally invariant interactions. However, when considering probabil... |

1 |
Characterizing Belief Propagation and Its Generalizations. [Online]. Available: http://www
- Yedidia, Freeman, et al.
- 2001
(Show Context)
Citation Context ...ece presented based on junction graphs [29]. We also recommend the elegant exposition of GBP presented by McEliece and Yildirim in [30]. We have also previously released a number of technical reports =-=[33]-=-–[35] that are largely superseded by this paper, as well as a somewhat more popular introduction [36]. The outline for the rest of the paper is as follows. In Section II, we review and introduce our n... |

1 |
Poset belief propagation—Experimental results
- Harel, McEliece, et al.
(Show Context)
Citation Context ...ctor graphs that do not have any symmetries. Besides the results that we now discuss, which were first reported in [17], the interested reader can find similar empirical results for GBP algorithms in =-=[58]-=-, [59], [56], [33], [49]. Readers who are more interested in rigorous bounds on the accuracy of marginals will want to consult the work of Wainwright et al. [24]. We studied factor graphs known in the... |