>Michael Nielsen (mnielsen@theory.caltech.edu)
>Fri, 29 Jan 1999 11:57:51 -0800 (PST)
>On 29 Jan 1999, Anders Sandberg wrote:
>
>> Probability Theory: the Logic of Science bt E. T. Jaynes
>>
>> Fulltext (but missing certain pieces) at
>> http://omega.albany.edu:8008/JaynesBook
>
>Sadly, Jaynes died a year or so ago, so this book will never be completed.
>
>Michael
To follow up on the Bayes/Jaynes discussion.
E.T Jaynes died last April '98, but his "spirit" is alive and thriving in the Maximum Entropy statistics field.
One of his old students (and a smart Bayesian scientist), Larry Bretthorst, is doing his best to complete Jaynes' book. He still has handwritten manuscripts from Jaynes that he is working to convert to a publishable form. As of last summer, Bretthorst was still deciding whether to complete the book with him (Bretthorst) filling in the remaining chapters with his own words, or to cut out those incomplete chapters. I believe that a book will be completed though, in the not-too-distant future, with mostly Jaynes' words.
I was fortunate last summer to attend the Eighteenth International Workshop on Maximum Entropy and Bayesian Methods (MaxEnt '98). It was one in a series of "MaxEnt" conferences that have been held every year since 1981 at different locations all over the world. (See: http://www.ipp.mpg.de/OP/maxent98/me98.html for the 1998 conference) The general scope of the annual conferences are the applications of the maximum entropy and Bayesian methods for diverse areas of scientific research. The workshop last year was dedicated to (and a special focus on) Edwin Jaynes. Because the 1998 meeting was a special meeting devoted to Edwin Jaynes, and because many of the meeting participants had a personal and professional relationship with Jaynes that lasted for decades, the presentations and coffee-break discussions provided a number of historical summaries that gave a nice historical framework for the conference newcomer and nice rememberences for the conference oldcomer.
I'm still a newcomer to the Bayesian Probability Theory field. I started reading the literature bit-by-bit last Fall, but I haven't learned enough to apply it to my own work yet. However, I've accumulated some really nice references since last Fall that I can share with you and written some text about the general ideas for my colleagues.
Here, let me give an overview of Bayesian Probability Theory. I'm an astronomer, so what follows will have an astronomy slant.
Overview
Bayesian Probability Theory is a rigorous mathematical theory constructed from a simple notion that "probability is a measure of degree of a proposition's plausibility."
The Bayesian approach to scientific inference takes into account not only the raw data, but also the prior knowledge that one has to supplement the data. That prior knowledge may be data or results from previous experiments, conservation laws or models, known characteristics of the assumed model, data filters, scientific conjecture, experience, or other objective or subjective data sources. The Bayesian approach assigns probabilities to all possible theories and to all possible evidence. Using a logical framework for prior and current information, the Bayesians infer a probabilistic answer to a well-posed question, using all of the information at one's disposal. And when one acquires new evidence, the Bayesians update their "priors" in the equation, resulting in a modified probabilistic answer that essentially reduces one's hypothesis space. Probability to the Bayesians represents a state of knowledge, conditional to some context.
The Bayesian probabilistic ideas have been around since the 1700s. Bernoulli, in 1713, recognized the distinction between two definitions of probability: (1) probability as a measure of the plausibility of an event with incomplete knowlege, and (2) probability as the long-run frequency of occurrence of an event in a sequence of repeated (sometimes hypothetical) experiments. The former (1) is a general definition of probability adopted by the Bayesians. The latter (2) is called the "frequentist" view, sometimes called the "classical", "orthodox" or "sampling theory" view.
Astronomers who rely on frequentist definitions, while assigning their uncertainties for their measurements, should be wary. The concept of sampling theory, or the statistical ensemble, in astronomy is often not relevant. For example, a gamma-ray burst is a unique event, observed once, and the astronomer needs to know what uncertainty to place on the one data set he/she actually has, not on thousands of other hypothetical gamma-ray burst events. And similarly, the astronomer who needs to assign uncertainty to the large-scale structure of the Universe needs to assign uncertainties based on _our_ particular Universe, because there are not similar Observations in each of the "thousands of universes like our own."
>From my readings so far, I have found Bayesian Probability Theory to
be a consistent, logical, elegant, probabilistic framework with
which to approach and calculate answers to scientific problems. I
believe that most scientists would find that they can better
formulate the solution of their scientific problems after being
introduced to Bayesian methods, and they would derive a more
methodical (and realistic) uncertainty to their results.
Bayesian Statistics Books and Papers
This section I list several sites that I have found extremely helpful for providing Bayesian statistics books and papers on the Internet.
Tom Loredo is an astronomer at Cornell, who is using Bayesian methods in his astronomy work. His persuasive papers can be found at:
http://astrosun.tn.cornell.edu/staff/loredo/bayes/tjl.html
where one can download gzipped, postscript versions. I think any astronomer seeking to begin to learn about Bayesian methods can do no better than starting with his seminal "From Laplace to Supernova SN 1987A" 60 page article, and then working through more of his papers, for example: "The Promise of Bayesian Inference for Astrophysics" and then "The Return of the Prodigal: Bayesian Inference in Astrophysics".
Edwin Jaynes was one of the founders of modern-day Bayesian ideas who died earlier last year. He had written 2/3 of a book before he died: _Probability Theory as Extended Logic_, that one of his former students: Larry Bretthorst is making available on the Internet, available at:
This site would appeal to any scientist, not only astronomers, because of the breadth and scope of Jaynes' ideas. Bretthorst's Washington University Web site also provides his own important book: _Bayesian Spectrum Analysis and Parameter Estimation_, as well as several other articles by himself, Jaynes, and other Bayesians.
Giulio D'Agnostini is a statistician from Rome, who is teaching statistics to high energy physicists at CERN. Last summer he completed teaching a course about Bayesian statistics, and you can find his 200 page book of detailed lecture material at a CERN site:
http://www.cern.ch/Training/ACAD/reglec_E.html
(Note: Scroll down the page to
"Bayesian Reasoning in High-Energy Physics - Principles and
Applications" by G. D'Agostini, INFN, Italy on 25, 26, 27, 28 & 29
May 1998.)
To read his lecture notes, you will need to download each gzipped, postscript part: 0) Introduction, 1) Part 1, 2) Part 2, 2) Part 3, 3) Bibliography. D'Agnostini has a lively way of presenting his material (if you have an opportunity to see his lectures in person, don't miss it) and his material is similarly humorous and interesting.
William Press, of the Harvard-Smithsonian Center for Astrophysics, is one of the authors of the influential work: _Numerical Recipes_. I was not aware of Press' interest in Bayesian methods until I saw a reference in one of Tom Loredo's papers (above). Press has a very interesting article: "Understanding Data Better with Bayesian and Global Statistical Methods" that I located on the Los Alamos National Laboratory's astro-ph server:
http://xxx.lanl.gov/list/astro-ph/9604?100
(and then scroll down to article: 9604126). This paper is a 14 page postscript paper that includes embedded figures, given at the Unsolved Problems in Astrophysics Conference, Princeton, April 1995.
And for some code:
http://www.astro.psu.edu/statcodes/
StatCodes is a metasite of over 200 links of on-line statistical software (in Fortran, C, executable binaries, others) for astronomy and related fields from the astrostatistics research group at Penn State, who also operate the Statistical Consulting Center for Astronomy. This site lists links to free source code, available on the Internet, from small, as well as large (for example, from StatLib at Carnegie Mellon University) software libraries.
This site is nicely organized in terms of topic from general statistics packages and information to Bayesian statistics to time series analysis to density estimation and smoothing to correlation and regression to visualization software and interactive Web software. You can search the StatCodes site by keywords, as well.
And some Non-Internet reference books:
Sivia, D.S. _Data Analysis: A Bayesian Tutorial_, Clarendon Press: Oxford, 1996.
Martz, Harry and Waller, Ray, chapter: "Bayesian Methods" in _Statistical Methods for Physical Science_, Editors: John L. Stanford and Stephen Vardeman [Volume 28 of the Methods of Experimental Physics], Academic Press, 1994, pg. 403-432.
Amara Graps
Amara Graps | Max-Planck-Institut fuer Kernphysik Interplanetary Dust Group | Saupfercheckweg 1 +49-6221-516-543 | 69117 Heidelberg, GERMANYAmara.Graps@mpi-hd.mpg.de * http://galileo.mpi-hd.mpg.de/~graps
"Never fight an inanimate object." - P. J. O'Rourke