Saturday, December 25, 2021

 

The Book of Why: The New Science of Cause and EffectThe Book of Why: The New Science of Cause and Effect by Judea Pearl
My rating: 3 of 5 stars

My notes are for the sake of future research that I may do. Statistics is field of endeavor I have avoiding out of my preference for axioms and theorems of pure mathematics. I gained most of my statistical experience from college physics experiments, the classic "least squares fit" that economists call "regression analysis."

This book is fascinatng. When I first read about Bayesian statistics about 1981, it was part of our fire control software, I judged it to be nonsense. Now I understand better. Judea Pearl does not mention Kalman filters, but they were critical to our success in the Navy.

His annotated bibliography is excellent. It is organized by corresponding chapters. I thought this adds to the difficulty of looking up specific items, but it is still manageable. An appendix of terms, acronyms and definitions would have been useful. The index is OK to assist with these kinds of questions.

His case studies included examples from genetics, cholera, smoking, college admissions, nurture vs nature, cause of scurvy, "algebra for all" education policy, use of tourniquet on the battle field, and others were all fascinating. Some of them were scary because they showed how good people could get things terribly wrong.

I didn't take the time and extra research to learn his methods but I found a lot of good information in his book:

Ladder of causation page 28.
Level one is observations e.g. data collection, this is classics statistics. What are the variables, how are they related?
Level two is intervention, what can we do to alter the observations? e.g. Would aspirin help my headache?
Level three is counter-factual, what if different choices or circumstances had existed before we made our observations? e.g. did X really cause Y? what if X had not been present?

Casual diagrams blew my mind! The power of such a simple tool to organize ideas, discover statistical bias, and understand their possible relationships. He clarifies from the findings of the courts the difference between discrimination and bias. Discrimination is an act of will, bias is something that happens and needs to be discovered.

Different variables in a casual diagram:
Confounding, deconfounding,

chain junction, fork junction, collider junction

The puzzle of Simpson's paradox where aggregated results contradict results from partitioned subsets. The lesson is that aggregating or partitioning data needs to be done with an awareness of the process that generated the data. {Personally I have seen too many software engineers and data people dive right in with their aggregating and partitioning tools without questioning the suitability of their tool. A good example is HADOOP and MapReduce.}

"Back-door adjustments" are not easy for me to describe, but vital to deciding interventions. Apparently the back-door adjustments is a means of eliminating confounding variables, i.e. "all else being held constant ...." (another phrase economists seem to like to use).

Moderating factors, interactions between possible causes before the effect is observed.



View all my reviews

1 comment:

Frederick Nissen Vogt said...

This work would support the idea of decentralized execution of the commander's intent. Explain this.
It also illustrates why individual can the game the system, if they now how management is trying to measure performance. Explain this.