The Master Algorithm, Pedro Domingos, 2015 - This is the Lord of the Rings of Machine Learning books. There is more reason than one for me to make such a tall claim. Like Tolkienâs Middle-Earth with its Elves, Dwarves, Orcs, Trolls, Men, Domingos covers the multiple-facets of AI/ML i.e Symbolists, Connectionists, Evolutionaries, Bayesians and Analogizers in reasonable depth and goes on to propose the one ring that will rule them all, the Master Algorithm. Alright will stop with the LOTR stuff here.
Symbolists - Humeâs problem with induction is the conundrum of generalizing from what weâve seen to what we havenât. Symbolists manage this via Inverse deduction with Logic as the formal language (driven by Decision Trees, Random Forests). Conjunctive and disjunctive concepts, ill-posed problems, no-free lunch theorem, overfitting (leaning towards non-recurring patterns - say deeper decision trees with too many leaf-nodes) and underfitting (missing out on patterns that exist - broader categories and shallow trees) are all explained very succinctly. Issue with Symbolist approach is that its driven too much by formal logic and by what we can âSeeâ, missing the submerged part of the iceberg. Decision Trees and Random Forests are used in a lot of classifiers (say spam) and in forecasting.
Connectionists - This is modeling learning after how the brain does it (Hebbian learning) via densely inter-connected Neurons with its Dendrites and Axons. Neural Networks are the way to approach the problem - with transistors catching up with neurons in the brain, NNs are now lot more feasible than ever before (they have been around for long) and Gradient-descent is how the model is optimized. Arguably this is also the hardest to grasp in-terms of internals for the human brain as it involves thinking in higher than 3 dimensions which our brain struggles with. Perceptrons, the significance of the Sigmoid curve, climbing mountains in hyperspace (for gradient descent and optimizing error), backprop, hopfield networks, sparse autoencoders are explained in a reasonably lay language. Understanding a simple thing like a dotproduct of a matrix doesnât come easily to us due to limitations of the human brain, so a lot of what goes under the hood in a NN will forever remain outside of our grasp unlike a Decision Tree.
Evolutionists - Genetic Algorithms are modeled after Evolution and just like Evolution are capable of leading to truly ânewâ outcomes. Exploration-Exploitation dilemma - Do you repeat what you are good at to maximise benefit or do you go exploring to gather more information? This is one thing Evolutionists are very good at as Connectionist algorithms can get stuck at local-maximas while Evolutionist algos can go exploring - sort of a Nature vs Nurture controversy (Both Connectionist and Evolutionsts are inspired by nature)
Bayesians - By far my favorite part of the book, not least because of my love for Bayes theorem where probability is a degree of prior beliefs only meant to be updated with new evidence for posterior beliefs. Naive Bayes classifier is one of my earliest exposures to ML and it still baffles me with its elegance and sophistication. Markov chains and HMMs or Hidden-Markov models are at the heart of some of the everyday tech we use, from Appleâs Siri to Googleâs PageRank and also in computational biology, economics and defence. There is a lot of overlap between how NNs go about their job and how conditional probability works (backprop is like updating posterior beliefs) - its just a different way of approaching the same problem.
Anologizers - Nearest Neighbor algorithms, Support-Vector Machines (SVMs) and Analogical reasoning are used often in predicting tastes in movies and music, image recognition, buying patterns etc. and other recommendation system used by Netflix and Amazon.
I was pretty much lost towards the end of unsupervised learning and the Master Algorithm parts so I have nothing much to summarize there. If your interest in piqued, do try the book. I donât think any other field attracts the sort of talent as ML does and some of the best minds here are well-versed in multiple disciplines from biology, philosophy, physics, politics, comp-sci, math, law, language, logic and so on.
This book is one of the best things I have read this year. 11/10