Reinforcement Learning, second edition: An Introduction (Adaptive Computation and Machine Learning series) 2nd Edition

I read the entire book cover to cover, doing every (non-programming) exercise, as part of a PhD involving RL. I was also familiar with the first edition (1998).The 2nd edition (2018) has been entirely reworked; it is much longer, the structure has changed, the notation has changed, many new topics are discussed. Same as the first edition, the second edition is both a reference book and a pedagogic manual. The main differences are:(1) Part I, which deals with the fundamentals of RL in a simplified setting, is carefully worded in order to convey understanding in the same accessible, intuitive manner as in the first edition, but also paying closer attention to mathematical rigor and systematicity.(2) Part II, which deals with RL with function approximation, is at times somewhat involved mathematically compared to the first edition, reflecting the general evolution of the field of Machine Learning. The most difficult sections are clearly marked as such, and can be skipped. But taking on the difficulties straight on is always (positively) rewarding, in the end.(3) Part III deals with related developments in psychology and neuroscience, which are a welcome addition to a field that has branched across disciplines. This should be useful to neuroscientists and computational neuroscientists, as well as to experimental/cognitive psychologists.Many exercises are included. I am no math buff and found that I could do every single one of them on my own, which is unfortunately not usually the case in other A.I. books. These exercises serve to help the reader understand key issues by working them out for themselves, in a guided manner; they make self-study possible and enjoyable.This is an indispensable book for anybody working in RL. My only issue is that I'm too late to be referenced in it, but I haven't lost hope of making it in the 3rd edition, which is expected in 2038.

Undertandable by anyone who knows high-school math and has some introduction to mathematical rigor (at the level of compelting a first-course in proofs). For a couple sections, some introduction to probability theory (Sigma algebras and martingales) can provide additional insight (if you want to prove some theorem's about alternate formulations of the Bellman Equations (action-values, ...)).The exercises are great and the amount of code to write is generally pretty small. By doing them I gained some new skills in setting up agents in some practical environments.

I was lucky enough to purchase this copy when it was first available on amazon and the print quality is definitely good, as opposed to the recent disappointing reviews. Perhaps buying directly from MIT Press is the surest way to go.With respect to the content: I really like how exercises and examples are strategically dispersed. In order to have a water-tight hold on the material, it is definitely recommended to code some of these on your own (e.g. the k-armed bandit test bed, MC, etc.) or refer to the Coursera Univ. of Albert RL course's programming assignments (that course uses this book).Note that applicability of RL to real-world problems *outside of finite-board games* is barely covered and only mentioned until Ch 16. However, this book does exceptionally well at meeting its goal: elucidating the fundamentals of modern RL.

I purchased this for a grad school class and it was exactly what we needed. In-depth but explained well. There's a reason this is the gold standard for reinforcement learning texts.

I like the book. It explains things adequately. The representations take a little bit to digest. I think that the field is more difficult than it needs to be. This is mostly because math is one of the most convoluted areas in education. IF we scrapped all of that crap about the names, and worked with names that had logical connections, then the learning would be greatly enhanced. As it is, when you want to learn the fundamentals of math, you are required to learn math history. We teach about some people as if they are extraordinary - as if no one else is working on the same things. But the first person to cross the finish line shall be immortalized like a god. This book is full of those references, but that is at least what I was looking for. Since that is how the world works, it makes some of the communications easier if we understand this convolution. I mean, a Kalman filter is really only a control system feedback loop of a Bayes equation. If you don't know about those people, then this book is either ripe for the reading, or a little bit out of reach. I am by no means claiming talent at taking these abstract ideas and presenting them in an easily consumable fashion. I would say that a class on Abstract ABC's so to speak would allow us to speak in a more abstract fashion and approach math form its constituent parts when building an algorithm. If anything, the symbolic crafting in this book (and the changes detailing how the symbolic representations have changed) is priceless. It details how refactoring can make things more understandable. Unfortunately, it is only done as a means of representing the author's work (which is the intent). I long for the author that is willing to rewrite the formulas, perhaps using artificial intelligence to compile optimal absorption. Until then, anyone learning more about math history than math will never be a leader in math. At some point, there are enough names to overload a lifetime of learning. This necessitates a simpler approach equivalent to the building blocks of atoms. Then, you learn how the relationship of the pieces affects the whole. I understand that mathematicians who are publishing new materials must arrive there, I just don't think that the process of getting there needs to be so difficult. Keep math history as a class called math history. Let math be just that - math.I included the rant as a reason why I liked this book. Revising the notation leaves me with hope that we will develop better means of conveying the pieces comprising the models that allow us to expand our imaginations on what is possible. It could be written with much more clarity and readability, but I think that will have to wait until we develop better semantics of mathematical expression. Cheers to all, comments welcome!