Q-Learning and Enhanced Policy Iteration in Discounted Dynamic Programming

Show simple item record

dc.contributor.author Bertsekas, Dimitri P.
dc.contributor.author Yu, Huizhen
dc.date.accessioned 2010-06-15T14:07:45Z
dc.date.available 2010-06-15T14:07:45Z
dc.date.issued 2010-06-15T14:07:45Z
dc.identifier.uri http://hdl.handle.net/10138/17117
dc.description.abstract We consider the classical nite-state discounted Markovian decision problem, and we introduce a new policy iteration-like algorithm for fi nding the optimal Q-factors. Instead of policy evaluation by solving a linear system of equations, our algorithm requires (possibly inexact) solution of a nonlinear system of equations, involving estimates of state costs as well as Q-factors. This is Bellman's equation for an optimal stopping problem that can be solved with simple Q-learning iterations, in the case where a lookup table representation is used; it can also be solved with the Q-learning algorithm of Tsitsiklis and Van Roy [TsV99], in the case where feature-based Q-factor approximations are used. In exact/lookup table representation form, our algorithm admits asynchronous and stochastic iterative implementations, in the spirit of asynchronous/modi ed policy iteration, with lower overhead and more reliable convergence advantages over existing Q-learning schemes. Furthermore, for large-scale problems, where linear basis function approximations and simulation-based temporal di erence implementations are used, our algorithm resolves e ffectively the inherent difficulties of existing schemes due to inadequate exploration. en
dc.language.iso en en
dc.relation.ispartofseries Report LIDS - 2831 en
dc.relation.ispartofseries Also as: Department of Computer Science Series of Publications C Report C-2010-10 en
dc.title Q-Learning and Enhanced Policy Iteration in Discounted Dynamic Programming en
dc.type Technical report en
dc.identifier.laitoskoodi Department of Computer Science en

Files in this item

Total number of downloads: Loading...

Files Size Format View
Enhanced_Policy_Iteration_BY.pdf 610.3Kb PDF View/Open

This item appears in the following Collection(s)

Show simple item record