Tata Institute of Fundamental Research
Batch Asynchronous Stochastic Approximation and Applications to Temporal Difference Learning
STCS Colloquium
Speaker:
M. Vidyasagar (Indian Institute of Technology Hyderabad.)
Organiser:
Sandeep K Juneja
Date:
Wednesday, 11 May 2022, 11:00 to 12:00
Venue:
In person @ A-201 and also via Zoom
(Scan to add to calendar)
Abstract:
(This talk consists of joint work with Prof. Rajeeva L. Karandikar) Reinforcement Learning (RL) algorithms such as Temporal Difference (TDL) or Q-learning update *just one component* of the value function (TDL) or the action-value function (Q) at each time step. This is known as asynchronous stochastic approximation. There are two issues with this. First, many of the "convergence proofs" in the literature are not always correct. Second, when the dimension of the state space is very high, learning requires a huge number of time steps. In effect, spatial complexity is replaced by temporal complexity. A compromise is to update *some but not all* components of the value, or the action-value, function at each time step. This may be called Batch Asynchronous Stochastic Approximation (BASA). In this talk, I will present a very general framework for proving the convergence of BASA, which includes both TD learning and Q-learning as special cases, and also leads to new algorithms with lower complexity.
Zoom Link - https://zoom.us/j/94084902566?pwd=KzNpb3NDQnorTWcvUlJvNXA3eHlWUT09
| Speaker: | M. Vidyasagar (Indian Institute of Technology Hyderabad.) |
| Organiser: | Sandeep K Juneja |
| Date: | Wednesday, 11 May 2022, 11:00 to 12:00 |
| Venue: | In person @ A-201 and also via Zoom |
(Scan to add to calendar)
Abstract:
(This talk consists of joint work with Prof. Rajeeva L. Karandikar) Reinforcement Learning (RL) algorithms such as Temporal Difference (TDL) or Q-learning update *just one component* of the value function (TDL) or the action-value function (Q) at each time step. This is known as asynchronous stochastic approximation. There are two issues with this. First, many of the "convergence proofs" in the literature are not always correct. Second, when the dimension of the state space is very high, learning requires a huge number of time steps. In effect, spatial complexity is replaced by temporal complexity. A compromise is to update *some but not all* components of the value, or the action-value, function at each time step. This may be called Batch Asynchronous Stochastic Approximation (BASA). In this talk, I will present a very general framework for proving the convergence of BASA, which includes both TD learning and Q-learning as special cases, and also leads to new algorithms with lower complexity.
Zoom Link - https://zoom.us/j/94084902566?pwd=KzNpb3NDQnorTWcvUlJvNXA3eHlWUT09
Zoom Link - https://zoom.us/j/94084902566?pwd=KzNpb3NDQnorTWcvUlJvNXA3eHlWUT09