Developing Continuous Reinforcement Learning in Distributed Spatial Problems (Case Study: Adaptive Traffic Control)

The Multi-agent systems has shown their usefulness as an efficient approach for modeling, analyzing as well as implementing complex, dynamic and distributed applications such as robotic teams, distributed control, resource management, traffic control, land use planning, crisis management, forest fire control and to name but a few. The main challenge in multi-agent systems is to find the suitable behavior for each agent that maximize the average utility rate of the whole system. Moreover, it is sometimes necessary that they learn new behaviors online, such that the performance of the whole system gradually improves. Thus, a learning mechanism is necessary so that agents gradually find the global optimal solution on their own. In this context, reinforcement learning as a promising approach for training agents could be useful such that the agent never sees examples of correct behavior but instead receives positive or negative rewards for the actions it tries. Thus, it allows agent to automatically determine the ideal behavior within a specific context, in order to maximize its performance. Each time the agent performs an action in its environment, a trainer may provide a reward to indicate the desirability of the resulting state and the agent tries to learn a control policy which is a mapping from states to actions that maximizes the expected sum of the received rewards. Continuous reinforcement learning algorithms which use generalization, the ability of a system to perform accurately on unseen data, perform properly in real-world problems. In practical point of view, there is a natural metric on the state space such that close states exhibit similar behavior so that the agents are able to deal with states never exactly experienced before and they can learn efficiently by generalizing from previously (similar, close) experienced states. The success of continuous reinforcement learning algorithms on real-world problems hinges on effective function approximator which maps states to values via a parameterized function. Among the many function approximator schemes proposed, tile coding which forms a piecewise-constant approximation of the value function and strikes an empirical balance between representational power and computational cost is applied in this research. The focus of this paper is to combine multi-agent systems with continuous state reinforcement learning by using tile coding. The proposed approached was validated using traffic signal control, in which traffic lights located at intersections can be seen as autonomous agents that learn while interacting with the environment. There are some challenging issues in traffic signal control such as high number of agents, nonstationarity of the multi-agent learning problem, the curse of dimensionality and continuity in state space which makes it as a suitable testbed. The reinforcement learning controller is benchmarked against optimized pretimed control. The results indicate that reinforcement learning agent achieves 21% less stop time compared to optimized pretimed control.