Developing a Reinforcement Learning Algorithm to Model Pavlovian Approach Bias on Bidirectional Planning

The decision- making process in the human brain is controlled by two mechanisms: Pavlovian and instrumental learning systems. The Pavlovian system learns the stimulus- outcome association independent of action; a process that manifests itself in the tendency to approach reward- associated stimuli. The instrumental controller, on the other hand, learns the action- outcome association. Instrumental learning is not limited to the current action's outcome and may evaluate a sequence of future actions in the form of forward planning. Nonetheless, forward planning may not be the only planning process used by instrumental learning. Humans may also use backward planning to evaluate actions sequences. However, backward planning has received less attention so far. Previous research has shown that despite the independence of Pavlovian and instrumental learning, they interact with each other such that the Pavlovian approach tendency biases forward planning, causing it to make decisions that may not be optimal actions from the instrumental learning perspective. Nevertheless, the effect of Pavlovian learning on backward planning has not yet been studied.

This paper designs a navigation experiment that allows investigating forward, backward, and bidirectional planning. Moreover, we embed Pavlovian approach cues into the maps to investigate how they bias the three forms of planning.

Statistical analysis of the collected data indicates the existence of backward planning and shows that the Pavlovian- approach cues bias the planning. This bias is stronger in forward planning compared to backward planning and is even stronger in bidirectional planning. In the context of reinforcement learning, we developed a bidirectional planning algorithm under the Pavlovian approach tendency.

The simulation results are consistent with the experimental results and indicate that the effect of Pavlovian bias can be modeled as pruning of decision trees.