Model-free adaptive optimal control of spacecraft formation flying reconfiguration using Q-Learning

This paper investigates an optimal adaptive controller based on reinforcement learning while considering orbital perturbations. The controller can achieve mission goals, online without any model. Reconfiguration capabilities provide great flexibility in achieving formation flying mission goals. In reconfiguration, it is desired that spacecrafts migrate from the current formation to a new formation, thus achieving mission goals. Orbital perturbations, difficulties in extracting exact mathematical models, and unknown system dynamics make the optimal reconfiguration problem challenging. Due to the digital nature of spacecraft computer systems, controllers have to be implemented digitally. Accordingly, this paper introduces an adaptive optimal digital controller for a discounted generalized cost function. The stability of the proposed controller is proven by the Lyapunov method. Then, using the Q-learning method, an algorithm is presented so that the controller can find the optimal control gains in a model-free fashion. Finally, numerical simulations of a formation flying mission scenario, confirm the effectiveness of this method.