Reinforcement-Learning-Based Autonomous Vehicle Navigation in a Dynamically Changing Environment

This dissertation, "Reinforcement-learning-based Autonomous Vehicle Navigation in a Dynamically Changing Environment" by Chi-kit, Ngai, 魏智傑, was obtained from The University of Hong Kong (Pokfulam, Hong Kong) and is being sold pursuant to Creative Commons: Attribution 3.0 Hong Kong License. The content of this dissertation has not been altered in any way. We have altered the formatting in order to facilitate the ease of printing and reading of the dissertation. All rights not granted by the above license are retained by the author. Abstract: Abstract of thesis entitled Reinforcement-Learning-Based Autonomous Vehicle Navigation in a Dynamically Changing Environment submitted by NGAI Chi Kit for the Degree of Doctor of Philosophy at The University of Hong Kong in November 2007 This thesis proposes a novel methodology for resolving autonomous vehicle navigation (AVN) using a multiple goal reinforcement learning framework. It is motivated by the fact that the real world environment is dynamically changing such that it is affected by other agents in the environment and is therefore difficult to model. The complexity of the navigation problem also increases if other functional tasks are involved. In this thesis, a methodology is developed upon the philosophy that the navigation problem should be considered as an intelligent decision making process that deals with high level strategies with the use of planning, learning, prediction, and the consideration of environment response, rather than a control issue. In essence, we propose a multiple goal reinforcement learning methodology that first divides the navigation task into sub-tasks such that each task is associated with a decision making method in order to achieve its goals. Each goal is achieved by using the reinforcement learning approach. Specifically, the Double Action Q-learning (DAQL) method is proposed to work in a dynamically changing environment by viewing the real world as an environment that also contains other multiple decision- making agents and by considering the decisions made by these agents in the learning and decision making process. An independent prediction model can be utilized in the proposed framework to further enhance the forward looking ability of the agent. To avoid inconsistency when combining different goals, a goal fusion function is employed to combine different goals together with predefined logics. The proposed DAQL method has been evaluated and compared with the traditional Q-learning method. The sum of negative rewards is found to be 48% less than that of Q-learning in a navigation task that contains 100 obstacles. Moreover, a two-goal navigation method and a multiple-goal navigation method for overtaking are proposed to demonstrate the effectiveness of the proposed multiple-goal reinforcement learning methodology. A comparison of the two-goal navigation method with the Artificial Potential Field method indicates that the proposed method improves path time and the number of collision-free episodes by 20.6% and 23.6% on average, and 27.8% and 115.6% at best, respectively. The simulation cases on overtaking demonstrated that the proposed method is able (1) to make correct action decisions for overtaking; (2) to avoid collision with other vehicles; (3) to reach the target within reasonable time; (4) to maintain an almost steady speed; and (5) to maintain an almost steady heading angle. It should also be noted that the proposed method performs lane following very well when not overtaking, and lane changing effectively when overtaking is in progress. The results demonstrate that the proposed method offers an effective solution to navigation problems involve multiple numbers of goals and other intelligent agents. It can learn to react to the dynamically changing environment in order to minimize the negative rewards received and thus cause less collision. It also has the p