关键词: 倒立摆, 强化学习, 平滑探索, 突变正则化项, 震荡正则化项, 近端策略优化算法, PPO算法, 虚实迁移

Abstract:

The nonlinear and underactuated nature of the inverted pendulum makes it a benchmark test case for RL(Reinforcement Learning) algorithms. When the simulation-learned RL strategy is deployed to the physical platform, the control signal has mutations and oscillations, which leads to the failure of the strategy deployment, and the problems of high power consumption, excessive system wear and hardware damage. To solve this problem, a regularization term for smoothing exploration of RL strategy is proposed in this study. In order to solve the policy mutation problem in the physical deployment stage, the mutation regularization term is designed to constrain the policy mutation in the exploration stage. Oscillation regularization term is designed to solve the small-range oscillation problem of the strategy, and the value functions of similar states are constrained. The smooth exploration regularization term is applied to the PPO(Proximal Policy Optimization) algorithm to carry out the virtual real transfer experiment of inverted pendulum. The experimental results show that the training speed of PPO algorithm for smooth exploration is increased by 40% in simulation, and the virtual-real transfer is successfully realized, which has strong smoothness and robustness.

Key words: inverted pendulum, reinforcement learning, smooth exploration, mutation regularization term, oscillation regularization term, proximal policy optimization algorithm, PPO algorithm, virtual-to-real migration