融合模仿学习的双足机器人全向行走步态生成方法

doi:10.16180/j.cnki.issn1007-7820.2025.01.005

电子科技 ›› 2025, Vol. 38 ›› Issue (1): 29-36.doi: 10.16180/j.cnki.issn1007-7820.2025.01.005

融合模仿学习的双足机器人全向行走步态生成方法

冯振¹^,², 牟海明¹^,², 薛杰¹^,², 李清都¹()

1.上海理工大学机器智能研究院,上海 200093
2.上海理工大学光电信息与计算机工程学院,上海 200093

收稿日期:2023-05-18 修回日期:2023-06-14 出版日期:2025-01-15 发布日期:2025-01-06
通讯作者: 李清都(1980-),男, E-mail:liqd@usst.edu.cn,博士,教授。研究方向:仿生机器人理论与技术、复杂系统的动力学与控制等。
作者简介:冯振(1997-),男,硕士研究生。研究方向:强化学习、机器人运动控制等。
牟海明(1989-),男,博士。研究方向:机器人运动控制、强化学习等。
基金资助:
东方学者计划(TP2019064)

Omnidirectional Gait Generation Method for Biped Robot with Fusion of Imitation Learning

FENG Zhen¹^,², MOU Haiming¹^,², XUE Jie¹^,², LI Qingdu¹()

1. Institute of Machine Intelligence,University of Shanghai for Science and Technology,Shanghai 200093,China
2. School of Optical-Electrical and Computer Engineering,University of Shanghai for Science and Technology,Shanghai 200093,China

Received:2023-05-18 Revised:2023-06-14 Online:2025-01-15 Published:2025-01-06
Supported by:
Oriental Scholars Program(TP2019064)

摘要/Abstract

摘要：

由于双足机器人具有复杂的高维动力学和高度动态特性,因此难以实现全向步态。为了实现双足机器人全向行走,文中基于深度强化学习算法提出了一种融合模仿学习的双足机器人步态训练方法。根据专家经验和双足行走具有周期性的特点,设计了可以实现不同步态风格的周期对称函数进行模仿学习。为了使双足机器人具有全向行走能力,使用ROS(Robot Operating System)中的脚步规划器生成目标落脚点进行模仿学习,并在自主设计的双足机器人上对所提方法进行验证。实验结果表明,所提方法可实现双足机器人前向、侧向、对角线和转向4种步态模式,实现了双足机器人的全向步态,并且可实现不同风格的周期对称步态。

关键词: 强化学习, 模仿学习, 专家经验, 周期步态, 全向运动, 转向运动, 对角线运动, 双足机器人

Abstract:

Due to the complex high-dimensional dynamics and highly dynamic characteristics of bipedal robots, achieving omnidirectional gait is a difficult problem. In order to achieve omnidirectional walking of bipedal robots, this study proposes a gait training method of biped robot based on deep reinforcement learning. Based on expert experience and the periodicity of bipedal walking, periodic symmetric functions that can achieve different gait styles are designed for imitation learning. In order to make the bipedal robot capable of omnidirectional walking, the footstep planner in ROS (Robot Operating System) is used to generate target foothold points for imitation learning. The proposed method is validated on a self-designed bipedal robot. The experimental results show that the proposed method can realize four gait modes of biped robot including forward, side, diagonal and turn, and realize omnidirectional gait of biped robot, and can realize different styles of cycles.

Key words: reinforcement learning, imitation learning, expert experience, periodic gait, omnidirectional movement, steering movement, diagonal movement, bipedal robot

中图分类号:

TP18

冯振, 牟海明, 薛杰, 李清都. 融合模仿学习的双足机器人全向行走步态生成方法[J]. 电子科技, 2025, 38(1): 29-36.

FENG Zhen, MOU Haiming, XUE Jie, LI Qingdu. Omnidirectional Gait Generation Method for Biped Robot with Fusion of Imitation Learning[J]. Electronic Science and Technology, 2025, 38(1): 29-36.

图/表 10

图1

表1

图2

图3

表2

图4

图5

表3

图6

图7

参考文献 20

[1]	黄艳龙, 徐德, 谭民. 机器人运动轨迹的模仿学习综述[J]. 自动化学报, 2022, 48(2):315-334.
	Huang Yanlong, Xu De, Tan Min. On imitation learning of robot movement trajectories:A survey[J]. ACTA Automatica Sinica, 2022, 48(2):315-334.
[2]	Xing B, Liu Y, Wang Z, et al. Dynamic walking locomotion framework based on impulse force design and ZMP principle[C]. Hefei: Chinese Control and Decision Conference, 2022:1995-2000.
[3]	Song H, Peng W Z, Kim J H. Partition-aware stability control for humanoid robot push recovery with whole-body capturability[J]. Journal of Mechanisms and Robotics, 2023, 16(1):11005-11012.
[4]	Kuindersma S, Permenter F, Tedrake R. An efficiently solvable quadratic program for stabilizing dynamic locomotion[C]. Hong Kong: IEEE International Conference on Robotics and Automation, 2014:2589-2594.
[5]	Dai H, Valenzuela A, Tedrake R. Whole-body motion planning with centroidal dynamics and full kinematics[C]. Madrid: IEEE-RAS International Conference on Humanoid Robots, 2014:295-302.
[6]	张佳鹏, 李琳, 朱叶. 基于强化学习的无人驾驶车辆行为决策方法研究进展[J]. 电子科技, 2021, 34(5):66-71.
	Zhang Jiapeng, Li Lin, Zhu Ye. A review of research on decision-making methods of autonomous vehicle based on reinforcement learning[J]. Electronic Science and Technology, 2021, 34(5):66-71.
[7]	董豪, 杨静, 李少波, 等. 基于深度强化学习的机器人运动控制研究进展[J]. 控制与决策, 2022, 37(2):278-292.
	Dong Hao, Yang Jing, Li Shaobo, et al. Research progress of robot motion control based on deep reinforcement learning[J]. Control and Decision, 2022, 37(2):278-292.
[8]	Peng X B, Berseth G, Yin K, et al. Deeploco:Dynamic locomotion skills using hierarchical deep reinforcement learning[J]. ACM Transactions on Graphics, 2017, 36(4):1-13.
[9]	Liu C, Lonsberry A G, Nandor M J, et al. Implementation of deep deterministic policy gradients for controlling dynamic bipedal walking[J]. Biomimetics, 2019, 4(1):28-30.
[10]	Li Z, Cheng X, Peng X B, et al. Reinforcement learning for robust parameterized locomotion control of bipedal robots[C]. Xi'an: IEEE International Conference on Robotics and Automation, 2021:2811-2817.
[11]	Siekmann J, Godse Y, Fern A, et al. Sim-to-real learning of all common bipedal gaits via periodic reward composition[C]. Xi'an: IEEE International Conference on Robotics and Automation, 2021:7309-7315.
[12]	Rodriguez D, Behnke S. DeepWalk:Omnidirectional bipedal gait by deep reinforcement learning[C]. Xi'an: IEEE International Conference on Robotics and Automation, 2021:3033-3039.
[13]	Peng X B, Coumans E, Zhang T, et al. Learning agile robotic locomotion skills by imitating animals[J]. Science and Systems, 2020, 47(10):117-120.
[14]	Ratliff N, Bagnell J A, Srinivasa S S. Imitation learning for locomotion and manipulation[C]. Pittsburgh: IEEE-RAS International Conference on Humanoid Robots, 2007:392-397.
[15]	Ross S, Gordon G J, Bagnell J A. A reduction of imitation learning and structured prediction to no-regret online learning[C]. Fort Lauderdale: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 2011:627-635.
[16]	Ho J, Ermon S. Generative adversarial imitation learning[J]. Advances in Neural Information Processing Systems, 2016, 29(9):4565-4573.
[17]	Xie Z, Berseth G, Clary P, et al. Feedback control for cassie with deep reinforcement learning[C]. Madrid: IEEE/RSJ International Conference on Intelligent Robots and Systems, 2018:1241-1246.
[18]	Peng X B, Abbeel P, Levine S, et al. Deepmimic:Example-guided deep reinforcement learning of physics-based character skills[J]. ACM Transactions on Graphics, 2018, 37(4):1-14.
[19]	Wu Q, Zhang C, Liu Y. Custom sine waves are enough for imitation learning of bipedal gaits with different styles[C]. Nanjing: IEEE International Conference on Mechatronics and Automation, 2022:499-505.
[20]	战忠丽, 王强, 陈显亭. 强化学习的模型、算法及应用[J]. 电子科技, 2011, 24(1):47-49.
	Zhan Zhongli, Wang Qiang, Chen Xianting. Rinforcement learning models,algorithms and its application[J]. Electronic Science and Technology, 2011, 24(1):47-49.

参数	符号	数值	单位
小腿长度	l_s	25	cm
大腿长度	l_l	25	cm
小腿质量	m_s	1	kg
大腿质量	m_l	2	kg
髋部质心高度	h_h	69	cm
整体质量	m	10	kg
机器人高度	h	130	cm

参数	数值
批大小	36 728(1 024×32)
最小批	8 192(1 024×8)
学习率	0.000 35
衰减因子	0.95
GAE折扣率	0.99
回合数	4.00

融合模仿学习的双足机器人全向行走步态生成方法

Omnidirectional Gait Generation Method for Biped Robot with Fusion of Imitation Learning

RichHTML

PDF (PC)

赞

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 20

相关文章 8

Metrics

本文评价

推荐阅读 0

方法	时间/s	前向	侧向	后向
本文方法	15	87	119	30
本文方法	30	77	89	36
文献[20]	15	76	106	33
文献[20]	30	71	71	36

[1]	方帅, 陈旭, 李康吉. Q学习差分进化算法求解热电动态经济排放调度[J]. 电子科技, 2024, 37(5): 9-17.
[2]	唐超, 张帆. 基于改进SAC算法的机械臂运动规划[J]. 电子科技, 2024, 37(11): 47-54.
[3]	肖雄, 刘鸿雁, 衣孟杰. 基于深度强化学习的异构融合网络接入算法[J]. 电子科技, 2024, 37(11): 7-12.
[4]	索芳菲,季云峰. 乒乓球机器人核心算法研究综述[J]. 电子科技, 2023, 36(12): 16-24.
[5]	张晔,鲍亮. 基于机器学习的服务组合技术研究进展[J]. 电子科技, 2022, 35(11): 58-63.
[6]	张佳鹏,李琳,朱叶. 基于强化学习的无人驾驶车辆行为决策方法研究进展[J]. 电子科技, 2021, 34(5): 66-71.
[7]	王泽学,万启东,秦杨梅,樊森清,肖泽仪. 基于多Agent强化学习的危险车辆预警算法[J]. 电子科技, 2020, 33(9): 44-49.
[8]	战忠丽, 王强, 陈显亭. 强化学习的模型、算法及应用[J]. , 2011, 24(1): 47-.