基于HOCBF安全强化学习的无人水面艇自主避障方法研究

HOCBF-Based Safe Reinforcement Learning Method for Autonomous Obstacle Avoidance of Unmanned Surface Vehicles

  • 摘要:目的】面向复杂海况下无人水面艇(Unmanned Surface Vehicle,USV)自主避障任务,构建基于控制障碍函数的安全强化学习框架,并针对安全层过度干预导致的策略失配、学习效率下降和任务性能退化等问题,研究面向可学习性的优化方法。【方法】基于Soft Actor-Critic(SAC)框架,建立USV三自由度动力学模型和避障任务马尔可夫决策过程,在执行层引入高阶控制障碍函数(Higher-Order Control Barrier Function,HOCBF)的二阶形式作为安全滤波器,构建HOCBF安全强化学习避障框架。在此基础上,设计兼顾安全约束与策略学习的安全层优化机制,形成LA-SafeSAC(Learnability-Aware Safe Soft Actor-Critic)方法,并在平静水域、波浪扰动水域和波浪-洋流耦合水域三类场景中与基线方法进行对比验证。【结果】仿真结果表明,在基于 HOCBF 的安全强化学习框架中引入可学习性优化后,LA-SafeSAC在3类场景中均优于标准SAC+CBF,并在复杂扰动场景下保持了与SAC基线相当的稳定阶段任务完成能力。与标准SAC+CBF相比,其训练末100回合成功率由38.7%~62.0%提升至79.7%~84.3%,超时率由37.7%~54.3%降至15.7%~19.3%,CBF激活率由35.9%~40.2%降至20.2%~24.1%。训练曲线与安全-学习协同性分析进一步表明,该方法能够在较低碰撞风险下更快形成有效策略,减弱策略输出与安全执行动作之间的长期失配,并使安全层逐步由持续接管转变为低频纠偏。【结论】研究表明,在基于HOCBF的安全强化学习框架中引入面向可学习性的安全层优化,能够在保持形式化安全约束的同时缓解长期保守退化,提升任务完成能力与训练稳定性,并促进安全层与策略学习之间的协同演化。

     

    Abstract: ObjectivesThis study presents a learnability-aware safe reinforcement learning method for unmanned surface vehicle (USV) obstacle avoidance under complex sea conditions. The method is intended to alleviate policy mismatch, reduced learning efficiency, and performance degradation caused by excessive safety-layer intervention.MethodsBuilt upon the Soft Actor-Critic (SAC) framework, a three-degree-of-freedom USV dynamic model and a Markov decision process for obstacle avoidance were formulated. A second-order HOCBF (Higher-Order Control Barrier Function) filter was introduced at the execution layer to enforce formal safety constraints, yielding an HOCBF-based safe reinforcement learning framework. On this basis, a learnability-aware optimization mechanism that jointly accounts for safety enforcement and policy learning was developed, resulting in LA-SafeSAC (Learnability-Aware Safe Soft Actor-Critic). Comparative experiments were conducted in calm-water, wave-disturbed, and wave-current-coupled scenarios. ResultsLA-SafeSAC outperformed standard SAC+CBF in all three scenarios while maintaining stable-stage task completion performance comparable to that of the SAC baseline under complex disturbances. Relative to standard SAC+CBF, over the final 100 episodes, the success rate increased from 38.7%-62.0% to 79.7%-84.3%, the timeout rate decreased from 37.7%-54.3% to 15.7%-19.3%, and the CBF activation rate decreased from 35.9%-40.2% to 20.2%-24.1%. Training curves and safety-learning coordination metrics further indicate that the proposed method facilitates faster formation of effective policies under low collision risk, mitigates the long-term mismatch between nominal actions and safety-filtered actions, and progressively shifts the safety layer from persistent takeover to low-frequency corrective intervention.ConclusionsThe proposed learnability-aware safety-layer optimization alleviates long-term conservative degradation, improves task completion capability and training stability, and enhances coordination between safety enforcement and policy learning while preserving formal safety guarantees.

     

/

返回文章
返回