Training a physics-based third-person character controller with residual RL + mocap

I’ve been experimenting with using reinforcement learning to build a self-balancing, physics-driven third-person character controller in Unity.

The character is a humanoid built with ArticulationBody joints and trained with Unity ML-Agents. Instead of asking the policy to learn locomotion entirely from scratch, I use a mocap-driven kinematic character as the reference trajectory.

The physics character continuously tracks the reference pose, while the policy outputs residual joint corrections on top of the mocap targets. The idea is that the animation provides the underlying gait, while RL learns the smaller corrections needed for balance, momentum control, foot placement, and recovery.

The reward currently combines things like joint-pose imitation, end-effector tracking, COM/root velocity, orientation, height, and foot-contact agreement, with additional penalties for excessive actions, jitter, and abrupt action changes.