Do As You Teach: A Multi-Teacher Approach to Self-Play in Deep Reinforcement Learning

A long-running challenge in the reinforcement learning (RL) community has been to train a goal-conditioned agent in sparse reward environment such that it also generalizes to unseen goals. We propose a novel goal-conditioned RL algorithm; Multi-Teacher Asymmetric Self-Play, which allows 1+ agents (i.e., the teachers) to create a successful curriculum for another agent (i.e., the student) and empirically demonstrate its effectiveness on domains like Fetch-Reach and a novel driving simulator designed for goal-conditioned RL. Surprisingly, results also show that training with multiple teachers actually helps the student learn faster by better covering the state space. Moreover, results show that completely new students can learn offline from the goals generated by teachers trained with a previous student. This is crucial in the context of application domains where repeatedly training a teacher agent is expensive or even infeasible.

AAMAS 2023: Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems