1

Robust Multi-Objective Preference Alignment with Online DPO

We introduce MO-ODPO, an efficient and robust algorithm for aligning large language models with multiple conflicting preferences, allowing flexible steerability at inference.

Ryan Sullivan, Raghav Gupta, Ryan Sullivan, Yunxuan Li, Samrat Phatale, Abhinav Rastogi

Robust Multi-Objective Preference Alignment with Online DPO

Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning

We propose Conditional Language Policy (CLP), a framework for finetuning steerable language models that effectively balance multiple conflicting objectives without maintaining separate models.

Ryan Sullivan, Kaiwen Wang, Rahul Kidambi, Ryan Sullivan, Alekh Agarwal, Christoph Dann, Andrea Michi, Marco Gelmi, Yunxuan Li, Raghav Gupta, Kumar Dubey

Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks

We applied the implementation tricks introduced by DreamerV3 to PPO, and identified cases where they help or harm reward robustness.

Ryan Sullivan, Akarsh Kumar, Shengyi Huang, John P. Dickerson, Joseph Suarez

Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks

Neural MMO 2.0: A Massively Multi-task Addition to Massively Multi-agent Learning

The paper describes Neural MMO 2.0, a massive multitask update for the multiagent NeuralMMO environment.

Joseph Suarez, David Bloomin, Kyoung Whan Choe, Hao Xiang Li, Ryan Sullivan, Nishaanth Kanna Ravichandran, Daniel Scott, Rose S Shuman, Herbie Bradley, Louis Castricato, Phillip Isola, Kirsty You, Yuhao Jiang, Qimai Li, Jiaxin Chen, Xiaolong Zhu

Neural MMO 2.0: A Massively Multi-task Addition to Massively Multi-agent Learning

Gradient Informed Proximal Policy Optimization

We introduce a novel policy learning method that integrates analytical gradients from differentiable environments with PPO.

Sanghyun Son, Laura Yu Zheng, Ryan Sullivan, Yi-Ling Qiao, Ming Lin

Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environments

We developed a new visualization technique for reinforcement learning and used it to demonstrate a failure mode of policy gradient methods.

Ryan Sullivan, J. K. Terry, Benjamin Black, John P. Dickerson

Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environments

PettingZoo: A Standard API for Multi-Agent Reinforcement Learning

This paper introduces the PettingZoo library and the accompanying Agent Environment Cycle (“AEC”) games model.

J. K. Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis Santos, Rodriguez Perez, Caroline Horsch, Clemens Dieffendahl, Niall L. Williams, Yashas Lokesh, Praveen Ravi