Dynamics as Prompts: In-Context Learning for Sim-to-Real System Identifications

Xilun Zhang^*1, Shiqi Liu^*1, Peide Huang¹, William Jongwon Han¹, Yiqi Lyu¹, Mengdi Xu¹, Ding Zhao¹

¹Carnegie Mellon University; ^*Equal Contribution

IEEE Robotics and Automation Letters (RA-L)

CAPTURE aims to take the history information to predict the next step environment parameters. SysID causal transformer adapts the simulation environment parameters to match the real-world performance on the fly via next-token prediction. CAPTURE takes three iterations to identify the correct center of mass of celery.

Abstract

Sim-to-real transfer remains a significant challenge in robotics due to the discrepancies between simulated and real-world dynamics. Traditional methods like Domain Randomization often fail to capture fine-grained dynamics, limiting their effectiveness for precise control tasks. In this work, we propose a novel approach that dynamically adjusts simulation environment parameters online using in-context learning. By leveraging past interaction histories as context, our method adapts the simulation environment dynamics to match real-world dynamics without requiring gradient updates, resulting in faster and more accurate alignment between simulated and real-world performance. We validate our approach across two tasks: object scooping and table air hockey. In the sim-to-sim evaluations, our method significantly outperforms the baselines on environment parameter estimation by 80% and 42% in the object scooping and table air hockey setups, respectively. Furthermore, our method achieves at least 70% success rate in sim-to-real transfer on object scooping across three different objects. By incorporating historical interaction data, our approach delivers efficient and smooth system identification, advancing the deployment of robots in dynamic real-world scenarios.