Generating Activity Snippets by Learning Human-Scene Interactions

George Mason University
ACM Transactions on Graphics (Proceeding of SIGGRAPH 2023)

Our approach generates an activity snippet referring to a sequence of high-level keyframe descriptions of multi-character, multi-object interactions represented as graphs, based on which our approach automatically instantiates virtual characters and objects in a 3D environment.


We present an approach to generate virtual activity snippets, which comprise sequenced keyframes of multi-character, multi-object interaction scenarios in 3D environments, by learning from recordings of human-scene interactions. The generation consists of two stages. First, we use a sequential deep graph generative model with a temporal module to iteratively generate keyframe descriptions, which represent abstract interactions using graphs, while preserving spatial-temporal relations through the activities. Second, we devise an optimization framework to instantiate the activity snippets in virtual 3D environments guided by the generated keyframe descriptions. Our approach optimizes the poses of character and object instances encoded by the graph nodes to satisfy the relations and constraints encoded by the graph edges. The instantiation process includes a coarse 2D optimization followed by a fine 3D optimization to effectively explore the complex solution space for placing and posing the instances. Through experiments and a perceptual study, we applied our approach to generate plausible activity snippets under different settings.




        title={Generating Activity Snippets by Learning Human-Scene Interactions},
        author={Li, Changyang and Yu, Lap-Fai},
        journal = {ACM Transactions on Graphics (TOG)},
        volume = {42},
        number = {4},
        year = {2023}