1

The paper proposes a condition-based multi-modal human-AI cooperation framework for enhancing user intent identification and information presentation in VR. It utilizes intent tuples and a 2-Large-Language-Models (2-LLMs) architecture, improving prompt length and response generation. A VR furniture purchasing system based on this framework outperforms in user study phases, promising personalized VR experiences.

This paper introduces a novel approach to AR dialogue systems, utilizing the SIMMC2-Point dataset to incorporate pointing modality. It employs BART and CLIP models to design multi-modal dialogues capturing spatial and attribute data. Ablation experiments underscore the pointing modality’s importance, advancing AR dialogue systems for immersive interactions.

Our paper proposes a novel approach to task assignment in human-AI cooperation, utilizing the capability-aware shared mental model with the unified form of tuples to represent task-specific capabilities of both human and AI. Results from our user study show that this approach improves accuracy and time efficiency while facilitating better understanding of each team member’s capabilities.