GAGhazanfar AliBehavioral Intelligence
PublicationsContext-aware behavior

Published · workshop paper

Auto-generating Virtual Human Behavior by Understanding User Contexts

Hanseob Kim · Ghazanfar Ali · Seungwon Kim · Gerard J. Kim · Jae-In Hwang

2021 · IEEE VR Abstracts and Workshops

Natural-language context activates grounded virtual-human actions.

Publisher / DOI Download BibTeX Code repository
Research illustration for Context-aware behavior
Research illustration

From the paper

Author abstract

Virtual humans are most natural and effective when it can act out and animate verbal/gestural actions. One popular method to realize this is to infer the actions from predefined phrases. This research aims to provide a more flexible method to activate various behaviors straight from natural conversations. Our approach uses BERT as the backbone for natural language understanding and, on top of it, a jointly learned sentence classifier (SC) and entity classifier (EC). The SC classifies the input into conversation or action, and EC extracts the entities for the action. The pilot study has shown promising results with high perceived naturalness and positive experiences.

Author-written abstract from the author manuscript.

In plain language

What this work does

A BERT-based model jointly classifies whether a sentence requests conversation or action and extracts action entities. An interaction module turns those predictions into virtual-human behaviors in a controlled room scenario.

  1. 01Natural-language input
  2. 02Sentence + entity classifier
  3. 03Grounded room behavior

At a glance

Method, evidence, and scope

Method diagram from Figure 2 of the context-aware-behavior paper
Method diagram from the paper · Figure 2, PDF page 2. View full size
Method and evidence for Context-aware behavior
InputNatural-language conversation and action requests
OutputAction intent, extracted entities, and virtual-human behavior
MethodJoint sentence classification and entity classification
Data and scopeControlled room scenario with a fixed object and interaction set
EvaluationPilot study of perceived naturalness and user experience
LimitationsThe controlled object and action inventory limits generalization to arbitrary environments.

Implementation and artifacts

Code and setup

Independent implementation of the paper’s core ideas, with setup instructions and data preparation documented in the repository README. The institute’s original source, datasets and trained models are not distributed.

Browse code and setup guide

Reference this work

Citation

Hanseob Kim, Ghazanfar Ali, Seungwon Kim, Gerard J. Kim, Jae-In Hwang. Auto-generating Virtual Human Behavior by Understanding User Contexts. IEEE VR Abstracts and Workshops, 2021. Pages 591-592. DOI: 10.1109/vrw52623.2021.00178.

Download BibTeX
@inproceedings{contextawarebehavior2021,
  title = {{Auto-generating Virtual Human Behavior by Understanding User Contexts}},
  author = {Kim, Hanseob and Ali, Ghazanfar and Kim, Seungwon and Kim, Gerard J. and Hwang, Jae-In},
  year = {2021},
  booktitle = {IEEE VR Abstracts and Workshops},
  pages = {591-592},
  doi = {10.1109/vrw52623.2021.00178},
  url = {https://ghazanfarali.com/research/context-aware-behavior/}
}