Published · journal article
Silhouettes from Real Objects Enable Realistic Interactions with a Virtual Human in Mobile Augmented Reality
2021 · Applied Sciences
Real-object silhouettes give virtual humans spatial context in mobile AR.
From the paper
Author abstract
: Realistic interactions with real objects (e.g., animals, toys, robots) in an augmented reality (AR) environment enhances the user experience. The common AR apps on the market achieve realistic interactions by superimposing pre-modeled virtual proxies on the real objects in the AR environment. This way user perceives the interaction with virtual proxies as interaction with real objects. However, catering to environment change, shape deformation, and view update is not a trivial task. Our proposed method uses the dynamic silhouette of a real object to enable realistic interactions. Our approach is practical, lightweight, and requires no additional hardware besides the device camera. For a case study, we designed a mobile AR application to interact with real animal dolls. Our scenario included a virtual human performing four types of realistic interactions. Results demonstrated our method’s stability that does not require pre-modeled virtual proxies in case of shape deformation and view update. We also conducted a pilot study using our approach and reported significant improvements in user perception of spatial awareness and presence for realistic interactions with a virtual human.
Author-written abstract from the author manuscript.
In plain language
What this work does
A lightweight segmentation pipeline constructs silhouette geometry from camera images of real objects. A virtual character can interact with this geometry and be occluded by it without a pre-modeled object proxy. A mobile animal-doll scenario tests changing views and object deformation.
- 01Camera image
- 02Segmentation + silhouette
- 03Spatially aware AR interaction
At a glance
Method, evidence, and scope
| Input | Device-camera images of real objects |
|---|---|
| Output | Silhouette geometry for virtual-human occlusion and interaction |
| Method | Segmentation and dynamic silhouette geometry |
| Data and scope | Mobile AR animal-doll scenario |
| Evaluation | Paper reports a 2.4M-parameter model, 0.971 mIoU, and a 24-person pilot study |
| Limitations | Silhouette geometry approximates visible shape; it is not a complete reconstruction of hidden 3D object geometry. |
Watch the system
Paper presentation / demo
Implementation and artifacts
Code and setup
Independent implementation of the paper’s core ideas, with setup instructions and data preparation documented in the repository README. The institute’s original source, datasets and trained models are not distributed.
Browse code and setup guideReference this work
Citation
Hanseob Kim, Ghazanfar Ali, Andréas Pastor, Myungho Lee, Gerard J. Kim, Jae-In Hwang. Silhouettes from Real Objects Enable Realistic Interactions with a Virtual Human in Mobile Augmented Reality. Applied Sciences, 2021. Volume 11. Issue 6. Pages 2763. DOI: 10.3390/app11062763.
@article{silhouettemobilear2021,
title = {{Silhouettes from Real Objects Enable Realistic Interactions with a Virtual Human in Mobile Augmented Reality}},
author = {Kim, Hanseob and Ali, Ghazanfar and Pastor, Andréas and Lee, Myungho and Kim, Gerard J. and Hwang, Jae-In},
year = {2021},
journal = {Applied Sciences},
volume = {11},
number = {6},
pages = {2763},
doi = {10.3390/app11062763},
url = {https://ghazanfarali.com/research/silhouette-mobile-ar/}
}