LUFA: Lightweight Upper-Face Animation for VR/MR Avatars

Hwang Youn Kim; Ghazanfar Ali; Jae-In Hwang

doi:10.1109/ismar-adjunct68609.2025.00217

Published · adjunct paper

LUFA: Lightweight Upper-Face Animation for VR/MR Avatars

Hwang Youn Kim · Ghazanfar Ali · Jae-In Hwang

2025 · IEEE ISMAR-Adjunct

Lightweight upper-face animation for VR and MR avatars.

Publisher / DOI Download BibTeX Code repository

Research illustration

From the paper

Author abstract

For virtual agents, realistic co-speech facial expressions are essential to enhance naturalness. Rule-based methods lack diversity and temporal consistency in generating emotional expressions. Additionally, generating facial animation using large-scale generative models requires substantial computational resources, making real-time deployment challenging. In this paper, we propose Lightweight Upper-Face Animation for VR/MR Avatars (LUFA), a co-speech facial expression framework for generating real-time animations from voice and text inputs. We fine-tune Wav2Vec2.0 and BERT encoders using a reconstruction loss. Our framework treats their outputs as latent representations, aligns them through contrastive learning, and retrieves facial animation sequences based on these representations.

Author-written abstract · Source record

In plain language

What this work does

LUFA encodes voice and text with fine-tuned Wav2Vec2.0 and BERT models. Reconstruction loss and contrastive learning align latent representations, which are used to retrieve facial-animation sequences for VR and MR avatars. The related journal manuscript studies a later emotion–liveness approach and is in final review.

01Voice + text
02Aligned latent representations
03Facial animation retrieval

At a glance

Method, evidence, and scope

Graphical abstract: aligned voice and text representations retrieve upper-face animation for VR and MR avatars — Graphical abstract diagram. Aligned speech and text representations support recorded facial-motion retrieval. View full size

Method and evidence for LUFA
Input	Voice and text
Output	Retrieved upper-face facial-animation sequences
Method	Wav2Vec2.0 and BERT encoders; reconstruction loss; contrastive latent alignment and retrieval
Data and scope	Conference abstract describes the representation and retrieval framework
Evaluation	Consult the full conference paper for evaluation details
Limitations	Retrieval selects existing animation sequences. The later emotion–liveness manuscript’s parameter counts and study results do not describe LUFA.

Implementation and artifacts

Code and setup

Independent implementation of the paper’s core ideas, with setup instructions and data preparation documented in the repository README. The institute’s original source, datasets and trained models are not distributed.

Browse code and setup guide

Reference this work

Citation

Hwang Youn Kim, Ghazanfar Ali, Jae-In Hwang. LUFA: Lightweight Upper-Face Animation for VR/MR Avatars. IEEE ISMAR-Adjunct, 2025. Pages 841-842. DOI: 10.1109/ismar-adjunct68609.2025.00217.

Download BibTeX

@inproceedings{lufa2025,
  title = {{LUFA: Lightweight Upper-Face Animation for VR/MR Avatars}},
  author = {Kim, Hwang Youn and Ali, Ghazanfar and Hwang, Jae-In},
  year = {2025},
  booktitle = {IEEE ISMAR-Adjunct},
  pages = {841-842},
  doi = {10.1109/ismar-adjunct68609.2025.00217},
  url = {https://ghazanfarali.com/research/lufa/}
}