ASAP: Auto-generating Storyboard and Previz

We present ASAP, a system that uses virtual humans to Automatically generate Storyboards And Pre-visualized scenes from movie scripts. In our ASAP system, virtual humans play the role of actors. To visualize the screenplay scene, our system understands the movie script, which is the text data, and then facilitates the automatic generation of the following virtual human’s non-/verbal behavior: (1) co-speech gesture, (2) facial expression, and (3) body movements. First of all, co-speech gestures are created from dialogue paragraphs using a text-to-gesture model trained with 2D videos and 3D motion-captured data. Next, for the facial expressions, we interpret the actors’ emotions in the parenthetical paragraphs and then adjust the virtual human’s face animation to reflect emotions such as anger and sadness. For body movements, our system extract action entities from action paragraphs (e.g., subject, target, and action) and then combine sets of animations to make animation sequences (e.g., a man’s act of sitting on a bed). As soon as possible, ASAP can reduce the amount of time, money, and labor-intensive work that needs to be done in the early stages of filmmaking.

Author-written abstract · Source record