Through Van Gogh’s Eyes: Global Style Transfer with Diffusion Model

Diffusion models have achieved strong performance in artis-006 tic image synthesis, yet they still suffer from stylistic bias: when instruct-007 ing the diffusion model to create ‘Van Gogh-style’ images, we observed008 that the model tends to repeatedly generate textures and compositions009 characteristic of a narrow subset of iconic works. This bias limits the010 model’s ability to fully represent an artist’s stylistic diversity. To ad-011 dress this, we introduce Global Style Transfer (GST), a text-independent012 framework that learns an artist’s unified style distribution by training013 global visual statistics from hundreds of artworks. GST guides the diffu-014 sion process in the intermediate feature space (h-space), enabling gener-015 ation that reflects an artist’s global stylistic spectrum rather than relying016 on prompt-specific cues or memorized exemplars. In addition, we propose017 Content Alignment Guidance (CAG), a training-free guidance that aligns018 the semantic structure of a given content image while permitting flex-019 ible, style-based deformation. CAG preserves content identity without020 constraining artistic variation, allowing structural reinterpretations that021 naturally arise in artistic expression. Experiments on WikiArt bench-022 marks demonstrate that our method produces images with improved023 style consistency, reduced prompt-induced bias, and greater fidelity to024 global artistic semantics compared to the vanilla diffusion model. Our025 findings establish GST as a new direction for bias-robust artistic image026 generation.027

Author-written abstract from the author manuscript.