Region-Based Semantic Factorization in GANs
Jiapeng Zhu1     Yujun Shen2     Yinghao Xu3     Deli Zhao4     Qifeng Chen1
1The Hong Kong University of Science and Technology         2ByteDance Inc
3The Chinese University of Hong Kong         4Ant Group.
[Paper] [Code] [Demo] [Colab]
Overview
In this work, we propose ReSeFa for Region-based Semantic Factorization in GANs. Specifically, we re-examine the task of local editing using pre-trained GANs. We argue that a good attribute vector for local control on the synthesized images will only affect the pixels landed in the target region while barely influencing the pixels outside this region. Based on this assumption, we obtain an optimization goal, which is happened to be a generalized Rayleigh quotient, which can be solved by a generalized eigenvalue problem. In short, the local control task is elegantly converted to solve an eigen-decomposition problem. Extensive experiments demonstrate the effectiveness of our proposed method.
Results
The following results are created by manipulating the versatile semantics found by ReSeFa.

Diverse semantics discovered from one sample.

FFHQ (StyleGAN2)

Eyes Eyebrows Mouth Nose

Car Wheels (StyleGAN2)

Generalizing semantics across samples.

Car Wheels (StyleGAN2)

Church (StyleGAN2)

AFHQ Eyes (StyleGAN3)

Demo Video

BibTeX
 @inproceedings{zhu2022resefa,
  title     = {Region-Based Semantic Factorization in {GAN}s},
  author    = {Zhu, Jiapeng and Shen, Yujun and Xu, Yinghao and Zhao, Deli and Chen, Qifeng},
  booktitle = {International Conference on Machine Learning (ICML)},
  year      = {2022}
}
Related Work
Zongze Wu and Dani Lischinski and Eli Shechtman. StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation. CVPR, 2021.
Comment: Explores the influences of the channels on the synthesized images.
Shen, Yujun and Zhou, Bolei. Closed-Form Factorization of Latent Semantics in GANs. CVPR2021.
Comment: Unsupervisedly factorizes the latent semantics in GANs.
Zhu, Jiapeng and Feng, Ruili and Shen, Yujun and Zhao, Deli and Zha, Zhengjun and Zhou, Jingren and Chen, Qifeng. Low-Rank Subspaces in GANs. NeurIPS 2021.
Comment: Explores the low rank subspaces in GANs to control the local region of the synthesized images.