Low-Rank Subspaces in GANs
Jiapeng Zhu1,  Ruili Feng2,3,  Yujun Shen4,  Deli Zhao2,  Zhengjun Zha3,  Jingren Zhou2,  Qifeng Chen1
1Hong Kong University of Science and Technology         2Alibaba Group
3University of Science and Technology of China             4ByteDance Inc.
[Paper] [Code] [Demo]
Overview
In this work, we propose LowRankGAN by introducing the low-rank subspaces of a pre-trained generator, which enables local control of the GAN generation. It mainly consists of three steps. First, we relate the image region (e.g., eyes as the foreground and the rest as the background for a face image) to the latent space via computing the Jacobian matrix on a pre-trained generator. Second, we perform low-rank factorization on the resulting Jacobian to get the principal space and the null space for the foreground and the background, respectively. Finally, we project the principal space of the foreground onto the null space of the background, resulting in the so-called low-rank subspace. Through altering the latent codes along the directions within the low-rank subspace, we manage to precisely control the foreground region yet barely affect the background region. Interestingly, the subspaces derived from local areas in one image can be convincingly applicable to other images. Moreover, our LowRankGAN is highly robust to the image region for obtaining those subspaces.
Results
The following results are created by manipulating the versatile semantics found by LowRankGAN.

FFHQ (StyleGAN2)

Eyes Mouth Nose Hair

MetFaces (StyleGAN2)

Eyes Mouth Nose Hair

Church (StyleGAN2)

Sky
Grass

Car (StyleGAN2)

Body
Grass

Various Categories on BigGAN

Pose
Size

Full Demo Video

BibTeX
  @inproceedings{zhu2021lowrankgan,
    title     = {Low-Rank Subspaces in {GAN}s},
    author    = {Zhu, Jiapeng and Feng, Ruili and Shen, Yujun and Zhao, Deli and Zha, Zhengjun and Zhou, Jingren and Chen, Qifeng},
    booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
    year      = {2021}
  }
Related Work
Ramesh, Aditya and Choi, Youngduck and LeCun, Yann. A spectral regularizer for unsupervised disentanglement. ICML 2019.
Comment: Explores the leading right-singular vectors of the Jacobian of the generator.
Y. Shen, J. Gu, X. Tang, B. Zhou. Interpreting the Latent Space of GANs for Semantic Face Editing. CVPR 2020.
Comment: Interprets the face semantics emerging in the latent space of GANs with the help of off-the-shelf classifiers.
C. Yang, Y. Shen, B. Zhou. Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis. IJCV 2020.
Comment: Explores the emergent semantic hierarchy in scene synthesis models.
A. Voynov and A. Babenko. Unsupervised Discovery of Interpretable Directions in the GAN Latent Space. ICML 2020.
Comment: Interprets meaningful directions in GAN latent space by unsupervisedly training a direction reconstructor.
E. Härkönen, A. Hertzmann, J. Lehtinen, S. Paris. GANSpace: Discovering Interpretable GAN Controls. NeurIPS 2020.
Comment: Unsupervisedly discovers the latent semantics learned by GANs using PCA.
Shen, Yujun and Zhou, Bolei. Closed-Form Factorization of Latent Semantics in GANs. CVPR2021.
Comment: Unsupervisedly factorizes the latent semantics in GANs.