Taming a Generative Model

Shimon Malnick1, Shai Avidan1, Ohad Fried2,
1Tel Aviv University, 2Reichman University
Paper arXiv Code
By taming a model we can control the color of the generated faces hair. In this case, to “forget” blond hair.

Abstract

Generative models are becoming ever more powerful, being able to synthesize highly realistic images. We propose an algorithm for taming these models — changing the probability that the model will produce a specific image or image category.

We consider generative models that are powered by normalizing flows, which allows us to reason about the exact generation probability likelihood for a given image. Our method is general purpose, and we exemplify it using models that generate human faces, a subdomain with many interesting privacy and bias considerations.

Our method can be used in the context of privacy, e.g., removing a specific person from the output of a model, and also in the context of de-biasing by forcing a model to output specific image categories according to a given target distribution. Our method uses a fast fine-tuning process without retraining the model from scratch, achieving the goal in less than 1% of the time taken to initially train the generative model. We evaluate qualitatively and quantitatively, to examine the success of the taming process and output quality.

BibTeX

@article{malnick2022taming,
  title={Taming a Generative Model},
  author = {Malnick, Shimon and Avidan, Shai and Fried, Ohad},
  journal={arXiv preprint arXiv:2211.16488},
  year={2022}
}