ProCreate, Don’t Reproduce!
Propulsive Energy Diffusion for
Creative Generation

Imagine a fashion designer attempting to brainstorm a new idea for a clothing line. Looking back on past runway shows that they found particularly stunning and influential, they attempt to draw inspiration from others’ work without copying it directly. They want to evoke similar concepts, expressed perhaps in the silhouettes and proportions showcased on the runway, the particular way the fabric drapes on each model’s frame, or the drama communicated by a specific cut or color palette. In other words, their goal is to take a reference set of images, which express a unique creative vision from a designer, and draw inspiration from them without direct reproduction. If they attempt to use a generative image model for this creative iteration, however, they would find that the model had either never acquired that ineffable concept before, and thus was unable to produce good examples, or had memorized examples from the reference set during adaptation (e.g. through fine-tuning or tuning new token embeddings (Gal et al., 2023)), and merely reproduces elements of the design exactly. Our paper aims to address this problem directly, propelling generated images away from those in a reference set while maintaining high-level conceptual inspiration. In that way, we jointly address the specific problem of exact replication and the general problem of low diversity among generative model samples.

Recent advances in generative image modeling, in particular the development of highly capable, and easier to train, denoising diffusion probabilistic models (DDPMs) (Ho et al., 2020), have enabled high-quality, complex image generation in both conditional (e.g. class- or text-conditional generation) and unconditional settings (Dhariwal and Nichol, 2021; Saharia et al., 2022b; Nichol et al., 2022; Rombach et al., 2022; Ramesh et al., 2022). With their widespread adoption, diffusion models are increasingly used in customized settings, where a smaller number of images are used to define a particular domain (Zhu et al., 2023; Benigmim et al., 2023), style (Lu et al., 2023; Wang et al., 2023; Zhang et al., 2023), subject/concept (Ruiz et al., 2023; Gal et al., 2023; Kumari et al., 2023).

Unfortunately, despite their improvements over GANs, diffusion models are similarly prone to training data memorization and a lack of sample diversity (Somepalli et al., 2023b, a; Corso et al., 2023), which is particularly damaging in light of their use in these customized, low-data settings—generated images are often similar to one another (Sadat et al., 2023). This behavior is particularly damaging when diffusion models are used to produce art or graphic design materials, where creativity is essential, since the models may replicate potentially copyrighted examples directly (Somepalli et al., 2023a, b), opening users and model developers up to potential liability.

To improve sampled image diversity, we propose an energy-based (Hinton, 2002; Song and Kingma, 2021; Du et al., 2023; Kingma et al., 2021; Song et al., 2021b) method which applies a propulsive force to latent representations during inference, pushing the generated image away from the images in the reference set. We consider two experimental setups: 1) few-shot creative generation and 2) training data replication prevention. For few-shot generation, we construct a new few-shot image generation dataset called FSCG-8 using collected images across eight different categories. While fine-tuning a diffusion model to a reference set quickly overfits in this low-data setting, ProCreate, by contrast, achieves greater sample diversity while retaining broad conceptual similarity, both being essential components of creative generation. In the training data replication experiments, we show that ProCreate is effective at preventing pre-trained diffusion models from replicating training data.

In summary, our contributions are as follows:

• To test sample diversity, memorization, and creative generation, we collected a few-shot generation dataset, FSCG-8, spanning across domains such as paintings, architecture, furniture, fashion, cartoon characters, etc.

• We propose ProCreate, a simple and easy-to-implement component that allows diffusion models to generate diverse and creative images using concepts from a reference set, without direct reproduction.

• In few-shot generation, ProCreate is found to have better sample diversity than prior arts, while maintaining a high similarity to the reference set.

• ProCreate addresses the training data replication issue with a significantly lower chance of replicating training images than pre-trained diffusion models.

In this section, we review related areas of work in guided diffusion models, few-shot image generation, sample diversity, and data replication issues.

In this section, we will cover the basics of diffusion denoising probabilistic models (DDPM) (Ho et al., 2020), denoising diffusion implicit models (DDIM) (Song et al., 2021a), and classifier guidance (Dhariwal and Nichol, 2021). Given samples from a data distribution $q\left(x_0\right)$ of images, we can use diffusion models to learn a model distribution $p_{\theta }\left(x_0\right)$ that approximates $q\left(x_0\right)$ and can be sampled from. To sample a new image, diffusion denoising probabilistic models (DDPMs) (Ho et al., 2020) start with a sample of Gaussian noise $x_T\sim N(0,\text{I})$, then repeatedly denoise $x_t$ into $x_{t-1}$ where $t$ decrements from $T$ to $1$. In this paper, we use a more performant sampling method than the default DDPM approach, termed denoising diffusion implicit models (DDIM) (Song et al., 2021a), which was shown to beat GAN models with only 25 sampling steps (Dhariwal and Nichol, 2021). At each denoising step with a noisy image $x_t$, the DDIM sampling process first makes a one-step prediction of ${\hat{x}}_0$ with Equation 1, then it denoises $x_t$ into ${\hat{x}}_{t-1}$ by Equation 2. ${ϵ}_{\theta }$ represents the learned diffusion model with weights $\theta$ and ${\alpha }_t$ are scalar parameters that define the diffusion noise schedule.

To generate samples from diffusion models that are conditioned on ImageNet class labels, Dhariwal and Nichol (2021) introduced classifier guidance. They used the gradient of a noise-aware ImageNet classifier $f_{\varphi }\left(y|x_t\right)$ to perturb the noise prediction of each denoising step. Following the formulation in Bansal et al. (2023), classifier-guidance of class label $c$ is applied by modifying the diffusion process with an additive gradient term:

where $l_{ce}$ is the cross-entropy loss and $f_{cl}$ is a noise-aware classifier, then replacing ${ϵ}_{\theta }(x_t,t)$ with ${ϵ}^{\prime }$ in Equation 2. Furthermore, Ho and Salimans (2022) formulated diffusion models as score-matching models and showed that Equation 3 works by lowering the energy of data for which the classifier $f_{cl}$ predicts to be of class $c$ with high likelihood. Note that here, lower energy corresponds to a higher probability of being sampled. In the next section, we will use the notations and basic concepts and notations from here to develop ProCreate.

In this section, we introduce the mathematical formulation of ProCreate and the techniques we use to improve ProCreate’s performance. The core idea behind ProCreate is to guide the generation away from images in the reference set using the gradient of an energy function computed with that set. As illustrated in Figure 2, at each denoising step $t$ with the current noisy image $x_t$, ProCreate uses the diffusion model to predict a clean image $x_0$, compute distances from $x_0$ to each reference image, then finally updates $x_t$ with a guidance gradient to maximize the distance between $x_0$ with the closest reference image to it. With the guided noisy version of $x_t$, ProCreate can resume the normal diffusion model sampling process to denoise it to $x_{t-1}$.

Given a diffusion model checkpoint that is fine-tuned on limited data, our goal is to improve the diversity of its generated samples while maintaining sample quality and conceptual similarity to the reference set. In this section, we describe our experiments on few-shot creative generation, comparing the samples generated from the default DDIM, CADS, and ProCreate sampling methods both qualitatively and quantitatively.

With the surging interest in generative AI in recent years, people use image generation models for a variety of entertainment or business purposes. However, recent studies show that even large-scale diffusion models are prone to replicating data inside its training set (Somepalli et al., 2023a, b), raising privacy and copyright concerns. To prevent large-scale diffusion models from replicating their training data, we follow the setup of Somepalli et al. (2023a) for this experiment to test ProCreate’s ability to guide generations of pre-trained Stable Diffusion models away from their LAION (Schuhmann et al., 2022) training dataset.

We propose ProCreate, a simple method for improving the generation diversity and creativity of diffusion models given a set of reference images. We focus on two setups to demonstrate the effectiveness of ProCreate. First, in the low-data fine-tuning regime, we introduce a new text-to-image few-shot generation dataset FSCG-8 and we show that ProCreate achieves the best performance in terms of sample diversity and distribution metrics compared to prior approaches. Second, for pre-trained diffusion networks, we show that ProCreate can effectively mitigate training data replication and improve sample quality on the LAION dataset. In the future, we expect that ProCreate can be applied to more modalities of data, including audio and video, to promote more diverse and creative generation of digital content.

We thank Zhun Deng and members of the NYU Agentic Learning AI Lab for their helpful discussions. The NYU High-Performance Computing resources, services, and staff expertise supported the compute.