Apps & Guides

Stable Diffusion: Generate repeatable faces

Repeatability is the most important aspect when creating graphical content with generative neural networks. This holds true regardless of the type of content you create, be it a cinema or game character, landscape, or scene environment. The main problem can be formulated as: “How can I repeat my result?”. Every time you start generating images with the same positive and negative prompts, you’ll get different results. Sometimes the differences are minor and acceptable, but in most cases, they could pose a problem.

Stable Diffusion is learned on a large dataset captured from the real world, which explains why repeatability isn’t a strong point of this neural network model. However, this rule doesn’t apply to celebrity photos. These photos are found much more frequently in the real world and, therefore, in the dataset on which Stable Diffusion was trained. You can use these photos as a “constant” or a “starting point” in the generating process.

Method 1. “Shaken, not stirred”

Of course, you don’t need to create only celebrity images, but you can use multiple relevant prompts to get more or less consistent results. For example, we can take two famous Greek singers: Elena Paparizou and Marina Satti, and get repeatable results:

Model: Realistic Vision v6.0 beta 1

Positive prompts:

Elena Paparizou, Marina Satti, fashion portrait, alone, solo, greek woman in beautiful clothes, natural skin, 8k uhd, high quality, film grain, Canon EOS

Negative prompts:

bad anatomy, bad hands, three hands, three legs, bad arms, missing legs, missing arms, poorly drawn face, bad face, fused face, cloned face, worst face, three crus, extra crus, fused crus, worst feet, three feet, fused feet, fused thigh, three thigh, fused thigh, extra thigh, worst thigh, missing fingers, extra fingers, ugly fingers, long fingers, horn, extra eyes, huge eyes, 2girl, amputation, disconnected limbs, cartoon, cg, 3d, unreal, animate, nsfw, nude, censored

It works with any celebrities, as Stable Diffusion tried to reproduce the most prominent facial features. Here, we use the same model and “shake” two Hollywood stars (Dwayne Johnson and Danny Trejo) into one new synthetic character.

Positive prompts:

Dwayne Johnson, Danny Trejo, fashion portrait, alone, solo, 8k uhd, high quality, film grain, Canon EOS

Negative prompts:

bad anatomy, bad hands, three hands, three legs, bad arms, missing legs, missing arms, poorly drawn face, bad face, fused face, cloned face, worst face, three crus, extra crus, fused crus, worst feet, three feet, fused feet, fused thigh, three thigh, fused thigh, extra thigh, worst thigh, missing fingers, extra fingers, ugly fingers, long fingers, horn, extra eyes, huge eyes, amputation, disconnected limbs, cartoon, cg, 3d, unreal, animate, nsfw, nude, censored

Every time you mix the same celebrities, you get similar results. Let’s look at another method to generate repeatable characters.

Method 2. Name anchor

Celebrities are a good start, but let’s consider other methods for achieving repeatable results. The answer is quite simple: we can use multiple human names. Every nation has unique names, related to linguistic features. For example, the Greek name Kostas can translate to “labor” or “effort”, while Nikos means “Victory of the people”. These two names create a unique image of a generated person, aiding neural network models in understanding our creation objectives.

Positive prompts:

Portrait of [Kostas | Nikos] on a white background, greek man, short haircut, beard

Negative prompts:

woman, bad anatomy, bad hands, three hands, three legs, bad arms, missing legs, missing arms, poorly drawn face, bad face, fused face, cloned face, worst face, three crus, extra crus, fused crus, worst feet, three feet, fused feet, fused thigh, three thigh, fused thigh, extra thigh, worst thigh, missing fingers, extra fingers, ugly fingers, long fingers, horn, extra eyes, huge eyes, 2girl, amputation, disconnected limbs, cartoon, cg, 3d, unreal, animate, nsfw, nude, censored

Let’s generate numerous images (80-100) for further dataset creation. The main prompt was selected to provide convenient images that can be easily cleared from the background. Negative prompts protect us from including random images with distortions in the dataset, as well as images of women.

Tip: if you receive very different images from each other, try changing the CFG Scale parameter from 7.5 to 15. This will force the neural network to follow the prompts more formally.

You can select your own unique names with a simple name generator, like Behind the Name. Also, you can use the ControlNet feature to gain more control.

Method 3. Teach appearance

We can’t directly influence the final result, but we observe that some tokens (such as celebrity image tokens) carry more weight than others. This means we can create our conditional “celebrity” token by creating an appropriate prompt for it and further training the model on it. This is how LoRA (Low-Rank Adaptation of Large Language Models) operates. You can use our step-by-step guide to train your own LoRA model based on a self-made dataset.

After removing the background, we obtain clear portraits and use them to create a specific LoRA model. This model helps to replicate a face with a few minor changes:

Now, we can generate this character in different locations, create stories, and place him in various roles: from gardener to businessman. His face will be consistent recognizable and repeatable:

This method isn’t ideal, but it works perfectly in a variety of situations. You don’t need to prepare a dataset from a real person, and it can be generated remotely:

You can attempt to create such a virtual character yourself, without the assistance of a professional designer or 3D-modeling specialist. All you need are fast GPUs, which you can find in dedicated servers by LeaderGPU.