The research on curating training data for diffusion models backs most of what careful fine-tuners already do, and then it puts a limit on each habit. Duplicated images and overly specific captions are what models learn to copy, and that copying can begin partway through a fine-tune while the quality scores are still improving. Filtering beats raw volume right up until the filtered set gets repeated too many times, and captions written by a vision model improve prompt following as long as some original text stays in the mix. At the small end, Meta found that a hundred hand-picked images were enough to turn a model that lost most visual-appeal comparisons into one that won most of them.

I train and publish FLUX checkpoints as BobsBlazed, and the curation rules I work by came out of finding what broke, release after release. They are written up in my checkpoint essay. I went through the peer-reviewed research on training data to see which of those rules the evidence supports and where I have been pushing a good habit further than any study can vouch for.

What does the research on diffusion training data cover?

Security researchers measure when a model reproduces its training images, and model builders measure what filtering and recaptioning do for quality. The large labs publish technical reports that describe their pipelines with fewer controlled comparisons behind them. I would start a fine-tuner on the papers in this table. The venue sits beside each one because a peer-reviewed result and a lab describing its own pipeline deserve different weight.

Paper Venue The finding
Extracting Training Data from Diffusion Models (Carlini et al.) USENIX Security 2023 Duplication is a major factor in which training images can be pulled back out of a model
Understanding and Mitigating Copying in Diffusion Models (Somepalli et al.) NeurIPS 2023 Caption specificity drives copying about as much as duplication does, and the memorization appears during fine-tuning
Why Diffusion Models Don’t Memorize (Bonnaire et al.) NeurIPS 2025 Memorization starts later the more distinct images a model trains on, while sample quality peaks early regardless
Scaling Laws for Data Filtering (Goyal et al.) CVPR 2024 Curated data loses value each time it repeats, so the right filter depends on compute
Scaling Rectified Flow Transformers (Esser et al.) ICML 2024 An even mix of original and synthetic captions beat original captions alone on GenEval
Emu (Dai et al.) Meta technical report, 2023 A couple of thousand hand-picked images lifted visual appeal sharply, and the first hundred did most of it

Do duplicate images make a diffusion model memorize?

Duplicates are a major driver of memorization, and the captions attached to those duplicates matter nearly as much as the pixels do. When Carlini and colleagues went looking for training images they could pull back out of Stable Diffusion, they aimed at the most duplicated examples in the training set and recovered more than a hundred near-copies, which led them to name duplication a major factor behind extraction.

Somepalli and colleagues followed with a NeurIPS 2023 paper arguing that text conditioning plays a similarly important role. A highly specific caption behaves like a key that retrieves one stored image, and in their experiments models trained without any text conditioning often did not replicate their training data at all.

The memorization in that study was acquired during fine-tuning, so a well-behaved base model does not stop a fine-tune from copying whatever its own dataset repeats. The same paper found that near-duplicates can stay in a set when their captions are resampled, because varying the text around a repeated image cut copying substantially. It also measured higher similarity to training images in every condition once the text encoder was trained as well.

Carlini’s team calls deduplication an effective mitigation and says in the same paper that it is not a perfect one. On a small CIFAR-10 model, retraining on deduplicated data still regenerated 986 training images, down from 1,280.

Can a diffusion model start copying while its scores improve?

It can, and one of the four best papers at NeurIPS 2025 explains the mechanism. Bonnaire, Urfin, Biroli, and Mézard found two timescales in diffusion training. Sample quality peaks at a point that does not depend on dataset size, and memorization begins at a later point that moves out in proportion to the number of training images.

In their runs on 1,024 images, FID improved from 30.1 to 19.1 between 100,000 and 1.62 million training steps, and across that same stretch the share of generated samples that were copies of training images went from zero to 34.1%.

  • 100,000 steps FID 30.1 0.0% copied
  • 1.62 million steps FID 19.1 34.1% copied
Share of generated samples that copied a training image, for a small unconditional diffusion model trained on 1,024 images, at two points in training. FID, where lower is better, improved across the same stretch. From Bonnaire et al., NeurIPS 2025, Figure 2.

Those experiments used small unconditional models on 32-pixel grayscale faces, so I would not carry the numbers over to a FLUX fine-tune, though the mechanism is general enough to plan around. A falling loss and an improving score say nothing about whether a model has started copying, and a run with fewer distinct images reaches that point sooner.

Does filtering a dataset beat adding more data?

Filtering wins until repetition catches up with it, and where that happens depends on how many passes the run will make. The DataComp benchmark at NeurIPS 2023 was built to answer this for image-text data, and it found that smaller, more stringently filtered subsets trained models that generalized better than the full pool did, with the best CLIP-score filter keeping roughly the top 30% of candidates.

Goyal and colleagues showed at CVPR 2024 that the strictest filter only wins on shorter runs, because curated data wears out.

In their experiments the top 10% of a pool fell below the 30-to-40% band after the fourth epoch, because each repeat of the same excellent images taught the model less than a first look at images that were merely decent. Both of those studies trained CLIP models, which learn by matching images to text, so they speak to image-text data broadly more than to diffusion in particular.

Diffusion-specific evidence points the same way. Li and colleagues at Amazon cut a billion-image source to a quarter of its size for a CVPR 2024 study, and the filtered set raised the aesthetic score of generated images from 5.07 to 5.20 while image-text alignment held level. How much repetition hurts diffusion training is less settled. A Princeton preprint from June 2026 found that a text-to-image model trained on a diverse mix lost only marginal performance when it used fewer unique images over more epochs.

Should you caption training images with a vision model?

A vision model should write most of the captions, and the strongest published results keep some original text in the mix. OpenAI trained DALL-E 3 on 95% synthetic captions after finding that higher synthetic blends always improved CLIP score, and the same report documents what that choice cost.

The captioner hallucinated details like a plant’s genus and species or a bird’s species, and the finished model became unreliable at generating those specific terms. It is a well-documented case of a captioner’s mistakes turning into the model’s mistakes.

Stability AI went to an even split for SD3, pairing 50% original with 50% synthetic captions, because synthetic text can make a model forget concepts the captioning model never knew. In their comparison the mix lifted GenEval from 43.27 to 49.78 compared with original captions alone. The Qwen-Image team kept raw captions in their pipeline for similar reasons, crediting them with robustness to short prompts and with real-world knowledge like plant names and cartoon IP that synthesized captions often miss.

Recaptioning earns its compute on that evidence, as long as someone audits the captioner on the names and terms the model will be asked for. Caption length deserves the same attention. The Princeton i1 study found that models trained on long captions came out stronger overall but performed poorly on short test prompts, so either the captions should resemble what people will type or the prompts need rewriting before they reach the model.

How many images does a quality fine-tune need?

Fewer than intuition suggests, provided each one is chosen by someone with a trained eye and captioned with care. Meta’s Emu report fine-tuned a model pretrained on 1.1 billion images with 2,000 hand-picked photographs whose captions were written by people, and raters preferred the tuned model’s visual appeal 82.9% of the time against the untuned version on PartiPrompts.

Its ablation against SDXL is the table I would show anyone assembling a fine-tuning set. The untuned model won 24.8% of visual-appeal comparisons, and a random hundred of the hand-picked images lifted that to 60.3%. The remaining 1,900 images added under seven points on top of that.

  • No quality-tuning 24.8%
  • 100 images 60.3%
  • 1,000 images 63.2%
  • 2,000 images 67.0%
Win rate on visual appeal against SDXL by size of the hand-picked fine-tuning set, from Meta's Emu report, Table 3. The 100 and 1,000 image sets were sampled at random from the 2,000 curated images.

Emu’s authors also stopped training by 15,000 iterations while the loss was still falling, because longer runs on a set that small overfit and lost generality. The DreamBooth paper at CVPR 2023 saw a related failure in subject fine-tuning, where runs that went long started snapping outputs to the few training views. It also measured what one word in a caption is worth, with the correct class noun scoring 0.744 on subject fidelity against 0.303 when the class noun was left out.

What should a fine-tuner change?

Mostly the thresholds on habits that already exist, in about this order:

  1. Deduplicate with a low bar, and give any near-duplicate you keep on purpose a different caption each time it appears.
  2. Set how hard to filter by how many passes the run will make, loosening the bar as the epoch count climbs.
  3. Recaption with a vision model but keep some original text in the mix, then audit the captioner on the names and terms people will type.
  4. Match caption length to real prompts, or rewrite prompts before they reach the model.
  5. Leave the text encoder frozen unless there is a specific reason to train it.
  6. Hand-pick the final quality set and write those captions yourself, since the first hundred images carried most of the gain in Meta’s tests.
  7. Stop small runs early and compare generations against the training set before every release, because neither the loss curve nor FID will flag copying.

What hasn’t the research answered yet?

I have not found a peer-reviewed curation study on FLUX-class fine-tunes, and Black Forest Labs has not published the kind of data recipe Stability published for SD3, so every number in this essay comes from other model families. Repetition is the most open question. The CLIP results say curated data wears out fast, and the one diffusion study I found says it wears out slowly. A careful study at LoRA scale, where a small set is seen many times over, would settle more arguments among fine-tuners than another architecture paper.

FAQ

Does deduplication stop a diffusion model from memorizing?

It reduces memorization, and the studies that measured it found copying continued at a lower rate afterward. Carlini’s deduplicated CIFAR-10 model still regenerated hundreds of training images, and Stable Diffusion 2.1 still replicated training data despite being trained on partly deduplicated images, so pair deduplication with varied captions and an early stop.

Should you train on synthetic captions or original captions?

Use both, because the strongest published results come from mixing synthetic captions with original text. Synthetic captions improve prompt following, while original text carries the names and concepts a captioning model does not know, and Stability’s SD3 comparison found that an even mix scored higher on GenEval than original captions alone.

How many images does it take to fine-tune a diffusion model’s look?

Far fewer than a pretraining run needs, as long as every image is exceptional and carefully captioned. In Meta’s Emu tests a hundred hand-picked images took the win rate on visual appeal against SDXL from 24.8% to 60.3%, and DreamBooth personalizes a single subject from three to five photos.

Can FID or training loss detect memorization?

Neither one flags memorization on its own, and the clearest evidence comes from the NeurIPS 2025 best-paper study, where FID kept improving on a 1,024-image dataset while more than a third of generated samples became copies of training images. Comparing generations against the training set directly is the check that catches it.

Where should you start reading?

Begin with Somepalli’s NeurIPS 2023 paper on copying in text-conditioned models, which is the most useful single read for anyone fine-tuning, and follow it with Meta’s Emu report for what a small hand-picked set can do. Every paper cited in this essay is linked with its venue in the Sources list below.

Sources