Fine-tuning images with a LoRA

Pick the base model first

Everything downstream depends on the checkpoint you attach to. SDXL remains the pragmatic default: it has been trained, retrained and debugged by thousands of people, the tooling assumes it, and a LoRA on SDXL runs comfortably in 12 to 16 GB of VRAM. Flux produces sharper, more coherent images and understands longer prompts, but it is heavier — expect 24 GB for a smooth run, or a long afternoon with offloading tricks if you have less.

One rule matters more than the brand name: train a LoRA against the checkpoint you actually intend to generate with, or at least one built on the same architecture. A LoRA tuned on SDXL 1.0 behaves strangely when loaded into a differently-fine-tuned SDXL variant. Weights that describe one model's internal vocabulary rarely translate cleanly to another's.

Build the dataset — this is the whole job

Ask anyone who trains these things and they will tell you the same thing: dataset quality decides the outcome before the first training step. Thirty carefully chosen images beat a hundred sloppy ones. Not sometimes. Always.

How many do you need? For a specific subject — one person, one product, one pet — start around 20 to 50. For a style, 30 to 60 works, because style needs breadth of subjects sharing a look rather than depth on one face. Below roughly 15 images the model memorises instead of learning; past 100 you are usually adding near-duplicates that push the run toward the same failure.

Curate ruthlessly. Reject any image with a visible watermark, a text overlay, a collage layout, or a screenshot's UI chrome. Reject blur, heavy compression and frames where the subject is buried in clutter. Two frames that differ only by a shift of a few pixels are one image wearing a disguise — keep one, delete the other, because near-duplicates teach the model to replay rather than generalise.

Then check the survivors for spread. A good subject set covers angles (front, three-quarter, profile), distances (close-up, half body, full body), and contexts (different backgrounds and lighting that still feel like the same world). If every photo was taken in the same hour in the same room, the LoRA will happily conclude that the room is part of the subject.

Resize everything to roughly your training resolution before you start — 1024 pixels on the long side is the usual target for SDXL. Keep the files as PNG or high-quality WebP; re-compressing JPEGs repeatedly compounds artefacts into the training data. Previews you exported from elsewhere should be re-rendered at a known width rather than screenshotted, and it is worth measuring the exact pixel dimensions of a few samples with a pixel ruler so a 977-pixel stray frame does not quietly break a 1024 assumption.

Caption every image by hand

Captions tell the model which parts of the picture are the lesson and which are scenery. Skip them entirely and the LoRA bakes in everything — lighting, background, camera angle — which is why caption-less subject LoRAs regenerate the same photo forever.

The workflow most people settle on: run an auto-captioner (any vision-language model works) to get a first draft, then edit every line yourself. Auto-captions misdescribe exactly the details you care most about. Read each one and ask, does this sentence describe what makes this image that image?

Write descriptively, not interpretively. "A woman in a red wool coat standing on a rain-wet street at dusk, reflections on the asphalt" teaches something. "A beautiful cinematic masterpiece" teaches nothing. Describe the subject, the clothing or material, the pose, the setting, the light. Leave out judgements.

Many trainers reserve a trigger word — an invented token like "mxcoatred" — that appears in every caption and in none of the base model's training data. Prompts that include it pull the LoRA's influence forward; prompts without it lean on the base model. Trigger words help when you plan to compose several LoRAs, and barely matter when yours is the only one loaded. Either convention is fine as long as you apply it consistently.

Choose the numbers

The defaults that ship with trainers are sensible starting points, not constraints. A first run on SDXL looks roughly like this:

• LoRA rank 16 to 32. Rank is the capacity of the adaptation — higher captures finer detail and overfits faster. Go to 64 for intricate styles, stay at 16 for simple subjects.
• Learning rate around 1e-4 with a cosine schedule and a short warmup. Halve it if samples degrade late in the run.
• Text encoder learning rate roughly half the unet's, if your trainer exposes the two separately.
• Batch size 1 to 4 with gradient accumulation to reach an effective batch near 8 to 16.
• Around 1500 to 3000 steps for a subject, fewer for a small style set. There is no magic number; sampling decides.

Train in bf16 or fp16 with gradient checkpointing on. Both roughly halve memory for a small speed cost, and on a 12 GB card they are the difference between running and not running.

Sample while it trains

Never start a run and walk away. Generate a fixed grid of test prompts — the same prompts, same seed, same settings — every few hundred steps and lay the grids side by side. You are watching for the shape of the curve, not a single good frame.

Early samples look like the base model with a vague nod toward your subject. Around the middle of the run they should start looking right at a range of prompts and angles. The failure mode arrives quietly: samples become pin-sharp copies of specific training photos, and prompt changes stop mattering. That is overfitting, and the fix is to stop earlier, drop the learning rate, lower the rank, or diversify the data. Keep the checkpoint from just before the samples got stale — most trainers save on a schedule precisely for this.

Use the thing

Load the base checkpoint plus your LoRA file in whatever you generate with — ComfyUI, the diffusers library, or the trainer's own inference tab. The strength (sometimes called scale) sits between 0 and 1. At 0.5 to 0.7 the LoRA leans on the base model; at 1.0 it speaks fully. Start at 0.8 and adjust: if identity drifts, raise it; if everything looks pasted-on, lower it.

The interesting part is that LoRAs compose. A character LoRA plus a style LoRA, each around 0.7 strength, frequently produces exactly what neither alone would. Test combinations before assuming they conflict.

Frequently asked questions