Fine-tuning video models with a LoRA

Choose a base model

Four open-weight families dominate at the moment. Wan 2.1 and 2.2 are the common default: strong motion, permissive licensing, and official LoRA training code. The 1.3B variant is the gentlest way into video LoRA training that exists — it fits in about 24 GB and produces respectable results. CogVideoX is older and slightly stiffer in motion, but its 2B model trains fast and the surrounding tooling is mature. HunyuanVideo gives some of the best quality in the open set and asks for 48 GB or aggressive memory tricks. LTX-Video trades a little fidelity for speed, which makes it a good laboratory while you are still learning what your dataset wants.

Start small. A LoRA you train in an evening on the 1.3B model teaches you more than a week-long run on the biggest checkpoint ever finishes, because the lessons — caption phrasing, rank, step counts — mostly transfer upward.

Prepare clips, not movies

Where an image dataset is a folder of photos, a video dataset is a folder of short single-idea shots. Two to ten seconds each, and one idea per clip: if the source video cuts to a new angle or a new scene, that is a new clip. Cut on the cuts. A video clipper keeps every shot of a long source on one timeline, so you can mark them all in one sitting and export each as its own frame-accurate MP4 at the source frame rate.

Count expectations run similar to images. A subject or character LoRA wants 20 to 50 clips; a motion or camera-move LoRA — the video-specific kind that teaches a push-in, a pan, a walking gait — can work with as few as 15 because the lesson is narrower. Style LoRAs sit in between.

Clean every clip before it enters the folder:

• Crop out watermarks, logos, letterboxing and burned-in captions. A corner watermark does not stay a corner watermark in the model's mind; it smears across the whole output. The clipper's per-clip frame does this without a re-scale, and can lock the crop to 9:16 or 1:1 if your training bucket expects one.
• Keep the native frame rate. Re-encoding 30 fps material at 60 fps fabricates frames that teach nothing.
• Trim fades, transitions and title cards entirely.
• Stay at or below the resolution you will train at — 480p or 720p covers most runs. Upscaling low-quality sources adds detail the model will happily learn as artefact.

Cropping does not change resolution, so normalise it afterwards. One ffmpeg pass handles the common case — trim to five seconds, normalise resolution, re-encode cleanly:

ffmpeg -i input.mp4 -t 5 -vf "scale=720:1280:force_original_aspect_ratio=decrease,pad=720:1280:(ow-iw)/2:(oh-ih)/2" -c:v libx264 -crf 18 -preset medium output.mp4

Check a few processed clips for their real dimensions rather than trusting the command. Frame-accurate size questions — is this clip actually 720 across, or 718? — are the same questions an image workflow asks, and a pixel measurement tool answers them the same way.

Caption motion, not just content

Image captions describe a scene; video captions describe a scene plus what happens in it. Every caption should carry the subject, the setting and the light, then the part images never have: the camera and the action. "Slow push-in on a coffee cup as steam rises" is a video caption. "Coffee cup, steam, wooden table" is half of one.

The camera vocabulary is worth building deliberately, because it is what separates a controllable motion LoRA from a random one: static shot, handheld, pan left, pan right, push-in, pull-back, tracking shot, orbit. Use the same phrase for the same move in every caption — the model learns your dictionary, not the dictionary.

Auto-captioning helps here too, and most vision-language models now process short clips directly, but hand-editing matters even more than with images. Temporal descriptions are exactly where auto-captioners hedge, writing "the camera moves slightly" when what happened was a decisive dolly.

Training setup and the VRAM reality

Finetrainers and DiffSynth-Studio are the two general-purpose trainers; Wan, CogVideoX and HunyuanVideo each also publish their own. The general-purpose ones spare you per-model ceremony and let you switch bases without relearning the workflow.

Reasonable first-run numbers, on the 1.3B or 2B class of model:

• LoRA rank 32 to 64. Video carries more variation per example than an image, so it spends capacity faster; rank 16 tends to underfit.
• Learning rate around 1e-4, cosine decay, short warmup.
• Batch size 1, with gradient accumulation to reach an effective batch of 4 to 8. Sequences eat memory — even batch 2 can be the difference between fitting and not.
• 1500 to 3000 steps for a subject LoRA, fewer for a narrow motion LoRA.

Memory is the honest constraint of this whole endeavour. bf16 mixed precision and gradient checkpointing are mandatory rather than optional; expect roughly 24 GB of VRAM for the small models, 48 GB and upward for the flagship ones, or multi-GPU setups with the big ones. The cloud-GPU escape hatch works exactly as it does for images, priced by the hour — but a video run takes real hours, so a debugging loop you could finish in an evening on images can cost real money on video. All the more reason to rehearse on a small checkpoint first.

Sample early, sample often

Generation is slow enough that you cannot eyeball the whole validation set every checkpoint. Pick three to five fixed prompts — same seeds, same settings, every time — that together probe what you taught: one prompt close to the training data, one that varies the subject's clothing or setting, one motion you never trained on. Generate that grid every 200 to 300 steps.

You are watching for the same curve as in image training, plus one video-only failure: motion collapse. Samples stay spatially pretty while the movement goes stiff, looping or absent — the model has learned what frames look like and given up on how they follow each other. Motion collapse usually means too many near-static clips, too many steps, or captions that under-describe the action. The remedies are more varied clips, fewer steps, and the camera vocabulary from the captioning section.

Also sample at LoRA strengths below 1.0. Video adapters bite harder than image ones at equal settings; a strength that looks perfect in stills can push a moving subject into mush.

Frequently asked questions