Lance 3B: one model for image & video, on a single RTX 4090

Lance ("Unified Multimodal Modeling by Multi-Task Synergy", ByteDance) is a 3 billion active-parameter model that does six things most stacks need six models for: text→image, text→video, image editing, video editing, and image / video understanding (visual question answering and captioning). The transformer backbone is a Qwen2-derived Mixture-of-Transformer with separate expert weights for "understanding" and "generation" tokens; it is trained from scratch (only the ViT and VAE encoders are pretrained) on a 128×A100 budget. The two visual front-ends are a Qwen2.5-VL ViT (for understanding/conditioning) and a Wan 2.2 video VAE (for generation). Two checkpoints ship: Lance_3B for images and Lance_3B_Video for video.

What makes it interesting for a home lab is the size. A 3B unified model that posts numbers next to 7B–20B systems is exactly the kind of thing that should run on a single consumer GPU — so I downloaded the weights to a data disk and pointed it at my RTX 4090. The rest of this article is what came out.

The CLI dispatches on a --task flag. Six modes are documented; a seventh — image_idip, subject-driven generation — lives in the code but isn't wired into the launcher, so I registered it to test reference-conditioned generation properly. Here's the map before we look at outputs.

Text-to-image is the most polished mode. The model renders clean in-image text (the cat's "STOP" sign on the project page is real), holds long compositional prompts together, and runs in about 10 seconds per 768² image on the 4090.

Instruction-guided editing

image_edit takes a source image and a free-form instruction. It preserves identity and pose while applying local edits (objects, relighting) or a whole-image restyle. Two examples — adding accessories to my own portrait, and a full 3D-cartoon restyle of a friend's photo:

Give Lance a single reference photo of a subject and a prompt, and it generates a new scene that keeps the subject's identity. This is the image_idip path (identity-preserving), and with one reference it works really well. Outputs are saved as a [reference | generated] pair:

Style transfer is just image_edit with the target style described in text. That is the "free-form manipulation" the project page shows, and it is excellent — the subject stays recognizable while the medium changes completely:

The understanding modes (x2t_image, x2t_video) take a visual plus a question and emit text. They're fast (~3 s per image) and accurate — including OCR and symbol recognition. A few real answers, verbatim:

Text-to-video runs from the Lance_3B_Video checkpoint at 848×480, 12 fps. Quality is strong and motion is coherent. Two clips — a tropical sunset coastline and a red-panda surfer:

Video editing

video_edit recolors subjects and replaces backgrounds while keeping motion. It is the most memory-hungry mode (it holds the reference video and the generated target at once), so on 24 GB it runs at video_360p on short clips. Input → edited:

The reason a 3B unified model is worth your disk space is the score-to-size ratio. From the Lance paper, here is where it lands against larger unified models on the four headline suites (Lance highlighted):

The standout is image editing: 7.30 on GEdit-Bench beats BAGEL (6.52) and InternVL-U (6.66) by a clear margin, and video generation at 85.11 VBench edges out the 14B Wan2.1-T2V. On GenEval it ties the 7B TUNA at 0.90 and beats the 20B Qwen-Image. DPG (84.67) is the one suite where the bigger models keep a small lead.

The README asks for a 40 GB GPU. The 4090 has 24. It fits, but only after two changes — both worth knowing if you try this yourself.

① Load in bf16 before moving to the GPU

The image checkpoint is 6.19 B parameters stored in fp32 (~24 GB). The stock loader moves the fp32 model to the GPU and only then casts to bf16 — which OOMs a 24 GB card at load time. Casting on CPU first, then moving the ~12 GB bf16 model to the GPU, fixes it:

# inference_lance.py — cast on CPU, defer the GPU move
model = model.to(dtype=torch.bfloat16)     # was: model.to(DEVICE)  [fp32 -> OOM]
# ... load checkpoint, resize embeddings ...
model = model.to(device=DEVICE, dtype=torch.bfloat16)   # GPU holds bf16 only

② Swap the flash-attn rotary kernel

flash-attn 2.8.3's Triton apply_rotary_emb calls torch.library.wrap_triton, which only exists in torch ≥ 2.6 (this repo pins 2.5.1). Any ViT pass — every edit, every understanding call — crashes with AttributeError. The pure-torch variant has identical math and is a drop-in; the fast attention kernel is untouched:

# modeling/vit/qwen2_5_vl_vit.py
from flash_attn.layers.rotary import apply_rotary_emb_torch as apply_rotary_emb

With those in place, plus PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to keep video generation from fragmenting, everything runs.

All seven modes, run back-to-back on the 4090. "Load" is checkpoint load + bf16 cast + GPU move; "Gen" is the denoise/decode (or autoregressive decode for understanding). Peak VRAM is the max sampled during the run.

Output is hard-coded to 12 fps and the model caps at 121 frames, so the absolute ceiling is 10.08 s. The surprise: t2v memory is flat at ~24 GB regardless of frame count — flash-attention keeps the long-sequence denoise cheap and the Wan VAE decodes frame-by-frame — so the full 121 frames fit even at native 480p. I swept it to confirm:

The one capability that consistently fails is composing two reference images into one output — whether that's two people, a person plus an object, or a content image plus a style image. The model anchors on one reference and reproduces or blends toward it. Three attempts, all on the correct subject-driven path:

The cause is mechanical: the position-id logic shifts all reference tokens to the same coordinate offset, so multiple references overlap in position space and the model can't keep them distinct. It is not a prompt problem and not a bug I introduced — multi-reference composition simply isn't a trained, exposed capability of the released 3B weights (the docs advertise only single-input tasks). The practical rule: one image reference for subject-driven generation; text instructions for style and edits.