An image-prompt adapter kept a model frozen and still worked

The image

The record here is Tencent AI Lab's paper "IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models," posted to arXiv on 13 August 2023 by Hu Ye, Jun Zhang, Sibo Liu, Xiao Han and Wei Yang, alongside the project's own repository. Both publish grids of a reference image next to generations that carry over its color, medium or composition while a separate text prompt supplies new content. As with any adapter paper, these are illustrative comparisons chosen by the authors, not independently verified production frames.

What the documents show

The paper's abstract states its central mechanism is a "decoupled cross-attention" that separates the attention layers handling text features from those handling image features, and reports the adapter itself holds only 22 million parameters while matching or exceeding fully fine-tuned image-prompt models. Because the base diffusion model stays frozen, the paper states the adapter generalizes to other custom checkpoints fine-tuned from the same base. The repository corroborates and extends this: it documents compatibility with both Stable Diffusion 1.5 and SDXL 1.0, states the method composes with ControlNet and T2I-Adapter for structural constraints, and adds a usage note the paper does not carry — that the adapter "works best for square images," with non-square inputs resized to 224x224.

Production context

Read editorially, a lightweight adapter that needs no full retraining lowers the cost of holding a client's brand palette or a location's lighting across many generated concept frames, since the paper states the same frozen base model can serve multiple such adapters. The repository's own note that scale is set near 1.0 with minimal text when the goal is an image-only prompt is a documented control setting, not a guarantee of any particular visual outcome; the paper frames the contribution as efficiency and compatibility, not as content fidelity to a specific photograph.

Reference versus imitation

The mechanism the paper describes conditions generation on image features extracted by an encoder, run alongside a text prompt, not on a claim to reproduce a named living artist's body of work. Crediting the method accurately means describing it as a documented technical adapter with a stated parameter count and a stated attention design, not as a style-transfer tool aimed at any one creator. Where the repository lists compatible derivative projects, such as face-identity variants, the entry should name that as the repository's own account of its ecosystem, not as evidence about any specific output's originality.

• Does the documentation name the base model and encoder the adapter was trained against?
• Is the claimed "22M parameters" figure the paper's own reported number, or an approximation?
• What does the source image control — its content, its style, or both — per the paper's own attention design?

A parameter count and an attention diagram are specific, checkable facts; they say nothing about whether a particular generated image was ever compared against a copyrighted source, which remains a separate and undocumented question.