CoViP: Contextualized Visual Personalization
in Vision-Language Models

VLMs Forget Who You Know

When users show a VLM a new image of someone they have talked about before—a friend, a pet, a familiar place—existing models respond as if it were the first time. They cannot connect the new visual input with the rich multimodal dialogue history that the user has already shared.

Figure 1. A qualitative comparison. Without contextualized visual personalization (Qwen3-VL, 8B), the model treats Martin Freeman as a generic celebrity. CoViP (8B, Ours) correctly recalls Jeffrey's personal details—his nickname, dietary preferences, and past encounter—from the user's accumulated dialogue history.

Overview

What is Contextualized Visual Personalization?

Given a user's accumulated multimodal dialogue history (visual-textual context), a VLM must: (1) visually recognize a concept in a new query image that was previously encountered in the context, and (2) internalize its associated personal memories from past dialogues into a grounded, context-aware caption—without relying on textual shortcuts.

Automated Pipeline for Personalized Contexts

Building a benchmark for contextualized visual personalization requires realistic multimodal contexts. We construct a fully automated three-stage pipeline:

Figure 2. The three-stage automated pipeline. Query Image Generation synthesizes multi-concept scenes; Text Dialogue Generation produces personalized dialogues per concept; Context Construction assembles interleaved positive/negative concept-dialogue pairs as the VLM's in-context input.

CoViP: RL Training with LLM-as-a-Judge Rewards

CoViP improves personalized image captioning capability through two complementary components: an RL-based post-training scheme with a verifiable reward signal, and Caption-Augmented Generation (CAG) for inference-time grounding.

Figure 3. The CoViP training and evaluation framework. The model receives interleaved image-text in-context demonstrations. Generated captions are evaluated offline by an LLM-as-a-Judge via auto-generated MCQA questions about each concept's personal details. Accuracy on these MCQAs serves as the verifiable reward (VR) for on-policy RL training.

Quantitative Results

We evaluate VLMs on the CoViP benchmark using an MCQA-based LLM-as-a-Judge protocol. Positive Accuracy measures how well a model recalls details of the target concept; Negative Accuracy evaluates whether the model avoids incorporating irrelevant contextual information.

Generalization to Real-World Personalization Scenarios

To assess whether CoViP learns genuine visual personalization—not just benchmark overfitting—we introduce three diagnostic downstream tasks that explicitly test memory recall from multimodal context.

Figure 4. Visualization of diagnostic personalization tasks. Each task explicitly precludes shortcut behaviors, requiring the model to ground visual input in user-specific contextual history.

CoViP consistently outperforms post-training baselines (RAP, RePIC) across all three tasks. CAG further amplifies performance by capitalizing on the fine-grained details of the generated personalized caption. Notably, proprietary VLMs show unstable CAG gains due to weaker captioning quality, reinforcing that personalized image captioning is a necessary prerequisite for reliable downstream personalization.

CoViP Enhances Retrieval, Not Just Recognition

A key question is what CoViP actually improves. Figure 5 analyzes the relationship between recognition (measured by F1 score of entity name inclusion in generated captions) and retrieval (measured by Acc⁺). Baseline models already achieve reasonable recognition capability (Avg F1 ≈ 0.810), yet their retrieval accuracy remains low—indicating that recognition alone is insufficient and that retrieval is the primary bottleneck.

CoViP addresses this gap directly. While the average F1 improves modestly (0.810 → 0.897), retrieval accuracy improves at a substantially larger margin for equivalent F1 levels, as shown by the steeper regression slope (m = 0.65 vs. 0.35 for Base and 0.39 for RePIC). This indicates that CoViP's gains are driven primarily by more effective integration of implicit personal cues through contextualized reasoning, rather than by improvements in recognition itself.

Figure 5. Scatter plot of recognition vs. retrieval on the proposed benchmark. Recognition is measured by the F1 score of entity name inclusion between generated captions and ground-truth dialogues; retrieval is measured by positive MCQA accuracy (Acc⁺). m denotes the slope of the linear regression line. CoViP's steeper slope (m=0.65) shows it converts recognition capability into retrieval far more effectively than Base (m=0.35) or RePIC (m=0.39).

BibTeX

If you find CoViP useful in your research, please cite our paper:

@article{oh2026contextualized,
  title     = {Contextualized Visual Personalization in Vision-Language Models},
  author    = {Oh, Yeongtak and Yu, Sangwon and Park, Junsung and
               Moon, Han Cheol and Mok, Jisoo and Yoon, Sungroh},
  journal   = {arXiv preprint arXiv:2602.03454},
  year      = {2026}
}