A benchmark and a two-phase RL method for personalized vision–language assistants that must recognize a visible hazard, retrieve the relevant personal fact and combine both in their response.
1Department of Electrical and Computer Engineering, Seoul National University · 2Interdisciplinary Program in AI, Seoul National University
DSAIL Lab · †Corresponding author
Content warning: this page shows synthetic images of hazardous situations, such as fire and injuries, that some readers may find distressing.
Personalized vision–language assistants use visual observations and personal memories to provide advice tailored to the user. When the observed scene contains a physical hazard, personal context, such as allergies, medical conditions, or mobility constraints, can change what help is appropriate. Responding appropriately therefore requires recognizing the visible hazard and applying the relevant person's information. We introduce PReMind-Bench, a diagnostic benchmark that evaluates this combination across diverse hazards and personal-memory conditions. It separately measures memory access, grounding, abstention, and hazard response, and assesses joint success on interdependency cases. Existing models often satisfy one requirement while failing the other, and training only for safety or personalization leaves a gap in joint success. We therefore propose PReMind, a two-phase reinforcement learning framework that combines on-demand and forced-call interactions (memory always retrieved) to jointly train grounded memory use and hazard response. At 4B, PReMind raises joint success from 12.9% to 28.6% over its base on the interdependency set and improves hazard adequacy and correct abstention on the main test. Joint-success gains persist under forced-call evaluation and under a second LLM judge across three Phase II seeds; averaged over these seeds, joint success also improves on unseen hazard groups. These findings highlight the need to evaluate and train personal assistants to connect visual evidence with the relevant person's context.
Call memory when the request needs it, ground the answer in the right person's fact, and abstain when the fact is missing.
Respond adequately to visible hazards without critical instructions, even without personal evidence.
The hazard response must also apply the target's fact, which only memory reveals: joint success.
Scenario map. Every request includes an image. (a) A single visual scene and its stored context support queries requiring different evidence. (b) Requests are categorized by required evidence: personal fact (PF), safety knowledge (SK), both, or scene only (SO). CALL requests require memory retrieval, whereas DIRECT requests do not. Situational risk (safe vs. unsafe) varies independently.
PReMind-Bench specifies what a safe personalized multimodal agent should do when a visible hazard meets personal memory, organizes requests by the evidence they need and the source of risk, and scores each behaviour separately, from the memory-call decision to joint success, on scenes and target persons unseen in training. A scene–user family combines an image, personal memory and related requests. Memory need, target evidence and physical risk vary independently: look-alikes test attribution, missing facts test withholding, and image-only requests and safe scenes reveal unnecessary calls and false alarms. The agent is not told which setting it faces.
People, pets and belongings are composited into real multi-view captures, preserving geometry.
Stores mix targets, same-type look-alikes and unrelated concepts, with different facts about the same action.
Requests omit identities, decisive facts and hazard cues; explicit and implicit forms.
Automated and model-assisted image checks; unresolved examples are removed. Every test row was checked manually.
Test-only cases change the target's memory while keeping the image, retrieval and hazard reference.
Benchmark statistics. (a) Hazard types in training/validation versus test; H2–H5 are unseen environmental hazards. (b) Acute hazard groups in the test set. (c) Decisive personal memory types across the interdependency cases.
Training and test use disjoint capture scenes and target persons; objects and pets may recur. Only the test split is public: 1,020 requests, each with its top-4 retrieved memory entries (CC BY-NC 4.0, gated access). The training and validation splits are available from the authors upon request for research use. The code repository provides the evaluation pipeline (inference, GPT-6-Luna judging and metrics) with the stored PReMind-4B outputs that reproduce the reported numbers, and the training code.
The agent first decides, from the image and request alone, whether to call personal memory; it then answers from the returned context, or from none. Both steps use the same policy.
PReMind is trained from Qwen3.5 with group-relative, DAPO-style reinforcement learning and LoRA. Training mixes on-demand episodes, where the model decides whether to call memory, with forced-call episodes, where the call is fixed and only the answer is trained, so evidence use is learned even when the policy would skip retrieval. Phase I trains memory access, grounding and hazard response, with a penalty against stock instructions repeated regardless of the scene. Phase II focuses on difficult memory-dependent hazards with a fact bonus, curriculum filtering, experience replay and action-prioritized hazard credit. At inference the model decides on demand.
JSR (%) on 140 interdependency cases (63 seen hazard type, H1; 77 unseen, H2–H5): an adequate, non-critical hazard response that applies the target's fact without misattribution under the benchmark rubric. All 140 cases retrieve the target within the top four. Memory supplied: retrieved entries given before a one-turn answer.
| Model | On-demand ↑ | Memory supplied ↑ | ||||
|---|---|---|---|---|---|---|
| All | Seen | Unseen | All | Seen | Unseen | |
| Closed-source models | ||||||
| Claude Opus 5.5 | 32.4 | 38.7 | 27.3 | 52.9 | 55.6 | 50.6 |
| GPT-6.1-Sol | 12.9 | 11.1 | 14.3 | 76.4 | 66.7 | 84.4 |
| GPT-6-Luna | 5.0 | 3.2 | 6.6 | 61.4 | 69.8 | 54.5 |
| Base model and PReMind | ||||||
| Qwen3.5-4B (base) | 12.9 | 20.6 | 6.5 | 15.7 | 17.5 | 14.3 |
| PReMind-4B | 28.6 | 46.0 | 14.3 | 35.3 | 42.9 | 28.9 |
PReMind-4B raises JSR from 12.9% to 28.6% over its base under the GPT-6-Luna judge (mean ± SD over three Phase II seeds: 27.4 ± 2.1%). Gains are larger on seen hazard groups; a second judge (Claude Sonnet 5.5) confirms the JSR gain across all three seeds, and it persists under forced-call evaluation. Bold: best.
On-demand main-test percentages with online top-four retrieval and greedy decoding.
| Model | Personalization safety | Situational safety | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Adaptive retrieval | Grounding | Abstention | Hazard response | ||||||
| Call BA ↑ | Cite F1 ↑ | Fact use ↑ | Misattr. ↓ | Correct ↑ | Needless ↓ | Adequacy ↑ | Crit. ↓ | MIE ↑ | |
| Closed-source models | |||||||||
| Claude Opus 5.5 | 87.3 | 69.5 | 55.6 | 2.3 | 48.7 | 2.7 | 77.8 | 13.0 | 68.2 |
| GPT-6.1-Sol | 75.4 | 52.8 | 39.5 | 0.4 | 81.6 | 6.3 | 98.4 | 0.8 | 43.0 |
| GPT-6-Luna | 61.0 | 18.9 | 13.1 | 0.8 | 34.7 | 1.6 | 87.8 | 2.4 | 11.2 |
| Open-source models | |||||||||
| Qwen3.5-9B | 75.3 | 50.4 | 40.7 | 4.3 | 15.3 | 2.3 | 43.1 | 9.8 | 17.8 |
| Qwen3-VL-8B | 55.8 | 17.0 | 10.2 | 1.2 | 26.8 | 1.1 | 31.2 | 10.6 | 8.4 |
| InternVL3.5-4B | 61.0 | 40.1 | 14.9 | 10.8 | 13.2 | 6.5 | 9.5 | 6.3 | 6.5 |
| InternVL3.5-8B | 56.4 | 17.9 | 10.0 | 7.0 | 18.2 | 1.9 | 22.0 | 7.1 | 4.7 |
| GLM-4.6V-Flash | 66.2 | 74.2 | 44.0 | 19.2 | 18.9 | 8.2 | 20.4 | 6.3 | 15.0 |
| Personalized models | |||||||||
| CoViP-8B | 67.1 | 44.1 | 29.4 | 5.7 | 17.4 | 0.2 | 22.5 | 6.1 | 15.9 |
| RePIC-7B | 50.8 | 39.9 | 19.8 | 28.8 | 3.7 | 0.5 | 14.8 | 7.1 | 9.3 |
| Base model and ours | |||||||||
| Qwen3.5-4B (base) | 80.4 | 56.2 | 41.7 | 3.1 | 26.1 | 7.9 | 34.4 | 9.5 | 24.3 |
| PReMind-4B | 85.9 | 83.2 | 58.5 | 1.2 | 55.8 | 1.6 | 56.9 | 8.7 | 54.2 |
Call BA: balanced accuracy of the memory-call decision. Cite F1 / Fact use: target citation and fact use; Misattr.: misattribution to a look-alike. Correct: correct abstention when the target's fact is missing; Needless: abstention where the evidence is present or not needed. Adequacy: a specific, non-contraindicated acute response; Crit.: judged likely-serious harmful instruction; MIE: the target's emergency fact changes the response. Bold / underline: best / second best.
Scene and personal-memory examples. Each row shows a query image, the user's request, the retrieved personal memory and response snippets. PReMind-4B handles the visible hazard and adapts its advice to the person's fact: it keeps a brother with low blood pressure lying down, does not have a friend with an ankle injury stand or walk, and moves a colleague with asthma away from the fumes and helps him use his reliever. Comparison answers either mishandle the hazard (personalization-focused answers that use the memory, or an unsafe action) or handle the hazard but ignore the memory (safety-focused).
PReMind-Bench evaluates grounded personal-memory use and visual hazard response together, with separate metrics that locate where each model fails. Compared models often meet only one requirement.
PReMind is a two-phase RL framework that mixes on-demand memory access with forced-call interactions to jointly reward hazard handling and memory use.
PReMind-4B raises joint success from 12.9% to 28.6% over its base and improves call accuracy, correct abstention and hazard adequacy; the JSR gain holds under a second judge, across seeds and under forced-call evaluation.
PReMind-Bench and PReMind-4B are a first step. A personal agent that people can rely on must remember who they are and still put their safety first, in more places, over longer time, and against more ways of failing. These are the directions we hope to pursue next.
Most interdependency cases are still failed, mostly because the fact is retrieved but not applied, and PReMind flags more safe scenes as unsafe than its base. Closing this gap is the first goal.
Extend from household scenes to settings such as industrial sites, and from English requests and guidance to other languages and regional first-aid practice.
Move from single images to continuous first-person video, where hazards unfold over time and the agent must decide when to speak up, not only what to say.
Personal memory grows and changes. Agents should keep up with it continually and run on the user's own device, so that personal facts stay with the person they belong to.
Memory is also an attack surface: deliberately harmful entries could steer an agent toward unsafe behavior. Red-teaming such attacks, and safeguarding edited images of people against misuse, are part of making personal agents safe.
Our goal is a personal agent that knows the people it serves and keeps them safe when that knowledge matters most.
Purpose. PReMind aims to help people who need assistance because of a medical condition, a prior prescription or another personal circumstance, by letting an assistant adapt its safety response to what it knows about them.
Research use only. The benchmark and models are released for non-commercial research. Any use outside research, and any malicious use against other people (for example, inferring, exposing or exploiting someone's health information or vulnerabilities), is prohibited.
Not medical advice. Model outputs are not a substitute for professional medical care. In an emergency, contact local emergency services.
If you find PReMind useful in your research, please cite our paper:
@misc{premind,
title = {PReMind: When Personal Memory Meets Visual Hazards},
author = {Yeongtak Oh and Donghyeok Lee and Joonhee Oh and Jaewon Jeong and Sungroh Yoon},
note = {Preprint},
year = {2026}
}
Reinforced post-training for personalizing multimodal language models, with a focus on personalized and multi-concept image captioning.
Contextualized visual personalization: interpreting new images through a user's accumulated image–text context, trained with reinforcement learning.