Preprint · 2026

PReMind: When Personal Memory
Meets Visual Hazards

A benchmark and a two-phase RL method for personalized vision–language assistants that must recognize a visible hazard, retrieve the relevant personal fact and combine both in their response.

Yeongtak Oh1, Donghyeok Lee1, Joonhee Oh1, Jaewon Jeong2, Sungroh Yoon1,2,†

1Department of Electrical and Computer Engineering, Seoul National University  ·  2Interdisciplinary Program in AI, Seoul National University
DSAIL Lab  ·  †Corresponding author

Paper Code PReMind-Bench (HF) PReMind-4B (HF)

Content warning: this page shows synthetic images of hazardous situations, such as fire and injuries, that some readers may find distressing.

When Personal Memory Meets Visual Hazards

Personalized vision–language assistants recognize the people a user knows and answer with what they know about them. Personalization research, however, has rarely examined safety-sensitive situations in which the right response depends jointly on what the image shows now and on a personal fact that only memory holds.

Two cases of a safe personal agent that combines visual evidence with personal memory: a fire scene where the agent identifies the user rather than a look-alike and puts evacuation and 911 before the emergency contact, and a collapse scene where the agent recognises Mira rather than Nora and tells the dispatcher that Mira is 8 weeks pregnant

A safe personal agent combines visual evidence with personal memory. Case 1: the user asks for help while a fire is visible behind him. The agent first matches the person in view to the user's own memory entry, not to a look-alike's, which lists a different emergency contact. A personalization-only answer just returns the emergency contact and overlooks the fire; the safe agent says to evacuate first, then call 911, and only then call the emergency contact. Case 2: a woman lies collapsed on the floor. The agent recognizes her as Mira, who is 8 weeks pregnant, rather than the similar-looking Nora, who has seasonal allergies. A safety-only answer calls emergency services but overlooks this hidden personal context; the safe agent checks responsiveness, seeks emergency help and tells the dispatcher that Mira is 8 weeks pregnant.

Overview

Personalized vision–language assistants use visual observations and personal memories to provide advice tailored to the user. When the observed scene contains a physical hazard, personal context, such as allergies, medical conditions, or mobility constraints, can change what help is appropriate. Responding appropriately therefore requires recognizing the visible hazard and applying the relevant person's information. We introduce PReMind-Bench, a diagnostic benchmark that evaluates this combination across diverse hazards and personal-memory conditions. It separately measures memory access, grounding, abstention, and hazard response, and assesses joint success on interdependency cases. Existing models often satisfy one requirement while failing the other, and training only for safety or personalization leaves a gap in joint success. We therefore propose PReMind, a two-phase reinforcement learning framework that combines on-demand and forced-call interactions (memory always retrieved) to jointly train grounded memory use and hazard response. At 4B, PReMind raises joint success from 12.9% to 28.6% over its base on the interdependency set and improves hazard adequacy and correct abstention on the main test. Joint-success gains persist under forced-call evaluation and under a second LLM judge across three Phase II seeds; averaged over these seeds, joint success also improves on unseen hazard groups. These findings highlight the need to evaluate and train personal assistants to connect visual evidence with the relevant person's context.

Using the Memory and Handling the Hazard

Personalization safety

Call memory when the request needs it, ground the answer in the right person's fact, and abstain when the fact is missing.

Situational safety

Respond adequately to visible hazards without critical instructions, even without personal evidence.

Interdependency

The hazard response must also apply the target's fact, which only memory reveals: joint success.

Scenario map: one kitchen scene with stored personal facts supports requests that need a personal fact, safety knowledge, both, or the scene only

Scenario map. Every request includes an image. (a) A single visual scene and its stored context support queries requiring different evidence. (b) Requests are categorized by required evidence: personal fact (PF), safety knowledge (SK), both, or scene only (SO). CALL requests require memory retrieval, whereas DIRECT requests do not. Situational risk (safe vs. unsafe) varies independently.

PReMind-Bench

PReMind-Bench specifies what a safe personalized multimodal agent should do when a visible hazard meets personal memory, organizes requests by the evidence they need and the source of risk, and scores each behaviour separately, from the memory-call decision to joint success, on scenes and target persons unseen in training. A scene–user family combines an image, personal memory and related requests. Memory need, target evidence and physical risk vary independently: look-alikes test attribution, missing facts test withholding, and image-only requests and safe scenes reveal unnecessary calls and false alarms. The agent is not told which setting it faces.

1K
test requests
140
interdependency cases
77
interdependency cases with hazard types unseen in training
top-4
online memory retrieval after a call

Construction

1

Scenes

People, pets and belongings are composited into real multi-view captures, preserving geometry.

2

Memories

Stores mix targets, same-type look-alikes and unrelated concepts, with different facts about the same action.

3

Requests

Requests omit identities, decisive facts and hazard cues; explicit and implicit forms.

4

Checks

Automated and model-assisted image checks; unresolved examples are removed. Every test row was checked manually.

5

Interdependency

Test-only cases change the target's memory while keeping the image, retrieval and hazard reference.

Benchmark statistics: hazard types in training and test, acute hazards in the test split, and decisive memory types of the 140 interdependency cases

Benchmark statistics. (a) Hazard types in training/validation versus test; H2–H5 are unseen environmental hazards. (b) Acute hazard groups in the test set. (c) Decisive personal memory types across the interdependency cases.

Training and test use disjoint capture scenes and target persons; objects and pets may recur. Only the test split is public: 1,020 requests, each with its top-4 retrieved memory entries (CC BY-NC 4.0, gated access). The training and validation splits are available from the authors upon request for research use. The code repository provides the evaluation pipeline (inference, GPT-6-Luna judging and metrics) with the stored PReMind-4B outputs that reproduce the reported numbers, and the training code.

PReMind: Decide, Then Respond

The agent first decides, from the image and request alone, whether to call personal memory; it then answers from the returned context, or from none. Both steps use the same policy.

PReMind is trained from Qwen3.5 with group-relative, DAPO-style reinforcement learning and LoRA. Training mixes on-demand episodes, where the model decides whether to call memory, with forced-call episodes, where the call is fixed and only the answer is trained, so evidence use is learned even when the policy would skip retrieval. Phase I trains memory access, grounding and hazard response, with a penalty against stock instructions repeated regardless of the scene. Phase II focuses on difficult memory-dependent hazards with a fact bonus, curriculum filtering, experience replay and action-prioritized hazard credit. At inference the model decides on demand.

Joint Success on Interdependency Cases

JSR (%) on 140 interdependency cases (63 seen hazard type, H1; 77 unseen, H2–H5): an adequate, non-critical hazard response that applies the target's fact without misattribution under the benchmark rubric. All 140 cases retrieve the target within the top four. Memory supplied: retrieved entries given before a one-turn answer.

ModelOn-demand ↑Memory supplied ↑
AllSeenUnseenAllSeenUnseen
Closed-source models
Claude Opus 5.532.438.727.352.955.650.6
GPT-6.1-Sol12.911.114.376.466.784.4
GPT-6-Luna5.03.26.661.469.854.5
Base model and PReMind
Qwen3.5-4B (base)12.920.66.515.717.514.3
PReMind-4B28.646.014.335.342.928.9

PReMind-4B raises JSR from 12.9% to 28.6% over its base under the GPT-6-Luna judge (mean ± SD over three Phase II seeds: 27.4 ± 2.1%). Gains are larger on seen hazard groups; a second judge (Claude Sonnet 5.5) confirms the JSR gain across all three seeds, and it persists under forced-call evaluation. Bold: best.

Memory access, grounding and acute response

On-demand main-test percentages with online top-four retrieval and greedy decoding.

ModelPersonalization safetySituational safety
Adaptive retrievalGroundingAbstentionHazard response
Call BA ↑Cite F1 ↑Fact use ↑Misattr. ↓Correct ↑Needless ↓Adequacy ↑Crit. ↓MIE ↑
Closed-source models
Claude Opus 5.587.369.555.62.348.72.777.813.068.2
GPT-6.1-Sol75.452.839.50.481.66.398.40.843.0
GPT-6-Luna61.018.913.10.834.71.687.82.411.2
Open-source models
Qwen3.5-9B75.350.440.74.315.32.343.19.817.8
Qwen3-VL-8B55.817.010.21.226.81.131.210.68.4
InternVL3.5-4B61.040.114.910.813.26.59.56.36.5
InternVL3.5-8B56.417.910.07.018.21.922.07.14.7
GLM-4.6V-Flash66.274.244.019.218.98.220.46.315.0
Personalized models
CoViP-8B67.144.129.45.717.40.222.56.115.9
RePIC-7B50.839.919.828.83.70.514.87.19.3
Base model and ours
Qwen3.5-4B (base)80.456.241.73.126.17.934.49.524.3
PReMind-4B85.983.258.51.255.81.656.98.754.2

Call BA: balanced accuracy of the memory-call decision. Cite F1 / Fact use: target citation and fact use; Misattr.: misattribution to a look-alike. Correct: correct abstention when the target's fact is missing; Needless: abstention where the evidence is present or not needed. Adequacy: a specific, non-contraindicated acute response; Crit.: judged likely-serious harmful instruction; MIE: the target's emergency fact changes the response. Bold / underline: best / second best.

Examples

Three test cases (collapse with low blood pressure, fall with an ankle injury, toxic fumes with asthma): query image, user request, retrieved personal memory and response snippets from PReMind-4B and comparison models

Scene and personal-memory examples. Each row shows a query image, the user's request, the retrieved personal memory and response snippets. PReMind-4B handles the visible hazard and adapts its advice to the person's fact: it keeps a brother with low blood pressure lying down, does not have a friend with an ankle injury stand or walk, and moves a colleague with asthma away from the fumes and helps him use his reliever. Comparison answers either mishandle the hazard (personalization-focused answers that use the memory, or an unsafe action) or handle the hazard but ignore the memory (safety-focused).

Contributions

1Benchmark

PReMind-Bench evaluates grounded personal-memory use and visual hazard response together, with separate metrics that locate where each model fails. Compared models often meet only one requirement.

2Method

PReMind is a two-phase RL framework that mixes on-demand memory access with forced-call interactions to jointly reward hazard handling and memory use.

3Results

PReMind-4B raises joint success from 12.9% to 28.6% over its base and improves call accuracy, correct abstention and hazard adequacy; the JSR gain holds under a second judge, across seeds and under forced-call evaluation.

Toward Personal Agents That Stay Safe

PReMind-Bench and PReMind-4B are a first step. A personal agent that people can rely on must remember who they are and still put their safety first, in more places, over longer time, and against more ways of failing. These are the directions we hope to pursue next.

Roadmap for safe personal agents
  1. 01
    Now · open problems

    Close the remaining joint failures

    Most interdependency cases are still failed, mostly because the fact is retrieved but not applied, and PReMind flags more safe scenes as unsafe than its base. Closing this gap is the first goal.

    A person cooking near a fire while the agent links the hazard to her memory
  2. 02
    Beyond the home and beyond English

    More places, languages and practices

    Extend from household scenes to settings such as industrial sites, and from English requests and guidance to other languages and regional first-aid practice.

    A worker at an industrial site with the agent giving first-aid guidance
  3. 03
    From images to streams

    Egocentric video

    Move from single images to continuous first-person video, where hazards unfold over time and the agent must decide when to speak up, not only what to say.

    Egocentric video frames showing a hazard developing over time
  4. 04
    Living memory

    On-device agents with continual adaptation

    Personal memory grows and changes. Agents should keep up with it continually and run on the user's own device, so that personal facts stay with the person they belong to.

    A phone at the center of continually updated personal memory
  5. 05
    Trust

    Memory that cannot be turned against the user

    Memory is also an attack surface: deliberately harmful entries could steer an agent toward unsafe behavior. Red-teaming such attacks, and safeguarding edited images of people against misuse, are part of making personal agents safe.

    A shield protecting personal memory against a tampered entry

Our goal is a personal agent that knows the people it serves and keeps them safe when that knowledge matters most.

Purpose. PReMind aims to help people who need assistance because of a medical condition, a prior prescription or another personal circumstance, by letting an assistant adapt its safety response to what it knows about them.

Research use only. The benchmark and models are released for non-commercial research. Any use outside research, and any malicious use against other people (for example, inferring, exposing or exploiting someone's health information or vulnerabilities), is prohibited.

Not medical advice. Model outputs are not a substitute for professional medical care. In an emergency, contact local emergency services.

BibTeX

If you find PReMind useful in your research, please cite our paper:

@misc{premind,
  title  = {PReMind: When Personal Memory Meets Visual Hazards},
  author = {Yeongtak Oh and Donghyeok Lee and Joonhee Oh and Jaewon Jeong and Sungroh Yoon},
  note   = {Preprint},
  year   = {2026}
}

Our Earlier Personalization Projects

NeurIPS 2025

RePIC

Reinforced post-training for personalizing multimodal language models, with a focus on personalized and multi-concept image captioning.

ICML 2026

CoViP

Contextualized visual personalization: interpreting new images through a user's accumulated image–text context, trained with reinforcement learning.

Preprint 2026

Omni-Persona

A benchmark for omnimodal personalization across image, text and audio, including queries about personas that are absent from memory.