Back to Bigger Picture

BIGGER PICTURE Accessibility & independence

A photograph becomes a conversation you can steer

A fixed image description cannot anticipate every question a blind person might have. Be My AI lets someone ask for the particular detail that matters to them, then ask again.

AI-generated illustration: A photographed everyday scene becomes an interactive description.
A photographed everyday scene becomes an interactive description.AI-generated illustration of the idea, not a photograph of the actual people, equipment or event.
Interaction
Photo, description, follow-up questions
User research
14 visually impaired interview participants
Human fallback
A volunteer remains reachable

How the idea works

  1. Share a view

    A phone photo or shared image supplies visual context.

  2. Ask about a detail

    The user directs the description with follow-up questions.

  3. Read, listen or verify

    Accessible text conveys the response; human help can resolve doubts.

Access includes ordinary curiosity

Understanding an outfit, a room or a shared photograph can matter as much as reading a label. A study of fourteen visually impaired users documented such everyday uses of Be My AI. Participants valued being able to explore a picture without always requesting another person’s description.

Sources: [1]

The useful part is the second question

The app sends a photo to a vision-language model and returns a description compatible with screen readers. Its follow-up function accepts a specific question or another picture. Someone can move from a broad overview to a clothing colour or a detail in a personal photograph.

Sources: [2]

The user supplies the purpose

The interview study found that users actively directed the system when general descriptions missed their goal. That interaction is the benefit: control over which visual information to explore. It is also work, especially when the system must be repeatedly redirected or checked against familiar surroundings.

Sources: [1]

A shorter answer can be the better assistance

A separate Maryland study analysed 418 interactions and found that lengthy replies often buried the useful detail. Human volunteers remain available in the app. These findings point to a concrete next improvement: prioritise the person’s question and make uncertainty easy to resolve, instead of merely describing more.

Sources: [2][3]

The original sources

  1. CHI 2025 / author manuscriptBeyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Models
  2. Be My Eyes Help CenterUsing Be My AI
  3. University of MarylandApp’s AI ‘Eyes’ for Blind Users Fall Short, UMD Researchers Find

What to explore next

A shop conversation becomes readable on a phoneFinding damaged buildings before every roof is checked
Explore all 99 applications