Jaehoon, a researcher of people × AI.

Multimodal Large Language Models (MLLMs) are increasingly deployed as social agents, yet their ability to integrate conflicting identity cues remains underexplored. We audit gender bias in ten recent MLLMs using a counterfactual cooperative gaming task that pairs synthetic voices with avatars of varying gender presentation and visual fidelity. Our analysis reveals distinct bias patterns that can occur independently: closed-source models (eg, Gemini 2.5/3) exhibit a near-deterministic “voice-matching” bias that enforces binary alignment between voice and appearance, whereas open-weight models (eg, Qwen-2.5-Omni-7B) show limited responsiveness to vocal cues and instead exhibit context-driven stereotypes, such as preferring male avatars in combat scenarios. We further find that reducing visual realism attenuates matching tendencies in some models. These findings demonstrate that multimodal fairness is not monolithic; models may appear unbiased on one dimension while enforcing strict identity congruence or role-based stereotypes on another. Code and data are available at https://github.com/halfhoon/whose-voice-whose-avatar.

Modern media consumption habits challenge ‘lean-back’ viewers—who prefer passive viewing—to stay engaged during the frequent periods of Non-Event Time in soccer matches. Existing commentary options often fail this audience, being either too dry or too interactive. To investigate their needs, we developed ARUA, a prototype that positions users as ‘directors’ of their own AI commentary. This approach serves as a probe to understand their preferences for a more engaging viewing experience. In a qualitative study with 32 fans, we found users craft the commentary into a relational tool, tailoring its social presence and emotional tone to maintain engagement. They created proxy voices for their own emotions and curated balanced conversational panels. Our work contributes an understanding of lean-back viewers, introduces a user-directed paradigm for personalized media, and provides design principles for creating engaging, low-effort experiences that support control over social presence, emotional tone, and cognitive load.

Early-career employees are particularly vulnerable to workplace stress due to psychosocial stressors such as high work demands and low job control. Despite the importance of addressing these stressors, existing digital interventions for early-career employees focus on information-oriented support like onboarding. This study presents an attribution reframing chatbot (AR chatbot) for workplace stress management among early-career employees. Attribution reframing offers a cognitive intervention point of psychosocial stressors by shifting how individuals interpret stress causes along locus and controllability dimensions. Following formative interviews with 11 early-career employees exploring stress situations and interpretation patterns, we evaluated the AR chatbot with 14 early-career employees across two common stress situations. In work-related situation, stress and controllability significantly improved alongside confidence and readiness, whereas in interpersonal situation, only confidence showed improvement. This study contributes to a deeper understanding of workplace stress among early-career employees and offers design implications for supporting early-career employees with the AR chatbot.

The transition to end-to-end Multimodal Large Language Models (MLLMs) has positioned these architectures as active social evaluators in high-stakes domains. However, it remains unclear whether these models maintain objective auditory perception or succumb to the "Hearing with Eyes" phenomenon, where visual racial cues distort linguistic proficiency evaluations. We investigate this cross-modal bias by constructing a controlled counterfactual dataset utilizing a Visual Matched-Guise Paradigm. By pairing identical native audio with diverse visual personas across English and Korean contexts, we reveal a distinct Cultural Asymmetry in model behavior. In Anglophone settings, most closed models exhibit Reverse Linguistic Stereotyping, hallucinating non-native accents for Asian speakers despite standard native audio. Conversely, in Korean settings, the same models assign baseline-relative competence premiums across all visual personas, with the largest gains for out-group (White/Black) speakers, consistent with Expectancy Violation Theory. Our findings demonstrate that MLLMs do not merely process sensory inputs but actively reproduce context-dependent sociolinguistic ideologies.

Recent Large Audio-Language Models (LALMs) integrate acoustic capabilities into reasoning, yet whether they reliably ground clinical judgments in audible evidence remains unproven. We introduce CliniCAST (Clinical Controlled Acoustic Synthetic Triage), a controlled benchmark that disentangles clinically meaningful acoustic cues from lexical content and speaker demographics. CliniCAST comprises 5,856 synthetic samples across 12 disease conditions: 4,800 audio samples forming 2,400 tagged–untagged pairs for five-level emergency triage, and 1,056 audio–text inconsistent samples in which reassuring speech is paired with high-risk acoustic cues. Evaluating a diverse suite of audio-capable foundation models, we find that LALMs exhibit fragile acoustic grounding and a pronounced “text dominance” failure mode: reassuring lexical content suppresses response to audible distress signals even under safety-critical conditions. Age and gender interactions are weak across conditions, indicating that the primary failure mode is insufficient cross-modal integration rather than demographic bias. These results suggest current LALMs are not yet robust enough for high-stakes medical triage, and motivate training objectives that explicitly enforce reliance on clinically grounded audible evidence.

Although Voice-based Conversational Agents (VCAs) continue to evolve, achieving natural interaction remains challenging. While studies have explored various aspects of conversational style, the impact of Pragmatic Markers (PMs)—linguistic elements that convey speaker intention and emotional context—remains underexplored in human-agent interactions. We conducted a formative study with eight participants using a prototype VCA system, which revealed that real-time sentiment analysis could support more effective PM usage by maintaining conversational context. Building on these insights, we developed a VCA system integrating sentiment-based PM selection and conducted a main study with 20 participants using a 2×2 within-subjects design (emotional/work-related topics × with/without markers). Analysis of conversation logs, surveys, and interviews revealed that moderate PM usage enhanced perceived naturalness and emotional engagement. However, excessive use negatively impacted user experience as participants expected VCAs to maintain a passive yet attentive listener role. Our findings inform design guidelines for context-aware PM integration in VCAs.

Ensuring visual content accessible on websites is essential for users with visual impairments. However, existing alt text authoring tools often provide alt text only in the primary language or use literal machine translations for other languages, often failing to capture cultural nuances and resulting in inaccurate or misleading descriptions. To overcome this limitation, we introduce AltCAT, the first Alt text authoring tool for web developers that incorporates Culturally-Aware Translation. AltCAT integrates culture-specific knowledge and region-appropriate image description conventions into alt text translations. Our tool consists of two main components: an Alt Text Processing Engine that detects images and alt text on a webpage, generates high-quality alt text, and provides culturally-aware translations using large multimodal models, and an Interactive Authoring Interface for refinement and HTML export. Our key contributions are introducing culturally-aware alt text translation and a practical authoring tool for multilingual web accessibility.

Web accessibility remains a persistent challenge, particularly in providing appropriate alternative text (alt text) for images that meet the needs of people with visual impairments. Existing tools often generate generic descriptions without taking into account image types and webpage context, leading to alt text that fails to align with established accessibility guidelines. To overcome these limitations, we present AltAuthor, an authoring tool designed to assist developers in managing alt text on existing websites. AltAuthor consists of three key components: (1) Image Classifier, which identifies visual content and its type, (2) Alt Text Generator, leveraging a Large Multimodal Model to generate compliant descriptions by incorporating identified image type, webpage context, and Section 508 accessibility guidelines, and (3) Interactive Authoring Interface, allowing developers to customize automated suggestions. Through this integrated approach, AltAuthor provides developers with tools to enhance accessibility management.