Creating and sharing knowledge for telecommunications

Project: A User-Centered Benchmark for Usability Assessment of Speech Translation Systems

Acronym: Ouvia
Main Objective:
This project aims to address disparities in speech translation (ST) technologies across different demographic groups by:

1. Creating a new multilingual speech translation benchmark that evaluates both technical performance and real-world usability
2. Investigating how ST tools perform across different sociodemographic groups, particularly focusing on gender and ethnicity
3. Simulating both high-stakes and low-stakes interaction scenarios
4. Collecting 3000 segments (25 hours) of English→Portuguese translations with quality and usability assessments from 450+ diverse participants

The research specifically examines whether users find ST tools usable in real scenarios, how well automatic metrics reflect actual usability, and whether there are systematic performance gaps across demographic groups.
Reference: Non Applicable (NA)
Funding: European Association for Machine Translation
Start Date: 02-05-2025
End Date: 02-05-2026
Team: Giuseppe Attanasio, André Filipe Torres Martins
Groups: Pattern and Image Analysis – Lx
Partners: Google
Local Coordinator: Giuseppe Attanasio

Associated Publications