Creating and sharing knowledge for telecommunications

DataWe: A neurocinematic system for studying collective decision-making


on 04-11-2019

... Researchers from the Pattern and Image Analysis group from IT in Lisbon (PIA-Lx), in collaboration with Tallinn University (TLU) and European Commission Joint Research Centre (JRC), have developed an audience psychophysiological assessment system, which was recently deployed at the European Commission JRC Resonances III Art-Science Festival, in the scope of the DataWe project.

Lead by Professor Aleksander Väljamäe from TLU & tunisian artist Amal Mokded, DataWe is a neurocinematics show, where the participants’ psychophysiological reactions influence the narrative, audio and visual media of a film. This particular performance, focused on the struggles of an European immigrant, to which the “neuronauts” (i.e. audience) reactions are measured using a novel wearable and unobtrusive Electrodermal Activity (EDA) sensor. This device, especially suitable for collective data acquisition in real time, measures the emotional arousal throughout the screening. Then, from the individual reactions, a collective affective waveform of the neuronauts is computed.

In its simplest form, the affective waveform can be determined as a weighted average of the amplitude values from each individual participant, normalized with respect to a calibration period. However, ongoing work is exploring the use of machine learning to determine the optimal feature set based on experimental datasets in which ground truth annotations are available.

In one mode of operation, DataWe uses the affective waveform to direct the audio-visual content to be shown, with the purpose of inducing emotional arousal to the audience through dynamically adapted content. In another mode of operation, DataWe detects strong emotional-manipulative connotation media, using features extracted from the affective waveform to serve as a demonstration system, warning against these media technologies and raising awareness about the challenge of processing media in an unbiased fashion, by highlighting and presenting to the neuronauts the segments of the screening in which the strongest responses were detected.

This was an action developed under project S-0003-LX-18 XiPhy, Xinhua Net Future Media Convergence Institute (FMCI) scientific research collaboration.


Photo: Electrodermal Activity (EDA) sensor