ACL 2024 (Main Conference)

MTP: A Dataset for Multi-Modal Turning Points in Casual Conversations

Gia-Bao Dinh Ho1 Chang Wei Tan2 Zahra Zamanzadeh Darban2 Mahsa Salehi2 Gholamreza Haffari2 Wray Buntine1
1VinUniversity
2Monash University
Paper arXiv Code Dataset Poster BibTeX
Five frames from a conversation; the middle frame, where Penny says she fell in love with a jerk, is marked as the turning point.
Emotional outbursts change the other speakers' states. The marked frame is the turning point: the mood of the scene shifts from the line before it to the line after.

Abstract

Detecting critical moments, such as emotional outbursts or changes in decisions during conversations, is crucial for understanding shifts in human behavior and their consequences. Our work introduces a novel problem setting that focuses on these moments as "turning points (TPs)," accompanied by a meticulously curated, high-consensus, human-annotated multi-modal dataset. We provide precise timestamps, descriptions, and visual-textual evidence highlighting changes in emotions, behaviors, perspectives, and decisions at these turning points. Additionally, we propose a framework, TPMaven, which utilizes state-of-the-art vision-language models to construct a narrative from the videos and large language models. Evaluation results show that TPMaven achieves an F1-score of 0.88 in classification and 0.61 in detection, with additional explanations aligning with human expectations.

Problem formulation: six utterances from a conversation; the fifth is the turning point, caused by Penny suddenly mentioning her ex, with each speaker's feeling, behaviour, perspective and decision before and after it.
A turning point. Penny suddenly mentions her ex (utterance i+4). Before it, both speakers are happy and chatting; after it, Penny is sad and Leonard is confused. Each turning point comes with a timestamp, its cause, and the change in feeling, behaviour, perspective and decision for every speaker.

Sample conversations

Conversation 1
Conversation 4

Dataset statistics

StatisticValue
Videos
Conversation videos340
Total duration (h)13.3
Utterance-level videos12,351
Turning-point videos214
Average length of conversation videos (s)1.9
Maximum length of conversation videos (m)2.5
Transcripts
Words in all transcripts81,909
Average length of conversation transcripts241.5
Maximum length of conversation transcripts460

Dataset access

The full dataset has 340 conversations, about 13.3 hours of video, with utterance-level videos, transcripts, speaker IDs and turning-point annotations.

  1. Sign the Data Usage Agreement.
  2. Email it to dinhhogiabao@gmail.com with the subject "Request Access for the MTP Dataset".
  3. We reply with the link to the full dataset.

Sample files are available here.

BibTeX

@inproceedings{ho-etal-2024-mtp,
    title = "{MTP}: A Dataset for Multi-Modal Turning Points in Casual Conversations",
    author = "Ho, Gia-Bao Dinh and Tan, Chang Wei and Darban, Zahra Zamanzadeh and
              Salehi, Mahsa and Haffari, Gholamreza and Buntine, Wray",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for
                 Computational Linguistics (Volume 2: Short Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-short.30",
    pages = "314--326",
}