Trope Dataset

Introduction

What is trope ?

A trope is a storytelling device, or a shortcut, frequently used in creative productions such as novels, TV series and movies to describe situations that storytellers can reasonably assume the audience will recognize. Beyond actions, events, and activities, they are the tools that the art creators use to express ideas to the audience without needing to spell out all the details.

For example, Heroic Sacrifice is when a character saves another/others from harm and is killed, crippled, or maimed as a result. Asshole Victim is normally when something bad happens to an individual you feel some degree of pity for them.

Deep Cognition

Trope is hide in story, named implicit meaning, in some kind of multimedia, it cannot be easily recognized from the surface.

Multi-Modal

TrUMAn consists 2423 videos with audio, and has total 132 tropes in 8 categories. It's one of fancy dataset for trope detection.

More semantics

Helping future learning AI system to be more human-like or more intelligent. As the proverb goes "It is much easier to earn a lot of money than to find a bosom friend". AI needs to understand the human's heart.

Downloads

{
  “train”:[
    {
      "data-video-sub": "The detected subtitles of the video",
      "data-video-tropename": "The trope name",
      "data-video-descrip": "The human-written description of the video",
      "data-video-name": "The video name"
    },
    ...
  ],
  “val”:[
    // The structure here is same as “train” show in above
  ],
  “test”:[
    // The structure here is same as “train” show in above
  ]
}

2. Video features

Resnet feature [TVtrope_resnet101.zip]

TVtrope_resnet101.zip file contains a HDF5 file, The 2048D features are extracted using imagenet pretrained resnet-101 model. For each video, we extracted at 3fps, and downsampled to 0.5 fps in our experiments. If the number of the downsampled frames exceeds 100, we simply truncate the exceeded frames.

S3D feature [truman_s3d.zip]

truman_s3d.zip file contains a HDF5 file, The 1024D features are extracted using Kinetics pretrained S3D model. For each video, we extracted at 1fps, and downsampled to 0.5 fps in our experiments. If the number of the downsampled frames exceeds 100, we simply truncate the exceeded frames.

3. Audio features

Soundnet feature [TVtrope_soundnet.zip]

TVtrope_soundnet.zip file contains a HDF5 file, The feature size is 1024D extracted from soundnet. For those audio lengths exceeding 256, we simply truncate them.

4. Object feature

Please follow the instructions here to do the extraction. Currently, we do not plan to release it, due to its size.

Explore

Annotation

Category:

Scene Identification

Trope:

Shock and Awe

Description:

One of Spider-Man's longtime enemies, Maxwell "Max" Dillon (AKA Electro) has the power to produce and manipulate electricity.

Papers

TrUMAn: Trope Understanding in Movies and Animations

Hung-Ting Su*, Po-Wei Shen*, Bing-Chen Tsai, Wen-Feng Cheng, Ke-Jyun Wang, Winston H. Hsu

The Conference on Information and Knowledge Management (CIKM) 2021

References

TVTrope website
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. 2018. Rethinking Spatio temporal Feature Learning: Speed-Accuracy Tradeoffs in Video Classification. In Proceedings of the European Conference on Computer Vision (ECCV)
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. 2016. SoundNet: Learning Sound Representations from Unlabeled Video. In Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS’16)
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R -CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume 1 (NIPS’15)