Google Group | Create an Issue | Join Discord | Contact Us

• Bridging the Gap in Text-Based Emotion Detection
• Languages
• Tracks
  • Track A: Multi-label Emotion Detection
  • Track B: Emotion Intensity
  • Track C: Cross-lingual Emotion Detection
• Dataset and Download Links
• Evaluation
• Important Dates and Phases of the Task
• How to Participate
• Competition Rules and Terms
• Dataset paper
• Communication
• FAQs
• Resources
• References
• Organizers

Emotions are simultaneously familiar and mysterious. On the one hand, we all express and manage our emotions every day. Yet, on the other hand, emotions are complex, nuanced, and sometimes hard to articulate. We also use language in subtle and complex ways to express emotion (Wiebe et al. 2005, Mohammad and Kiritcheko 2018, Mohammad et al. 2018). Further, people are highly variable in how they perceive and express emotions (even within the same culture or social group). Thus, we can never truly identify how one is feeling based on something that they have said with absolute certainty.

Emotion recognition is not one task but an umbrella term for several tasks such as detecting the emotions of the speaker, identifying what emotion a piece of text is conveying and detecting emotions evoked in a reader (Mohammad 2021, Mohammad 2023).

This task is on perceived emotion and focuses on:

• Determining what emotion most people will think the speaker may be feeling given a sentence or short text snippet uttered by the speaker.

The task is not about:

• The emotion evoked in the reader.
• The emotion of someone else mentioned in the text.
• Or even the true emotion of the speaker (which cannot be definitively known from just a short text snippet).

We acknowledge the importance of this distinction as perceived emotions can differ from actual emotions due to various factors such as cultural context, individual differences in emotional expression, and the limitations of text-based communication (Van Woensel and Nevil 2019, Wakerfield 2021).

We include a large number of languages with many predominantly spoken in regions characterised by a relatively limited availability of NLP resources (e.g., Africa, Asia, Eastern Europe and Latin America):

• Afrikaans, Algerian Arabic, Amharic, Hausa, Igbo, Kinyarwanda, Moroccan Arabic, Mozambican Portuguese, Nigerian-Pidgin, Oromo, Setswana, Somali, Swahili, Tigrinya,Xitsonga, isiXhosa, Yoruba, isiZulu Arabic, Chinese, Hindi, Indonesian, Javanese, Marathi English, German, Romanian, Russian, Spanish, Tatar, Ukrainian, Swedish Brazilian Portuguese

Participants can participate in one or more of the following tracks:

Given a target text snippet, predict the perceived emotion(s) of the speaker. Specifically, select whether each of the following emotions apply: Joy, Sadness, Fear, Anger, Surprise, or Disgust. In other words, label the text snippet with: joy (1) or no joy (0), sadness (1) or no sadness (0), anger (1) or no anger (0), surprise (1) or no surprise (0), and disgust (1) or no disgust (0).

A training dataset with gold emotion labels will be provided for this track.

Given a target text and a target emotion, predict the intensity for each of the target emotion classes.

The set of target emotions includes: Joy, Sadness, Fear, Anger, Surprise, or Disgust.

The set of ordinal classes of intensity includes: 0 for no emotion 1 for a low degree of emotion 2 for a moderate degree of emotion 3 for a high degree of emotion.

Given a labeled training set in one of the languages given above, predict the emotion labels of a new text instance in a different target language using the set of six emotion classes:Joy, Sadness, Fear, Anger, Surprise, or Disgust.

For each track, we provide the sample, training, and evaluation datasets. Find the links to download the datasets for each track below:

Download the dataset using gdown:

1. Install gdown if you haven't already:

   pip install gdown

2. Use the following commands to download the datasets using provided IDs:

   gdown --folder https://drive.google.com/drive/folders/<file_id>

The table below shows the languages of the different datasets, their sizes and the release status of their pilot samples, training, and development (dev) sets. Note that ✓ means released and can be found in the data folder.

The performance of each submission will be evaluated using F1-score based on the predicted labels and the gold ones. Participants will be provided with an evaluation script and a starter kit that includes a simple baseline.

The task will be divided into three phases: Development, Evaluation, and Post-Evaluation. The following summarize the phases and their timelines.

1. Register: Sign up on the Codalab competition platform (link to be provided).
2. Track: Decide on the track(s) you want to participate in (Track A, B and/or C)
3. Download: Access to the datasets for each track will be provided in this repository.
4. Develop: Build your models using the provided data.
5. Submit: Submit your predictions on the Codalab competition platform (link to be provided).

We will soon release a dataset paper that describes the data collection, annotation process, and baseline experiments. This paper will provide additional details and information that will be useful for the task participants.

• Join our Discord Channel to ask questions and receive updates (coming soon).
• If you have any questions or issues, please feel free to create an issue.
• Contact organizers at: emotion-semeval-2025-organisers[at]googlegroups[dot]com

1. SemEval 2025 Shared Tasks
2. Frequently Asked Questions about SemEval
3. Paper Submission Requirements
4. Guidelines for Writing Papers
5. Paper style files
6. Previous shared-tasks on emotion detection (to be added)
7. Resources for Beginners (to be added)
8. Paper submission link (to be added)

Janyce Wiebe, Theresa Wilson, and Claire Cardie. "Annotating expressions of opinions and emotions in language." Language resources and evaluation 39 (2005): 165-210.

Saif M. Mohammad,, and Svetlana Kiritchenko. "Understanding Emotions: A Dataset of Tweets to Study Interactions between Affect Categories." Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). 2018.

Saif M. Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. In Proceedings of the International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, USA, June 2018.

Lieve Van Woensel and Nissy Nevil. 2019. What if your emotions were tracked to spy on you? European Parliamentary Research Service, PE 634.415. https://www.europarl.europa.eu/RegData/etudes/ATAG/2019/634415/EPRS_ATA(2019)634415_EN.pdf.

Jane Wakefield. 2021. AI emotion-detection software tested on Uyghurs. BBC. https://www.bbc.com/news/technology-57101248.

Saif M. Mohammad "Ethics sheet for automatic emotion recognition and sentiment analysis." Computational Linguistics 48.2 (2022): 239-278.

Saif M. Mohammad "Best Practices in the Creation and Use of Emotion Lexicons." Findings of the Association for Computational Linguistics: EACL 2023. 2023.

Shamsuddeen Hassan Muhammad, Seid Muhie Yimam , Nedjma Ousidhoum, Idris Abdulmumin, Ibrahim Said Ahmad, Alham Fikri Aji, Vladimir Araujo, Abinew Ali Ayele, Tadesse Destaw Belay, David Ifeoluwa Adelani, Daniela Teodorescu, Jan Philip Wahle, Terry Ruas, Nirmal Surange, Yi Zhou, Alexander Panchenko, Vukosi Marivate, Saif M. Mohammad