Automated Semantic Understanding of Human Emotions in Writing and Speech

Affective Human Computer Interaction (A-HCI) will be critical for the success of new technologies that will be prevalent in the 21st century. If cell phones and the internet are any indication, there will be continued rapid development of automated assistive systems that help humans to live better, more productive lives. These will not be just passive systems such as cell phones, but active assistive systems like robot aides in use in hospitals, homes, entertainment room, office, and other work environments. Such systems will need to be able to properly deduce human emotional state before they determine how to best interact with people. This work explores and extends the body of knowledge related to Affective HCI. New semantic methodologies are studied for reliable and accurate detection of human emotional states and magnitudes in written and spoken speech; and for mapping emotional states and magnitudes to 3-D facial expression outputs. This is a dissertation on affective human computer interaction. The topics include: * Applications of Affective Human Computer Interaction * Applications to Health, Applications to Cognitive Ergonomics, Applications to Information Retrieval, Applications to Movie Animation and Video Game Design * Emotion Recognition Systems * Emotion Recognition from Text * Lexical Based Approaches * Sentiment vs. Commonsense Knowledge Approaches * Emotion Recognition from Speech * Automatic Speech Recognition (ASR) and Speech Characteristics * Multimodal Emotion Recognition * Actor Detection * Actor Detection Using Low-level Features * Sentient Actor Detection Using Higher Semantic-level Features * Corpora * Inter-annotator Metrics * Features and Feature Selection * Text Features * Speech Features * Feature Selection Techniques * Overview of Machine Learning Approaches * Classifiers * Regression Approaches for Magnitude Prediction * Non-Sequential vs. Sequential Methods * System Response to Emotions * Environment Responses * Facial Expression Responses * Speech Response * Performance Assessment * System Accuracy on Test Corpora * Text Annotator Agreement * Generalization of the Method * Ranking Analysis of Emotion Triggers * EMOTION FEATURE EXTRACTION AND EMOTION RECOGNITION IN TEXT * Methodology * Automatic Feature Extraction Approach * Classification: Support Vector Machines * Assessment of the Feature Extraction and Classification Methodologies * Analysis and Results * Preliminary Data Analysis * Kernel-based Data Analysis and Class Imbalance * Improved Results Analysis * EMOTION CORPORA * Affect Corpus 2.0 Annotation and Evaluation Methodology * Multimodal Health Care Related Corpus Annotation and Methodology (LSU-MD) * Analysis and Results * Affect Corpus 2.0 * DETECTION OF AFFECTIVE STATES FROM TEXT AND SPEECH * Speech Features * Text Features * Classification Model * Analysis and Results * Speech Affect Detection * Multimodal Affect Detection for 2 Classes (Emotion vs. Neutral) * Multimodal Affect Detection for 3 Classes (Positive, Negative, and Neutral) * Multimodal Affect Detection for Positive vs. Negative * Multimodal Affect Detection for 5 Emotion Classes * Application of the Methodology on a Medical Drama Corpus * ACTOR AND ENVIRONMENT DETECTION * Automatic Detection of Sentient Nominal Entities