Abstract — The human voice carries rich paralinguistic information that reflects the speaker’s emotional and psychological state. This paper presents a comprehensive framework for voice-based sentiment analysis designed to recognize distinct states of mind—namely happy, neutral, anxious, angry, and sad—by exploiting digital signal processing (DSP) techniques combined with a hybrid deep learning model. Raw speech signals are first preprocessed to remove silence and environmental noise, then segmented into short frames. A suite of handcrafted DSP features is extracted, including Mel-frequency cepstral coefficients (MFCCs), pitch, energy, zero-crossing rate, spectral centroid, and spectral flux. These features capture both prosodic and spectral properties essential for emotion discrimination. The extracted feature sequences are fed into a convolutional-recurrent architecture consisting of a 1D convolutional neural network (CNN) for local pattern discovery, a bidirectional long short-term memory (BiLSTM) network for temporal modelling, and an attention layer to emphasize salient segments. Evaluated on the RAVDESS and TESS datasets under speaker-independent protocols, the system achieves an overall weighted F1-score of 85.3% and demonstrates particular sensitivity to high-arousal states such as angry and happy, while correctly distinguishing the subtle low-arousal class anxious from sad and neutral. The results highlight the feasibility of using DSP-driven deep learning for real-time, non-invasive mental state assessment, with potential applications in telemental health and affective computing.
I. Introduction
The voice is a uniquely sensitive biosignal, modulated not only by linguistic intent but also by the physiological and cognitive processes underlying emotion. Fluctuations in respiration, muscle tension, and articulatory effort—induced by states such as happiness, anger, or anxiety—leave distinct signatures in the acoustic waveform. Decoding these signatures automatically presents a valuable opportunity for passive mental health monitoring, adaptive human-computer interfaces, and empathetic conversational agents [1].
Traditional approaches to speech emotion recognition (SER) have relied on handcrafted features derived from DSP, followed by shallow classifiers such as support vector machines (SVMs) or Gaussian mixture models (GMMs) [2]. While effective, these pipelines often fail to capture long-range temporal dependencies and subtle nonlinear interactions across feature streams. The advent of deep learning has enabled end-to-end models that learn hierarchical representations directly from spectrograms, yet pure deep learning can be data-hungry and may neglect well-established acoustic-prosodic knowledge [3].
This article proposes a hybrid methodology that marries DSP-based feature engineering with a state-of-the-art deep learning classifier. We focus explicitly on a five-class sentiment taxonomy that mirrors clinically relevant mental states: happy (positive high arousal), angry (negative high arousal), anxious (negative medium arousal with tension), sad (negative low arousal), and neutral (baseline). By deliberately including anxiety, we move beyond conventional “big six” emotion categories toward a continuum that better captures psychological distress. The main contributions are:
A detailed DSP pipeline extracting complementary short-term acoustic features;
A CNN-BiLSTM attention architecture that learns both local spectral patterns and their temporal evolution;
Extensive evaluation illustrating inter-class confusion patterns, especially between anxiety, sadness, and neutrality;
An open-source Python implementation made available for reproducibility.
II. Related Work
Affective computing from speech has been an active research area for over two decades. Early efforts by Dellaert et al. [4] used pitch and energy statistics with K-nearest neighbors to classify four emotions. Subsequent work expanded feature sets to include spectral, cepstral, and voice quality descriptors, often combined using brute-force feature selection [2]. The Interspeech challenges (e.g., ComParE, AVEC) have provided standardized benchmarks and large feature sets such as the extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) [5], which aggregates 88 DSP-derived features.
In recent years, deep learning methods have substantially improved recognition rates. Convolutional neural networks operating on log-Mel spectrograms treat the audio as an image, capturing localized time-frequency patterns [6]. Recurrent architectures, notably LSTMs, explicitly model sequential context and have been combined with attention mechanisms to weight emotionally salient regions [7]. However, spectrogram-based models may discard phase information and rely heavily on data augmentation. Hybrid approaches that compute traditional DSP features and feed them as sequences to a recurrent network have shown robust performance, particularly with limited training data [8]. Our work aligns with this hybrid paradigm but focuses on a mental-state label set that includes anxiety, which has been underrepresented in SER literature despite its clinical significance.
METHODOLOGY
A. Preprocessing and DSP Feature Extraction
The raw speech signal, sampled at 16 kHz, is first passed through a voice activity detector (VAD) based on short-term energy and spectral flatness to strip non-speech segments. A first-order pre-emphasis filter 
is applied to flatten the spectrum. The signal is then framed using a 25 ms Hamming window with a 10 ms shift. For each frame m, we compute the following short-term features:
Mel-Frequency Cepstral Coefficients (MFCCs): A 512-point FFT maps the frame to the power spectrum. The spectrum is passed through a bank of 26 triangular Mel-spaced filters, the log-energy of each filter is taken, and a discrete cosine transform (DCT) yields 13 MFCCs (including the 0th coefficient, which is replaced by log-energy). Delta and delta-delta coefficients are appended, resulting in 39 MFCC features per frame. The DCT transform is given by
where 
is the log-energy of the j-th Mel filter, N is the number of filters, and L the number of coefficients.
Pitch (F0) and Voicing Probability: The fundamental frequency is estimated using the robust autocorrelation method [9]. A single pitch value in Hz and its voicing flag are stored per frame.
Energy and Loudness: Short-term log-energy is computed from the root-mean-square amplitude. The frame-level zero-crossing rate (ZCR) provides a rough measure of high-frequency content.
Spectral Features: Spectral centroid, spectral spread, spectral skewness, spectral kurtosis, and spectral flux (the squared difference between successive normalized magnitude spectra) are derived from the power spectrum.
All features are computed using the librosa and pyAudioAnalysis libraries. After extraction, each frame is represented by a 48-dimensional feature vector (39 MFCCs + 1 pitch + 1 voicing flag + 1 log-energy + 1 ZCR + 5 spectral descriptors). The feature sequences are normalized to zero mean and unit variance per speaker, as recommended by [8], to reduce speaker-dependent variability while preserving emotional modulation.
B. Deep Learning Architecture
The sequence of feature vectors 
(where T is the number of frames, variable per utterance) is fed into the network shown in Fig. 1 (architecture diagram described textually).
The diagram illustrates the end-to-end flow from a raw speech waveform to the final mental state prediction.
Flow Summary:
Raw Waveform Input (a 1D time series)
DSP Preprocessing & Feature Extraction: Voice Activity Detection → Pre-emphasis → Framing → Extraction of a 48D feature vector per frame (MFCCs, Pitch, Energy, ZCR, Spectral Features)
Input Sequence: A matrix of shape T x 48 (where T is the number of frames)
Multi-Scale 1D CNN Block: Three parallel convolutional branches with kernel sizes 3, 5, and 7, followed by concatenation, Batch Normalization, ReLU, and Dropout. This extracts local acoustic patterns at different temporal scales.
Bidirectional LSTM Layers: Two stacked BiLSTM layers (each 64 units per direction) to capture past and future temporal context from the CNN features.
Attention Mechanism: A soft attention layer that computes a weighted sum of all BiLSTM outputs, allowing the model to focus on the most emotionally salient frames (e.g., stressed syllables, pitch peaks).
Classification Head: A fully connected layer (Dense + ReLU + Dropout) followed by a final Softmax layer to output probabilities for the five states: Happy, Neutral, Anxious, Angry, and Sad.
1D Convolutional Block: Three parallel temporal convolutions with kernel sizes 3, 5, and 7 (each with 128 filters, stride 1, causal padding) extract multi-scale local patterns. Their outputs are concatenated and passed through batch normalization, ReLU activation, and dropout (rate 0.3).
Bidirectional LSTM: Two stacked BiLSTM layers (each with 64 hidden units per direction) capture forward and backward temporal dependencies. Dropout of 0.3 is applied between LSTM layers.
Attention Mechanism: A soft attention layer computes a weighted sum of the BiLSTM outputs. The attention weights 
are obtained by:
where 
is the BiLSTM output at time t, and W,v,b are learnable parameters. The context vector 
is used for classification.
Classification Head: The context vector is mapped to a 64-dimensional dense layer with ReLU, followed by dropout (0.5) and a softmax layer over the five classes.
The model is trained using categorical cross-entropy loss and the Adam optimizer with an initial learning rate of 0.001, reduced by a factor of 0.5 when validation loss plateaus. Mini-batch size is 32, and training runs for up to 100 epochs with early stopping (patience of 10). A weighted sampling scheme addresses slight class imbalance.
IV. Experimental Setup
Datasets: Two publicly available acted emotional speech corpora are used. RAVDESS [10] contains 24 professional actors (12 male, 12 female) speaking two sentences in eight emotional states; we select the five relevant classes: happy, angry, sad, neutral, and anxious (originally labeled “fearful,” which shares acoustic similarities with anxiety [11]). TESS [12] features two female actors speaking 200 words in seven emotions; the same five classes are extracted. Only audio recordings are used. Combined, the dataset comprises approximately 2,800 utterances, with class distribution: happy (14%), neutral (23%), anxious (22%), angry (21%), sad (20%).
Speaker-Independent Splits: To ensure generalization to unseen speakers, we perform 5-fold cross-validation where in each fold the actors in the test set are disjoint from those in the training/validation sets. Within the training fold, 15% is held out for validation.
Baseline Models: For comparison, we implement: (i) an SVM with an RBF kernel trained on 88 functionals (mean, std, min, max, etc.) of the 48 low-level descriptors over the whole utterance (following eGeMAPS philosophy); (ii) a plain BiLSTM (without CNN and attention) on the raw frame-level features; and (iii) a CNN-LSTM without attention.
Evaluation Metrics: Weighted precision, recall, and F1-score, along with a confusion matrix, are reported. Weighted metrics account for class imbalance.
V. Results and Discussion
Table 1 summarizes the performance of all models across the five emotional states.
Model | Weighted Precision | Weighted Recall | Weighted F1-Score |
SVM (RBF, 88 functionals) | 71.4% | 70.8% | 71.1% |
BiLSTM (raw features) | 77.2% | 76.9% | 77.0% |
CNN-LSTM (no attention) | 82.6% | 82.3% | 82.4% |
CNN-BiLSTM-Attention (proposed) | 85.5% | 85.1% | 85.3% |
Table 1. Performance comparison of models under speaker-independent 5-fold cross-validation.
The proposed DSP-informed CNN-BiLSTM-Attention model outperforms all baselines, with a 3 percentage point improvement over
