Sentiment analysis has become increasingly vital for modern businesses, offering insights into customer opinions, satisfaction, and potential issues. With interactions frequently happening via text (like social media, chat, and e-commerce reviews) or voice (such as call centers), organizations require effective ways to interpret these signals at scale. Accurately identifying and classifying a customer’s emotional state allows companies to provide more proactive, personalized experiences, which can enhance customer satisfaction and loyalty.
Despite its strategic importance, deploying thorough sentiment analysis solutions involves several challenges. These include language ambiguity, cultural nuances, regional dialects, sarcasm, and large volumes of real-time data, all of which necessitate scalable and flexible architectures. For voice-based sentiment analysis, crucial features like intonation and prosody can be lost if audio is merely transcribed and processed as text. Amazon Web Services (AWS) offers a range of tools to tackle these issues, including services for audio capture and transcription (Amazon Transcribe), text sentiment classification (Amazon Comprehend), intelligent contact center solutions (Amazon Connect), and real-time data streaming (Amazon Kinesis).
This article, developed through a scientific partnership between AWS and the Instituto de Ciência e Tecnologia Itaú (ICTi), a P&D hub supported by Itaú Unibanco, examines the technical aspects of sentiment analysis for both text and audio. It presents experiments comparing various machine learning (ML) models and services, discusses the trade-offs and challenges of each method, and demonstrates how AWS services can be combined to create robust, end-to-end solutions. The article also provides insights into future possibilities, such as advanced prompt engineering for large language models (LLMs) and broadening audio-based analysis to detect emotional cues that text data might overlook. Audio-based sentiment analysis is explored in two stages:
- Stage 1 – Transcribe audio into text and perform sentiment analysis using LLMs
- Stage 2 – Analyze sentiment directly from the audio signal using audio models
Sentiment analysis in text
This section discusses the method of transcribing audio into text and performing sentiment analysis using LLMs.
Challenges and characteristics
This method involves the following challenges:
- Variety of data sources – Textual interactions emerge from numerous channels—social networks, ecommerce platforms, chatbots, and helpdesk tickets—each with unique formats and constraints. For instance, social media text might contain hashtags, emojis, or character limits, whereas chat messages might include acronyms and domain-specific jargon. A robust text-processing pipeline must therefore include appropriate data cleaning and preprocessing steps to normalize these variations.
- Ambiguity of natural language – Human language is often ambiguous and context-dependent. Sarcasm, irony, and figurative expressions complicate classification by superficial natural language processing (NLP) techniques. Although deep neural networks—such as BERT, RoBERTa, and Transformers-based architectures—have proven more adept at capturing nuanced semantics, it remains an ongoing challenge to fully account for creative or context-dependent language usage.
- Multilingual and dialect considerations – Global enterprises like Itaú Unibanco encounter multiple languages and regional dialects, each requiring specialized models or additional training data. A sentiment model trained primarily on one language or dialect might fail when confronted with slang, colloquialisms, or distinctive grammatical structures from another.
Tested models and rationale
Experiments involved evaluating several LLMs for sentiment classification. These included popular foundation models (FMs) accessible via Amazon Bedrock and Amazon SageMaker JumpStart, such as Meta’s Llama 3 70B, Anthropic’s Claude 3.5 Sonnet, Mistral AI’s Mixtral 8x7B, and Amazon Nova Pro. Each service provides distinct advantages depending on specific requirements. For instance, Amazon Bedrock simplifies large-scale experimentation by offering a unified, serverless interface to various LLM providers through API access. SageMaker AI delivers a managed experience for popular FMs with a user-friendly UI or API-based deployment. Both Amazon Bedrock and SageMaker AI optimize operational aspects like model hosting, scalability, security, and cost, which are crucial for enterprise adoption of generative AI.
- Zero-shot or few-shot prompting – Using generic prompts to classify sentiment in text
- Fine-tuning – Adapting the model on domain-specific sentiment data to assess whether this specialized training improved performance or risked overfitting
AWS services for text analysis
Amazon provides a suite of services to streamline text analysis. The following services were utilized to build a text analysis service:
- Amazon Bedrock – Facilitates serverless access to pre-trained FMs from different providers within a single, secure interface—particularly access to closed weights models like Anthropic’s Claude. This allows rapid testing of multiple models without managing underlying infrastructure.
- Amazon SageMaker AI – Provides access to the latest open-source FMs like Llama, Mistral, DeepSeek, and more. With SageMaker AI, you have the option to simplify deployment of FMs using Amazon SageMaker JumpStart—an ML and generative AI managed hub that provides simple UI or API based deployment of hundreds of FMs or alternatively helping you deploy your preferred FM and architecture on managed NVIDIA GPU infrastructure with ease.
- Amazon Comprehend – An AI service with text analytics capabilities including sentiment analysis, entity recognition, and topic modeling. It can serve as a baseline or be integrated with advanced LLM workflows for a more comprehensive pipeline.
- Amazon Kinesis – Handles real-time ingestion and streaming of text data from diverse sources (such as social media feeds, log streams, or real-time customer chat sessions).
A simplified architecture might consist of the following components:
- Data ingestion using Kinesis to capture text from various sources
- Data preprocessing using AWS Lambda or Amazon EMR for normalization, tokenization, and filtering.
- Model inference using either an LLM accessed through Amazon Bedrock or SageMaker AI
- Storage and analytics in Amazon Simple Storage Service (Amazon S3) or Amazon Redshift for long-term analysis, reporting, and visualization
Experimental results for text
Performance metrics (accuracy, precision, recall) were summarized across different models tested. Each model was evaluated on the same text dataset to classify sentences as positive, negative, or neutral. The results showed overall low performance, as detailed in the analysis below.
Analysis of findings
Observations from the results include:
- Overall low performance – All models exhibited relatively low accuracy in detecting sentiment polarity. This suggests that purely text-based inputs may not provide sufficient contextual or emotional cues, particularly for subtle expressions like sarcasm or irony.
- Impact of fine-tuning – The two fine-tuned OpenAI models achieved higher metrics compared to most other configurations, though this performance increase could suggest overfitting. These models consistently labeled sentences as non-neutral only when strong emotional indicators were present.
- Model variation – Meta’s Llama 3 70B and Anthropic’s Claude 3.5 Sonnet performed better than some other base models but still lagged behind the fine-tuned OpenAI solutions. This could be due to differences in their pre-training objectives and the domain of their original training data compared to the sentiment classification task.
Future directions for text-based analysis
Expanding text-based analysis could involve the following approaches:
- Advanced prompt engineering – Current experiments employed straightforward chain-of-thought prompts. Future work could explore more refined few-shot or zero-shot prompt designs, including advanced reasoning strategies like “buffer of thoughts,” or carefully targeted domain-specific prompting.
- Multimodal inputs – Incorporating paralinguistic information (such as intonation or speaker emphasis) might boost text-based classification. Such data could be encoded as metadata or extracted by auxiliary models to enrich the textual context.
- Language coverage – Extending to non-English corpora and training domain-specific or multilingual models would likely improve generalization in real-world deployments.
Sentiment analysis in audio
This section discusses the method of analyzing sentiment directly from the audio signal using audio models.
Challenges and characteristics
This method involves the following challenges:
- Intonation and prosody – Spoken language carries acoustic cues (tone, pitch, volume, tempo, and rhythm) that greatly influence perceived sentiment. A simple greeting such as “Hi, how are you?” can be genuinely enthusiastic or passively sarcastic, depending on the intonation. Traditional speech-to-text pipelines discard these non-verbal cues, potentially weakening the sentiment signal.
- Speech-to-text conversion – Many audio sentiment analysis systems rely on ASR (Automatic Speech Recognition) to generate transcripts, which are then fed into text-based sentiment models. Though beneficial for content understanding, purely textual analysis ignores prosodic features—one reason direct audio-based sentiment classification has garnered research interest.
- Noise and recording quality – Real-world audio often contains background noise, overlapping dialogue, or low-fidelity recordings. Models must be robust to such conditions to be viable in environments like call centers or customer support lines.
Experimental datasets
Two distinct types of datasets were used, each focusing on different aspects of emotion in speech:
- Type 1 – A curated collection of short utterances recorded with different emotional intonations. Initially labeled by arousal (such as, happy, angry, disgusted), the data was then re-labeled by valence (positive, negative, neutral). Recordings labeled as “surprise” were removed because it can manifest as either positive or negative.
- Type 2 – Contains more varied sentences, each labeled as positive, negative, or neutral. The diversity and complexity of utterances make this dataset significantly more challenging.
- Sources include the Audio Speech Sentiment dataset and MELD.
Tested models and rationale
Three prominent speech-based models were evaluated:
- HuBERT (Hidden Unit BERT) – Employs a self-supervised Transformer that learns hidden cluster assignments in the audio signal. HuBERT excels at capturing prosodic and acoustic patterns crucial for emotion detection.
- Wav2Vec – Similar in philosophy to HuBERT, Wav2Vec learns powerful representations directly from raw audio using a Transformer-encoder backbone. Its self-supervised training scheme is highly effective with limited labeled data.
- Whisper – A Transformer-based encoder-decoder originally designed for robust speech recognition. Although its emphasis is on transcription and translation, its ability to extract embeddings for downstream sentiment classification tasks was tested.
AWS services for audio analysis
To streamline the training and inference pipeline, the following AWS services were used:
- Amazon SageMaker Studio – Allows quick setup of training jobs on purpose-built instances (for example, GPU-enabled) without significant infrastructure overhead. Each model (HuBERT, Wav2Vec, Whisper) was trained and validated in separate SageMaker sessions.
- Amazon Transcribe – For those workflows requiring speech-to-text conversion, Amazon Transcribe provides scalable and accurate ASR. Though not the focus of direct audio-based sentiment methods, it’s commonly integrated into contact center architectures, where text transcripts are also used for analytics or compliance checks.
A representative architecture could involve Kinesis for audio ingestion, Lambda for orchestrating pre-processing or route selection (such as direct audio-based sentiment vs. text-based after transcription), and Amazon S3 for storing final results. The following diagram illustrates this example architecture.
Experimental results for audio
Classification accuracy was evaluated on separate test splits for Type 1 and Type 2 datasets. Generally, all three models performed better on Type 1 than on Type 2. The analysis below details these findings.
Analysis of findings
Observations from the results include:
- Type 1 – Because the same phrases were repeated with different emotional intonations, models focused more on acoustic cues rather than content. This led to higher accuracy—especially in distinguishing high-arousal (anger, excitement) from low-arousal (sadness, calm) states.
- Type 2 – Performance dropped significantly when faced with more varied sentences. Here, the differences in lexical content and context overshadowed purely prosodic features. The models struggled to generalize across diverse sentence structures, speaker styles, and emotional expressions.
Future directions for audio-based analysis
Expanding audio-based analysis could involve the following approaches:
- Data diversity – Expanding the datasets to include more languages, regional accents, and environmental conditions might improve the generalizability of these models.
- Multimodal fusion – Combining direct audio embeddings (prosody, intonation) with textual analysis (lexical content) might yield richer sentiment representations. This is especially pertinent in customer service scenarios where semantic content and emotional tone both matters.
- Real-time inference – For applications like live contact center support using Amazon Connect, real-time inference pipelines are crucial. Researchers can investigate methods such as streaming-based model inference (for example, chunk-by-chunk or frame-level processing) to get immediate feedback on customer sentiment and adapt responses accordingly.
Conclusion
Sentiment analysis—whether performed on text or audio—offers powerful insights into customer perceptions, enabling more proactive and empathetic engagement strategies. However, the technical hurdles are non-trivial:
- Text – Ambiguity, irony, and limited context can hinder purely text-based classification. LLMs, even those fine-tuned, might underperform without careful data curation, advanced prompt engineering, or additional metadata.
- Audio – Directly analyzing audio captures prosodic and acoustic cues often lost in transcription. However, environmental noise, overlapping speech, and speaker diversity complicate training robust models.
AWS provides an extensive suite of services that cover the end-to-end sentiment analysis pipeline:
- Data ingestion – Kinesis for real-time text and audio streaming
- Preprocessing – Lambda and Amazon EMR for data cleansing, feature extraction, and transformations
- Transcription (Optional) – Amazon Transcribe to convert audio to text if a combined text and audio approach is needed
- Sentiment classification – AWS offers the following:
- Text – Amazon Comprehend or FMs accessed through Amazon Bedrock and SageMaker AI
- Audio – Custom models (such as HuBERT, Wav2Vec, Whisper) trained in SageMaker AI
- Customer Engagement – Amazon Connect for intelligent contact centers with potential for real-time sentiment feedback loops
Ultimately, the choice between audio-based, text-based, or hybrid approaches depends on the use case and available data. Direct audio-based methods might capture emotional subtleties crucial in call center interactions—particularly during greetings or highly charged conversations—whereas text-based methods are often more straightforward to deploy at scale for chats, social media, and review-based analysis. By using AWS Cloud-based capabilities alongside rigorous ML methodologies, enterprises can tailor sentiment analysis solutions that balance accuracy, scalability, and cost-effectiveness. Future explorations might further integrate multimodal streams, advanced prompt engineering, and domain-specific fine-tuning, continuously refining our ability to interpret and act on the “voice of the customer.”
