Speech: recognition, synthesis, prosody, and vocoders

radio_button_uncheckedshort_text1. Introduction to speech-enabled solutions

Speech is a natural form of human communication. Adding it to AI applications can make assistants, accessibility features, and conversational agents more intuitive, inclusive, and engaging.

Two capabilities provide the foundation: speech recognition turns spoken words into text, while speech synthesis turns text into natural-sounding audio. Used together, they create continuous voice interactions.

General benefits

• Accessibility for people with visual impairments or mobility limitations.
• Productivity through hands-free work and multitasking without a keyboard or screen.
• More natural, human-like, and engaging user experiences.
• Global reach across multiple languages, accents, and regional dialects.

The original learning module supports video or text-and-image study. Its written version usually contains extra detail and can supplement the videos.

radio_button_uncheckedshort_text2. Speech recognition scenarios

radio_button_uncheckedshort_text3. Speech synthesis scenarios

radio_button_uncheckedshort_text4. Full conversations and implementation choices

Combined solutions listen, reason, and reply. Voice customer service recognizes a request and synthesizes help; IVR guides callers through natural dialogue; language apps evaluate a spoken phrase and speak corrections; vehicles accept hands-free commands and confirm actions.

A practical approach is to prove one high-value speech capability first, then expand into more complex conversational flows.

Always keep alternative input and output methods because some people prefer or require a text interface.

radio_button_uncheckedshort_text5. Recognition: capture and pipeline overview

Speech-to-text coordinates six stages: audio capture, feature preparation, acoustic modeling, language modeling, decoding, and final refinement.

A microphone converts analog sound waves to numeric samples. Speech systems commonly use 16,000 samples per second, or 16 kHz. Music rates such as 44.1 kHz preserve more detail at greater processing cost; 8–16 kHz usually balances speech clarity and efficiency.

Noise, microphone quality, and speaker distance affect later accuracy. Early filters may remove hums, clicks, and other interference.

radio_button_uncheckedshort_text6. Preprocessing and MFCC features

Raw samples carry too much information for efficient pattern matching. Preprocessing keeps speech characteristics while discarding less useful details such as absolute volume.

Mel-Frequency Cepstral Coefficients (MFCCs) approximate human hearing by emphasizing the frequency ranges where speech energy is concentrated.

1. Split the signal into overlapping 20–30 millisecond frames.
2. Apply a Fourier transform to expose pitches in the frequency domain.
3. Map frequency bins to the Mel scale, reflecting stronger human discrimination at lower pitches.
4. Compute a compact summary, commonly 13 coefficients, for the spectral shape of each frame.

Frame 1: [-113.2, 45.3, 12.1, -3.4, 7.8, ...]  // 13 coefficients
Frame 2: [-112.8, 44.7, 11.8, -3.1, 7.5, ...]
Frame 3: [-110.5, 43.9, 11.5, -2.9, 7.3, ...]

radio_button_uncheckedshort_text7. Acoustic models and phonemes

An acoustic model learns how features correspond to phonemes, the smallest sound units that distinguish words. English has about 44; “cat” consists of /k/, /æ/, and /t/.

Modern transformers process MFCC sequences and estimate phonemes over time. Attention examines nearby frames to resolve ambiguity, parallel processing handles many frames at once, and contextual learning favors patterns that occur in natural speech.

The output is a probability distribution: one frame might score /æ/ at 80%, /ɛ/ at 15%, and other phonemes at 5%. Phonemes are language-specific, so an English model cannot recognize Mandarin tones without retraining.

radio_button_uncheckedshort_text8. Language modeling and decoding

Phonemes alone cannot resolve homophones such as “their” and “there.” A language model adds vocabulary, grammar, and frequent patterns: it favors “The weather is nice,” expects a verb after “I need to,” and can be adapted with medical or legal terminology.

Decoding searches millions of word sequences for the hypothesis that best balances acoustic evidence and readable language. Beam search retains a shortlist of strong partial transcripts, extends each one, prunes weak paths, and continues with the best candidates.

A three-second utterance may produce thousands of candidates before “Please send the report by Friday” wins over similar-sounding alternatives. Real-time systems limit beam width and hypothesis depth to trade some search breadth for lower latency.

radio_button_uncheckedshort_text9. Post-processing and recognition output

Speech can return the transcript with word timestamps and confidence scores so an application can highlight uncertainty or trigger fallback behavior. Separating the pipeline also makes troubleshooting easier: poor input, weak domain training, and overly aggressive formatting rules have different remedies.

radio_button_uncheckedshort_text10. Synthesis: normalization and linguistic analysis

Text-to-speech builds natural audio in four incremental stages.

Text normalization expands abbreviations, numbers, dates, times, and symbols and resolves homographs from context. “Dr. Smith ordered 3 items for $25.50 on 12/15/2023” becomes a speakable form such as “Doctor Smith ordered three items for twenty-five dollars and fifty cents on December fifteenth, two thousand twenty-three.” Medical doses need different rules from financial currency and percentages.

Linguistic analysis segments words and syllables, consults pronunciation lexicons, applies G2P rules or neural models to unknown words, marks boundaries and stress, and accounts for neighboring sounds. Grapheme-to-phoneme conversion is essential because “though,” “through,” and “cough” share “ough” but become /ðoʊ/, /θruː/, and /kɔːf/.

Neural G2P models trained on dictionaries handle uncommon words, proper names, and regional variation. Transformers use sentence context to distinguish present /riːd/ from past /rɛd/ in “read”.

radio_button_uncheckedshort_text11. Prosody generation

Prosody is the rhythm, stress, and intonation that determines how an utterance is delivered. It includes pitch contours, sound duration, intensity, pauses, and stressed syllables. Changing the emphasized word in “I never said he ate the cake” changes the implied meaning.

1. Encode phonemes together with punctuation, part of speech, and sentence structure.
2. Use self-attention to connect words, references, and sentence boundaries.
3. Predict pitch, duration, and energy for each phoneme.
4. Apply neutral, expressive, or conversational style and speaker characteristics.

Thousands of hours of paired speech and transcripts teach patterns: questions often rise, commas pause, emphasis lengthens, and sentence endings fall. Syntax, semantics, discourse contrast, speaker identity, and emotional tone all influence the choices.

A target could request /æ/ at 180 Hz for 80 ms with moderate intensity, followed by a 200 ms pause. Flat prosody, rather than incorrect phonemes, is often what makes synthesis sound robotic.

radio_button_uncheckedshort_text12. Vocoders and audio generation

An acoustic model, often a transformer, converts phonemes and prosody into a mel-spectrogram. A neural vocoder turns it into a waveform of 16,000–48,000 amplitude samples per second, followed by filters, normalization, or effects.

WaveNet, WaveGlow, and HiFi-GAN are well-known vocoder architectures. Neural vocoders offer near-studio fidelity, subtle vocal detail, real-time generation on modern hardware, and flexibility across speakers, languages, and styles. Their direction is effectively the reverse of recognition: linguistic representation becomes audio.

For “Dr. Chen’s appointment is at 3:00 PM,” normalization expands the title and time, linguistic analysis creates phonemes, prosody raises “appointment,” pauses after “is,” and emphasizes “three,” and synthesis renders the requested waveform. Modern hardware usually completes the pipeline in under a second.

radio_button_uncheckedshort_text13. Exercise, assessment, and summary

The exercise uses voice mode in Chat Playground to talk with a model, ask questions, and hear responses, demonstrating recognition and synthesis in one experience.

• Open the original speech exercise

Reworded knowledge check

1. During preprocessing, does the system convert audio to WMV, add noise, or extract feature vectors from the waveform?
2. Are phonemes removed artifacts, the smallest speech sounds, or audio-generating models?
3. Does prosody maximize volume, translate languages, or create natural pronunciation and cadence?

Answers

• Preprocessing extracts feature vectors for later modeling.
• Phonemes are the smallest units of sound that distinguish speech.
• Prosody supplies natural rhythm, pitch, stress, and cadence.

In summary, recognition converts sound to text through capture, MFCC features, acoustic and language models, decoding, and cleanup. Synthesis uses normalization, phonemes, prosody, and a vocoder. Together they support two-way voice experiences in service, accessibility, healthcare, education, agents, and many other scenarios.

• Get started with speech in Azure