Azure Speech: Speech to Text, Text to Speech, and Voice Live

radio_button_uncheckedshort_text1. Speech interfaces for applications and agents

Speech capabilities let people control systems by voice, receive answers to spoken questions, create captions from audio, and talk with agents without relying on a keyboard or screen. This natural interaction also improves accessibility and inclusion. A complete experience must recognize what a person says and synthesize an audible response.

radio_button_uncheckedshort_text2. Azure Speech in Foundry Tools

Azure Speech in Foundry Tools provides speech to text, text to speech, and speech translation. Prebuilt and custom models support transcription, speaker identification, custom voices, and other scenarios. A Microsoft Foundry resource supplies endpoint and credentials, while the project organizes experiences, agents, and integrations.

In the new Microsoft Foundry portal, open Build, Models, and AI services. The playground lets you record or upload audio, try voices, tune parameters, and inspect results before writing code. Video can provide an introduction, while the written material includes more implementation detail.

• Speech to Text converts microphone, file, or stream audio into text.
• Text to Speech produces natural audio from text and a selected voice.
• Speech translation recognizes speech and delivers content in another language.
• Voice Live combines audio input, reasoning, and spoken output in real time.

radio_button_uncheckedshort_text3. How speech recognition works

Speech recognition, or speech to text (STT), converts spoken words into data, usually text. An acoustic model examines the signal and represents it as phonemes, which are units of sound. A language model then maps those phonemes to plausible words and sequences.

Recognized text can drive captions, call transcripts, note dictation, voicemail, and agents. The Azure Speech Speech to Text API accepts microphone audio or an audio file. An API defines the rules and endpoints that let one software application use another application’s functionality or data.

radio_button_uncheckedshort_text4. Speech-to-text playground and SDK

In the Speech to Text playground, upload a file or record your voice. The transcription demonstrates how an application would respond. To integrate the capability into a product, use the Speech SDK, a client library that abstracts networking, authentication, audio streaming, and response parsing.

• Capture or supply audio from a microphone, file, or stream.
• Send audio securely to Azure Speech.
• Receive text near real time or after processing.
• Use the SDK in a client or service layer as the bridge to the endpoint.

For Python, install azure-cognitiveservices-speech in the Visual Studio Code terminal. The Foundry resource supplies an endpoint and key; Microsoft Entra ID can also authenticate. Keep secrets out of source code in production.

pip install azure-cognitiveservices-speech python-dotenv

radio_button_uncheckedshort_text5. Python client for continuous recognition

The application initializes and authenticates the SDK, captures or loads audio, sends it securely, runs recognition in the cloud, and receives text plus optional metadata. SpeechConfig contains connection details, AudioConfig selects the source, and SpeechRecognizer performs recognition.

import os
import azure.cognitiveservices.speech as speechsdk
from dotenv import load_dotenv

load_dotenv()
speech_config = speechsdk.SpeechConfig(
    subscription=os.environ['FOUNDRY_KEY'],
    endpoint=os.environ['AZURE_SPEECH_ENDPOINT']
)
microphone = speechsdk.audio.AudioConfig(use_default_microphone=True)
recognizer = speechsdk.SpeechRecognizer(
    speech_config=speech_config,
    audio_config=microphone
)

recognizer.recognizing.connect(
    lambda event: print('Partial:', event.result.text)
)
recognizer.recognized.connect(
    lambda event: print('Recognized:', event.result.text)
)

recognizer.start_continuous_recognition()
input('Speak; press Enter to stop...')
recognizer.stop_continuous_recognition()

The recognizing event reports interim hypotheses; recognized provides the confirmed result. For voicemail, AudioConfig can select a file instead of the microphone. SpeechRecognizer transcribes the audio and the application displays or stores the text.

radio_button_uncheckedshort_text6. Real-time and batch transcription

The Speech to Text API supports real-time and batch processing. In real time, an application listens to a microphone or another source, streams audio, and receives text while speech is underway. This fits presentations, demos, captions, and conversations.

Batch transcription processes existing recordings from a file share, remote server, or Azure Storage asynchronously. An application can provide a shared access signature (SAS) URI. Jobs are scheduled on a best-effort basis: they normally begin within minutes, but there is no exact guarantee for when a job enters the running state.

radio_button_uncheckedshort_text7. How speech synthesis works

Speech synthesis, or text to speech (TTS), vocalizes data by converting text into audio. The solution needs the text and a selected voice. Processing tokenizes the content, assigns phonetic sounds, organizes the transcription into prosodic units such as phrases and sentences, produces phonemes, and synthesizes the signal.

Output can answer a user, read messages, or broadcast announcements. Voice, language, and regional pronunciation shape the experience; rate, pitch, and volume control delivery. Neural voices use neural networks to reduce common intonation limitations and sound more natural. The platform offers predefined voices and supports custom voices.

radio_button_uncheckedshort_text8. Text-to-speech playground and SDK

In the Text to Speech playground, enter text, select a synthetic voice, and adjust settings such as rate and pitch. Audio can play through speakers or be written to a file. The SDK sends text to Azure Speech, uses neural voices, and returns audio while handling authentication, networking, formatting, and playback.

Client applications can speak immediately on desktop and mobile devices; backend services can generate audio files for later playback. Python uses the same azure-cognitiveservices-speech package as recognition.

radio_button_uncheckedshort_text9. Python client for synthesis

SpeechSynthesizer receives SpeechConfig and AudioOutputConfig. The example uses the default speaker, reads console input, and waits for the asynchronous result. The application checks success or cancellation and can display error details.

import os
import azure.cognitiveservices.speech as speechsdk
from dotenv import load_dotenv

load_dotenv()
config = speechsdk.SpeechConfig(
    subscription=os.environ['FOUNDRY_KEY'],
    endpoint=os.environ['AZURE_SPEECH_ENDPOINT']
)
config.speech_synthesis_voice_name = 'en-US-Ava:DragonHDLatestNeural'
speaker = speechsdk.audio.AudioOutputConfig(use_default_speaker=True)
synthesizer = speechsdk.SpeechSynthesizer(
    speech_config=config,
    audio_config=speaker
)

message = input('Text to speak: ')
result = synthesizer.speak_text_async(message).get()
if result.reason == speechsdk.ResultReason.SynthesizingAudioCompleted:
    print('Speech synthesized.')
elif result.reason == speechsdk.ResultReason.Canceled:
    details = result.cancellation_details
    print('Synthesis canceled:', details.reason)
    if details.error_details:
        print(details.error_details)

A message-reading application can load a text file, create SpeechSynthesizer, and generate speech. A multilingual neural voice can adapt to the input language. Configure file output when audio should be saved instead of played.

radio_button_uncheckedshort_text10. Speech-to-speech conversations with Voice Live

Speech to speech accepts spoken audio and produces spoken audio, enabling conversation without reading or typing. A traditional pipeline recognizes speech, passes text through translation, summarization, application logic, or agent reasoning, and synthesizes the response. Assistants, spoken translation, kiosks, navigation, industrial tools, accessibility, and support bots are common scenarios.

Azure Speech Voice Live combines recognition, generative AI, and text to speech in a managed low-latency service. Instead of connecting several components manually, the application sends audio and receives speech. The service can also return visuals such as avatars and trigger actions.

• Azure Speech provides recognition and synthesis.
• An agent or application logic decides what the response should contain.
• Foundry Tools or MCP servers can expose speech as callable tools.
• A selected generative model works with acoustic models to support the conversation.

radio_button_uncheckedshort_text11. Build and integrate a speech-capable agent

The Voice Live playground includes ready-made voice samples and supports creating a solution. Select the agent’s generative model and configure voice, instructions, and behavior. Proactive engagement lets the agent start a conversation. Voice mode can also be enabled on a Microsoft Foundry agent, encapsulating speech configuration in the agent definition and reducing client code.

For Python, install azure-ai-voicelive, pyaudio, python-dotenv, and azure-identity. Portal sample code starts the session, connects microphones and speakers, processes incoming and outgoing audio streams, and handles interruptions. At runtime, the assistant streams microphone audio to Voice Live and plays the returned speech.

pip install azure-ai-voicelive pyaudio python-dotenv azure-identity

radio_button_uncheckedshort_text12. Exercise and knowledge check

The approximately 25-minute exercise uses Azure Speech in Foundry Tools and Voice Live to create an agent capable of real-time conversation. An Azure subscription is required; new accounts may include free credits for 30 days.

• Start the original Microsoft Learn exercise

Reworded questions

1. Why integrate the Speech to Text SDK instead of relying only on the playground?
2. Which responsibilities does the Text to Speech SDK handle for a developer?
3. What role does azure-ai-voicelive play in a voice-enabled agent?

Explained answers

• The SDK embeds recognition in application code and workflows; the playground is for experimentation.
• It handles authentication, network communication, formatting, audio generation, and configured playback or file output.
• The Voice Live SDK opens the real-time connection, streams audio, and handles spoken responses and interruptions; it does not replace local devices or permanently store recordings by default.

radio_button_uncheckedshort_text13. Chapter summary and official references

• Speech to Text converts microphones, files, or streams into text for captions, transcripts, dictation, and agent input.
• Recognition can run in real time or through asynchronous batch jobs.
• Text to Speech uses text, voice, phonetics, prosody, and neural models to produce natural audio.
• The Speech SDK connects applications to Azure Speech and abstracts authentication, networking, audio, and responses.
• Voice Live combines recognition, reasoning, and synthesis in a managed service for responsive voice agents.

• Azure Speech documentation
• Speech SDK documentation
• Voice Live documentation