Information extraction from documents, audio, and video in Microsoft Foundry

radio_button_uncheckedshort_text1. Why extract information from unstructured content

Invoices, forms, receipts, contracts, recorded calls, and video meetings contain valuable data, but manually reading, typing, and checking every item is slow and error-prone. AI-assisted extraction turns this material into consistent, searchable data ready for automation.

radio_button_uncheckedshort_text2. Azure Content Understanding in Foundry Tools

Azure Content Understanding in Foundry Tools is a Microsoft Foundry resource tool that combines generative AI and specialized models to understand documents, images, audio, and video. It extracts content, entities, fields, relationships, and meaning into a user-defined format, typically JSON.

The workflow stays consistent: ingest the file; use OCR, speech recognition, language understanding, and multimodal models; apply a schema; and return machine-readable results for storage, search, analytics, or downstream systems. Current documentation also covers segmentation, classification, confidence scores, and grounding extracted fields in the source.

radio_button_uncheckedshort_text3. OCR, layout, and semantic understanding

Optical Character Recognition (OCR) identifies characters in photographs and scans and turns them into editable, searchable text. Layout analysis preserves paragraphs, sections, tables, selection marks, formulas, and hierarchy. Those capabilities are essential, but they do not by themselves explain business meaning.

Content Understanding adds semantic interpretation and schema mapping. “Invoice No.,” “Invoice #,” or even an unlabeled number can therefore populate the same InvoiceNumber field. The result represents concepts, relationships, and predictable types instead of a flat string of words.

radio_button_uncheckedshort_text4. Invoice schema and nested fields

A schema describes exactly what a process must receive. For an invoice, simple fields can include vendor, invoice number, date, customer, address, subtotal, tax, shipping, and total. The Items collection contains nested objects for description, unit price, quantity, and line total.

radio_button_uncheckedshort_text5. Prebuilt and custom analyzers

An analyzer is the reusable unit that determines modality, extraction behavior, field schema, output structure, and models. A prebuilt analyzer works without configuration for a known scenario; a custom analyzer applies an organization’s schema and rules.

• prebuilt-invoice understands common invoice fields.
• prebuilt-imageSearch describes images for search and retrieval.
• prebuilt-audioSearch transcribes audio, diarizes speakers, and generates conversational insights.
• prebuilt-videoSearch combines frames, audio, time, and summaries.
• Current documentation also lists base analyzers such as prebuilt-document, prebuilt-image, prebuilt-audio, and prebuilt-video.

radio_button_uncheckedshort_text6. Validate documents in the Microsoft Foundry portal

Before writing code, the portal lets you select an analyzer, use sample files or upload your own, and verify that text, layout, and fields match expectations. The new and classic portals look different but support the same experimentation purpose.

radio_button_uncheckedshort_text7. Document client application with the Python SDK

A client application authenticates to the endpoint, submits content, tracks the operation, and processes the result. The azure-ai-contentunderstanding package provides ContentUnderstandingClient. The endpoint follows https://<resource>.services.ai.azure.com/, and credentials can use AzureKeyCredential or Microsoft Entra ID when configured.

python -m pip install azure-ai-contentunderstanding

import os
from azure.ai.contentunderstanding import ContentUnderstandingClient
from azure.core.credentials import AzureKeyCredential

client = ContentUnderstandingClient(
    endpoint=os.environ['FOUNDRY_ENDPOINT'],
    credential=AzureKeyCredential(os.environ['FOUNDRY_KEY'])
)
poller = client.begin_analyze(
    analyzer_id='prebuilt-invoice',
    inputs=[{'url': os.environ['INVOICE_URL']}]
)
result = poller.result()

for content in result.contents:
    print(getattr(content, 'markdown', None))
    print(getattr(content, 'fields', None))

The response includes status, analyzerId, apiVersion, and contents. Fields may contain type, value, and confidence, while markdown preserves readable content for review or RAG. Keep secrets out of source code and review low-confidence values before sensitive decisions.

radio_button_uncheckedshort_text8. Asynchronous operation and JSON

Analysis can take time, so the API uses an asynchronous operation. The initial request returns an Operation-Location address, which the client polls until the status becomes Succeeded or Failed. The SDK wraps this cycle in the poller returned by begin_analyze and the result() call.

1. Submit a file or accessible URL to the analyzer.
2. Store the identifier or Operation-Location.
3. Poll at controlled intervals rather than in a tight loop.
4. After completion, read contents, markdown, fields, and confidence scores.
5. Handle failure, timeout, and unsupported content.

radio_button_uncheckedshort_text9. Structured extraction from audio

For audio, the service can produce transcription, speaker diarization, a summary, and schema-defined fields. A voicemail schema could request caller, message summary, actions, callback number, and alternative contact details.

radio_button_uncheckedshort_text10. Structured extraction from video

Video combines visual cues, audio, and time. For a meeting, an initial schema might request location, in-person attendees, remote attendees, and total attendees. The sample frame depicts a conference room with one in-person and three remote participants, for a total of four.

A richer recording schema could count attendance at intervals, identify who spoke and what they said, summarize the discussion, and list assigned actions. Temporal relationships among frames, speech, and events distinguish full video analysis from inspecting one image.

radio_button_uncheckedshort_text11. Client for audio and video analyzers

The document-client pattern also works for media. Select prebuilt-audioSearch or prebuilt-videoSearch, provide an accessible URL, and wait for the poller. Depending on the analyzer and schema, contents can include transcript/markdown, fields, segments, and timing metadata.

analyzer_id = 'prebuilt-audioSearch'
inputs = [{'url': 'https://example.org/samples/voicemail.wav'}]

poller = client.begin_analyze(analyzer_id=analyzer_id, inputs=inputs)
result = poller.result()

for content in result.contents:
    print('TRANSCRIPT:', getattr(content, 'markdown', None))
    print('FIELDS:', getattr(content, 'fields', None))

radio_button_uncheckedshort_text12. Exercise and explained assessment

The approximately 25-minute exercise uses Azure Content Understanding in Microsoft Foundry to analyze documents. It requires an Azure subscription; new accounts may include free credits for the first 30 days. Compare source content with fields and JSON before automating.

• Start the original Microsoft Learn exercise

radio_button_uncheckedshort_text13. Summary and official references

Content Understanding applies one workflow to documents, images, audio, and video: ingest, extract content, reason with AI, map to a schema, and return structured data. Prebuilt analyzers accelerate common scenarios; custom analyzers add domain consistency; the portal, REST API, and Python SDK span validation through scaled automation.

• Use OCR for text and layout; use schema and semantics for business meaning.
• Model nested collections for repeated groups such as invoice items.
• Test in the portal and integrate only after checking fields, JSON, and confidence.
• Treat analysis as asynchronous and protect endpoints, keys, personal data, and recordings.

• Azure Content Understanding overview
• Content Understanding glossary
• Analyzer reference
• REST API quickstart
• Azure Content Understanding SDK for Python