Documents, optical character recognition, semantic interpretation, normalization, validation, and integration
Suggested study time: 42 minutes • Beginner level • Original rewrite based on Microsoft Learn objectives
By João Ricardo Dutra••Complete original content
1. Information extraction and its scenarios
AI information-extraction solutions turn unstructured content into organized fields. This chapter focuses on documents and images, while speech recognition and multimodal techniques can also recover information from audio and video.
Scenarios range from reading a name, role, email address, and phone number from a business-card photo to complex automation for financial, legal, healthcare, and logistics documents. Structured results can feed databases, APIs, and workflows.
The same extraction discipline supports very different media and business processes.
Common information in four document families.
Domain
Document
Potential fields
Finance
Invoice
Vendor, address, contact, number, dates, payment terms, line items, quantities, prices, totals, taxes, and exemptions.
A comprehensive solution usually performs two major tasks. Computer vision and optical character recognition (OCR) first locate visual text and convert it to machine-readable data. Machine learning or generative AI then interprets meaning and assigns each value to a defined field.
For an expense claim, the system can read a receipt and return vendor, date, subtotal, tax, and total. This structure reduces typing, speeds review, and can start an automated approval process.
OCR answers what is written; field extraction determines what each value means and where it belongs.
Example receipt converted into fields.
Field
Value
Vendor
Fourth Coffee
Date
2024-08-15
Subtotal
$6.48
Tax
$0.49
Total claim
$6.97
3. Choosing the right approach
Documents drive the architecture. Stable forms favor templates, while varied formats and layouts call for flexible models. High volume benefits from automation and optimized hardware; critical uses may require human-in-the-loop validation.
Security and privacy: protect confidential documents, control access, and satisfy industry storage and processing rules.
Compute: deep-learning and generative models can require substantial resources.
Latency: real-time responses can constrain model complexity.
Scale: cloud services can adapt to variable workloads.
Integration: plan API compatibility and data formats from the beginning.
Managed services reduce implementation work and provide proven scale, accuracy, and integration. Examples include and Azure Content Understanding in Foundry Tools.
4. OCR input and preprocessing
OCR converts visible text in images into editable, searchable content. Inputs include scanned invoices and receipts, document photos, image-based PDFs, screenshots, forms, handwritten notes, video frames, and rendered pages.
Input quality directly affects results. Preprocessing removes noise with Gaussian and median filters, morphological operations, denoising autoencoders, or CNNs. Contrast can be enhanced through histogram equalization, adaptive thresholding, gamma correction, or learned enhancement models.
Skew can be corrected with the Hough transform, projection profiles, connected components, or regression CNNs that estimate an angle. Bicubic, bilinear, and Lanczos interpolation adjust resolution; GANs and residual networks can super-resolve poor text.
Input, enhancement, detection, recognition, and post-processing turn an image into usable text.
5. Region detection and reading order
After enhancement, layout analysis separates text, images, graphics, and white space. Classical methods use connected components, run-length encoding, and projection-based segmentation. U-Net, Mask R-CNN, models trained on PubLayNet, and architectures such as LayoutLM add learned visual understanding.
Characters are grouped into words, lines, and paragraphs through distances, white-space analysis, and morphology, or with graph neural networks and transformers. Reading order can come from bounding-box geometry and spatial rules or from sequence and graph models.
Headers, body text, captions, and tables are also classified. SVMs can use handcrafted font, position, and formatting features; CNNs and Transformers learn the categories from labeled documents.
6. Character recognition and context
At the core of OCR, the system examines shape, size, loops, endpoints, and intersections. Traditional features include moments, Fourier descriptors, and structural measurements; CNNs learn discriminative features directly from pixels.
Classification can correlate stored templates, apply HMMs, SVMs, or k-nearest neighbors, or use multilayer perceptrons, LeNet, ResNet, DenseNet, and EfficientNet. The models must handle many fonts, sizes, and writing styles.
Context resolves ambiguity through n-grams, lexicons and Levenshtein distance, LSTMs, BERT-like transformers, and attention. Confidence can be estimated through Bayesian models, softmax probabilities, or ensembles.
7. OCR output and post-processing
Rules based on proximity and confidence thresholds, RNNs/LSTMs, or attention-based transformers assemble recognized characters into words and sentences. Geometry, graph networks, document-AI models, and multimodal transformers such as LayoutLM can preserve paragraphs, line breaks, spacing, and layout.
The output also records coordinates. Mathematical transformations map pixels to document positions, R-trees and quad-trees support spatial queries, and regression models can predict exact placement. Dictionaries, domain vocabularies, n-grams, probabilistic parsers, GPT/BERT-like models, and ensembles then check spelling, grammar, and recognition errors.
8. The field-extraction pipeline
OCR says which text exists. Field extraction explains what that text represents and where it belongs in business systems. Its input includes text, bounding boxes, page locations, reading order, confidence, lines, paragraphs, and other layout metadata.
Position matters: “12345” could be an invoice number, customer ID, or phone number depending on its label and surroundings. Processing continues through candidate detection, schema association, normalization, and integration.
OCR output is interpreted, standardized, and delivered to a business process.
9. Templates, machine learning, and generative AI
Templates use known positions, anchor words, label-value pairs such as “Invoice Number:”, regular expressions, and string matching. They are fast, explainable, and accurate for stable layouts, but require manual maintenance and struggle with position or naming changes.
Machine-learning models learn relationships from example documents. Training can be supervised with labeled fields, self-supervised on large corpora, or multimodal across text, appearance, and position. Graph neural networks model spatial connections, attention selects important regions, and sequence-to-sequence models transform free text into field assignments.
LLMs support schema-guided extraction: a prompt supplies document text and field definitions, and the model performs semantic matching. Few-shot learning adapts custom fields from a small set of examples, while stepwise reasoning can guide identification.
10. Association, tables, confidence, and validation
Simple values can be paired with keys through spatial clustering, reading order, alignment, indentation, and positioning. Named entity recognition detects dates, amounts, and names; part-of-speech tagging and dependency parsing help connect labels and values.
Tables can be found with specialized CNNs, object detection, or graphs representing cells. Row-column association, header detection, and hierarchical processing recover line items, subtotals, and nested structures.
Final confidence combines OCR certainty, pattern fit, and contextual consistency. Cross-field checks can verify relationships, such as line-item subtotals adding up to the invoice total.
Geometry, language, cross-field rules, and confidence cooperate to produce trustworthy data.
11. Normalization and business integration
Raw values need consistent formats. Dates such as MM/DD/YYYY and DD-MM-YYYY are detected, disambiguated, and converted to a standard such as ISO. Currency processing handles symbols and thousands or decimal separators; measurements may need unit conversion. Text can be standardized for case, encoding, and abbreviations.
Quality checks validate phone and email patterns, numeric ranges, and required fields. Outlier detection, distribution analysis, historical comparison, and cross-document validation expose unlikely results.
Integration maps fields to database columns and tables, REST API payloads, or asynchronous queue messages. Renaming, type conversion, and conditional business rules adapt the target schema. Reports record field confidence, document-level quality, and error categories and causes.
12. Exercise and assessment
The exercise combines OCR with a large language model to extract and interpret receipt fields. The image below is the original interface capture and remains unchanged because it is a real screen.
The screenshot is preserved as PNG; only conceptual diagrams were recreated.
Does information extraction query SQL, copy files, or analyze unstructured content to identify fields and values?
In the extraction flow, does OCR retrieve online content, turn visual text into machine-readable text, or convert JSON into images?
Does generative AI enhance extraction by creating documents, requiring one manual rule set per layout, or semantically matching extracted values to fields?
Answers
It analyzes unstructured content and identifies relevant fields and values.
OCR converts images of text into machine-readable textual data.
Semantic language models accurately associate extracted values with the field schema.
13. Chapter summary
AI extraction converts unstructured documents, images, and other media into organized data.
OCR combines acquisition, enhancement, detection, recognition, and post-processing.
Field extraction adds meaning, schema association, normalization, and validation.
Templates, machine learning, and generative AI fit different levels of variety, scale, and explainability.
A production design must consider security, accuracy, latency, scale, integration, and human review.
and Azure Content Understanding in Foundry Tools accelerate delivery.
