Multimodal vision and image and video generation in Microsoft Foundry

radio_button_uncheckedshort_text1. Computer vision: interpreting and creating visual content

Computer vision enables AI systems to interpret images, videos, and live camera feeds. Models automate time-consuming work by locating objects, recognizing patterns, reading text, and understanding scenes. Generative models extend that ability so an application can create original images and videos as well as analyze them.

radio_button_uncheckedshort_text2. Multimodal models and visual reasoning

A multimodal model works with more than one data type, such as text, images, audio, or video. With an image and a text instruction in the same context, it can describe a scene, answer questions, interpret charts, read documents, and explain screenshots. This combination is often called vision-enabled GPT or GPT with vision.

Applications use visual understanding to improve user workflows, while agents use images as evidence for better decisions. Examples include reviewing uploaded documents, examining customer support photos, and explaining diagrams in plain language. One multimodal experience reduces the need for separate vision and language pipelines.

radio_button_uncheckedshort_text3. Multimodal models in Microsoft Foundry

The Microsoft Foundry catalog offers first-party and partner models that accept images. GPT-4.1, GPT-4.1-mini, and GPT-4.1-nano cover descriptions, visual questions, documents, screenshots, charts, and diagrams. The GPT-5 family adds long-context reasoning, structured outputs, and tool use for more demanding enterprise agents and applications.

Partner offerings can include multimodal models from providers such as Anthropic. Because catalogs, versions, and regional availability change, verify that the selected deployment accepts visual input and pass the deployment name assigned in your resource rather than assuming the base model name.

• Describe objects, text, and relationships in an image.
• Answer questions about photos, documents, interfaces, and scenes.
• Extract meaning from charts and combine visual evidence with text instructions.
• Return natural or structured responses and, where supported, invoke tools.

radio_button_uncheckedshort_text4. Image analysis in the Playground

The new Microsoft Foundry Model Playground lets you chat with a vision-enabled deployment. Attach one or more images, enter a question, and examine the interpretation before building the application. The classic portal uses a different interface, but serves the same validation purpose.

radio_button_uncheckedshort_text5. Responses API with text and images

To move from testing to code, the OpenAI Responses API in Foundry accepts native multimodal input. One request contains an instruction and one or more images supplied as URLs or Base64-encoded data. The model processes them together and returns text, supporting applications where users upload images and ask questions in real time.

Base64 converts binary bytes into ASCII text that can travel in JSON or a data URL. For large images, account for transport, cost, and context limits. Store keys in environment variables or use Microsoft Entra ID where supported.

radio_button_uncheckedshort_text6. Python client for visual analysis

Install openai, obtain the resource endpoint and credential, and supply the deployment name. The client points to the OpenAI-compatible endpoint; responses.create receives a multipart message containing input_text and input_image.

pip install openai

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ['FOUNDRY_KEY'], base_url=os.environ['FOUNDRY_ENDPOINT'])
response = client.responses.create(
    model=os.environ['MODEL_NAME'],
    input=[{'role': 'user', 'content': [
        {'type': 'input_text', 'text': 'Describe the image in three bullet points.'},
        {'type': 'input_image', 'image_url': os.environ['IMAGE_URL']}
    ]}]
)
print(response.output_text)

radio_button_uncheckedshort_text7. Image generation models

Analysis models associate visual information with text; generation models reverse the direction and create pixels from a description. In Foundry, the GPT-Image family supports text-to-image, variations, and editing. GPT-Image-1.5 emphasizes fidelity, prompt alignment, consistency, and enterprise workflows; GPT-Image-1 is a broadly integrated general model; GPT-Image-1-mini lowers cost and latency for experiments and volume.

The catalog may also include third-party options such as the FLUX family from Black Forest Labs for photorealistic and stylistically flexible output. Select by quality, editing support, latency, cost, availability, and responsible-use requirements.

radio_button_uncheckedshort_text8. Playground and API for image generation

After deploying a model, describe the desired image in the Playground and wait for the result. Through APIs, an application can create new images or edit existing ones. The model parameter receives the deployment name configured in the resource; authentication can use an API key or Microsoft Entra ID.

import base64
import os
from openai import OpenAI

client = OpenAI(api_key=os.environ['FOUNDRY_KEY'], base_url=os.environ['FOUNDRY_ENDPOINT'])
response = client.responses.create(
    model=os.environ['IMAGE_DEPLOYMENT'],
    input='A clean vector illustration of a robot holding a potted plant, pastel colors.',
    tools=[{'type': 'image_generation'}]
)
encoded = next(item.result for item in response.output if item.type == 'image_generation_call')
with open('generated-image.png', 'wb') as output:
    output.write(base64.b64decode(encoded))

radio_button_uncheckedshort_text9. Video generation with Sora

Video models turn instructions and visual references into original clips. Sora 1 was OpenAI’s first text-to-video model offered in Foundry; it creates short videos, accepts a guiding image, and supports multiple resolutions and durations. It suits storyboards, short animations, and creative prototypes.

Sora 2, presented as preview content, extends the workflow to text-to-video, image-to-video, and video remix. It adds audio, improved realism, and targeted edits. Uses include marketing, concept trailers, and educational media. Sora models include Responsible AI protections and restrictions covering real people, protected characters, and sensitive content.

radio_button_uncheckedshort_text10. Video Playground and REST interface

In the Video Playground, choose a deployment, describe the scene, and set dimensions and duration. Generation takes several minutes and the interface provides code samples. Sora is the native video-output family covered in this module; other multimodal models may understand text, image, or audio without generating video.

REST is an HTTP interface between programs; an SDK is a developer-friendly layer over those operations. When a language lacks an SDK, curl can send requests, data, and headers directly to the API.

radio_button_uncheckedshort_text11. Asynchronous job: create, poll, and download

Rendering video is resource intensive and should not block a synchronous request. The client creates a job, keeps the returned identifier, polls until completed or failed, and downloads the MP4 only after completion. The source material estimates one to five minutes in typical cases, depending on duration, resolution, and capacity.

• Prerequisites: a Foundry/Azure OpenAI resource in a supported region and a Sora deployment.
• Authentication: API key or Microsoft Entra ID.
• Separate endpoints start the job, report its status, and return the completed content.

curl -X POST "$FOUNDRY_ENDPOINT/videos"   -H "Content-Type: application/json" -H "api-key: $AZURE_OPENAI_API_KEY"   -d '{"model":"sora-2","prompt":"Rain on a neon-lit window","size":"1280x720","seconds":"8"}'

curl -X GET "$FOUNDRY_ENDPOINT/videos/{video_id}" -H "api-key: $AZURE_OPENAI_API_KEY"
curl -L "$FOUNDRY_ENDPOINT/videos/{video_id}/content?variant=video"   -H "api-key: $AZURE_OPENAI_API_KEY" --output result.mp4

radio_button_uncheckedshort_text12. Exercise and knowledge check

The approximately 30-minute exercise uses generative models in Microsoft Foundry with visual data. It requires an Azure subscription; new accounts may include free credits for 30 days. The lab covers image analysis and visual-generation experiences.

• Start the original Microsoft Learn exercise

radio_button_uncheckedshort_text13. Chapter summary and official references

Multimodal models connect vision and language to analyze documents, photos, interfaces, and charts. GPT-Image creates and edits images, while Sora creates and transforms video through an asynchronous workflow. Playgrounds, the Responses API, image APIs, and REST expose these abilities to visual assistants, accessibility tools, agents, and creative applications.

• Use text and image together when an answer depends on visual context.
• Pass the deployment name and keep credentials out of source code.
• Choose the model by the required output modality: understanding, image, or video.
• Validate quality, usage rights, content filters, cost, latency, and regional availability.

• Image analysis quickstart
• Vision-enabled chat models
• Azure OpenAI image generation
• Video generation with Sora