Azure Event Grid: CloudEvents, filters, retries, and custom AI events

radio_button_uncheckedshort_text1. The event-driven AI scenario and learning goals

A content-moderation platform receives images and text in Azure Blob Storage, creates embeddings, runs classifiers, and asks reviewers to inspect risky results. Polling every source and pipeline stage adds idle requests, delays reactions, couples services, and becomes inefficient when upload volume spikes. The design needs a lightweight signal whenever data arrives, inference completes, a model changes, or a stage hands work to the next one.

Azure Event Grid provides that routing layer. This chapter explains how to model events, select topics and delivery modes, filter subscriptions, handle transient failures, preserve undeliverable events, observe outcomes, and publish application events through the SDK or REST API.

• Identify events, sources, topics, subscriptions, and handlers in an AI workflow.
• Choose system, custom, or namespace topics and push or pull delivery.
• Create interoperable CloudEvents with useful types, subjects, and compact payloads.
• Configure filters, retries, dead-lettering, batching, and monitoring.
• Publish securely with Microsoft Entra ID, managed identity, Python, or REST.

Topic summary

Event Grid replaces wasteful polling with explicit, routable state-change notifications that let AI stages react independently.

radio_button_uncheckedshort_text2. Understand the Event Grid routing model

Event Grid is a fully managed, low-latency publish-subscribe service with usage-based event operations. An event is a compact fact that something changed; it is not the full resource or result. A source publishes the fact to a topic, an event subscription selects matching events, and a handler performs the reaction. Azure Blob Storage, Key Vault, Container Registry, Event Hubs, custom applications, and partner SaaS systems can all act as sources.

Topic summary

Separate the fact, its producer, the routing topic, each filter, and each reaction; that separation is the foundation of loose coupling.

radio_button_uncheckedshort_text3. Choose system, custom, or namespace topics

Use system events to react to platform state, custom events to announce application milestones, and namespace topics when consumers need pull control, private endpoints, or a unified namespace model. A moderation flow can start with BlobCreated on a system topic and publish com.contoso.ai.ContentClassified to an application topic after inference.

Topic summary

Match the topic to the publisher and consumption model: Azure resource events, application-defined milestones, or namespace-based push and pull.

radio_button_uncheckedshort_text4. Apply event-driven patterns and select handlers

Reactive data processing starts work when data arrives. Pipeline coordination publishes a new event only at meaningful boundaries such as embeddings completed or index refreshed. Model lifecycle events announce training, validation, promotion, and deployment. These patterns allow a new audit, notification, or quality service to subscribe without modifying the publisher.

Push delivery sends an HTTP request as soon as an event matches. Pull delivery is available for namespace topics: the consumer chooses when and how fast to receive and can acknowledge, release, reject, or renew a lock. Prefer pull when the consumer cannot expose an endpoint, needs private connectivity, or must control the intake rate.

Topic summary

Publish only meaningful state transitions, use push for immediate callbacks, and use namespace pull when the consumer needs timing, rate, or network control.

radio_button_uncheckedshort_text5. Standardize events with CloudEvents 1.0

Event Grid supports its original schema and CloudEvents 1.0. CloudEvents is the recommended format for new designs because it standardizes event context across products and protocols and allows extension attributes. Event Grid schema remains supported for compatibility. A topic that accepts Event Grid events can deliver them as CloudEvents, but CloudEvents input cannot be converted to Event Grid output because the older schema cannot represent CloudEvents extensions.

Topic summary

Use CloudEvents 1.0 end to end for new implementations and plan schema compatibility before creating topics and subscriptions.

radio_button_uncheckedshort_text6. Design useful CloudEvents for AI operations

A CloudEvent requires specversion, type, source, and id. The reverse-DNS type names the business fact; source identifies the producing context; id must uniquely identify the occurrence. Subject, time, datacontenttype, and data add routing and business context. Subject works well as a hierarchical path such as /pipelines/moderation/image-classifier.

{
  "specversion": "1.0",
  "type": "com.contoso.ai.InferenceCompleted",
  "source": "/services/content-moderation",
  "id": "evt-7e18",
  "subject": "/pipelines/moderation/image-classifier",
  "time": "2026-08-24T14:30:00Z",
  "datacontenttype": "application/json",
  "data": {
    "requestId": "req-78901",
    "modelName": "content-classifier",
    "modelVersion": "3.2.1",
    "processingDurationMs": 1250,
    "resultLocation": "https://results.example/output/req-78901.json",
    "status": "completed",
    "summary": { "flagged": true, "confidence": 0.97 }
  }
}

Keep data compact. Include correlation identifiers, model and version, duration, status, a decision summary, and a secured URI for the full result. Useful types include InferenceCompleted, EmbeddingsRefreshed, BatchProcessingStarted, AnomalyDetected, ModelRetrained, and ContentClassified. Do not publish every internal variable change; publish facts another component can act on.

Topic summary

Build small, self-describing CloudEvents around meaningful business facts and reference large results instead of embedding them.

radio_button_uncheckedshort_text7. Filter by event type and subject

A subscription can restrict included event types so a handler receives only the facts it understands. Subject prefix and suffix filters route hierarchical resources, file families, tenants, or pipeline stages without opening the data object. When combined, type and subject conditions must both match.

az eventgrid event-subscription create \
  --name flagged-content \
  --source-resource-id <topic-resource-id> \
  --endpoint <handler-endpoint> \
  --included-event-types com.contoso.ai.ContentClassified \
  --subject-begins-with /pipelines/moderation/ \
  --advanced-filter data.status StringIn flagged review

Stable naming is operational infrastructure. Define event type and subject conventions before producers multiply, preserve casing, and test positive and negative examples. Type tells what happened; subject tells where or to which business object it happened.

Topic summary

Use type for semantic routing and subject prefixes or suffixes for path routing; combine them to reduce unnecessary handler work.

radio_button_uncheckedshort_text8. Route on data with advanced filters

Advanced filters inspect CloudEvents context or data fields. Operators include StringIn, StringContains, StringBeginsWith, NumberGreaterThan, BoolEquals, and IsNotNull. A moderation subscription can use StringIn on data.status for flagged or review, while another can select confidence above a threshold or a particular model version.

A subscription supports up to 25 advanced filters and 25 total filter values, with string values limited to 512 characters. Conditions across filter clauses use AND; multiple values within one condition provide alternatives. Keys containing a dot cannot be escaped, so design filterable field names accordingly. Filters are routing controls, not an authorization boundary, and handlers must still validate the event.

Topic summary

Advanced filters move lightweight selection into the router, but contracts, limits, field names, and handler validation still require deliberate design.

radio_button_uncheckedshort_text9. Acknowledge push delivery correctly

For push delivery, Event Grid sends HTTP POST requests. Only 200, 201, 202, 203, and 204 acknowledge success. The endpoint has 30 seconds to respond; a timeout is a failed attempt. Each request contains an array, with one event by default. Delivery is at least once, so duplicate events can occur and the handler must use the event id or a business key for idempotent effects.

Return 503 for temporary capacity or dependency failures instead of mislabeling them as 400. A 401 from a webhook is not retried; the special delayed retry for 401 and 404 applies to Azure resource endpoints. If inference cannot finish within 30 seconds, persist accepted work, return 202, and complete it asynchronously.

Topic summary

Acknowledge quickly with the right 2xx status, classify transient failures accurately, and make every side effect safe under at-least-once delivery.

radio_button_uncheckedshort_text10. Configure retry lifetime and attempt limits

Retry uses exponential backoff with randomization. The nominal sequence is 10 seconds, 30 seconds, 1 minute, 5 minutes, 10 minutes, 30 minutes, 1 hour, 3 hours, 6 hours, and then every 12 hours through the 24-hour window. The subscription controls two stopping conditions: maximum delivery attempts from 1 to 30, and event TTL from 1 to 1,440 minutes. Both default to 30 attempts and 1,440 minutes; the first limit reached ends delivery. The schedule itself is not configurable.

A short TTL can expire before a high attempt limit becomes relevant. Choose both from the event business value, recovery time of the handler, and downstream capacity. Event Grid can skip attempts when an endpoint appears unhealthy. If the endpoint later responds within three minutes, best-effort removal from the retry queue can still race with another attempt and produce a duplicate.

az eventgrid event-subscription create \
  --name moderation-subscription \
  --source-resource-id <topic-resource-id> \
  --endpoint <handler-endpoint> \
  --max-delivery-attempts 5 \
  --event-ttl 30 \
  --deadletter-endpoint <blob-container-resource-id> \
  --max-events-per-batch 100 \
  --preferred-batch-size-in-kilobytes 512

Topic summary

Tune TTL and attempt count together because whichever expires first wins, while idempotency remains necessary despite backoff.

radio_button_uncheckedshort_text11. Preserve undeliverable events with dead-lettering

Dead-lettering is disabled until the subscription points to an existing Azure Blob Storage container. When retries or TTL are exhausted, Event Grid can store the original event plus diagnostic properties such as deadLetterReason (for example, MaxDeliveryAttemptsExceeded or MaxRetryDurationExceeded), deliveryAttempts, lastDeliveryOutcome (such as NotFound, TimedOut, Busy, or Forbidden), publishTime, and lastDeliveryAttemptTime. Without a dead-letter destination, expired events are dropped.

Treat the container as an operational queue. Secure it, set retention, alert on new blobs, inspect patterns such as NotFound or TimedOut, correct the root cause, and replay through a controlled idempotent process. A second event subscription on the dead-letter storage container can trigger notification or reprocessing.

Topic summary

Dead-lettering converts exhausted delivery into inspectable evidence, but only an owned alert, diagnosis, retention, and replay process makes it useful.

radio_button_uncheckedshort_text12. Batch output without breaking reliability

Output batching is off by default. A subscription can request 1 to 5,000 events per batch and a preferred batch size from 1 to 1,024 KB. These are best-effort targets: Event Grid does not wait to fill a batch, and one large event can exceed the preferred size while remaining deliverable.

Delivery is all or none. The endpoint must process the entire batch and return one result within 30 seconds; partial success is unsupported and any failure retries the whole batch. Select a size the handler can validate and process safely, and retain per-event idempotency because previously completed items can reappear.

Topic summary

Batching lowers HTTP overhead, but the handler must finish the complete batch within the timeout and safely tolerate whole-batch redelivery.

radio_button_uncheckedshort_text13. Monitor event flow and delivery outcomes

Azure Monitor exposes delivery success, delivery failure attempts, matched events, dropped events, and dead-lettered events. Correlate these with handler latency, availability, logs, model capacity, and the event source publication rate. A failure metric counts attempts rather than only final losses, so interpret it with success and dead-letter data.

Alert on sudden dead-letter growth, sustained delivery failures, unexpected drops, disappearance of matched events, and changes in success rate. Include event id, type, subject, correlation id, subscription, handler, and attempt outcome in logs without recording secrets or sensitive payloads.

Topic summary

Observe publication, matching, attempts, success, drops, and dead-letter together so an event-driven pipeline has end-to-end health rather than isolated metrics.

radio_button_uncheckedshort_text14. Authenticate publishers with least privilege

Publishers can use an access key, a SAS token, or Microsoft Entra ID. For Azure-hosted production workloads, prefer a system-assigned or user-assigned managed identity and grant the Event Grid Data Sender role at the narrowest topic scope. The role supplies Microsoft.EventGrid/events/send/action without embedding a secret.

Access keys use the aeg-sas-key header and are convenient for a short lab. SAS narrows time and resource exposure but is still a secret. Microsoft Entra ID supports token-based controls, including applicable Conditional Access policy, and avoids application-managed key rotation. Ingress identity is separate from Event Grid identity used to authenticate delivery to downstream Azure services.

Topic summary

Use managed identity and Event Grid Data Sender for production publishing; reserve keys or SAS for explicitly controlled cases and rotate them.

radio_button_uncheckedshort_text15. Publish CloudEvents with the Python SDK

The current azure-eventgrid package provides EventGridPublisherClient, and azure.core.messaging provides CloudEvent. The client accepts AzureKeyCredential, AzureSasCredential, or a token credential such as DefaultAzureCredential. It can send one event or a homogeneous list. For a namespace topic, supply its topic context as required by the current client API.

import os
from azure.core.messaging import CloudEvent
from azure.eventgrid import EventGridPublisherClient
from azure.identity import DefaultAzureCredential

client = EventGridPublisherClient(
    os.environ["EVENTGRID_TOPIC_ENDPOINT"],
    DefaultAzureCredential(),
)

event = CloudEvent(
    type="com.contoso.ai.InferenceCompleted",
    source="/services/content-moderation",
    subject="/pipelines/moderation/image-classifier",
    data={
        "requestId": "req-78901",
        "modelName": "content-classifier",
        "status": "completed",
        "resultLocation": "https://results.example/output/req-78901.json",
    },
)

client.send(event)  # A list publishes a batch of one event type.

Create one long-lived client where practical, validate event data before sending, propagate correlation, and publish at natural checkpoints such as inference completion, model promotion, anomaly detection, or pipeline transition. A successful send means Event Grid accepted the event; it does not mean every handler completed.

Topic summary

The SDK handles serialization and authentication, while the application remains responsible for stable contracts, meaningful checkpoints, and acceptance-versus-processing semantics.

radio_button_uncheckedshort_text16. Publish by REST and design repeatable event patterns

A custom topic also accepts HTTP POST. For a structured CloudEvent, use application/cloudevents+json and authenticate with Microsoft Entra bearer tokens, aeg-sas-key, or SAS as appropriate. A 200 response confirms acceptance for routing; malformed JSON, invalid credentials, or schema mismatch returns a non-2xx response.

curl -X POST "$EVENTGRID_TOPIC_ENDPOINT" \
  -H "Content-Type: application/cloudevents+json; charset=utf-8" \
  -H "aeg-sas-key: $EVENTGRID_TOPIC_KEY" \
  --data-binary @inference-completed.json

Topic summary

REST offers a portable publication path; use the same versioned event patterns and security rules as SDK publishers.

radio_button_uncheckedshort_text17. Guided lab: route moderation events

The source exercise provisions an Event Grid namespace and namespace topic, creates subscriptions filtered for flagged, approved, and all content, then runs a Python Flask application that publishes moderation events. A pull consumer receives them and explicitly acknowledges, releases, or rejects each delivery. Allow about 30 minutes after preparing an Azure subscription, Visual Studio Code, Python 3.12 or later, and the current Azure CLI.

az group create --name ai200-eventgrid-rg --location eastus
az eventgrid namespace create \
  --resource-group ai200-eventgrid-rg \
  --name <globally-unique-namespace> \
  --location eastus
az eventgrid namespace topic create \
  --resource-group ai200-eventgrid-rg \
  --namespace-name <namespace> \
  --name moderation-events \
  --event-retention-in-days 1

python -m venv .venv
python -m pip install --upgrade azure-identity azure-eventgrid flask

1. Download the starter project, create the namespace topic with CloudEvents, and deploy three subscriptions with nonoverlapping, tested filters.
2. Publish approved and flagged events containing stable type, subject, request ID, model, status, and result reference.
3. Receive in pull mode, inspect event and broker properties, acknowledge success, release transient work, and reject invalid input.
4. Verify each subscription receives only its intended events and correlate the publisher, Event Grid, and consumer logs.
5. Trigger a controlled delivery failure, observe retry or dead-letter behavior, then remove the resource group after the lab.

Topic summary

The lab proves publication, filtered routing, pull settlement, and observability with real events instead of relying on configuration alone.

radio_button_uncheckedshort_text18. Assessment review and production checklist

Before production, confirm topic type, CloudEvents contract and versioning, subject and type taxonomy, filter tests, push or pull choice, endpoint validation, 30-second behavior, idempotency store, retry TTL and attempts, Blob dead-letter ownership, batch limit, monitoring and alerts, replay runbook, managed identity and RBAC scope, private connectivity, quotas, cost, and load tests against real handler capacity.

1. Microsoft Learn: What is Azure Event Grid?
2. Microsoft Learn: Event Grid push delivery concepts
3. Microsoft Learn: Event Grid namespace concepts and pull delivery
4. Microsoft Learn: CloudEvents 1.0 with Event Grid
5. Microsoft Learn: Event filtering
6. Microsoft Learn: Message delivery, retry, and batching
7. Microsoft Learn: Dead-letter location and retry policy
8. Microsoft Learn: Authenticate publishers with Microsoft Entra ID
9. Microsoft Learn: Python EventGridPublisherClient

Topic summary

A production Event Grid design combines an interoperable event contract, precise routing, safe delivery behavior, observable failure recovery, and identity-based publishing.