Deploy and expose AI inference APIs on Azure Kubernetes Service

radio_button_uncheckedshort_text1. From a container image to a highly available AI endpoint

Azure Kubernetes Service (AKS) operates a managed Kubernetes control plane on Azure. You supply cluster capacity and workload definitions while Azure manages the control-plane infrastructure. This makes AKS suitable for containerized model-inference APIs, vector-search services, and other AI components that need repeatable deployment, resilience, scaling, and network access.

A containerized FastAPI inference service becomes an application only after Kubernetes knows what to run, how many copies to maintain, which resources and configuration it needs, and how clients reach it. YAML manifests describe that desired state; the cluster continuously reconciles reality with it.

• A Pod is the smallest deployable unit and normally wraps one application container.
• A Deployment creates and replaces Pods and maintains the requested replica count.
• A Service gives changing Pods a stable IP or DNS endpoint and distributes traffic by label.
• kubectl submits manifests, inspects state, reads logs, and exposes diagnostic events.

Topic summary

AKS manages orchestration while Pods run containers, Deployments preserve the desired replica state, Services stabilize access, and kubectl controls and diagnoses the workload.

radio_button_uncheckedshort_text2. How Pods, Deployments, Services, and labels work together

Running a Pod directly is possible but fragile. A Deployment owns a ReplicaSet, the ReplicaSet maintains Pods, and the Deployment replaces a failed Pod automatically. A Service does not point to fixed Pod addresses; its selector discovers Pods whose labels match and sends connections to those endpoints.

The label contract is critical. If the Pod template says app: inference-api but the Service selects a different value, the Service exists yet has no endpoints. Names of the Deployment and Service do not need to match, and Service port 80 may legitimately forward to container port 8080.

Topic summary

The Deployment controls Pod lifecycle, while the Service independently finds Pods by matching labels; matching labels, not matching object names, connect the two.

radio_button_uncheckedshort_text3. Anatomy of a Kubernetes Deployment manifest

A Deployment uses apiVersion apps/v1 and kind Deployment. metadata names the object and may select a namespace. spec declares replicas, a selector, and the Pod template. The template repeats the selected label and contains the container definitions. This declarative file can be reviewed, versioned, and reapplied rather than reproducing manual commands.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-inference-api
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: inference-api
  template:
    metadata:
      labels:
        app: inference-api
    spec:
      containers:
      - name: api
        image: myregistry.azurecr.io/inference-api:v1.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "4Gi"
            cpu: "2000m"
        env:
        - name: MODEL_NAME
          value: "gpt-4"
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: api-secrets
              key: api-key

The image reference follows registry.azurecr.io/repository:tag. Build and push the image before deployment, ensure it is accessible to the cluster, include application code and dependencies, and prefer an immutable version tag over latest when reproducibility matters.

Topic summary

A Deployment manifest combines identity, replica intent, matching labels, Pod template, container image, ports, resources, and configuration in one versionable desired-state document.

radio_button_uncheckedshort_text4. Replicas, availability, and resource requests and limits

Two replicas provide basic tolerance for a single Pod failure; three are a stronger production baseline; four or more should be justified by traffic and load testing because each copy consumes compute. Kubernetes restarts failed Pods, but several replicas let surviving Pods continue serving during that recovery.

Requests reserve the amount needed for scheduling. If no node has the requested CPU or memory, the Pod remains Pending. Limits cap consumption: CPU is throttled when constrained, while exceeding the memory limit can produce an OOM kill and restart. For a small inference model, 2–4 GiB plus roughly 20 percent headroom and one to two CPU cores may be a starting point, not a universal rule.

Topic summary

Use replicas for resilience and measured throughput; set requests for schedulability and limits for containment, then refine all values from real model behavior.

radio_button_uncheckedshort_text5. Environment variables, Secrets, and registry access

Use literal environment variables for nonsensitive settings such as model names and API endpoints. Reference a Kubernetes Secret through valueFrom and secretKeyRef for API keys and credentials. Create it separately, for example: kubectl create secret generic api-secrets --from-literal=api-key=<value>. Never commit the secret value in a manifest or source history.

AKS must also authenticate to Azure Container Registry. For a normal RBAC registry, AKS-to-ACR integration grants AcrPull to the kubelet managed identity. An ABAC-enabled registry instead requires Container Registry Repository Reader. Private registries outside that integration can use an image pull Secret. Verify the applicable model rather than assuming image access is automatic.

Topic summary

Keep ordinary configuration in environment variables, sensitive values in referenced Secrets, and grant the AKS kubelet identity the correct least-privilege registry pull role.

radio_button_uncheckedshort_text6. Choose the right Kubernetes Service type

Pod IP addresses are ephemeral. A Service creates a persistent virtual endpoint and routes traffic to matching Pods even after replacements. The correct Service type depends on who should call the workload.

For production HTTP routing across several applications, TLS termination, or host/path rules, an Ingress design may be more appropriate than one public LoadBalancer per Service. The module focus remains the four Service types above.

Topic summary

Use ClusterIP internally, NodePort for limited direct-node scenarios, LoadBalancer for Azure-managed exposure, and ExternalName as a DNS alias to an external dependency.

radio_button_uncheckedshort_text7. Service manifest, selectors, and port mapping

The Service manifest uses apiVersion v1 and kind Service. type controls exposure, selector binds the Service to Pod labels, port is where clients connect, and targetPort is where the application listens inside each container. For HTTP, port 80 commonly forwards to FastAPI on 8080; HTTPS often uses port 443 when termination is configured.

apiVersion: v1
kind: Service
metadata:
  name: inference-api-service
spec:
  type: LoadBalancer
  selector:
    app: inference-api
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

Inside the cluster, DNS follows servicename.namespace.svc.cluster.local, such as inference-api-service.default.svc.cluster.local. NodePort clients use node-ip:nodeport, usually a port above 30000. LoadBalancer clients use the assigned external address shown by kubectl get svc.

Topic summary

A Service selector must match Pod labels, while port receives client traffic and targetPort forwards it to the application process.

radio_button_uncheckedshort_text8. Apply manifests and understand reconciliation

kubectl apply reads YAML and creates or updates the declared objects. Kubernetes then pulls images, schedules Pods, starts containers, creates the Service, and provisions networking. The command is asynchronous: a successful response means the desired state was accepted, not that every Pod or public IP is already ready.

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
# Equivalent options:
kubectl apply -f deployment.yaml -f service.yaml
kubectl apply -f .

Applying a directory is convenient but can submit unrelated YAML, so keep deployment artifacts scoped and review changes. Reapplying is idempotent at the declarative level: the control plane compares the submitted configuration with live state and reconciles the difference.

Topic summary

Use kubectl apply to create or update declarative resources, then separately observe reconciliation because Pods, image pulls, and Load Balancer provisioning finish asynchronously.

radio_button_uncheckedshort_text9. Verify Pods, Deployments, Services, logs, and connectivity

A healthy deployment shows the intended READY count, Running Pods, an AVAILABLE replica count that reaches the target, and a Service address appropriate for its type. Pending may be transient during image pulling or capacity waits; CrashLoopBackOff means repeated application startup failure. A LoadBalancer EXTERNAL-IP can remain pending while Azure provisions it.

kubectl get pods
kubectl get deployment
kubectl get svc inference-api-service
kubectl logs -l app=inference-api

# Test an internal ClusterIP Service
kubectl run -it --rm debug --image=alpine:latest --restart=Never -- sh
wget http://inference-api-service:80

# Test a public LoadBalancer address
curl http://<EXTERNAL-IP>
curl http://<EXTERNAL-IP>/predict -X POST -d '{"input":"test"}'

For internal access, launch a disposable debug Pod and call the Service name. For external access, test the public IP and a real inference endpoint. Health, readiness, and inference requests prove different layers: a process can be alive but not ready to serve a loaded model.

Topic summary

Check object status and logs, wait for the appropriate Service address, and test health, readiness, and real inference rather than treating successful manifest submission as completion.

radio_button_uncheckedshort_text10. Diagnose ImagePullBackOff and CrashLoopBackOff

ImagePullBackOff means the node cannot retrieve the image. Inspect Pod events; verify registry host, repository, and tag; confirm the image exists with az acr repository list or show-tags; and validate the AKS-to-registry role assignment. A manual docker pull can isolate an invalid image reference from a cluster permission problem.

CrashLoopBackOff means the container starts and repeatedly exits. Read current and --previous logs, then describe the Pod for events. Common causes include missing environment variables, unavailable Secrets, incorrect command or port, model startup failure, and unmounted configuration. Run the image locally to prove whether the fault is in the container or cluster configuration.

# Image pull or scheduling events
kubectl describe pod <pod-name>

# Current and previous-container logs
kubectl logs <pod-name>
kubectl logs <pod-name> --since=10m
kubectl logs <pod-name> --previous

# Capacity and Service selectors
kubectl get nodes
kubectl top nodes
kubectl get pods --show-labels
kubectl get pods -L app
kubectl describe svc inference-api-service

Topic summary

Use events for image and scheduling failures, current and previous logs for application crashes, and compare registry permissions, manifest configuration, and local container behavior.

radio_button_uncheckedshort_text11. Diagnose Pending Pods and Services with no endpoints

A persistently Pending Pod normally cannot be scheduled because requests exceed available CPU or memory, a node constraint cannot be met, or a node is unhealthy. Describe the Pod and nodes, inspect capacity with kubectl top nodes when Metrics Server is available, reduce unrealistic requests, or add cluster capacity with az aks scale after validating cost and quota.

A Service with Endpoints: <none> cannot forward traffic. Compare kubectl get pods --show-labels or -L app with kubectl describe svc. Correct either the Pod template label or Service selector and ensure the Pods are Ready. A Service object alone does not prove a working path.

Topic summary

Pending is primarily a scheduling-capacity investigation; no endpoints is primarily a label, readiness, or selector investigation.

radio_button_uncheckedshort_text12. Guided exercise: deploy an AI inference API

The source laboratory takes about 30–40 minutes. It deploys a Microsoft Foundry model, Azure Container Registry, and an AKS cluster; completes deployment.yaml and service.yaml with container settings, probes, resource limits, and load balancing; then uses a Python client to test health, readiness, and inference.

1. Download the starter project and review the application and manifest placeholders.
2. Deploy the Azure resources with an identity permitted to create the required services.
3. Complete container image, health probes, requests, limits, labels, selectors, and port mapping.
4. Apply both manifests, wait for Pods and the LoadBalancer address, and inspect failures before continuing.
5. Run the client tests against health, readiness, and model-inference endpoints; remove disposable resources afterward.

Prerequisites are an Azure subscription, Visual Studio Code, Python 3.12 or later, the latest Azure CLI, and kubectl. Azure Container Registry task runs may not be covered by Azure free credits, so the exercise can require pay-as-you-go or another paid plan.

Topic summary

The lab joins Microsoft Foundry, Azure Container Registry, AKS, YAML, probes, resources, load balancing, and Python endpoint tests in one end-to-end deployment.

radio_button_uncheckedshort_text13. Assessment review and exam decisions

Topic summary

Remember the operational contract: LoadBalancer exposes, requests affect scheduling, labels connect Services to Pods, --previous retrieves crash logs, and replicas controls Pod copies.

radio_button_uncheckedshort_text14. Production checklist and official references

• Use immutable image tags and verify AKS-to-registry authentication.
• Keep sensitive values out of Git and reference Secrets or an external secret solution.
• Set realistic requests, limits, replicas, and health probes from load tests.
• Match labels and selectors and validate every port-to-targetPort path.
• Observe Pods, Deployment availability, Service endpoints, events, logs, health, readiness, and inference.
• Prefer private exposure, TLS, Ingress, network policies, and least privilege when the architecture requires them.

1. Azure Kubernetes Service documentation
2. AKS core concepts
3. Deploy an application to AKS
4. Services in Azure Kubernetes Service
5. Authenticate AKS with Azure Container Registry
6. Troubleshoot image pull errors in AKS
7. kubectl command reference

Topic summary

Production readiness combines repeatable images, least-privilege access, resource and probe evidence, stable networking, and layered observability backed by current Microsoft documentation.