The interesting story in ML infrastructure right now isn’t which managed Kubernetes service to pick — it’s that the tooling running on top of Kubernetes has converged across all three major clouds. Kueue, Karpenter, and Ray aren’t AWS things or Google things anymore; they’re Kubernetes things that happen to run on EKS, AKS, and GKE alike. The real differences between clouds are narrower than they used to be, and increasingly come down to what managed layer each vendor bundles on top of the same primitives. Skipping cluster bootstrap entirely — here’s what’s actually shared, and what’s not.
The shared foundation: scheduling, autoscaling, and distributed compute
Kueue handles job queueing, quotas, and fair-share scheduling across teams on a single cluster. It’s the piece that stops one team’s training run from starving another team’s inference workload, and it’s used natively both on Amazon EKS (via SageMaker HyperPod’s Task Governance) and on Google GKE (paired directly with JobSet for multi-job orchestration).
Karpenter used to be an AWS-only autoscaler. That’s no longer true: Azure shipped an official Karpenter provider for AKS, and it reached general availability as AKS Node Auto Provisioning in early 2026. On both AKS and EKS, Karpenter now does the same job — reading pending pod requirements (CPU, memory, GPU, zone) and provisioning matching nodes on demand instead of relying on pre-defined node pools per instance type.
Ray is the distributed training and serving layer showing up across all three. On GKE, Ray Train has alpha support for JAX (via JaxTrainer) and PyTorch specifically on TPUs, and Ray Serve now supports TPU-backed vLLM for inference. On EKS, Ray runs natively through SageMaker HyperPod with built-in observability and direct Jupyter/Code Editor connections to live clusters.
vLLM and KServe round out the picture as the vendor-neutral serving layer: a high-throughput inference engine (vLLM) paired with a standard model-serving interface (KServe), usable on any of the three without cloud-specific glue code.
What EKS adds: SageMaker HyperPod
HyperPod is AWS’s managed layer over EKS — automatic node health checks, tiered checkpointing for long training runs, and the HyperPod Inference Operator, which collapses the usual stack of Deployments/Services/autoscaling policies into a single custom resource you submit via console, CLI, SDK, kubectl, or Terraform.
What AKS adds: KAITO
The Kubernetes AI Toolchain Operator (KAITO) is Azure’s equivalent move: it automatically provisions right-sized GPU nodes and stands up an inference endpoint for a model in one step, coordinating GPU compute, the inference runtime, and the underlying Kubernetes resources. It integrates with vLLM directly, and exposes an OpenAI-compatible API out of the box — useful if you want a drop-in replacement for a hosted model API that keeps data inside your own cluster. It ships as a managed AKS add-on, so it’s a checkbox rather than a build.
What GKE adds: JobSet, Kueue, and TPU-native orchestration
Google’s differentiator is depth on its own hardware. JobSet, paired with Kueue, orchestrates TPU Multislice workloads — splitting a single large training job across multiple TPU slices and scheduling them as a coordinated unit rather than a pile of independent jobs. Combined with Dynamic Resource Allocation and LeaderWorkerSet (both part of GKE’s AI-native orchestration roadmap), this is the piece that doesn’t have a direct EKS or AKS equivalent, because it doesn’t need to — it exists specifically to make TPU-scale training tractable on Kubernetes.
The practical shape of a 2026 Kubernetes ML stack
However you assemble it, the pattern is now consistent across clouds:
- Scheduling/governance: Kueue, everywhere
- Autoscaling: Karpenter, on both AWS and Azure (GKE leans on its own Dynamic Resource Allocation plus Kueue/JobSet for TPU workloads)
- Distributed compute: Ray, with cloud-specific integration depth (TPU support on GKE, native HyperPod integration on EKS)
- Serving: vLLM/KServe as the common layer, wrapped by a cloud-specific operator (HyperPod Inference Operator on EKS, KAITO on AKS) when you want a managed one-step deploy
The practical takeaway: pick the Kubernetes-native primitives first — they’re portable, and increasingly identical across providers — and treat the cloud-specific managed layer (HyperPod, KAITO, JobSet) as a convenience wrapper, not a lock-in decision. None of this requires solving cluster bootstrap perfectly first; every layer here is designed to sit on top of a cluster you already have.
Sources: Amazon SageMaker HyperPod, Task governance for HyperPod, Best practices to run inference on Amazon SageMaker HyperPod, Microsoft Learn — AI toolchain operator (KAITO) on AKS, Microsoft Learn — Onboard custom models with KAITO, Azure — karpenter-provider-azure, Google Cloud — AI/ML orchestration on GKE, Google Cloud — Orchestrate Multislice workloads using JobSet and Kueue, Google Cloud Blog — Ray on TPUs with GKE, Google Open Source Blog — Transforming Kubernetes and GKE into the leading platform for AI/ML
Opinions are my own.