Kubernetes can advertise and schedule GPUs through vendor device plugins. Production reliability still depends on the driver, runtime, node image, topology, storage, workload framework, quotas, monitoring, and upgrade process surrounding that resource.
Create explicit GPU node classes
Separate accelerator type, memory, topology, driver, architecture, and workload policy with labels, taints, and node pools. Prevent incompatible jobs from landing on superficially similar capacity.
Control the software stack
Pin container, framework, CUDA, libraries, and model-serving dependencies. Validate combinations before node upgrades. Keep compatibility evidence and a rollback path.
Design queues, quotas, and placement
Use workload priority, team quotas, topology awareness, disruption policy, and fair scheduling appropriate to batch and serving. Avoid holding scarce GPUs while jobs wait for data or manual input.
Observe device and workload health
Track allocatable and unhealthy devices, utilization, memory, power, temperature, errors, queue age, job completion, restarts, and application-level throughput. Connect infrastructure signals to useful work.
Treat scheduling as capacity policy
Device plugins expose resources, but production scheduling must also consider GPU type, memory, topology, sharing mode, node image, driver, runtime, priority, and preemption. Label and taint nodes intentionally. Avoid allowing a workload to request “a GPU” when its compatibility and memory needs are specific.
Decide whether workloads can time-slice, use partitioning, or require exclusive devices. Isolation, predictability, and utilization differ. Validate the chosen mechanism with the model and service-level objective rather than assuming higher packing is free.
Plan capacity for queue time and failure domains. Autoscaling GPU nodes is slower and more supply-constrained than typical CPU scaling. Keep critical inference and interruptible training under separate policies.
Own the node software chain
Pin and test driver, CUDA, runtime, operator, framework, and model compatibility. Roll node changes through a canary pool with performance and quality checks. A successful container start does not prove numerical or throughput equivalence.
Monitor device health, memory errors, temperature, throttling, utilization, allocation, pod retries, and queue delay. Define drain, replacement, and workload recovery procedures before a hardware fault affects a production queue.
Executive decision record
The decision is which scheduling, sharing, node-software, and failure policies provide acceptable utilization without violating workload isolation or latency. Write that decision before selecting a model, vendor, framework, or implementation pattern. A written boundary keeps technical exploration connected to the operating outcome and makes it possible to explain why the organization advanced, revised, or stopped the work.
Approval should depend on same-workload compatibility tests, sustained load, queue and scale behavior, device health monitoring, canary upgrades, and recovery exercises. The evidence does not need to remove every uncertainty, but it should address the uncertainty capable of changing value, architecture, risk, or ownership. Record the baseline, assumptions, unresolved questions, and the person accepting the next stage.
Failure boundary and operating ownership
The central failure to guard against is treating device-plugin exposure as a complete GPU platform while topology, drivers, supply, and workload priorities remain unmanaged. Treat that condition as a testable scenario. Define how the system detects it, what users experience, which action is prevented or reversed, and what evidence reaches the person responsible for recovery.
Long-term accountability sits with the GPU platform team with application owners responsible for resource requests, performance evidence, and service objectives. Supporting specialists can provide platforms, research, review, or delivery capacity, but they cannot substitute for an owner who controls policy and operating change. Name that owner before production and include the ownership path in release evidence and incident procedure.
A practical 90-day application plan
During the first 30 days, convert same-workload compatibility tests, sustained load, queue and scale behavior, device health monitoring, canary upgrades, and recovery exercises into a bounded evidence plan. Assign each artifact to a named contributor, identify the representative inputs required, and agree on the comparison baseline before implementation expands. The objective of this period is to expose the assumption most likely to invalidate the work while the cost of changing direction is still low.
During days 31 through 60, build or instrument the smallest complete workflow that can support the decision about which scheduling, sharing, node-software, and failure policies provide acceptable utilization without violating workload isolation or latency. Include the real data and authorization path where feasible, record exceptions, and review difficult cases with the people who own the underlying process. Resist adding breadth until the team can explain the measured behavior of this narrow slice.
During days 61 through 90, test the boundary represented by treating device-plugin exposure as a complete GPU platform while topology, drivers, supply, and workload priorities remain unmanaged. Exercise degraded dependencies, ambiguous inputs, recovery, and handoff rather than demonstrating only successful cases. End the period with a written advance, revise, or stop decision that cites evidence, residual exposure, expected operating cost, and the next authority boundary.
The review should be accepted by the GPU platform team with application owners responsible for resource requests, performance evidence, and service objectives. That group should confirm not only that the system can work, but that ownership, support capacity, monitoring, and change control are credible. If those conditions are absent, the responsible outcome is another bounded learning stage rather than an unsupported production commitment.
Practical checklist
- GPU node-class and compatibility matrix
- Vendor device plugin lifecycle
- Pinned workload images
- Storage and data-locality design
- Queues, priorities, and quotas
- Device and job telemetry
- Upgrade, drain, failure, and rollback runbooks
Continue reading
- [GPU cloud cost planning](/gpu-cloud-cost-planning-ai-workloads)
- [Cloud architecture for AI workloads](/cloud-architecture-ai-workloads)
Sources and further reading
- [Kubernetes: schedule GPUs](https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/)
- [Kubernetes device plugins](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)