GPUHedge is an open-source tool that uses speculative execution across serverless GPU providers to reduce cold start latency from 117 seconds to 30 seconds, achieving a 4x improvement in p95 latency. Cold start latency is a major pain point in deploying large AI models on serverless GPUs, and this hedging approach offers a practical, cost-effective solution that can significantly improve user experience and reduce wasted compute. In benchmarks, a fixed RunPod → Cerebrium hedge with a 10-second launch delay reduced p95 latency from 116.6s to 29.4s, eliminated all requests over 60 seconds, and lowered modeled active-compute cost from $0.0114 to $0.0083 per request.
Background
Serverless GPU providers experience cold start latency when a GPU instance must be initialized before processing a request, which can take over a minute. GPUHedge treats this as a speculative execution problem: it starts a request on a primary provider, monitors progress, and conditionally launches a backup on another provider, canceling the losing job via the provider's API.