We migrated our background email notification dispatcher from Node.js to Rust Axum with Tokio. While cold start time plummeted from 450ms down to 4ms, our first live hackathon spike of 10,000 concurrent requests resulted in 504 Gateway Timeouts. Here is how we diagnosed and resolved the bottleneck.
1. The Diagnosis: Connection Pool Starvation
Our Tokio async tasks were spawning freely without bounding the maximum concurrent database connections. When 10,000 requests hit simultaneously, SQLx connection pool queues exhausted, causing incoming HTTP threads to deadlock waiting for available sockets.
// Fixed connection pool with semaphore backpressure
let pool = PgPoolOptions::new()
.max_connections(50)
.acquire_timeout(Duration::from_secs(3))
.connect(&database_url)
.await?;KEY TAKEAWAY:Always bound your concurrent async tasks with semaphores when communicating with external databases.
TAGS:#Rust#Tokio#Post-Mortem#System Design