41 regions in: how the scheduler picks your node in under 30ms

The intuitive design would be to ping every node when a job arrives and hand back whichever answers first. That is not what happens, and the difference is worth understanding before trying to reason about why a job landed where it did.

A probe round trip across 3,180 nodes, even fanned out in parallel, costs more latency than any placement decision could recover. So the measuring happens continuously and out of band, and the request path only reads the result. The median scheduling decision finishes in under 30ms because it is a filter and a sort over state that was already in memory before the job existed.

Everything below therefore describes two systems on very different clocks: a probe loop that maintains the candidate sets, running on a seconds-scale interval, and a scheduler that reads them in milliseconds. Almost all of the engineering is in the first one.

Every job declares a VRAM class: 16, 24, 48 or 80 GB. It is the first thing the scheduler applies and it is absolute. A job that declared 48 GB is never placed on a 24 GB card no matter how idle, close or healthy that card is, because the failure mode is not slowness. It is an allocation error forty minutes into a run.

The scheduler will place a job on a larger class than it asked for, but only once its own class is exhausted in the region. Over-placement strands capacity: an 80 GB card running a 16 GB job is 64 GB of nothing, and the jobs that genuinely needed 80 GB have nowhere else to go. So it is a fallback, it is applied late, and it always prefers the smallest class that still fits.

This is also why the hardware bar sits at 16 GB. A card below that has no class to serve, so there is nothing for the scheduler to route to it and no way for it to earn.

A node has to be ACTIVE and hold a health score above 70 to appear in a candidate set at all. That threshold is a floor, not a target. It exists so a degraded node drops out of rotation automatically instead of continuing to accept jobs until somebody notices the failures piling up.

The same filter defines what the network claims publicly. When the site says a region is served, it means at least one ACTIVE node above that floor is in it, in at least one VRAM class. A region whose nodes are all sitting at 62 is not a served region, and it is not counted as one.

Inside the eligible set, nearest means measured rather than geographic. Distance on a map predicts round-trip time poorly once transit paths, peering and congestion are involved. A node 200 km away behind a bad path routinely loses to one 800 km away on a clean one.

Each node keeps a rolling histogram of probe round-trip times per client region, bucketed on a fixed logarithmic scale. Histograms rather than running averages, for one reason: a mean hides the tail. A node that answers in 8ms nine times out of ten and 400ms on the tenth has a respectable average and a miserable experience, and only a percentile makes that visible.

The scheduler reads p95, not the median, and it reads it as a bucket lookup rather than a computation. Fixed buckets are cheap to merge across probe workers and cheap to query, which is a large part of why the decision fits inside 30ms at all. It is also why the histograms are kept per client region instead of globally: one global distribution would average away exactly the regional differences the scheduler exists to exploit.

The probe loop is the part that actually knows anything. Every node runs an agent, and the control plane checks it on a fixed interval for agent liveness, whether the driver still enumerates the GPU, board temperature and clock throttling, ECC error counts, free VRAM against what the node claims, disk headroom, and whether the container runtime will genuinely start something rather than merely reporting that it is up.

Those feed a decayed rolling score. Decay matters in both directions. One bad probe should not evict a node that has been solid for a month, and a solid month should not buy indefinite credit for a node that has just started throttling. A failing node slides toward the floor over minutes rather than instantly, and it climbs back the same way.

Crossing below 70 removes the node from every candidate set it was in. It keeps being probed, because eviction is not deregistration, and it is re-added the moment the score recovers. None of that needs a human, which is the design goal. At 3,180 GPUs and 2.4M GPU-hours a month, anything requiring a person in the loop is a thing that does not happen at 3 a.m.

The one thing the scheduler deliberately never measures is the job itself. Placement reads VRAM class, region, health and latency, and nothing else. Jobs run in isolated containers, workload contents are never inspected or retained, and only scheduling and billing metadata is kept.

If a node stops answering probes mid-job, the job is rescheduled. There is no special failover path. The scheduler runs the same filter again with the failed node no longer in the candidate set, which is why failover is uneventful rather than a distinct mode carrying its own distinct bugs.

Because containers are isolated and leave nothing on the host that outlives them, rescheduling is a restart on a different node rather than a migration. That is a real constraint on job design: anything long-running should checkpoint to its own storage, because the network guarantees a place to run, not an uninterrupted process.

The operator side of that is worth stating plainly. GPU-hours are credited when they are verified as delivered, so a node that drops a job halfway is credited for the half it served and the replacement node is credited for the rest. 99.94% pool uptime is a property of the network rather than a promise about any single node, and it holds precisely because individual nodes are allowed to fail and be replaced without ceremony.

Uptime is the highest-leverage thing you control, because the health score is a gate and not a ranking term. Above 70 you are in the candidate set; below it you earn nothing. There is no partial credit for a node sitting at 68.

Network path is second, and it is more tunable than most people expect. The scheduler reads p95 round-trip time, so the thing to fix is jitter rather than peak throughput. A saturated uplink with no queue management produces a tail that quietly costs you placements even though a speed test looks fine. Symmetric 100 Mbit/s is the floor, and a clean 100 is worth more than a congested gigabit.

Third is honest capacity. The probes compare what a node claims against what it can demonstrate, free VRAM in particular, and a node advertising more than it can deliver fails those checks and slides toward the floor. Declaring one card fewer than you have is a better trade than declaring one more.

What you cannot tune is demand in your region. A card where jobs outnumber eligible nodes is scheduled far more than an identical card in a saturated region, and that gap is larger than anything the other three levers produce. It is the thing to check before adding hardware, not after.