Reinforcement Learning across Regions

Reinforcement learning is how today's strongest reasoning models are built. But the systems that run it assume something most teams don't have: a rack of GPUs stitched together with hundred-gigabit RDMA. AuroraRL removes that assumption, reaching nearly the same speed using flexibly rented GPU capacity across providers and regions.

Who it's forWhy cross-region RL matters

For the last few years, RL post-training (the step that turns a capable base model into a strong reasoner) has effectively belonged to whoever could reserve a hundred-gigabit RDMA supercluster. Individual GPUs can often be rented flexibly across providers and regions. What is much harder to rent on demand is the tightly-coupled fabric that fuses them into one cluster. Without that fabric, the weight-sync wall (next section) puts RL training out of reach. AuroraRL removes that barrier, so serious RL post-training can run on flexible, loosely-connected GPUs.

The numbers make this concrete. On AWS, a single p5.48xlarge node (8×H100) lists at $55.04 per hour on demand, roughly $6.88 per GPU-hour. A 512-GPU cluster therefore costs about $3,500 per hour, or over $590,000 for a one-week run. Availability is the second hurdle: EC2 Capacity Blocks, the service AWS built for exactly this need, allocates tightly-coupled clusters on a scheduled future start date up to eight weeks out, and caps a single reservation at 64 instances (512 GPUs). Anything larger, or anything needed today, leaves self-service entirely.

That makes cross-region RL more than a deployment trick. It is a cost and access question: can we make RL post-training cheap enough, flexible enough, and reliable enough for the broader research community? If the answer requires reserved RDMA clusters, most teams are locked out. If ordinary GPUs can be stitched together, RL becomes a practical tool for many more groups.

For academic labs and small teams, AuroraRL turns a procurement wall into a systems problem. Teams no longer have to chase provider relationships, reserve a fixed RDMA-enabled GPU cluster block, or pay for a long slot before the experiment is proven. They can assemble the GPUs they can actually get: local machines, on-demand cloud instances, and capacity from partner institutions. Start small, add GPUs when they appear, and stop when the run is done. AuroraRL makes this patchwork practical for serious RL post-training: cheaper infrastructure, flexible scheduling, and no private supercluster required.

For large-scale deployments, disaggregated RL is becoming important for a practical reason: Trainer and rollout pools do not want the same hardware. Policy optimization is dominated by dense training compute and benefits from tightly connected high-end GPUs. Rollout decoding is often memory-bandwidth and capacity bound, so it can use cheaper and more heterogeneous accelerators. Big providers and large labs already have GPUs spread across regions, clouds, and hardware generations. The opportunity is to turn that fragmented fleet into one RL system, but policy refresh is the blocker: full-weight synchronization means every remote actor must pull the entire model again each step, so distant GPUs spend more time waiting for weights than generating rollouts. AuroraRL cuts that network overhead: the trainer stays on the best-connected cluster, while rollout capacity is pulled from wherever GPUs are available.

Figure 1 sets the two regimes side by side: a reserved RDMA supercluster on one side, on-demand GPUs scattered across regions on the other.

RL post-training follows a simple loop: a Trainer updates the policy, then broadcasts the new weights to a fleet of Rollout Actors that generate samples and send them back. The catch is that broadcast. Because RL refreshes the entire model every single step, the weight transfer sits squarely on the critical path, and it scales with model size. On a datacenter RDMA fabric this is invisible. On a cross-region network, it becomes the dominant cost of every step.

The wallWhy RL needs a supercomputer

Consider an 8B model. In BF16 that's ~16 GB of weights that must reach every actor, every step. On a 100 Gbps RDMA link the sync takes about 1.3 seconds, comfortably hidden inside the ~45-second window the actors spend generating (Fig. 2). Move the same job to a 1 Gbps cross-cloud link and that sync balloons to 128 seconds, dwarfing generation entirely. Even one-step asynchronous pipelining can't hide a transfer that's longer than the work it's supposed to overlap with, and GPU utilization collapses into the single digits.

This is why state-of-the-art RL systems (OpenRLHF, veRL, StreamRL) are built for tightly-coupled RDMA clusters. And those clusters are exactly what most of the field can't easily get:

• A network premium. On one provider, an identical 16×H100 allocation costs ~$39/hr with an RDMA fabric versus ~$31/hr on standard Ethernet. Same GPUs, same count, a 26% surcharge for the interconnect alone.
• Minimum commitments. Providers gate these clusters behind 24-hour to one-week reservations, so an exploratory 48-hour run can inflate into a one-week bill, up to 3.5× the cost you needed.
• Procurement gatekeeping. Reserving large RDMA blocks often requires sales relationships and credit lines that academic labs, government groups, and startups simply don't have.

Meanwhile, the same 16 GPUs stitched together on demand from two clouds cost ~$26–27/hr with flexible billing. The capacity is right there, sitting behind cross-region links of 1–10 Gbps. The only thing standing between that on-demand capacity and real RL training is the weight-sync wall. So we asked: can ordinary cross-region networks deliver RDMA-class RL throughput?

The insightRL barely changes the model

The usual response to a bandwidth problem is to throw more bandwidth at it. We took the opposite view: stop treating the RL loop as a black box, open it up, and look at what is actually being sent. The answer is striking. Unlike pretraining, where gradient descent nudges nearly every parameter every step, a single RL update leaves the overwhelming majority of weights bit-for-bit unchanged.

Measuring the element-wise nonzero ratio (the fraction of scalar parameters that differ between consecutive checkpoints), only about 1% of elements actually change value per step (Fig. 3). The other 99% stay bit-for-bit identical. The update is fine-grained-sparse: almost every tensor sees some change, but within each tensor only a tiny scatter of elements is touched.

Why is RL so sparse? First, RL post-training uses tiny learning rates around 1e-6, about two orders below pretraining, so each step only nudges the model. Second, online RL updates are in-distribution, which recent theory connects to less forgetting and smaller parameter movement. Third, KL penalties, gradient clipping, and related regularizers further bound each update. The effect persists in training: the nonzero-update ratio quickly falls below 1% and stays there across 800 steps.

BF16 rollout serving amplifies this sparsity. Trainers may update FP32 master weights, but actors receive BF16 checkpoints; FP32 changes below the BF16 representation gap round away before inference. AuroraRL therefore measures and transfers the BF16 inference-side delta that actors actually need. FP16 has a finer mantissa, but its narrower dynamic range makes it less common for this serving path because it raises underflow risk and often requires loss scaling.

The systemHow AuroraRL works

AuroraRL turns that one observation into a system. It reuses some training and inference code from PRIME-RL, including PyTorch FSDP2 training and vLLM inference components. The implementation is ~4.3K lines of Python and does not change the underlying RL algorithm. Three mechanisms work together (Fig. 4).

1 · Lossless sparse deltas

Instead of an ephemeral broadcast, every step produces a versioned, immutable artifact with only the changed elements at full precision. There is no quantization and no dropped update; actors receive exactly the bits the trainer computed. Unifying checkpoint storage and network transfer into one object also makes retries unambiguous: a rollout either matches a verified version hash or the result is rejected.

Naively, each change is an (index, value) pair, and the 4-byte index takes twice the space of the 2-byte value it locates. AuroraRL cuts this overhead in two steps. It stores differences between sorted indices, then encodes those differences with LEB128 variable-length integers (Fig. 5). Small gaps take one byte; rare large gaps spill into a second. The index footprint drops from 4 bytes to under 2 per entry, shaving another 30–50% off the checkpoint.

2 · Streaming transfer

Sparse deltas are decomposable, so AuroraRL never waits for a whole file. It packetizes each delta into segments and ships the first segment while the rest is still being extracted, creating a cut-through path from trainer to relay to actor (Fig. 6). Segments are striped round-robin across several TCP streams. A loss-induced stall on one stream does not block the others, so the WAN link stays saturated. In our US–Canada deployment, multi-stream transfer cuts a 202 MB delta from 4.71 s to 2.90 s. Crucially, transfer overlaps rollout generation. Actors stage the next version while generating on the current one, then commit it in-place at a safe point between batches. A rollout is never served from a half-applied model.

3 · Scheduling for messy fleets

Loosely-coupled actors differ by 2–3× in speed (H100 vs A100 vs L40), and those gaps drift as links congest and spot instances vanish. AuroraRL assigns each actor work in proportion to measured throughput, using an EMA of tokens/s. A fast H100 at 5,000 tok/s and an A100 at 2,500 tok/s split a 300-prompt batch 200/100 and finish together. One feedback signal absorbs GPU throttling, network congestion, and resource contention without tracking them separately. Version-aware gating keeps stale actors from poisoning a step. Time-bounded leases handle failure: if an actor or region misses its deadline, its prompts return to the pool for healthy actors. Results are admitted only when the lease is valid and the version and checkpoint hash match.

Together, sparse deltas, relay fan-out, and version-aware scheduling let the system use more of the global fleet instead of leaving remote capacity stranded behind slow cross-region links.

EvidenceDoes it hold up?

The main end-to-end evaluation runs Qwen3 4B/8B/14B with the Trainer on H100s in the US and A100 rollout actors in Canada. The real cross-cloud link fluctuates between ~500 Mbps and 1 Gbps. Separate stress studies emulate 0.25–10 Gbps links, spread actors across Canada, Japan, the Netherlands, and Iceland, evaluate relay fan-out on a Canada–Australia deployment, and mix A100/L40 actors for heterogeneity. Baselines are vanilla cross-DC full-weight broadcast, its multi-stream variant, and Ideal-SingleDC, an idealized 800 Gbps RDMA upper bound. Each step now ships a 202 MB sparse delta instead of 15.6 GB of full weights, a 79× cut. Even at 0.25 Gbps, the delta crosses in seconds; a full broadcast needs 566 seconds.

Throughput tracks the ideal

End-to-end, AuroraRL nearly matches the RDMA upper bound, as shown in Fig. 7.

It scales across regions

The advantage widens as regions are added, as shown in Fig. 8. Note that the reference bar changes between the two figures. Figure 7 compares against Ideal-SingleDC, an idealized RDMA upper bound, so AuroraRL sits just below it. Figure 8 compares against the vanilla cross-DC baseline doing full-weight broadcast over the same WAN, so AuroraRL sits far above it.

Three smaller ablations round out the picture, all on the same testbed:

More tokens per dollar

It is also cheaper at matched throughput, as shown in Fig. 9.

Lossless by construction

AuroraRL sends a lossless delta, not a compressed gradient or quantized approximation. Under the same one-step asynchronous schedule, actors receive the same policy update that a full-weight refresh would deliver. The system changes how bytes move, not the RL objective, rollout policy semantics, or optimizer update.

This is why the paper frames AuroraRL as improving throughput without sacrificing accuracy. Sparse-delta transfer preserves full precision for nonzero values and activates updates only at safe version-switch points. You get most of the throughput of a high-bandwidth fabric on commodity links, without making model quality a compression trade-off.

Acknowledgments

• Reuses some training and inference code from PRIME-RL, including PyTorch FSDP2 training and vLLM inference components.
• Benchmarked on Hendrycks MATH, GSM8K, and DeepScaleR.

Read more & cite

@article{aurorarl2026,
  title   = {RL over Commodity Networks: Overcoming the
             Bandwidth Barrier with Lossless Sparse Deltas},
  author  = {Ruan, Chaoyi and Luo, Geng and Wan, Xinyi and Zhao, Long
             and Wang, Qinghe and Zhu, Jiaan and Xu, Duling and Xu, Guanbin
             and Wei, Dehui and Liu, Xiang and Li, Cheng and Sun, Haifeng
             and Miao, Congcong and Li, Jialin},
  journal = {arXiv preprint arXiv:2602.11456},
  year    = {2026}
}