Turn the algorithms from agentic-and-long-horizon-rl into a runnable systems engineering reality: how to set up an async rollout cluster, manage environment sandboxes, budget GPU memory, and choose a training stack. Read the agentic page first for the why; this page gives you the how to build.

0. TL;DR

• The essential difference between agent RL training and standard RLHF: multi-turn trajectories (not single-turn responses) + interactive environments (not static datasets) + async rollout (inference decoupled from training).
• The three-pool architecture = the core mental model: Rollout Pool (inference engines + environments) → Reward Pool (verification / judging) → Training Pool (policy updates); three pools are async and independently scalable.
• Training stack selection along four axes: native multi-turn support / async rollout support / environment-management approach / GPU memory strategy (hybrid engine vs separated deployment).
• verl (Volcano Engine): hybrid engine (inference + training on same GPUs, time-multiplexed), SPMD 3D parallelism, agent RL support in active development.
• OpenRLHF (Ray ecosystem): Ray actors are naturally async; mature PPO / DPO / Rejection Sampling pipelines; agent extension via custom Environment wrappers.
• AReaL (Ant Group): fully async agent-native design, generation decoupled from training; systems-level, not an algorithmic variant (see agentic §5).
• The environment is the bottleneck, not the GPU: a single SWE test takes 10–60s, a web page load takes seconds — environment latency >> inference latency; throughput ceiling is capped by N_parallel_envs × per-env speed.
• Multi-turn KV cache memory explosion: $T$ turns, $L$ tokens each → cache $\propto T \times L$; mitigations = KV eviction (lossy) / summarization (error-prone) / truncation-restart (loses history).
• Trajectory storage = the new data engineering: one agent trajectory can reach tens of thousands of tokens; training on thousands means GB-scale data; raw trajectories + rewards + metadata must be stored, and format design affects replay and filtering efficiency.
• Fault tolerance is a hard requirement: environments are flaky (network timeouts / sandbox crashes / non-deterministic tests), GPUs OOM, rollout workers die — the training framework must handle retries / degradation / checkpointing, or it won't survive one epoch.

1. Why agent RL infra ≠ RLHF infra

Every assumption of standard RLHF breaks in the agent setting:

This directly dictates the core design of an agent RL training stack:

1. Inference and training usually need to be decoupled (async): for slow interactive envs (code execution / web interaction, env step in seconds >> GPU inference in milliseconds), synchronous waiting = idle GPU; fast verifiers or generation-dominated scenarios may keep GPU as the bottleneck — the bottleneck shifts with task and scale (see Q9).
2. Environment management is a first-class citizen: you need environment pools (docker sandbox pool / browser pool), health checks, timeout kills, and automatic resets.
3. Trajectories are long sequences: KV cache management, truncation strategies, and storage formats all need rethinking from scratch.

2. The three-pool architecture

The core mental model for agent RL training: three independently-scalable async pools, each with different resource profiles.

                    ┌─────────────────────┐
                    │   Training Pool       │
                    │  (GPU: policy update) │
                    │  - consumes trajectory │
                    │    batches             │
                    │  - gradient accum +    │
                    │    update              │
                    │  - periodically pushes │
                    │    new weights         │
                    └──────────┬──────────┘
                               │ new weights
                               ▼
┌─────────────────────┐  ┌─────────────────────┐
│   Rollout Pool       │  │   Reward Pool        │
│  (GPU: inference +   │  │  (CPU: verification) │
│   CPU: environment)  │  │                      │
│                      │  │  - run unit tests /   │
│  - inference engine   │  │    check terminal     │
│    (vLLM/SGLang)      │  │    state              │
│    generates actions  │  │  - rule-based checks  │
│  - environment executes│  │  - LLM-judge (opt.)  │
│    actions            │  │                      │
│  - collects full       │──▶  - produces reward    │
│    trajectories       │  │    vector             │
│                      │  │                      │
└─────────────────────┘  └─────────────────────┘

Rollout Pool (inference + environment, GPU+CPU hybrid):

• Inference engines (vLLM5PagedAttention + continuous batching. Kwon 2023 ↗/SGLang6RadixAttention + structured generation. Zheng 2024 ↗) deployed with TP/DP, serving token output for multi-turn generation
• Each rollout worker is bound to one environment instance (docker container / browser instance)
• Loop: inference engine outputs action → worker executes environment step → observation injected into context → continue inference until termination
• Throughput key: N_parallel_workers × inference_speed_per_worker; the GPU does not directly wait for the environment (during the environment step the GPU serves other workers)

Reward Pool (verification, CPU-intensive):

• Receives complete trajectories, executes reward computation: run unit tests / check environment terminal state / run verification scripts
• Can be physically co-located with the Rollout Pool but logically independent (for independent scaling)
• Bottleneck: SWE unit-test execution can take tens of seconds; WebArena terminal-state checks require browser interaction

Training Pool (policy update, GPU-intensive):

• Pulls batches from the trajectory buffer, computes PPO/GRPO loss + gradient update
• Hybrid engine mode: training GPUs also serve inference (GPU time-multiplexed: train one batch → switch to inference → generate new rollouts → switch back to training)
• Separated mode: inference GPUs and training GPUs physically separated; suitable for large-scale continuous training

From-scratch implementation (async three-pool rollout skeleton, interview hand-tear standard):

import threading, queue, time
from concurrent.futures import ThreadPoolExecutor

class AsyncRolloutPipeline:
    """Async three-pool skeleton: Rollout worker → Reward worker → Trajectory buffer → Training."""
    def __init__(self, env_factory, reward_fn, policy_model, buffer_size=1000):
        self.env = env_factory
        self.reward_fn = reward_fn
        self.model = policy_model                         # synced from Training pool periodically
        self.buffer = queue.Queue(maxsize=buffer_size)

    def rollout_worker(self, prompts, num_envs=8):
        """Rollout Pool: sample trajectories in parallel against interactive environments."""
        def run_one(prompt):
            env = self.env()
            obs, traj = env.reset(prompt), []
            for _ in range(max_steps := 50):
                action = self.model.generate(obs["history"])
                next_obs, done = env.step(action)
                traj.append({"obs": obs, "action": action, "done": done})
                if done: break
                obs = next_obs
            return traj
        with ThreadPoolExecutor(max_workers=num_envs) as ex:
            return list(ex.map(run_one, prompts))

    def reward_worker(self, trajectories):
        """Reward Pool: compute final reward per trajectory (env verifier / judge, pure CPU)."""
        for traj in trajectories:
            traj.append({"reward": self.reward_fn(traj[-1]["obs"])})
        return trajectories

    def run_async(self, prompts):
        """Launch async loop: rollout+reward produce → buffer → train consumes."""
        def producer():
            while True:
                for t in self.reward_worker(self.rollout_worker(prompts)):
                    self.buffer.put(t)
                time.sleep(0.1)
        threading.Thread(target=producer, daemon=True).start()

Key: ① Three pools decoupled — no blocking ② Environment is bottleneck (env delay ≫ GPU)

③ ThreadPoolExecutor = toy; production → Ray actors / k8s pods ④ Buffer decouples cadence

3. Training stack comparison

3.1 verl (Volcano Engine)

verl1Initiated by ByteDance Seed, community-maintained RL training framework: hybrid engine time-multiplexes GPUs (inference ↔ training switch), SPMD 3D parallelism, multi-algorithm support. github.com/verl-project/verl (formerly volcengine/verl)'s core design is the hybrid engine: the same set of GPUs time-multiplexes between inference and training modes — train one batch → switch to inference mode to generate new rollouts → switch back to training. Saves GPUs (no need for two clusters), but switching incurs overhead (weight sync + memory reallocation).

• Parallelism strategy: SPMD (Single Program Multiple Data), supporting DP/TP/PP 3D combinations
• Algorithm support: PPO / GRPO / DPO / ReMax; agent RL support in active development
• Agent adaptation: community verl-agent extension embeds environment interaction steps into the RL loop
• Suitable for: medium scale (tens to hundreds of GPUs), teams already experienced with FSDP/Megatron

3.2 OpenRLHF

OpenRLHF2Ray-native RLHF framework: naturally async actors, PPO/DPO/Rejection Sampling support. Hu 2024 ↗ is built on Ray's distributed actor model:

• Naturally async: each Ray actor is independently scheduled; rollout workers / trainer / reward model communicate through Ray — no manual synchronization needed
• Flexible scheduling: can assign different GPUs to rollout and trainer, or go hybrid
• Algorithms: mature PPO / DPO / Rejection Sampling / Conditional SFT implementations
• Agent extension: any interactive environment can be plugged in via a custom Environment abstraction; Ray's actor model makes env-worker scaling and fault tolerance relatively natural
• Suitable for: rapid prototyping, medium scale, teams wanting scheduling flexibility

3.3 AReaL (Ant Group)

AReaL3Ant Group's fully async RL system (for LLM reasoning): generation and training fully separated; paper experiments focus on math/code reasoning, but the async architecture is equally applicable to agentic long-horizon settings. arXiv:2505.24298 is a systems-level design, not a GRPO algorithmic variant (see agentic §5):

• Fully async: generation (rollout) and training fully decoupled, each on an independent GPU pool; the generation pool continuously produces trajectories, the training pool continuously consumes them
• Agent applicability: paper experiments are in the reasoning domain, but its fully async architecture imposes no single-turn assumption and is equally applicable to long-horizon, multi-turn agent RL
• Suitable for: large-scale continuous training, high-throughput long-horizon agent RL

3.4 SkyRL-Agent (NovaSky/Berkeley)

SkyRL-Agent4NovaSky/Berkeley's async training stack for long-horizon multi-turn tools: overlaps CPU-side runtime with GPU generation. arXiv:2511.16108's core contribution is an async pipeline dispatcher:

• Overlaps CPU-side environment init / reward computation with GPU generation in time
• Not stalled by slow trajectories when mixing short and long ones (via dynamic scheduling)
• Reports roughly 1.55× throughput improvement over naive async batching
• Pluggable into VeRL / Tinker and other backends

4. Environment management

This is the most underestimated layer of agent RL infra — environments are slower, less reliable, and harder to scale than GPUs.

4.1 Three environment types and their challenges

4.2 Environment pool design pattern

          ┌──────────────────────────────┐
          │     Environment Pool           │
          │                                │
          │  ┌─────┐ ┌─────┐ ┌─────┐     │
          │  │ Env │ │ Env │ │ Env │ ...  │  ← N pre-warmed instances
          │  │ #1  │ │ #2  │ │ #3  │     │
          │  └─────┘ └─────┘ └─────┘     │
          │       ↑  acquire               │
          │       │  release               │
          └───────┼────────────────────────┘
                  │
          ┌───────┴────────┐
          │ Rollout Worker │ → acquire env → run one trajectory → reset → return
          └────────────────┘

Key mechanisms:

1. Pre-warm: environment instances are started ahead of time in the pool (docker image already pulled, web server already running); a worker can use one immediately. Cold-starting a docker container can take tens of seconds.
2. Health check: periodically or on-acquire, check that the environment is reachable (docker ps, HTTP ping); bad instances are automatically replaced.
3. Auto-reset: after a trajectory completes, the worker calls reset to restore the initial state (git checkout back to initial commit, DB rollback, VM snapshot restore); must reset before returning to pool.
4. Timeout kill: single-step or whole-trajectory timeout → forcefully terminate the env process → recreate a fresh instance (rather than reuse), preventing residual state contamination.
5. Pool size = N_env: determined by N_concurrent_workers + env_reset_latency; too small → workers idle; too large → wasted resources.

4.3 Matching environment throughput to inference

Core formula: to avoid the GPU waiting on environments, you need N_env × throughput_per_env ≥ GPU_inference_throughput.

Concretely: if one environment trajectory averages 20 steps, per-step env latency is 5s, and GPU inference per step is 0.2s, then the GPU spends 0.2s on one worker's step but waits 5s for the environment → GPU utilization is only 0.2/5.2 ≈ 3.8%. So you need many parallel env workers: at least 5.2/0.2 ≈ 26 workers so the GPU always has a request to serve at any moment. In practice you often need hundreds of env workers to saturate a single GPU.

5. GPU memory & multi-turn KV cache

5.1 Multi-turn KV cache growth model

Single-turn inference: encode prompt once → decode $L$ tokens → KV cache $\propto \text{prompt\_len} + L$. Multi-turn agent: $T$ turns of dialogue, each turn's context includes all prior-turn history. At turn $t$ the context length is roughly $t \times L$, so total KV cache grows as $O(T \times L)$.

Concrete numbers: if each turn is 500 tokens (think 200 + act 100 + observe 200), 20 turns gives a 10k-token context. For a 7B model (FP16), per-layer KV cache ≈ 2×2×num_heads×head_dim×seq_len bytes; summed across all layers this can reach several GB — a single trajectory's KV cache could fill one GPU's memory.

5.2 Mitigation strategies

5.3 Hybrid engine memory management

In hybrid engine mode, the same GPU serves inference and training in time slices, and memory must switch between two modes:

• Inference phase: memory holds model weights + KV cache (multi-turn! Large!)
• Training phase: memory holds model weights + optimizer states + gradients + activations (roughly 3–4× weight memory vs inference)
• Switch overhead: free KV cache → allocate optimizer states → load; seconds per switch. Frequent switching (once per trajectory) significantly reduces throughput.

6. Trajectory data pipeline

The metadata structure of one agent trajectory:

trajectory = {
  task_id,              # task identifier
  trajectory_id,        # unique trajectory ID
  turns: [              # multi-turn list
    {think, act, obs, reward_step, done},
    ...
  ],
  total_reward,         # terminal reward
  metadata: {           # metadata
    env_version,        # environment version (for reproducibility)
    total_steps,        # step count
    total_tokens,       # token consumption
    wall_time,          # wall-clock time
    truncated,          # whether truncated
    env_errors,         # environment error count
  }
}

Storage magnitude: one 20-turn trajectory ≈ 10k–20k tokens (think + act + obs); one thousand trajectories ≈ 10M–20M tokens → roughly 50–100MB in raw format (with metadata + reward). Training at the ten-thousand-trajectory scale needs GB-level storage; offline datasets of millions of trajectories can reach TB scale.

Filtering pipeline (before training, which trajectories enter a batch):

1. Completeness filter: discard truncated / env-errored trajectories (optionally keep for diagnostics)
2. Quality filter: discard trajectories with zero or anomalously low/high reward (task-dependent; all-zero trajectories carry no signal in GRPO but keep a few as negatives)
3. Diversity filter: discard trajectories with too-high duplication vs already-collected ones (based on action n-gram or embedding similarity) — prevents Echo Trap (see agentic Q14)
4. Balanced sampling: ensure a reasonable ratio of positive (reward > 0) to negative examples in each batch, avoiding one side dominating the gradient

7. Async staleness & flow control

In a fully async architecture, policy lag is inherent between rollout and training:

Rollout pool generates trajectories with θ_t → Reward pool computes → Training pool has since updated to θ_{t+k} → trajectories were sampled with an old policy

Impact of staleness:

• Trajectories are sampled from an old policy version; under the current policy θ_{t+k}, their log-probs and advantages have changed → the variance of IS (importance sampling) correction grows with staleness
• Off-policy degree = (trainer step at rollout completion) - (policy version at rollout start); if too large the trajectories come from a nearly different policy → training signal degrades

Control mechanisms:

1. Weight versioning: each trajectory is tagged with its generating policy_version; at training time, discard trajectories whose version is too old (set a max_policy_lag)
2. IS correction (V-trace / truncated IS): correct with $\frac{\pi_{\theta_{\text{new}}}(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}$, but variance grows with lag; need clipping ($\rho = \min(\frac{\pi_{\text{new}}}{\pi_{\text{old}}}, \bar{c})$) or ESS (effective sample size) monitoring
3. Queue backpressure: if Training Pool consumption rate < Rollout Pool production rate → trajectory queue backlog → staleness grows → need backpressure (throttle rollout or accelerate training), maintaining steady-state queue depth
4. Stale trajectory reuse policy: slightly stale trajectories are not entirely useless — they can be downweighted (weight = $e^{-\lambda \cdot \text{lag}}$) rather than discarded outright, trading off throughput against signal quality

8. Fault tolerance

The agent RL training chain is long; every link can fail. Production-grade training must handle these failures:

Checkpoint strategy: at minimum save (model_weights, optimizer_state, LR_scheduler_state, global_step) → resume from the most recent checkpoint. Recommend checkpointing every N training steps or every M minutes. If using hybrid engine, ensure you switch to training mode before checkpointing to save the full optimizer state.

9. Cost estimation

Given the following parameters, you can roughly estimate the cost of one agent RL training run:

Inference cost (rollout): $$\text{Tokens}_{\text{rollout}} = N_{\text{task}} \times N_{\text{rollout-per-task}} \times \bar{T} \times \bar{L}_{\text{turn}}$$

For 5k tasks, 4 rollouts per task, average 20 turns, 500 tokens per turn: roughly 200M tokens. With vLLM on 8×A100 at roughly 5k tokens/s inference speed, this takes about 11 wall-clock hours (≈89 GPU-hours = 8 GPU × 11h).

Training cost: $$\text{Tokens}_{\text{train}} = \text{Tokens}_{\text{rollout}} \times N_{\text{epoch}}$$

Training throughput depends on model size + parallelism strategy. A 7B model on 8×A100 doing PPO training achieves roughly 2k–5k tokens/s (including advantage computation + gradient update). Training 200M tokens for 1 epoch ≈ 11–28 wall-clock hours (≈89–222 GPU-hours).

Environment cost (often ignored): SWE docker sandboxes need CPU instances; 100 concurrent environments running for 1 week — the CPU instance cost can approach the GPU cost.

Stratified follow-ups

L1 Basics

L2 Intermediate

L3 Deep

References

All are primary system/framework sources, web-verified. Framework versions and performance numbers change fast; this page records only architectural design ideas.

1. Volcano Engine. verl: Volcano Engine Reinforcement Learning for LLMs. github.com/volcengine/verl — hybrid engine, SPMD 3D parallelism, agent RL support in active development. ↩
2. Hu et al. OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework. 2024. arXiv:2405.11143 — Ray-native async RLHF framework. ↩
3. Ant Group. AReaL: A Large-Scale Asynchronous RL System for Language Reasoning. 2025. arXiv:2505.24298 — fully async RL system (systems-level, not an algorithmic variant; paper focused on reasoning, architecture applicable to agents). ↩
4. NovaSky/Berkeley. SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent. 2025. arXiv:2511.16108 — CPU/GPU pipeline overlap + long-horizon tool scheduling. ↩
5. Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. 2023. arXiv:2309.06180 — vLLM: PagedAttention + continuous batching. ↩
6. Zheng et al. SGLang: Efficient Execution of Structured Language Model Programs. 2024. arXiv:2312.07104 — SGLang radix attention + structured generation. ↩
7. Duan et al. SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution. 2025. arXiv:2502.18449 — SWE-domain RL training data pipeline (this page cites its environment design). ↩
8. Qi et al. WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning. 2024. arXiv:2411.02337 — Web-domain RL training environment and self-evolving curriculum. ↩