夜雨聆风学习主题：VERL 源码精读 03
理解 ppo_trainer.yaml、分组 yaml、dataclass config、命令行 override 和配置校验如何共同决定一次训练。
阅读目标：建立可复述、可定位、可调试的源码理解，不停留在 API 名称层面。
1. 本篇学习范围
理解 ppo_trainer.yaml、分组 yaml、dataclass config、命令行 override 和配置校验如何共同决定一次训练。
这一篇关注的不是“怎么把命令跑起来”，而是读清楚源码中每个对象为什么存在、它和前后模块如何传递数据，以及出问题时应该从哪里开始定位。
2. 源码入口文件
本篇建议按下面顺序阅读：
verl/trainer/config/ppo_trainer.yamlverl/trainer/config/algorithm.pyverl/trainer/config/config.pyverl/workers/config/actor.pyverl/workers/config/critic.pyverl/workers/config/rollout.pyverl/workers/config/engine.pyverl/base_config.py这些文件覆盖了本篇主题的主路径。阅读时不需要一开始就把所有分支展开，先抓主干，再回头看特殊配置、后端差异和异常处理。
3. 核心调用链
本篇主调用链可以压缩为：
`ppo_trainer.yaml` 用 defaults 列表组合 data、actor、critic、ref、reward、algorithm、trainer 等配置块。命令行 override 修改 OmegaConf 中的具体字段，例如 `algorithm.adv_estimator=grpo`。worker 配置在运行时通过 `omega_conf_to_dataclass()` 转成 `ActorConfig`、`CriticConfig`、`RolloutConfig` 等 dataclass。dataclass 的 `validate()` 负责检查 batch size、micro batch、dynamic bsz、GPU 数和策略后端是否匹配。trainer 根据配置决定是否创建 critic、reference policy、reward model 和不同 resource pool。这条链路是读源码时的地图。后续遇到新的类、函数或配置字段，都可以先判断它属于链路中的哪一段。
4. 核心概念
VERL 配置是源码行为的入口，不读配置就无法判断实际代码分支。 actor_rollout_ref是最核心的复合配置，里面同时包含 model、actor、rollout、ref。
# specify the default per-component configsdefaults:- model_engine: dp# <folder_name>@<field_name>.<field_name>: <yaml_file_name># actor_rollout_ref.actor: trainer/config/actor/dp_actor.yaml- actor@actor_rollout_ref.actor: ${model_engine}_actor# data: trainer/config/data/legacy_data.yaml- data@data: legacy_data# Reference model config.# Reference model will be enabled when actor.use_kl_loss or/and algorithm.use_kl_in_reward is/are True.- ref@actor_rollout_ref.ref: ${model_engine}_ref# Rollout model config.- rollout@actor_rollout_ref.rollout: rollout# Model config.- model@actor_rollout_ref.model: hf_model# Critic model config.- critic@critic: ${model_engine}_critic- model@critic.model: hf_model# legacy reward impl config, for backward compatibility- legacy_reward_impl# Reward config.- reward@reward: reward# Rollout correction config.- algorithm@algorithm.rollout_correction: rollout_correction# distillation config- distillation@distillation: distillation# load the reference default config, then apply the fields in the current yaml# self config override anything above- _self_# config for actor, rollout and reference modelactor_rollout_ref:# Whether it's a hybrid engine, currently only supports hybrid enginehybrid_engine: true# Timeout for operations executed against the process groupnccl_timeout: 600
algorithm决定 advantage estimator、KL 位置、filter group、rollout correction 等算法行为。 
# config for the algorithmalgorithm:# Required when using verl.utils.omega_conf_to_dataclass to instantiate dataclass configs_target_: verl.trainer.config.AlgoConfig# Discount factor for future rewardsgamma: 1.0# Trade-off between bias and variance in the GAE estimatorlam: 1.0# Advantage estimator type: "gae", "grpo", "reinforce_plus_plus", etc.adv_estimator: gae# Whether to normalize advantages by std (specific to GRPO)norm_adv_by_std_in_grpo: True# Whether to enable in-reward KL penaltyuse_kl_in_reward: False# How to estimate KL divergence: "kl", "abs", "mse", "low_var_kl", or "full"kl_penalty: kl# KL control configurationkl_ctrl:# Required when using verl.utils.omega_conf_to_dataclass to instantiate dataclass configs_target_: verl.trainer.config.KLControlConfig# KL control type: "fixed" or "adaptive"type: fixed# Initial coefficient for KL penaltykl_coef: 0.001# Horizon value for adaptive controller (if enabled)horizon: 10000# Target KL divergence (used for adaptive controller)target_kl: 0.1# Whether to enable preference feedback PPOuse_pf_ppo: False# Preference feedback PPO settingspf_ppo:# Method for reweighting samples: "pow", "max_min", or "max_random"reweight_method: pow# Power used for weight scaling in "pow" methodweight_pow: 2.0
trainer决定训练步数、validation、checkpoint、logger、use_v1、资源规模。 
# config for the trainertrainer:# Whether to balance batch sizes across distributed workersbalance_batch: True# Number of epochs in trainingtotal_epochs: 30# Total training steps (can be set explicitly or derived from epochs)total_training_steps: null# Project name for experiment tracking (e.g., wandb)project_name: verl_examples# Experiment name for run identification in tracking toolsexperiment_name: gsm8k# Logging backends to use: "console", "wandb", etc.logger: ["console", "wandb"]# Number of generations to log during validationlog_val_generations: 0# Directory for logging rollout data; no dump if nullrollout_data_dir: null# Directory for logging validation data; no dump if nullvalidation_data_dir: null# Number of nodes used in the trainingnnodes: 1# Number of GPUs per noden_gpus_per_node: 8# Save frequency (by iteration) for model checkpointssave_freq: -1# ESI refers to the elastic server instance used during training, similar to the training plan. For example,# if you purchase 10 hours of computing power, the ESI will automatically shut down after 10 hours of training.# To ensure a checkpoint is saved before ESI shuts down, the system will start saving a checkpoint in advance.# The advance time is calculated as: Advance Time = Longest historical step duration + Checkpoint save duration + esi_redundant_time.# Here, esi_redundant_time is a user-defined value that further extends the advance time for added safety.esi_redundant_time: 0# Resume mode: "auto", "disable", or "resume_path"# "auto": resume from last checkpoint if available# "disable": start from scratch# "resume_path": resume from a user-defined pathresume_mode: auto# Path to resume training from (only used when resume_mode is "resume_path")resume_from_path: null# Whether to run validation before training beginsval_before_train: True# Whether to run validation onlyval_only: False# Validation frequency (in training iterations)test_freq: -1# Number of iterations to warm up the critic before updating policycritic_warmup: 0# Default path to distributed filesystem for saving checkpointsdefault_hdfs_dir: null# Whether to delete local checkpoints after loadingdel_local_ckpt_after_load: False# Default local directory for saving checkpointsdefault_local_dir: checkpoints/${trainer.project_name}/${trainer.experiment_name}# Maximum number of actor checkpoints to keepmax_actor_ckpt_to_keep: null# Maximum number of critic checkpoints to keepmax_critic_ckpt_to_keep: null# Timeout (in seconds) for Ray worker to wait for registrationray_wait_register_center_timeout: 300# Device to run training on (e.g., "cuda", "cpu")device: cuda# Whether to use V1 traineruse_v1: true# V1 trainer mode configsv1:# Trainer mode: "sync", "colocate_async", "separate_async"trainer_mode: sync# Synchronous PPO trainersync: {}# Asynchronous PPO trainer with colocated trainer and rolloutcolocate_async:# Number of warmup batches to add before training loop startsnum_warmup_batches: 1# Asynchronous PPO trainer with separate trainer and rolloutseparate_async:# Number of warmup batches to add before training loop startsnum_warmup_batches: 1# Frequency of parameter synchronization between trainer and rolloutparameter_sync_step: 4# Replay buffer sampling strategysampler:# Maximum number of model versions that trajectory can spanmax_off_policy_threshold: 8# How to handle trajectory that exceeds the maximum number of model versions# drop: drop the trajectory# wait: dropless, wait all trajectories that reach threshold to finishmax_off_policy_strategy: drop# Custom sampler configcustom_sampler:# Path to the custom sampler classpath: null# Name of the custom sampler classname: null# Additional kwargs for the custom samplersampler_kwargs: {}
这些概念共同决定了本篇源码的设计方式。VERL 的一个重要特点是，很多类名看起来像普通工程封装，但背后实际是在解决大模型 RL 的分布式执行问题。
5. 数据在这一层如何流动
在 VERL 中，几乎所有训练阶段都可以用同一套数据流语言描述：
输入 batch-> 按并行度或任务类型拆分-> 分发到本地函数或远程 worker-> 执行高成本计算或轻量控制逻辑-> 收集结果-> 写回 DataProto / TensorDict / TransferQueue-> 进入下一阶段本篇主题对应的数据流重点是：
ppo_trainer.yaml用 defaults 列表组合 data、actor、critic、ref、reward、algorithm、trainer 等配置块。 命令行 override 修改 OmegaConf 中的具体字段，例如 algorithm.adv_estimator=grpo。worker 配置在运行时通过 omega_conf_to_dataclass()转成ActorConfig、CriticConfig、RolloutConfig等 dataclass。dataclass 的 validate()负责检查 batch size、micro batch、dynamic bsz、GPU 数和策略后端是否匹配。trainer 根据配置决定是否创建 critic、reference policy、reward model 和不同 resource pool。
读代码时要始终跟踪字段而不是只跟踪函数名。典型字段包括：
promptsresponsesattention_maskposition_idsold_log_probsref_log_probvaluesrm_scorestoken_level_rewardsadvantagesreturnsmetrics不是每一天都会出现全部字段，但这些字段构成了 VERL PPO/GRPO 训练的共同词汇表。
6. 和前后模块的关系
本篇主题通常不是孤立工作的。它至少会连接三类模块：
上游： 配置、数据、controller 状态、已有 batch 字段。本层： 当前主题负责的调度、计算、转换或封装逻辑。下游： rollout、reward、advantage、loss、metrics、checkpoint 或异步队列。因此读源码时要避免只看单个函数。更稳的方式是：
1. 找到谁调用它。2. 找到它读取哪些字段。3. 找到它写出哪些字段。4. 找到这些字段下一步被谁使用。5. 找到配置项如何改变它的分支。这个五步法适用于 VERL 的大多数文件。
7. 实现细节拆解
本篇源码中最值得关注的细节包括：
ppo_trainer.yaml用 defaults 列表组合 data、actor、critic、ref、reward、algorithm、trainer 等配置块。 
命令行 override 修改 OmegaConf 中的具体字段，例如 algorithm.adv_estimator=grpo。
worker 配置在运行时通过 omega_conf_to_dataclass()转成ActorConfig、CriticConfig、RolloutConfig等 dataclass。
def omega_conf_to_dataclass(config: DictConfig | dict, dataclass_type: Optional[type[Any]] = None) -> Any:"""Convert an OmegaConf DictConfig to a dataclass.Args:config: The OmegaConf DictConfig or dict to convert.dataclass_type: The dataclass type to convert to. When dataclass_type is None,the DictConfig must contain _target_ to be instantiated via hydra.instantiate API.Returns:The dataclass instance."""
validate() 负责检查 batch size、micro batch、dynamic bsz、GPU 数和策略后端是否匹配。
这些步骤背后通常有两类逻辑：
控制逻辑： 判断当前训练需要哪些角色、哪些字段、哪些分支。计算逻辑： 真正执行模型推理、训练、reward、advantage 或 loss。HybridFlow 的设计要求我们把这两类逻辑区分开。控制逻辑更适合在 trainer 或 manager 中读；计算逻辑更适合在 worker、engine 或 core_algos 中读。
8. 配置如何影响本篇路径
VERL 的同一段源码经常会被配置切到不同路径。阅读本篇时尤其要关注这些配置类型：
algorithm: 决定 PPO、GRPO、KL、advantage、rollout correction 等算法行为。actor_rollout_ref: 决定 actor、rollout、reference policy、model path、训练后端和推理后端。critic: 决定是否启用 value model，以及 critic 的训练后端。reward: 决定使用规则 reward、reward model、remote reward 还是 sandbox reward。trainer: 决定训练步数、资源规模、logger、validation、checkpoint 和 V1/V0 模式。读源码前最好先打印 resolved config。否则很容易在一个未启用的分支里浪费时间。
9. 常见误区和调试要点
同一个字段可能在 yaml、dataclass 和运行时 open_dict 修改中出现，最终值以 resolved config 为准。 配置中 critic.enable=None不等于启用 critic，VERL 会根据 advantage estimator 推断是否需要 critic。读报错时要分清是 Hydra 解析失败、配置校验失败，还是 worker 初始化阶段失败。
调试 VERL 时，不建议一上来就改源码。更稳的顺序是：
1. 确认命令行 override 是否真的进入 resolved config。2. 确认当前走 V0 还是 V1 trainer。3. 确认 DataProto / TensorDict 里字段是否存在。4. 确认 batch 维、response 长度、mask 是否一致。5. 确认对应 worker 是否真的被创建。6. 确认 Ray worker 日志里的原始异常。7. 最后再判断是不是算法公式或 loss 本身的问题。这个顺序能避免把配置错误、数据错误、分布式调度错误误判成算法错误。
10. 本篇压缩总结
读懂这一篇后，应该能够回答三个问题：
1. 这一层在 VERL 训练链路中负责什么？2. 它接收哪些字段，又产出哪些字段？3. 它的行为主要由哪些配置项改变？如果这三个问题能答清楚，就说明不是在背目录，而是在按 VERL 的真实执行路径读源码。
11. 下一步阅读
读完本篇后，建议继续沿着训练链路向后走：
入口与配置-> 数据和 DataProto-> WorkerGroup 和 Ray 调度-> Worker 与模型引擎-> Rollout-> Reward-> Advantage-> Actor/Critic loss-> Metrics、Checkpoint、异步扩展VERL 源码量很大，但主线并不乱。只要始终围绕“一个 batch 如何从 prompt 变成 actor update”这条线阅读，就能把分散目录组织成一张完整图。
