verl-project/verl

vllm cannot load model after megatron training

Open

#1,757 opened on May 29, 2025

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Python (3,940 forks)auto 404
help wanted

Repository metrics

Stars
 (21,533 stars)
PR merge metrics
 (Avg merge 5d) (146 merged PRs in 30d)

Description

After megatron training and convert to hf model, i want to infer using vllm, which meet problem when loading. File "/python3.11/site-packages/vllm/model_executor/models/utils.py", line 250, in _load_module raise ValueError(msg) ValueError: There is no module or parameter named 'decoder' in Qwen3ForCausalLM After model_merger.py process, the name of parameters changed.

Contributor guide

Research direction
Inspect the model conversion script (model merger.py) to understand how parameter names are changed, then verify that the vllm loader expects original HuggingFace names. Check if there is a misalignment between the converted model's state dict keys and vllm's expected module structure.
Tech stack
python
Domain
aimachine learningbackend
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonMegatron trainingHugging Face model conversion
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
75