[Klaud Cold] Add MI300X Qwen3.8-27B-FP8 vLLM MTP 1k1k + 8k1k / 新增 MI300X Qwen3.8-27B-FP8 vLLM MTP 1k1k + 8k1k 配方 - #3294
functionstackx wants to merge 5 commits into
Conversation
…TP 1k1k recipe on MI300X Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
functionstackx
added
full-sweep-fail-fast
qwen3.8-27b
b5f172e to
75ddac3
Compare
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Beyond the inline finding, I checked the MAX_MODEL_LEN fallback (MODEL_LEN="${MAX_MODEL_LEN:-$((ISL + OSL + 256))}") against the rest of the codebase: benchmark_lib.sh and several sibling recipes (e.g. the deprecated gptoss_fp4_*.sh scripts) already treat MAX_MODEL_LEN as an optional override with a derived default rather than a required caller input, so this isn't a new deviation from convention.
Extended reasoning...
The confirmed inline finding (missing cleanup of the GPU monitor process/CSV when a post-benchmark accuracy eval fails under set -eo pipefail) is a real gap worth a human's attention before merge, so I did not approve. I independently investigated the two ruled-out MAX_MODEL_LEN concerns from the candidate list by grepping the broader benchmarks tree: benchmark_lib.sh itself defaults eval_ctx="${MAX_MODEL_LEN:-16384}" and treats an unset MAX_MODEL_LEN as a valid "derive from ISL/OSL" signal ([ -z "${EVAL_MAX_MODEL_LEN:-}" ] branch), and the deprecated gptoss_fp4_h200.sh/gptoss_fp4_b200.sh recipes use the identical ${MAX_MODEL_LEN:-N} shape. This confirms the new script's fallback is consistent with an established, intentional "optional override with a runtime-derived default" pattern rather than a caller-config value that should instead be enforced via check_env_vars, so I did not raise it as a separate finding.
| set -eo pipefail | ||
|
|
||
| # Qwen3.8-27B-FP8 (fp8 e4m3 dynamic-activation checkpoint of the dense hybrid | ||
| # attention model: 48 linear-attention and 16 full attention layers) on one | ||
| # MI300X, served by vLLM with the checkpoint's own MTP head (one MTP layer, | ||
| # mtp_num_hidden_layers=1) drafting three tokens per step. | ||
| # https://recipes.vllm.ai/Qwen/Qwen3.8-27B | ||
| # https://huggingface.co/Qwen/Qwen3.8-27B-FP8 | ||
| source "$(dirname "$0")/../../benchmark_lib.sh" | ||
|
|
||
| check_env_vars MODEL TP CONC ISL OSL RANDOM_RANGE_RATIO RESULT_FILENAME | ||
|
|
||
| if [[ -n "${SLURM_JOB_ID:-}" ]]; then | ||
| echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME" | ||
| fi | ||
|
|
||
| if [[ "$TP" -ne 1 ]]; then | ||
| echo "This recipe serves Qwen3.8-27B-FP8 on a single GPU; got TP=$TP" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| # The vLLM recipe's MTP setting for this model: the in-checkpoint head (fp8 | ||
| # acceptance 0.771 on the recipe page) recursed for three draft tokens. | ||
| NUM_SPEC_TOKENS=3 | ||
|
|
||
| # gfx942: the ROCm vLLM arms run Triton attention and keep the KV cache in bf16 | ||
| # (no calibrated fp8 attention scales on this card); breakable piecewise graph | ||
| # capture is off as in the MiniMax-M3 and DSv4.1 Flash gfx942 arms. | ||
| export VLLM_USE_BREAKABLE_CUDAGRAPH=0 | ||
| export PYTHONNOUSERSITE=1 | ||
|
|
||
| rocm-smi --showmeminfo vram || true | ||
|
|
||
| # Complete/resume partial downloads instead of trusting nonempty directories. | ||
| if [[ "$MODEL" != /* ]]; then hf download "$MODEL"; fi | ||
|
|
||
| SERVER_LOG=/workspace/server.log | ||
|
|
||
| # Serve the matrix context (isl + osl + slack), not the checkpoint's 262K; | ||
| # accuracy evals use the eval context benchmark_lib derives. | ||
| MODEL_LEN="${MAX_MODEL_LEN:-$((ISL + OSL + 256))}" | ||
| if [[ "${EVAL_ONLY:-false}" == true ]]; then | ||
| setup_eval_context | ||
| MODEL_LEN="$EVAL_MAX_MODEL_LEN" | ||
| fi | ||
|
|
||
| # vLLM's default max-num-seqs (1024) exceeds the GDN/Mamba cache blocks that fit | ||
| # next to the weights on the smaller cards (472 next to the bf16 weights on an | ||
| # 80 GB H100, run 35357364404) and engine start aborts before graph capture. | ||
| # Size the scheduler batch to the sweep point instead; the accuracy eval serves | ||
| # up to 256 concurrent requests. | ||
| MAX_NUM_SEQS=$(( CONC > 16 ? CONC : 16 )) | ||
| if [[ "${EVAL_ONLY:-false}" == true ]]; then | ||
| MAX_NUM_SEQS=256 | ||
| fi | ||
|
|
||
| # Pyxis shares the host network; port 8888 can already belong to a host service. | ||
| select_available_server_port | ||
|
|
||
| # Native MTP: no draft model, the head ships in the checkpoint (mtp.* tensors). | ||
| SPEC_CONFIG=$(printf '{"method":"mtp","num_speculative_tokens":%d}' "$NUM_SPEC_TOKENS") | ||
|
|
||
| start_gpu_monitor | ||
|
|
||
| VLLM_CMD=( | ||
| vllm serve "$MODEL" --served-model-name "$MODEL" | ||
| --host 0.0.0.0 --port "$PORT" | ||
| --tensor-parallel-size 1 | ||
| # Text-only serving: skip the vision tower of Qwen3_5ForConditionalGeneration. | ||
| --language-model-only | ||
| --trust-remote-code | ||
| --attention-backend TRITON_ATTN | ||
| --max-model-len "$MODEL_LEN" | ||
| --max-num-seqs "$MAX_NUM_SEQS" | ||
| # Every 1k1k request prefills its full random prompt; no prefix-cache hits. | ||
| --no-enable-prefix-caching | ||
| --reasoning-parser qwen3 | ||
| --enable-auto-tool-choice --tool-call-parser qwen3_xml | ||
| --speculative-config "$SPEC_CONFIG" | ||
| --disable-uvicorn-access-log | ||
| ) | ||
| printf '%q ' "${VLLM_CMD[@]}" | tee /workspace/vllm_command.txt | ||
| printf '\n' | tee -a /workspace/vllm_command.txt | ||
| "${VLLM_CMD[@]}" > "$SERVER_LOG" 2>&1 & | ||
| SERVER_PID=$! | ||
|
|
||
| wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID" | ||
|
|
||
| if [[ "${EVAL_ONLY:-false}" == true ]]; then | ||
| run_eval --framework lm-eval --port "$PORT" | ||
| # Non-agentic evals must stage lm-eval's output into the workspace root. | ||
| append_lm_eval_summary | ||
| else | ||
| pip install -q datasets pandas | ||
| run_benchmark_serving \ | ||
| --model "$MODEL" \ | ||
| --port "$PORT" \ | ||
| --backend vllm \ | ||
| --input-len "$ISL" \ | ||
| --output-len "$OSL" \ | ||
| --random-range-ratio "$RANDOM_RANGE_RATIO" \ | ||
| --num-prompts "$((CONC * 10))" \ | ||
| --max-concurrency "$CONC" \ | ||
| --result-filename "$RESULT_FILENAME" \ | ||
| --result-dir /workspace/ \ | ||
| `# Chat-templated prompts: raw random tokens tank MTP acceptance.` \ | ||
| --use-chat-template \ | ||
| --server-pid "$SERVER_PID" | ||
| if [[ "${RUN_EVAL:-false}" == true ]]; then | ||
| run_eval --framework lm-eval --port "$PORT" | ||
| append_lm_eval_summary | ||
| fi | ||
| fi | ||
|
|
||
| stop_gpu_monitor |
There was a problem hiding this comment.
🟡 (optional) A failed post-benchmark accuracy eval now leaves the GPU-metrics monitor orphaned and its CSV unfinalized, unlike sibling _mtp.sh recipes. This script adds set -eo pipefail (line 2); when run_eval/append_lm_eval_summary (lines 90-93, 110-113) return non-zero, the script exits before line 116's stop_gpu_monitor, which kills the monitor, repairs a truncated tail, and writes the AMD energy-end sidecar. Sibling scripts (e.g. qwen3.5_fp8_h100_mtp.sh) lack set -e and always reach that cleanup. Fix: trap EXIT (or guard the eval calls) so stop_gpu_monitor and server teardown run on every exit path, not just success.
Extended reasoning...
start_gpu_monitor (line 64) forks a background nvidia-smi/amd-smi loop tracked by GPU_MONITOR_PID, exported globally. The throughput run_benchmark_serving at lines 96-109 succeeds and writes RESULT_FILENAME. RUN_EVAL=true triggers run_eval --framework lm-eval at line 111 (a real flag used for accuracy validation runs); lm-eval hits a transient failure (network, model timeout, _install_lm_eval_deps) and returns non-zero. Because line 2 sets -eo pipefail and this call is a plain statement inside the if-body (not an if/while condition), bash aborts the whole script immediately. append_lm_eval_summary (line 112) and stop_gpu_monitor (line 116) never run. The amd-smi|awk pipeline keeps running orphaned on the shared MI300x runner. GPU_METRICS_CSV never gets _repair_truncated_gpu_metrics_tail or the AMD energy-end sidecar (per stop_gpu_monitor's own comments these are needed for auditors to cross-check integrated energy), so the power data for an otherwise-successful benchmark run is corrupted/missing even though the throughput JSON itself is fine. Same failure path applies to EVAL_ONLY's…
Verification: nit. The scenario is real and reachable but low-severity. Line 2 sets set -eo pipefail; start_gpu_monitor (line 64) forks the sampler into GPU_MONITOR_PID and there is no trap ... EXIT in the script, so only the trailing stop_gpu_monitor (line 116) reaps it. run_eval genuinely returns non-zero on failure (benchmark_lib.sh:2905/2940/2949 return 2), so a failed eval at line 111…
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35501520852 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35501520852 |
|
/reuse-sweep-run 35465808674 |
Sync with origin/main after the green sweep run 35465808674; the reuse gate authorizes that run on this head. 在绿色 sweep 运行 35465808674 之后与 origin/main 同步；reuse gate 在此 head 上授权该运行。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
应用 thinking-on 黄金 AL 2.52，保持原始 BF16 原生 MTP 头及草稿 KV cache，并在精度评测路径使用真实验证。
同步 main，逐字节保留 changelog 历史，并在末尾保留本 PR 的原始条目和黄金 AL 更新。
在 MI300X 的 Qwen3.8-27B-FP8 原生 MTP 配方中新增 8192/1024、TP1、并发 1-128；保留 1k1k、thinking-on AL 2.52 和真实精度验证。
Oseltamivir
changed the title
This branch has not been deployed
2 participants
Goal: Add
qwen3.827b-fp8-mi300x-vllm-mtpon MI300X, using native MTP3 with the measured thinking-on synthetic AL 2.52 for throughput.Baseline: N/A — new model/SKU recipe; the acceptance reference is #3304 and SPEED-Bench run 35492787451.
vllm/vllm-openai-rocm:nightly-a8d1aa9c99b8698a2a78b611b7a10c30e6b3995b· TP1 · 1k/1k + 8k/1k · concurrency 1–128. The recipe readsthinking_on[3]fromgolden_al_distribution/qwen3.827b_fp8_mtp.yamland sendsrejection_sample_method=synthetic,synthetic_acceptance_length=2.52to vLLM. The pinned Qwen chat template enables thinking; the script requiresTHINKING_MODE=thinking_on.The FP8 target is pinned to
017b9c7af6b5689d5dd426a76e0bc077eb5ca20a. Its native MTP head comes from the original BF16Qwen/Qwen3.8-27Bat1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, with BF16 draft KV cache and explicit MTP quantization exclusions. Eval-only and combined throughput+eval paths use real standard verification. The shared helper rejects unmeasured draft lengths outside 1–4 and missing or invalid AL values.Validation: CI Tests and Lint, exact-key matrix generation, Bash syntax, and changelog byte-prefix preservation passed. The fresh full sweep passed all 18 GPU jobs: eight 1k/1k points, eight 8k/1k points, and two separate accuracy evals. Both curves cover concurrency 1, 2, 4, 8, 16, 32, 64 and 128. All eight 8k/1k artifacts were checked for the workload identity, positive throughput, synthetic AL 2.52, and unquantized native MTP linear modules. Real-verification GSM8K scored 97.12% at concurrency 64 and 97.04% at concurrency 128, with all 1,319 questions in each eval. Validated full sweep 35501520852.
The existing reuse authorization predates the AL and 8k/1k changes; select the new validated full sweep before merging.
AI model disclosure
AL integration, 8k/1k coverage and validation: GPT-6; the exact runtime model/version identifier could not be verified. The original PR's AI model/version was not recorded and could not be verified. No delegated agents contributed to this update.
中文
**目标：**在 MI300X 上新增
qwen3.827b-fp8-mi300x-vllm-mtp，使用原生 MTP3，并在吞吐测试中应用实测的 thinking-on 合成 AL 2.52。**基线：**N/A——新增模型/SKU 配方；接受长度依据 #3304 和 SPEED-Bench 运行 35492787451。
vllm/vllm-openai-rocm:nightly-a8d1aa9c99b8698a2a78b611b7a10c30e6b3995b· TP1 · 1k/1k + 8k/1k · 并发 1–128。配方从golden_al_distribution/qwen3.827b_fp8_mtp.yaml读取thinking_on[3]，向 vLLM 传入rejection_sample_method=synthetic、synthetic_acceptance_length=2.52。固定版本的 Qwen chat template 默认开启 thinking，脚本要求THINKING_MODE=thinking_on。FP8 目标固定为 revision
017b9c7af6b5689d5dd426a76e0bc077eb5ca20a。原生 MTP 头来自原始 BF16Qwen/Qwen3.8-27B，revision 为1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0；草稿 KV cache 保持 BF16，并显式排除 MTP 量化。仅精度评测以及吞吐后紧接精度评测的路径均使用真实的标准验证。共享辅助程序拒绝超出 1–4 范围的草稿长度，以及缺失或无效的 AL。验证：CI Tests 和 Lint、精确 key 矩阵生成、Bash 语法和 changelog 历史字节保留检查均通过。新的完整 sweep 的 18 个 GPU 任务全部通过：八个 1k/1k 点、八个 8k/1k 点，以及两个独立精度评测。 两条曲线均覆盖并发 1、2、4、8、16、32、64、128。八个 8k/1k artifact 均已核实工作负载标识、有效吞吐、合成 AL 2.52 和未量化的原生 MTP 线性层。真实验证 GSM8K 在并发 64 时准确率为 97.12%，并发 128 时为 97.04%，每次评测均完成全部 1,319 道题。验证通过的完整 sweep 35501520852。
现有复用授权早于 AL 和 8k/1k 更新；合并前应选择新验证通过的完整 sweep。
AI 模型披露：AL 集成、8k/1k 覆盖和验证由 GPT-6 完成；无法核实运行时精确模型及版本标识。原始 PR 未记录 AI 模型及版本，无法核实。本次更新未使用委派代理。