Skip to content

Model accuracy alignment 2#2

Open
zhengshengning wants to merge 1 commit into
PFCCLab:mainfrom
zhengshengning:minimax_model_new_0713
Open

Model accuracy alignment 2#2
zhengshengning wants to merge 1 commit into
PFCCLab:mainfrom
zhengshengning:minimax_model_new_0713

Conversation

@zhengshengning

Copy link
Copy Markdown

问题9: permute 反向 index_select 的 fp32 反向累积实现diff
描述:forward 与原生 tokens.index_select(0, sorted_indices) 完全一致。backward 时,多个 permuted 行会回写到同一个原始 token 行(一个 token 被 topk 个 expert 选中),原生 index_select 的 backward 走 bf16 index_add/scatter_add,atomic 累加顺序不可复现,与 PaddleFleet 末位 diff。这里把 grad 提升到 fp32 后 scatter_add 累加,再 cast 回原 dtype,单次 round,复刻 PF 侧确定性 fp32 累加路径。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant