[moe] sync and shard expert grads over moe_dp * sp with all_to_all sequence parallelism - #6468

[moe] sync and shard expert grads over moe_dp * sp with all_to_all sequence parallelism#6468

yuxuan-lou merged 1 commit into

mainhpcaitech/ColossalAI:mainfrom

fix/moe-sp-expert-grad-synchpcaitech/ColossalAI:fix/moe-sp-expert-grad-syncCopy head branch name to clipboard