diff --git a/docs/source/en/features/sequence_parallelism.md b/docs/source/en/features/sequence_parallelism.md index 70fd2eb10970..e422a24a9c8f 100644 --- a/docs/source/en/features/sequence_parallelism.md +++ b/docs/source/en/features/sequence_parallelism.md @@ -101,6 +101,7 @@ plugin = HybridParallelPlugin( sequence_parallelism_mode="split_gather", ) ``` +Example of startup command parameters: ```--tp 2 --sp 8 --sp_mode split_gather``` #### Using DeepSpeed-Ulysses Define the plugin. In the DeepSpeed-Ulysses sequence parallelism, the tp group and sp group are orthogonal. @@ -113,6 +114,7 @@ plugin = HybridParallelPlugin( sequence_parallelism_mode="all_to_all", ) ``` +Example of startup command parameters: ```--tp 2 --sp 8 --sp_mode all_to_all``` #### Using Ring Attention Define the plugin. In ring attention sequence parallelism, the tp group and sp group are orthogonal, and sp_size must be set to the correct parallel size. @@ -125,6 +127,8 @@ plugin = HybridParallelPlugin( sequence_parallelism_mode="ring_attn", ) ``` +Example of startup command parameters: ```--tp 2 --sp 8 --sp_mode ring_attn``` + #### Using Booster ```python booster = Booster(plugin=plugin) @@ -151,6 +155,15 @@ Currently, the `MoeHybridParallelPlugin` only supports DeepSpeed-Ulysses sequenc ### Conclusion -Among the sequence parallelism methods mentioned, ring attention has no requirements for the number of attention heads and can train ultra-long sequences. However, due to the division of computation, its performance may decrease. TP+SP and DeepSpeed-Ulysses have requirements for the number of attention heads, which must be divisible by the sp group size. These sequence parallelism methods are all compatible with high-performance attention mechanisms like flash attention. Sequence parallelism can also be used with Gemini to train extremely large-scale models, and it can be combined with TP, PP, and DP to form 4D parallelism. +Among the sequence parallelism methods mentioned, both ring attention and Ulysses have their pros and cons, and we need to choose the appropriate sequence parallelism method based on the situation: + + Communication: Ulysses has lower communication overhead compared to ring attention, as it primarily involves three All-to-All communication ops, whereas the communication cost of ring attention grows quadratically with the sequence length. However, on the other hand, All-to-All op also demands dense network topologies, e.g. NVLink + NVSwitch, so it doesn't scale well across multiple nodes. + + Memory usage: Both are similar in terms of memory consumption. + + Model structure generalization: Ring attention is better than Ulysses in terms of generalization. Ulysses requires that the model config need to meet ```the head number // (tp group size * sp group size)``` condition, while ring attention has no such restrictions. + +Due to its simplicity and non-intrusive modification to attention calculation, Ulysses is currently the mainstream for sequence parallelism. All sequence parallel methods can be compatible with other high-performance attention methods such as Flash Attention, and can also be combined with other parallel training strategies like ZeRO, TP, PP, and DP. +Overall, we recommend using Ulysses. You only need to specify ```--sp_mode all_to_all``` during startup. Based on testing, in a two-node, 16-GPU setup, using the startup parameters ```--tp 2 --sp 8 --sp_mode all_to_all```, it's easy to train sequences of up to 128k length, and the performance is the best among all sequence parallelism methods，can reach approximately 480+ TFLOPS on dual H800s. However, if you're aiming for extreme performance optimization or training long texts on a larger scale of machines, you might want to consider using the ring attention. diff --git a/docs/source/zh-Hans/features/sequence_parallelism.md b/docs/source/zh-Hans/features/sequence_parallelism.md index 534035cb5abf..ddf1ae293d5e 100644 --- a/docs/source/zh-Hans/features/sequence_parallelism.md +++ b/docs/source/zh-Hans/features/sequence_parallelism.md @@ -101,6 +101,8 @@ plugin = HybridParallelPlugin( ) ``` +启动参数举例：```--tp 2 --sp 8 --sp_mode split_gather``` + #### 使用DeepSpeed-Ulysses 定义plugin， 在DeepSpeed-Ulysses的序列并行种，tp group与sp group 是正交的， ```python @@ -112,6 +114,7 @@ plugin = HybridParallelPlugin( sequence_parallelism_mode="all_to_all", ) ``` +启动参数举例：```--tp 2 --sp 8 --sp_mode all_to_all``` #### 使用ring attention 定义plugin， 在ring attention的序列并行种，tp group与sp group 是正交的，sp_size必须传入准确的并行大小。 @@ -124,6 +127,8 @@ plugin = HybridParallelPlugin( sequence_parallelism_mode="ring_attn", ) ``` +启动参数举例：```--tp 2 --sp 8 --sp_mode ring_attn``` + #### 使用booster ```python booster = Booster(plugin=plugin) @@ -150,6 +155,17 @@ for step, batch in enumerate(tqdm(dataloader, desc="Step", disable=not dist.get_ ### 结论 -在上述序列并行方法中，ring attention对head number没有要求，可训练超长文本，但是由于细分了计算，计算性能会有所下降。TP+SP， DeepSpeed-Ulysses对于head number有要求，需要可被sp group size 整除。这些序列并行都可与其他高性能注意力兼容，如flash attention。sp可与Gemini一起使用训练超大规模模型，也可以与TP，PP，DP等组成4D并行。 +在上述序列并行方法中，ring attention和Ulysses各有优劣，我们需要根据情况来选择合适的序列并行方法： + + 通信方面：Ulysses通信量优于ring attention，Ulysess主要包含三次All2All通信量,而ring attention的通信会随着序列长度增长而平方增长。不过另一方面，all2all op由于需要更复杂的网络拓扑，例如NVLink和NVSwitch，因此在多机情况时，并不会随着机器数量增加而有较好的性能提升。 + + 内存占用：二者类似。 + + 模型结构泛化：ring attention优于Ulysses。Ulysses模型泛化性一般，对于head number有要求，需要满足: `head number // (tp group size * sp group size)` ，而ring attention没有此限制。 + +由于使用简单，对Attention计算不侵入修改，Ulysses目前是序列并行的主流。这些序列并行都可与其他高性能注意力兼容，如flash attention，还可以与ZeRO、TP、PP、DP等多种并行训练策略混合使用。 + +总的来说，我们更推荐您使用Ulysses，只需要在启动时指定```--sp_mode all_to_all```即可。经过测试，在双机16卡的情况下，使用```--tp 2 --sp 8 --sp_mode all_to_all```的启动参数可以很轻松训练128k长度的序列，同时它的性能表现也是所有序列并行模式中最好的，在双机H800上能够达到约480以上的tflops。但如果追求极致性能优化，或者使用较多机器训练长文本，可以考虑使用ring attention模式的序列并行。 +