data-parallelism

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Making large AI models cheaper, faster and more accessible

Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.

A state-of-the-art multithreading runtime: message-passing based, fast, scalable, ultra-low overhead

飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。

LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training

Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.

Distributed Keras Engine, Make Keras faster with only one line of code.

Ternary Gradients to Reduce Communication in Distributed Deep Learning (TensorFlow)

Orkhon: ML Inference Framework and Server Runtime

Distributed training (multi-node) of a Transformer model

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

Educational distributed training and inference library for heterogeneous local hardware: Mac minis, Raspberry Pis and GPUs over plain Python sockets. FSDP, GRPO, parameter-server, expert and data parallelism.

SC23 Deep Learning at Scale Tutorial Material

Multi-GPU training for Keras

Deep Learning at Scale Training Event at NERSC

Deep learning for science school material 2025

WIP. Veloce is a low-code Ray-based parallelization library that makes machine learning computation novel, efficient, and heterogeneous.

♨️ Optimized Gaussian blur filter on CPU.