Yandex, a global tech company, has recently introduced YaFSDP, an open-source method that aims to revolutionize the training of large language models (LLMs). This tool is currently the most effective publicly available method for enhancing GPU communication and reducing memory usage in LLM training, offering a speedup of up to 26% compared to its predecessor, FSDP. The potential savings in GPU resources could reach up to 20%, translating to substantial financial benefits for users.
"Currently, we're actively experimenting with various model architectures and parameter sizes to expand YaFSDP’s versatility," noted Mikhail Khruschev, a senior developer at Yandex. "We are thrilled to share our developments in LLM training with the global ML community, contributing to increased accessibility and efficiency for researchers and developers worldwide."
LLM training is a resource-intensive process that consumes significant GPU resources and time. The introduction of YaFSDP addresses these challenges by eliminating GPU communication inefficiencies and ensuring that training requires only the necessary processor memory. This optimization makes GPU interactions uninterrupted, thus speeding up the learning process and reducing costs.
For instance, in a pre-training scenario involving a model with 70 billion parameters, using YaFSDP can save the resources of approximately 150 GPUs, which translates to roughly $0.5 to $1.5 million in potential monthly savings, depending on the virtual GPU provider or platform. Such savings are not just theoretical; YaFSDP has demonstrated impressive results on models ranging from 13 to 70 billion parameters, with particularly strong performance in the 30 to 70 billion range.
YaFSDP is an enhanced version of FSDP and outperforms it in the most communication-heavy stages of LLM training, such as pre-training, alignment, and fine-tuning. The final speedup shown by YaFSDP on LLaMA 2 and LLaMA 3 models demonstrates significant improvements in training speed, reaching up to 26% on LLaMA 3 70B.
YaFSDP isn't Yandex’s first foray into open-source tools. The company has previously shared several other tools that have become popular within the ML community, including CatBoost, a high-performance library for gradient boosting on decision trees; YTsaurus, a big data platform for distributed storage and processing; and AQLM, one of the most advanced quantization algorithms for extreme compression of LLMs.
YaFSDP is freely available on Github, making it easily accessible for AI developers worldwide. This move is expected to democratize access to advanced LLM training methods, enabling more researchers and developers to build and train sophisticated models without incurring prohibitive costs.
In summary, YaFSDP by Yandex represents a significant advancement in the field of LLM training. By optimizing GPU communication and memory usage, it promises to reduce training times and operational costs substantially. This development is poised to benefit the global machine learning community by making high-efficiency LLM training accessible and affordable.
