Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More
Nous Research attracted attention earlier this month with the release of its permissive, open-source Llama 3.1 variant Hermes 3.
Now, the small research team devoted to creating personalized, unrestricted AI models has unveiled another significant breakthrough: DisTrO (Distributed Training Over-the-Internet), a new optimizer that minimizes the amount of information that needs to be transmitted between different GPUs (graphics processing units) during each step of training an AI model.
The DisTrO optimizer by Nous enables powerful AI models to be trained outside of large companies, over the open web on consumer-grade connections, potentially by individuals or institutions collaborating globally.
DisTrO has been tested and shown in a Nous Research technical paper to provide an 857 times efficiency increase compared to the popular existing training algorithm, All-Reduce, while reducing the amount of information transmitted during each training step significantly (86.8 megabytes compared to 74.4 gigabytes) with only a slight performance loss. Refer to the results in the table below from the Nous Research technical paper:
Ultimately, the DisTrO method could empower more people to train powerful AI models as needed.
What if you could use all the computing power in the world to train a shared, open-source AI model?
Preliminary report: https://t.co/b1XgJylsnV
Nous Research is proud to release a preliminary report on DisTrO (Distributed Training Over-the-Internet) a family of… pic.twitter.com/h2gQJ4m7lB
— Nous Research (@NousResearch) August 26, 2024
The problem with AI training: steep hardware requirements
As previously covered on VentureBeat, Nvidia’s GPUs are highly sought after in the generative AI era for their powerful parallel processing capabilities necessary for efficient AI model training. This blog post at APNic provides a good description of the process.
A significant aspect of the AI training process relies on GPU clusters—multiple GPUs—exchanging information about the model and the knowledge gained from training data sets.
This “inter-GPU communication” necessitates that GPU clusters be set up in a specific manner in controlled conditions to minimize latency and maximize throughput. This is why companies like Tesla are investing in physical “superclusters” with large numbers of GPUs located in the same facility.
Training generative AI, especially the largest and most powerful models, is typically a capital-intensive endeavor, accessible primarily to well-funded companies like Tesla, Meta, OpenAI, Microsoft, Google, and Anthropic.
Although the training process differs for each of these companies, they follow similar basic steps and hardware components. They tightly control their AI model training processes, making it challenging for others to compete by training similarly sized models.
However, Nous Research, with its approach of creating powerful and capable AI openly and affordably for all to use and customize, has found an alternative.
What DisTrO does differently
Unlike traditional AI training methods that require synchronizing full gradients across all GPUs and high bandwidth connections, DisTrO reduces communication overhead significantly.
The authors of the paper have not yet fully disclosed how their algorithms reduce information during training steps while maintaining overall model performance, but they plan to share more details soon.
This reduction was achieved without relying on amortized analysis or compromising training convergence rates, enabling large-scale models to be trained over slower internet connections available to many consumers around the world.
The authors tested DisTrO using the Meta Llama 2, a 1.2 billion large language model (LLM) architecture and achieved comparable training performance to conventional methods with significantly reduced communication overhead.
They mention that they are unsure if the bandwidth reduction ratio scales up, down, or remains constant as model size increases.
However, they indicate that preliminary tests show a possible bandwidth requirement reduction of 1000x to 3000x during pre-training of LLMs and up to 10000x without noticeable degradation in loss for post-training and fine-tuning.
Furthermore, they speculate that this research, initially conducted on LLMs, could be applied to train large diffusion models (LDMs) like the Stable Diffusion open-source image generation model and related image generation services.
Still need good GPUs
Despite DisTrO’s innovation, it still relies on GPUs, which are now distributed across the world and communicate over the consumer internet instead of being clustered in a single location.
DisTrO was evaluated using 32x H100 GPUs under the Distributed Data Parallelism (DDP) strategy, with each GPU having the entire model loaded in VRAM.
This setup allowed the team to rigorously test DisTrO’s capabilities, demonstrating that it can match the convergence rates of AdamW+All-Reduce with significantly reduced communication requirements.
This result suggests that DisTrO could potentially replace existing training methods without compromising model quality, offering an efficient solution for large-scale distributed training.
By reducing the need for high-speed interconnects, DisTrO could facilitate collaborative model training across decentralized networks, even with participants using consumer-grade internet connections.
The report also delves into the implications of DisTrO for applications like federated learning and decentralized training.
Additionally, DisTrO’s efficiency could help reduce the environmental impact of AI training by optimizing existing infrastructure and minimizing the need for large data centers.
Moreover, these breakthroughs could revolutionize how large-scale models are trained, shifting from centralized, resource-intensive data centers to more distributed, collaborative approaches utilizing diverse computing resources worldwide.
What’s next for the Nous Research team and DisTrO?
The research team invites others to join them in exploring DisTrO’s potential. The preliminary report and supporting materials are available on GitHub, and the team is actively seeking collaborators to help enhance and expand this groundbreaking technology.
Several AI influencers, such as @kimmonismus on X (aka chubby), have hailed the research as a significant breakthrough in the field, stating, “This could change everything!”
With DisTrO, Nous Research is not just advancing the technical aspects of AI training but also fostering a more inclusive and resilient research ecosystem that could lead to unprecedented advancements in AI.
VB Daily
Stay in the know! Get the latest news in your inbox daily
Thanks for subscribing. Check out more VB newsletters here.
An error occurred.
FAQs
Q: What is DisTrO in AI training?
A: DisTrO, or Distributed Training Over-the-Internet, is a new optimizer developed by Nous Research that reduces the information exchange between GPUs during AI model training, making it more efficient and accessible.
Q: How does DisTrO differ from traditional AI training methods?
A: DisTrO minimizes communication overhead by several orders of magnitude, enabling large-scale model training over slower internet connections and reducing the reliance on high-bandwidth interconnects.
Q: What are the potential implications of DisTrO for the AI industry?
A: DisTrO could democratize AI training by allowing individuals and institutions globally to train powerful models collaboratively without the need for massive infrastructure, fostering innovation and progress in the field.
Credit: venturebeat.com
