On March 18th, reports emerged that Elon Musk's AI startup, xAI, officially announced the open-sourcing of its large language model, Grok-1, making it available for public download.
It is understood that Grok-1 is developed using the Mixture-of-Experts (MoE) technology and is equipped with 314 billion parameters, surpassing the 175 billion parameters of OpenAI's GPT-3.5. This makes it the largest open-source language model to date in terms of parameter count, and it adheres to the Apache 2.0 license, which includes the model weights and architecture.
xAI mentions that Grok-1 was entirely trained in-house and completed its pre-training phase in October 2023. The release includes the original baseline model checkpoint from the end of Grok-1’s pre-training phase, indicating the model has not been fine-tuned for any specific applications.
One of xAI's goals is to compete in the large model domain against other giants like OpenAI, Google, Microsoft, with a team comprising members from renowned companies and research institutions such as OpenAI, Google DeepMind, Google Brain, and Microsoft Research.
As of now, xAI has not released specific performance metrics for Grok-1. The competition between it and other companies' large models is expected to be a focal point of industry attention.
