San Francisco, CA – The rapid pace of artificial intelligence (AI) development, fueled by massive datasets and computational power, may be slowing down due to a looming data crisis. OpenAI, the company behind ChatGPT, is facing significant challenges in developing its next-generation AI model, GPT-5.
According to a recent report in The Wall Street Journal, OpenAI's ambitious project, codenamed "Orion," has encountered unexpected delays and escalating costs. Despite multiple rounds of extensive training on massive datasets, the results have fallen short of expectations. While the new model outperforms its predecessor, GPT-4, it hasn't demonstrated sufficient improvements to justify the immense computational resources consumed. It's estimated that six months of large-scale AI training can cost upwards of $500 million.
This setback has cast doubt on the long-held belief in the "scaling law," which posits that larger models trained on more data will inevitably yield better performance. Experts now suggest that this law may have reached its limits.
The primary bottleneck is the dwindling supply of high-quality data. Companies like OpenAI have previously relied on vast quantities of text data scraped from the internet, including news articles, social media posts, and scientific papers. However, as AI models become more sophisticated, the demand for novel and informative data is outpacing supply.
To address this data scarcity, AI developers are exploring alternative approaches. One strategy involves training models on synthetic data, generated by AI itself. Another involves creating specialized datasets, such as by hiring mathematicians to solve complex problems and using the resulting data to train AI models to reason and think more like humans.
The focus is now shifting towards improving the reasoning capabilities of AI models. Instead of relying solely on pattern recognition, researchers are developing models that can understand context and make inferences. OpenAI CEO Sam Altman recently unveiled "o3," a new AI model specifically designed for reasoning tasks, and plans to release a public version early next year. However, the company has been hesitant to announce a timeline for the release of GPT-5.
Google, another major player in the AI race, has also made strides in developing reasoning-based AI models, releasing a test version of its Gemini 2.0 model.
Demis Hassabis, CEO of DeepMind, expressed concerns about the future of AI scaling in an interview with The New York Times. While acknowledging the significant progress made in recent years, Hassabis noted that "we're not seeing the same level of progress anymore."
Ilya Sutskever, a co-founder of OpenAI who was instrumental in the development of ChatGPT, has warned that the internet's supply of human-generated content is finite. "Data is the fossil fuel of AI," Sutskever said in a recent talk. "We will undoubtedly run out of the kind of pre-training that we know and love."
As the AI industry grapples with these challenges, it's clear that a new approach is needed to continue driving innovation. While the scaling law may have reached its limits, the potential for AI to revolutionize industries and society remains vast.
[Copyright (c) Global Economic Times. All Rights Reserved.]
