Introduction

Last week, Raza Habib, CEO of Humanloop, had the privilege of sitting down with Sam Altman and 20 other developers to discuss OpenAI's API and their product roadmap. Sam was remarkably candid. The conversation covered both practical developer concerns and broader questions related to OpenAI’s mission and AI’s societal impact. Here are the key takeaways:

OpenAI Is Currently Heavily Dependent on GPUs

A recurring theme throughout the discussion was that OpenAI is currently extremely reliant on GPUs, which has delayed many of their short-term plans. The biggest customer complaints revolve around the reliability and speed of the API. Sam acknowledged these concerns and explained that most issues stem from GPU shortages.

Longer 32k context isn’t yet scalable to more users. OpenAI hasn’t overcome the O(n^2) attention scaling problem, so while they may soon have context windows of 100k–1M tokens (this year), anything larger will require a research breakthrough.

Fine-tuning API is limited by GPU availability. They aren’t yet using efficient fine-tuning methods like Adapters or LoRa, making fine-tuning runs and management highly compute-intensive. Better fine-tuning support is coming. They might even host a marketplace for community-contributed models.

Dedicated capacity offerings are constrained by GPU availability. OpenAI also offers dedicated capacity, providing customers with private copies of models. To use this service, customers must commit upfront to spending $100,000.

OpenAI’s Short-Term Roadmap

2023:

Cheaper, faster GPT-4—This is their top priority. Overall, OpenAI aims to minimize the cost of “intelligence,” so they’ll continue driving down API costs.

Longer context windows—Context windows up to one million tokens are possible in the near future.

Fine-tuning API—The fine-tuning API will expand to the latest models, but its exact form will depend on what developers actually want.

A stateful API—When you call the chat API today, you must repeatedly pass the same conversation history and pay for the same tokens again. Future versions of the API will remember conversation history.

2024:

Multimodality—This was demonstrated as part of the GPT-4 release, but it can't be scaled to everyone until more GPUs come online.

Plugins "haven’t achieved PMF" and likely won’t appear on the API soon

Many developers are interested in externally calling ChatGPT Plugins via the API, but Sam said he doesn’t think they’ll release this functionality anytime soon. Aside from browsing, current plugin use cases suggest they haven’t yet achieved product-market fit (PMF). He noted that many people assumed they wanted apps to be used within ChatGPT, but what users really want is ChatGPT capabilities embedded within their own applications.

OpenAI will avoid competing with its customers—except for ChatGPT

Many developers expressed concern about building applications using OpenAI’s API, fearing OpenAI might launch competing products. But Sam stated that OpenAI won’t release additional products beyond ChatGPT. He emphasized that a great platform company typically has one killer app, and ChatGPT allows them to improve the API by being their own customer. The vision for ChatGPT is to become a super intelligent assistant at work, but OpenAI won’t venture into other GPT use cases.

Regulation is needed, but so is open source

Although Sam is calling for regulation of future models, he does not believe existing models are dangerous and considers regulating or banning them a major mistake. He reiterated his belief in the importance of open source and mentioned that OpenAI is considering open-sourcing GPT-3. One reason they haven’t open-sourced yet is his skepticism about how many individuals and companies have the capability to host and serve large models.

Scaling laws still hold

Recently, many articles have claimed that “the era of big models is over.” This misrepresents the original intent.

Internal data at OpenAI shows that scaling laws for model performance still hold—making models larger continues to yield better performance. The rate of scaling cannot be sustained indefinitely, as OpenAI has already increased model size by millions of times in just a few years, and continuing this pace is unsustainable. This doesn’t mean OpenAI won’t keep trying to scale up, only that increases may be limited to doubling or tripling per year rather than growing by orders of magnitude.

The continued validity of scaling laws has significant implications for the AGI development timeline. The scaling hypothesis suggests we may already possess most of the components needed to build AGI, with the remaining work largely involving scaling existing methods to larger models and datasets. If the era of scaling were truly over, we should expect AGI to be much further away. The fact that scaling laws still hold strongly suggests a shorter timeline to AGI.