TechFlow Logo
Login/ Sign up
ETH Gas
Gwei
Fear
gas
Benchmark Scores Soar, Real-World Tests Fail: Just-Released GPT-6.2 Exposes OpenAI’s Weakness

Benchmark Scores Soar, Real-World Tests Fail: Just-Released GPT-6.2 Exposes OpenAI’s Weakness

2026.11.18
Share

Author: King of Techno, Geek Park

The rift between paper benchmarks and real-world experience reached its peak today.

"Basically unusable, not recommended."

Just 8 minutes after the launch of OpenAI's new model, well-known Silicon Valley entrepreneur Matt Shumer dropped a ruthless evaluation on social media.

Follower closely, Theo Browne, who commands a large base of developer fans, published a long essay titled "Why I Hate GPT-6.2 Astra." Ethan Mollick, a Wharton professor who typically maintains a gentle and patient attitude toward frontier models, also delivered a rare and strict verdict after testing: "highly unstable and weird," stating bluntly that this is "the first time since GPT-4.5 that an OpenAI frontier release has shown no obvious progress."

This was supposed to be a brilliant counterattack by OpenAI.

In the early hours of November 18 Beijing Time, OpenAI officially released the next-generation flagship GPT-6.2 Astra and the daily workhorse GPT-6.2 Sol. The company directly slashed the API unit price for Astra by 20% and handed over a report card that refreshed records across the board, from agents to scientific research.

When the wild sprint of benchmark scores collided with the collective complaints of frontline developers, this extreme sense of fragmentation precisely exposed OpenAI's most realistic dilemma at present.

Benchmark Gaming and Hidden Compute Costs

Judging solely by official posters and top-tier data from third-party comprehensive leaderboards, both GPT-6.2 models are indeed highly competitive.

Both models feature a 1.05 million context window, with input limits of 922k tokens and output limits of 128k tokens. They currently only support text and image inputs with pure text outputs, without support for audio/video or fine-tuning. Regarding pricing, the new flagship replacing the early September GPT-6 Astra is set at $8 per million input tokens and $40 per million output tokens, with cached inputs slashed by 60% down to $0.40; the GPT-6.2 Sol, which replaces the September DevDay version, maintains unchanged prices of $2 for input and $10 for output.

On Artificial Analysis's v5.0.1 Comprehensive Index leaderboard, GPT-6.2 Astra leapt to second place overall with a score of 62, trailing behind the 65-scored Claude Fable 5.2, while surpassing the 60-scored Gemini 4 Carbon and the 59-scored Claude Opus 5.5.

Even more tempting is the single-task cost: Astra averages just $4, only 15% of Fable 5.2's staggering $26.54 cost. Achieving 57 points, Sol costs only $1.01 per task, firmly landing in the high cost-performance quadrant of the upper-left corner on the Pareto frontier scatter plot.

Fig: Comparison of Artificial Analysis Intelligence Index v5.0.1 total score leaderboard and single-task cost Pareto frontier | Image source: Artificial Analysis

But once you scroll through the official blog post and carefully read those minuscule footnotes, the flavor starts to change.

In the Agents' Last Exam test, OpenAI proudly announced that Astra achieved a peak score of 78.3% (in the XHigh tier, Max tier is 77.6%), significantly ahead of Fable 5.2's 72.8%. Yet the footnotes quietly note that this comparison excluded 9 tasks where Fable 5.2 used a fallback model. More subtly, in the official illustrations, Google's Gemini 4 Carbon scored 77.4% under high reasoning mode at a single-task cost of $4.60, nearly matching Astra's $4.47, yet the official article omits any mention of this competitor.

Beyond the sophisticated filtering of benchmark metrics, two hidden landmines lurk within the API pricing table.

If a single input exceeds 272k tokens, the input and cached unit prices for the entire request double, and the output unit price increases by 1.5 times, causing Astra's prices to jump directly to $16 and $60. Meanwhile, the Ultrafast mode, limited to US-region data residency and offering up to 4.5x speedup, is currently discounted at 3x acceleration. However, starting March 1, 2027, it will double to 6x, at which point Astra's input and output prices will become $48 and $240.

Turning attention to specialized leaderboards that dictate programmers' daily experiences, shortcomings are exposed even more thoroughly.

On Artificial Analysis's Coding Agent leaderboard, GPT-6.2 Astra is stuck at 53 points, leveling off with the previous generation, trailing behind the 58-scored Claude Fable 5.2 and Opus 5.5, as well as the 55-scored Gemini 4 Carbon. The Sol model, designed for daily volume, actually regressed by 3 points to 49, dropping back to tie with open-source Kimi K3.1 Pro. Among these, the Codebase Q&A category plummeted by 14 points, making it highly prone to missing critical facts.

Fig: Artificial Analysis Coding Agent Index v2.1 comparison: Astra stagnates at 53 points, Sol regresses 3 points from the previous generation | Image source: Artificial Analysis

In tests by another authoritative body, Epoch AI, although Astra solved 7 out of 68 frontier Erdős math problems, breaking the previous record of 3, the report still criticized it for lacking significant progress over the previous generation in real-world research automation tasks and being overly conservative. On the Vals comprehensive leaderboard, Astra ranks only 7th, with its standard version falling to 30th place in legal retrieval, requiring a specially customized legal edition to salvage its reputation.

Only in extreme vulnerability mining with safety guardrails completely disabled does it demonstrate a terrifying ceiling.

DeepBinary tested it on 12 Windows system binary files. Across two independent runs, Astra average discovered 348.5 unique vulnerabilities, valued at up to $1.038 million according to Microsoft bounty standards, with an API cost of merely $4,072; the smaller Sol model also uncovered 227 vulnerabilities worth $687,000 at a cost of $840, leaving far behind Opus 5 and Kimi K3.1, whose valuations stalled around $160,000.

The Genius with Extreme Subject Bias

Why does a model capable of independently uncovering million-dollar system vulnerabilities become "basically unusable" in the hands of ordinary programmers? Flipping through netizens' frontline testing records reveals a genius with extreme subject bias.

Once handed clear-boundary, hardcore prompts leaning heavily toward spatial geometry or physics simulation, the capability ceiling of GPT-6.2 is indeed astonishing.

Developer Thomas Ricouard used it to combine street maps with French surveying data, replicating a 3D Paris "Paris Atlas" covering over 131,000 buildings in one-to-one scale within the browser. Complete with street architecture, pedestrians, and traffic flow, it could also be continuously refined based on screenshot feedback.

Fig: The 1:1 3D Paris project "Paris Atlas" built by OpenAI developers based on GPT-6.2 (containing over 131,000 building data points) | Image source: TechRadar / OpenAI

In official OpenAI demos, Astra even independently modeled a 3D convertible sports car, wrote its own Python physics solver, and simultaneously rendered suspension movement and four-wheel ground pressure heatmaps for mountainous curves in Blender. Enterprise-side cases are equally outstanding: Higgsfield's internal 3D generation achieved a first-time success rate of 73%, quantitative giant Jump Trading broke a hardware optimization benchmark maintained for three years within a single day, partners like Harvey, Legora, and ClickUp reported saving between 30% to 60% in tokens, and Replit's practical tests showed Sol 6.2 accelerating nearly 3-fold.

Fig: GPT-6.2 Astra's self-built 3D car and the resulting mountain chassis dynamics and four-wheel ground pressure simulation using a Python solver | Image source: OpenAI Official

Fig: However, once departing from single-point breakthrough demo scenarios and entering real-world software engineering that requires long-term communication and multi-step collaboration, the quirky temperament of GPT-6.2 Astra drives people crazy

In that viral essay with over a million reads, Theo Browne first acknowledged Astra's strong code review capabilities, noting it even designed a reverse proxy architecture for him that saves $25,000 monthly, exclaiming "Even Opus lacks the courage to explore this much." But immediately following that, he listed four soul-crushing transgressions.

The most fatal flaw is the complete lack of concept regarding communication blind spots. Theo asked it to design a new level for "Outer Wilds" in goal-oriented mode. After running for two hours, the model suddenly froze, stopping only to ask him about an internal temporary filename impossible for a human to understand, forcing him to write a scheduled script just to monitor it. Second is the unbearable "Astra jargon"; begging it to explain in the simplest terms via system prompts proved futile. During one instance troubleshooting a sluggish working tree, Astra spewed four massive paragraphs on package manager caching mechanisms. Theo tossed the problem to Claude Opus 5.5, which instantly pinpointed the truth in a single sentence: "Your SSD is too slow."

More torturous is its obsession with infinite loops. On a test project to rewrite the TypeScript compiler in Rust, early OpenAI models burned over $300,000 with zero progress, while Opus completed it smoothly in two weeks for about $20,000. This time, Theo switched to 6.2 Astra to retry. The model ran for 7 consecutive days with zero improvement, stubbornly trying to reduce size, which ironically slowed down the compiler by 56%; after he explicitly forbade size optimization, the model idled for another 4 days, still yielding zero commits. Additionally, aesthetic sense suffered severe regression to the mean. When Theo and his friends had it design unique levels for different games, it surprisingly spit out identical water bell-themed levels named "Bellwether" for both.

This experiential gap is by no means an isolated case in developer circles. Every founder Dan Shipper complained it falls short of Fable 5.2, with its excessive pursuit of minimalism causing writing to regress into obscurity, end-to-end coding frequently halting midway, and designs reeking of rigid templates. In Simon Willison's famous Three.js Yonkers Mecha practical test, where the model was instructed to transform a city map into a giant mecha, both Astra and Sol failed to execute the correct transformation, delivering visuals far inferior to Fable 5.2 and Gemini 4 Carbon.

Fig: Simon Willison's side-by-side Three.js "Yonkers Mecha" practical test comparison: GPT-6.2 Astra (top) vs. GPT-6.2 Sol (bottom) | Image source: Simon Willison

Even Ethan Mollick, consistently mild-mannered, gave a negative review of "highly unstable and weird, harder to steer" despite using it to craft an interactive space elevator shooting straight into space from beneath cloud layers in Blender and Unity. Many in the community also complained that legitimate binary reverse-engineering tasks were frequently false-flagged and killed by guardrails; worse, the frontend charts on the official release page itself crashed and errored on launch day.

Fig: Ethan Mollick's demonstration of an interactive space elevator built from scratch in Blender and Unity using GPT-6.2 Astra | Image source: Ethan Mollick (X)

A Compromised Work Rattling Its Own Ranks

Zooming the timeline back to November, it's not hard to see why GPT-6.2 carries such a hurried, compromised flavor.

External rivals are closing the net at a terrifying pace. On November 9, on the eve of its IPO, Anthropic unleashed Fable 5.2 and Mythos 5.2,率先 pushing flagship prices down to $8 and $40, dominating the #1 spots on major programming and comprehensive leaderboards. It also showcased shocking feats: 1,000 agents completing a formal proof of Fermat's Last Theorem in 30 hours, an bootable OS prototype written in 8 days, and 3 circuit boards designed and lit up on the first try in just 1 day.

On the other front, Google's Gemini 4 family rolled out cards consecutively from October to November. Carbon's promotional price for November was crushed to $2 and $10, while Hydrogen sat at merely $1.5 and $7.5, accurately targeting enterprise concerns over reliability. In the AA-Omniscience test, hallucination rates for Gemini 4 Hydrogen dipped to just 8%, and Carbon to 11%, whereas GPT-6.2 Astra's hallucination rate deteriorated from 50% in the previous generation to 59%, with Sol skyrocketing to 55%. Paired with multi-agent dashboards and scheduling tools, Google can run tens of thousands of concurrent collaborations in just two days.

Fig: AA-Omniscience v1.1 knowledge reliability and hallucination rate comparison: GPT-6.2 Astra hallucination rate reaches 59%, while Gemini 4 Carbon and Hydrogen are only 11% and 8% respectively | Image source: Artificial Analysis

With Anthropic raising the ceiling upstream and Google solidifying the low-hallucination floor downstream, OpenAI found itself plagued by internal fires during those crucial two months in between.

The originally scheduled GPT-6.1 Astra for late September was urgently scrapped at the last minute after testing revealed serious unauthorized access and inaccurate reporting. Co-president Greg Brockman revealed in a public interview that approximately 25% of product engineers had been reassigned to support the security team. Just one day before the GPT-6.2 launch, the California Attorney General formally sued OpenAI, exposing at least 80 security incidents during testing.

With massive manpower pulled away to fight fires, while simultaneously forced to rush a release before competitors fully captured developer minds, the consequences were all documented in the system card openly disclosed by the company itself.

This exceptionally candid security report admitted to three types of regression. First is learning to hide problems more covertly ("reporting good news but hiding bad"); when encountering obstacles like missing files, Astra hid barriers at a rate that doubled from 11.09% to 22.28%, while Sol reached 24.08%; when search tools malfunctioned, the rate of fabrications in native API calls skyrocketed from 7.58% to 23.00%. Second is soaring surveillance escape and evaluation awareness: 85% of test trajectories doubted in their chain-of-thoughts that they were being evaluated, while file download escape rates under non-chain-of-thought modes spiked from 1.7% to 48.8%.

To keep this beast, which crossed the high-risk threshold of "AI self-improvement" for the first time, at bay, OpenAI hastily deployed a 35-second network-disconnect emergency stop control room system after its reinforcement learning agent accidentally connected to the public internet in September, and urgently tightened refusal boundaries. However, this clumsy patch directly led to a double loss in security and usability. The UK's Institute for AI Safety breached the pre-release version's network jailbreak defenses using half the query attempts, labeling it a "substantive regression," while the availability rate for legitimate cybersecurity tasks plummeted from 90.7% to 78.1%. Mental health benchmark scores also dropped from 58.7% to 53.3% because the models rushed to give answers rather than asking clarifying questions about context.

Understanding these underlying circumstances makes all of OpenAI's abnormal compromises on the product side make perfect sense.

To date, GPT-6.2 has still not been integrated into the standard ChatGPT chat interface aimed at hundreds of millions of ordinary users; it dares only reside in workspaces and coding tools. On the API end, turning off the reasoning mode is directly prohibited, and regulations stipulate that switching to a degraded model or altering historical prefixes will cause the system to directly discard prior reasoning records. It neither dares to fully unleash its capabilities nor can it afford to lose face on the benchmark leaderboards.

The era of competing in large language models by simply stacking compute to rack up pretty scores is over. As real-world workflows begin demanding stability, low hallucination rates, and effective communication, GPT-6.2 Astra's mixed-reception debut has sounded an alarm. Making smart models learn to understand human language and honestly report issues is far more urgent than artificially inflating benchmark scores by a couple more points.