TechFlow Logo
Login/ Sign up
ETH Gas
Gwei
Fear
gas
IOSG | AI at the Crossroads: Why Is Wall Street Saying "No" to ChatGPT and Claude?

IOSG | AI at the Crossroads: Why Is Wall Street Saying "No" to ChatGPT and Claude?

2026.07.14
Share
In the field where a company alpha operates, expert-tuned open models have already surpassed the frontier in both accuracy and cost, while the infrastructure to build it in a privacy-preserving environment is arriving node by node.

Author: Jeff @IOSG

Why Private AI Is Needed

On July 1, Palantir CEO Alex Karp gave a 20-minute interview on CNBC that some media outlets dubbed a "mental breakdown." According to Karp, enterprises are paying a premium for tokens to frontier labs while watching their own IP flow to model providers. He called this leak an alpha transfer, occurring at the architecture layer: every request sent to a closed-source model arrives as plaintext at the provider's servers. Just days before the broadcast, Palantir announced a collaboration with NVIDIA to run the open Nemotron model in a customer-controlled environment, accompanied by a nine-point AI sovereignty manifesto. Following the CNBC broadcast, PLTR stock jumped 8%.

The Crossroads of AI: Why Wall Street Is Saying “No” to ChatGPT and Claude?

For the past two decades, enterprises adopted cloud software based on trust at the protocol level, and it worked. Each SaaS vendor only saw slices of enterprise data, and most had little incentive to feed client data back into core products. Salesforce sees sales pipelines, Workday sees HR, Jira sees development iterations, and AWS provides the storage and compute foundation. However, today's AI workflows advocate uploading everything at once, along with the structured context linking various departments, to maximize productivity. Setting aside good intentions, upstream service providers can now use this data to build new features instead of letting it gather dust on servers.

No one is slowing down. Anthropic's annualized revenue hit $47 billion in May, a sharp increase from $9 billion at the end of 2025. OpenAI broke 900 million weekly active users in February. Both companies completed new rounds of financing this spring, with valuations nearing $1 trillion, and are expected to IPO at even higher market caps. Years of privacy and IP allegations haven't slowed either company down even a bit.

Some enterprises have already taken action. Within less than three months of ChatGPT's release in February 2023, major Wall Street banks had already restricted its use. In May 2023, after Samsung engineers leaked chip source code into ChatGPT, the company banned generative AI across its network. In response, OpenAI launched ChatGPT Enterprise in August that year, promising not to train on commercial data and adding a zero-data-retention (ZDR) agreement, which later became a standard requirement for enterprise procurement.

But contracts only lock corporate accounts. IBM found that by 2025, shadow AI (employees feeding company data into unapproved AI tools via personal accounts) was involved in one-fifth of all data breaches, with heavy shadow AI usage adding an average of $670,000 to breach costs. In Anagram's 2025 survey, four out of ten employees said they would violate AI usage policies to complete tasks faster.

Enterprises can at least buy their way out with ZDR contracts, non-training service tiers, and sovereign deployments if you're a government or Palantir client. As for regular users like us, whether privacy AI matters remains debated until subpoenas come knocking.

A court order in May 2025 forced OpenAI to retain consumer-grade chats even after users deleted them. In November, a judge ordered the handover of 20 million of these records to The New York Times lawyers as evidence during discovery. Then came criminal cases: ChatGPT records of a defendant in the Palisades fire arson case entered evidence, and an affidavit in a Florida double-homicide case cited the suspect's questions about how to dispose of bodies. Sam Altman also admitted in a July 2025 interview that ChatGPT conversations are not protected by legal privilege, and OpenAI "may be required to turn over" user chat logs in litigation.

The point isn't that only criminals need private conversations. People's dialogues with AI are archived and subpoenaable—a surveillance surface most users don't know exists. Kolmogorov Law's October 2025 survey of 1,000 US AI users found that 50% didn't know these conversations could be subpoenaed, while two-thirds believed these chats should enjoy the same protections as consulting a lawyer or doctor.

Self-hosted or open-source models running in verifiable environments are catching up quickly, but the top tier still lags behind leading closed-source models by about 4 months in general capabilities. This puts tokenmaxxing enterprises and individuals at a crossroads: give up several months of model quality for this privacy, or continue uploading sensitive materials to Anthropic's servers because competitors are seizing productivity advantages this way.

The Crossroads of AI: Why Wall Street Is Saying “No” to ChatGPT and Claude?

There is no perfect solution on the market yet. This report outlines various parties' attempts to bridge the gap, assessing how far provably private frontier intelligence remains from reaching enterprises and everyday users.

How Privacy Is Currently Achieved

Privacy AI is not a single engineering discipline, but currently, every mechanism in the market addresses the same event: a prompt leaves your device, travels across the network, lands on the machine running the model, and returns with a reply. The differences between mechanisms lie in where plaintext exists along this path, who can read it there, and what verifies the confidentiality of the response.

Protocol-Level Privacy

At this layer, someone other than you reads your plaintext prompt. What happens next relies entirely on a promise.

The Crossroads of AI: Why Wall Street Is Saying “No” to ChatGPT and Claude?

  • Contractual zero-data retention is the enterprise-grade solution. The provider knows who you are, processes your prompt, and promises not to retain it; enforcement relies on contracts and reputation.
  • Anonymous proxies erase who you are but do not encrypt what you said; downstream providers still process plaintext according to their own policies. Terms vary by provider—for example, Duck.ai (DuckDuckGo's chatbot product) negotiates deletion agreements with model providers, while Venice simply lets users assume the provider stores everything, neither being verifiable.

Every hop between machines runs over TLS, which only encrypts the pipe, allowing the receiving party to read all information. Relays typically split this visibility using Oblivious HTTP (RFC 9458), functioning like asking a friend to pass a note. The friend knows who passed it but can't read the content; the recipient reads the content but doesn't know who wrote it. OHTTP has been an IETF standard since January 2024, and many companies are already routing production traffic through OHTTP relays leased from Cloudflare and Fastly.

This is the privacy ceiling available when accessing closed-source models, due to a simple arithmetic reality. A single flagship training run now costs around a billion dollars, and these labs' multi-trillion-dollar valuations bet squarely on exclusive model weights. How long the capability gap lasts dictates how long the premium lasts, so labs guard weight files like state secrets.

Meta has already passively conducted this experiment. LLaMA, released in February 2023, was initially open only to researchers. But within a week, weights leaked to 4chan in seed form. Another week later, llama.cpp enabled the smallest 7B model to answer locally on a MacBook. Three days later, Stanford fine-tuned a chat assistant named Alpaca on the same model for under $600. This leak drove Llama's inference cost down to electricity bills; anyone with the file could run it at home. By July 2023, Meta officially open-sourced Llama 2 under a commercial license with a monthly active user exclusion clause exceeding 700 million. Weights ran, and the premium ran with them.

Frontier labs could theoretically perform remote attestation for closed-source model inference, but attestation only proves which code segment read the prompt, not what it did with it. To determine if a server retained data, we must audit the serving code and reconstruct it to the hash reported by the hardware. But handing over serving code means labs surrender batch processing and caching techniques that support their profit margins, which would migrate to every future model generation. Apple and Meta can remotely attest to the service stacks behind iPhone and WhatsApp because their profits lie in devices and ads; publishing serving code costs them virtually nothing.

This is why flagship model weights and serving code never reach external operators. Without external operators, there is no third-party attestation. Without attestation, verifiable privacy only exists atop open-source models.

Machine-Level Privacy

In this category, each mechanism replaces trust-based promises with hardware-based, cryptographic, or physical proofs. However, each demands different costs for upgrading privacy, primarily that they can only run open-source models.

The Crossroads of AI: Why Wall Street Is Saying “No” to ChatGPT and Claude?

  • TEE (Trusted Execution Environment) confidential computing runs inference inside a hardware enclave (a sealed chamber on the chip even inaccessible to the machine operator). The chip signs an attestation specifying exactly which model and code segment ran.
  • The prompt is only sealed at the destination. The path relayed through platform proxies still leaves a role that can read plaintext. Only the protocol prevents proxies from logging or leaking relayed content.
  • E2EE (End-to-End Encryption) seals off readable relays. User devices encrypt prompts using the enclave's key; every hop along the way carries a sealed envelope only the enclave can open.
  • Trust falls on the client. The code responsible for encrypting prompts and verifying attestation also holds the power to revoke this guarantee. Therefore, verifiable E2EE requires both proven enclaves and open, reproducible client-side code.
  • Compared to TEE's simplicity, E2EE's cost is engineering overhead, which slows feature integration. E2EE turns the proxy into a blind courier, forcing all features reliant on reading plaintext to be rebuilt around client-side keys or solely within the enclave.
  • FHE (Fully Homomorphic Encryption, and MPC variants) removes trusted parties entirely. Servers compute on ciphertext inside a locked box they can never open; only you hold the key. MPC (Multi-Party Computation) splits the prompt into secret shares distributed among multiple parties, yielding equivalent results unless all participants collude.
  • The cost is speed. FHE natively only supports addition and multiplication, so the nonlinear steps required for transformers must be reconstructed at high cost. Inference on ciphertext costs 10,000 to 100,000 times more than plaintext, taking seconds to minutes per token on small models, versus milliseconds without encryption.
  • Chips customized for encrypted computation are expected to close the gap, but the first prototype demo wasn't completed until early 2026, with commercial versions still years away.
  • Local inference completely deletes this path. Models run on your own hardware—no relays, no servers, no providers, and no verification requirements.
  • The obvious cost is expense and model capability. gpt-oss-120b scores roughly half of GLM-5.2 on the Artificial Analysis index, but at 65GB volume, it exceeds the combined VRAM of two flagship gaming GPUs on the market. Meanwhile, full-precision GLM-5.2 can only run on an 8-GPU data center node, requiring over $300,000 in GPUs alone.

Yet beyond these structural limits, the cost of running inference in enclaves is compressing. For single-GPU inference, baseline tests from enclave cloud provider Phala show H100 throughput loss in enclave mode averages under 7%, approaching zero for large models, since the primary cost lies in moving data into the chip rather than computing inside it. For multi-GPU inference, NVIDIA's next-gen GPU Blackwell now supports direct encryption of inter-chip traffic, while older H100s achieve the same effect only by routing at one-seventh bandwidth to the CPU host. NVIDIA's own benchmarks on Blackwell show less than 8% throughput loss for a 397B model in enclave mode. With these advances, performance degradation from privacy inference itself is no longer the decisive constraint.

In fact, enclaves add virtually no extra operational cost for operators. Every H100 since 2023 includes native enclave mode; the additional cost is throughput loss from encryption, not extra chips. Confidential H100 SKUs on Azure still lease at $8.90/hour, versus $6.98 without enclave—an effective 27% markup on traditional cloud infrastructure. Meanwhile, on dedicated enclave operators like Phala, confidential-mode H100s lease starting at $3.80/hour, undercutting Lambda's standard SXM card range of $3.99–$4.29. For managed API solutions, NEAR AI offers gpt-oss-120b via attested endpoints at $0.15/million input tokens and $0.55/million output tokens, matching plaintext routes on Amazon Bedrock, Together, and Groq. Even for models requiring multi-GPU parallelism, NEAR AI prices GLM 5.2 identically to Fireworks, and inputs 15% cheaper and outputs 4% cheaper on the larger Kimi K2.6.

While these new privacy inference providers may be burning profit to grab share (true for any growth-focused company in this market), the structural trajectory points toward declining costs for consumers and operators alike.

How Can Open-Source Models Win?

Despite compressing performance overhead, a visible gap remains between frontier models and SOTA open-source models. Any entity seeking maximum productivity staying at the forefront still must trust frontier labs not to steal its IP.

The gap persists, but Bridgewater-affiliated AIA Labs and Thinking Machines presented a case on June 30: an open model fine-tuned with expert annotations simultaneously defeated frontier models in accuracy and cost.

In the study, the team fine-tuned Qwen3-235B on Tinker (Thinking Machines' hosted fine-tuning API service). They first purchased annotations from vendors, trained the first round with this data, then sent divergent samples to the firm's investment personnel for re-annotation. Training used reinforcement learning (GRPO) plus three modifications: round-robin batching, CISPO loss (capping how far a single answer can pull the model), and on-policy distillation (anchoring the current best checkpoint to prevent the model from learning from weaker clones).

Tasks were entirely drawn from the daily workflows of investment personnel: whether a news article matters to C-suite investment professionals, whether a central bank document hints at future interest rate directions, and where boilerplate starts in a document or email. Scoring came from independent test sets. Frontier models averaged ~50% on simple prompts, peaking at 78.2% with expert prompts, falling short of the 80% threshold set by investment staff. The fine-tuned Qwen scored 84.7%, which, by the paper's metrics, equates to a 29.8% reduction in errors compared to the frontier optimum, at 13.8x lower inference costs.

The Crossroads of AI: Why Wall Street Is Saying “No” to ChatGPT and Claude?

https://thinkingmachines.ai/news/learning-to-replicate-expert-judgment-in-financial-tasks/

This case proves open-source models can win on accuracy and cost, but the training process remains non-private. The expert annotations used are Bridgewater's proprietary data, routed through Tinker's third-party service, landing on the same trust layer as ZDR agreements. The fund also rented compute power, running the entire training on machines it never controlled. Buyers wanting this recipe without assuming trust have few options today. Renting bare-metal GPU clusters leaves the training process readable by cloud operators. Buying clusters solves data hosting but skyrockets costs.

The attestation route is just arriving. In March, Workshop Labs and Tinfoil released Silo, a post-training stack running inside a Tinfoil enclave on a single 8-GPU node, with keys held solely by the customer. The article cites enclave costs as an extra 11 minutes of training time. Furthermore, this stack fits a trillion-parameter model (Kimi K2 Thinking) by freezing base weights and training only small adapters on top. The challenge is that reinforcement learning requires shuttling data back and forth between components, and data movement is precisely where enclave costs accumulate.

Less than a month after Silo's release, Workshop Labs was acquired by Thinking Machines. Components needed to run the next Bridgewater-style RL loop inside an enclave now reside under one company.

Harness-Level Privacy

Another problem sits outside all private inference mechanisms. These mechanisms each govern the prompt-to-model path, but every external tool call initiated by an agent opens a route untouched by the inference layer. The recent harness engineering trend magnifies this exponentially. Every tool, memory store, and data source attached to the model becomes another destination reading its slice of the workflow in plaintext. Calendar servers read schedules; database servers read queries. A fully local agent still needs to submit search terms in plaintext to a search engine if it wants anything beyond its training set. If the server can't read plaintext, it can't answer.

Mainstream solutions still default to the protocol layer. Companies like Runlayer and MintMCP use a central gateway to manage all tool traffic, masking PII before requests leave. The gateway simultaneously decides which servers receive traffic, blocks unvetted requests, and logs destinations and contents for forensics. Even with independent audits (SOC 2) attached to these controls, tool servers must still read plaintext queries to respond. Whether they retain copies depends on their own retention terms, multiplied by every tool in the harness. Additionally, the gateway itself is another trust-dependent reader in the path, not a verifier.

Structural-level solutions strike the middle layer. For example, Phala hosts MCP servers directly inside TEEs, with directories covering wallets, code execution, and data sources. Users can verify privacy claims with an attestation instead of trusting the operator. However, tools hosted in TEEs ultimately hand queries in plaintext to the service provider; the enclave only seals the messenger, not the destination.

The Crossroads of AI: Why Wall Street Is Saying “No” to ChatGPT and Claude?

Few destinations have learned to answer without reading, and only for structured queries. Apple offers private information retrieval for iPhones, allowing caller ID checks against spam databases without exposing numbers. Microsoft applies the same scheme to passwords in Edge. MongoDB's Queryable Encryption allows clients to encrypt fields before they leave, enabling the server to perform equality and range matches using only ciphertext.

But for open-ended search, today's best answer stops at trust. Verifiable encrypted search hasn't left the lab. Brave promises zero-data retention on its 400-billion-page index (rather than Google's), but it still rests on the protocol layer. Exa built a neural index embedding user keywords semantically, sorting results by semantic match, but the embedding step still computes from plaintext on Exa's servers. MIT's 2023 Tiptoe paper ranked 360 million webpages without exposing queries, but each search burns massive server compute, and ranking quality trails unencrypted search. Apple's 2024 Wally paper hides real queries inside decoys, capping communication costs at 31x reduction, but this math only becomes cheap at millions of concurrent queries—a scale no private search system currently possesses.

Encrypted search is feasible, but performance and price aren't yet commercially viable.

Outlook

Demand for private AI is growing. Venice AI recently surpassed 3.5 million registered users and 1.3 trillion tokens/month in throughput, subsequently closing a new Series A valuation at $1 billion. Proton is its direct competitor; its chat product Lumo crossed 10 million users within a year of launch. On the infrastructure side, Phala currently routes 2–3 billion tokens daily on OpenRouter alone. Duck.ai routes gpt-oss-120b and Gemma into Tinfoil enclaves, offering verifiable privacy beyond user proxies. This doesn't even count self-hosting, likely the largest channel for private inference, since models run on personal hardware, leaving no usage footprint.

Yet placed against the mainstream AI surge, privacy AI occupies a tiny fraction. This gap will only close if frontier labs intentionally meet this demand. In May, Google processed 32 quadrillion tokens across all products; by that calculation, Venice's monthly throughput equals roughly 18 minutes of Google's. Last November, Google launched Private AI Compute (PAC), routing select Gemini-driven features into sealed TPU enclaves isolated from the company's own systems, designed for independent audit by NCC Group. The catch: PAC only covers a handful of Pixel features like personalized recommendations and audio summaries, not the billions using Gemini apps. Google dares to hand designs to auditors because these features monetize via devices and ads, not token sales.

Current managed solutions remain imperfect. Users seeking maximum privacy via E2EE must wait for new features to be rebuilt where the provider can't read them. Private harnesses still rely on protocols at the service layer. For affordably priced post-training, achieving the finest fine-tuning results still requires trusting third-party vendors. Self-hosting temporarily sheds all providers, but running the strongest open-source models locally may cost more than the house plugging it in.

Flaws notwithstanding, private AI is already a real and affordable option, with remaining gaps narrowing. For everyday consumers, open-model private chats under no-log commitments are free on Lumo and Venice. Venice or Tinfoil's $18–$20 subscriptions seal identical chats inside enclaves, costing no more than a ChatGPT subscription. For enterprise workflows, attested endpoints are now cheaper than plaintext routes even. Endpoints like NEAR's E2EE API can already carry encrypted context into enclaves; memory, file uploads, and custom instructions can run atop E2EE today. As for attested post-training, NVIDIA's upcoming Vera Rubin NVL72 will expand confidential computing from Blackwell's 8-GPU nodes to 72-GPU racks, making frontier RL loops more viable without exposing IP.

Yet the critical value capture lies beyond these price-compressing layers. Privacy is nearly free where it already exists, but hasn't yet covered mainstream agentic workflows. Operators specializing in renting enclaves hold a switch on standard chips, not a moat, while protocol-layer gateways compete alongside traditional middleware. Defensible ground lies in the other half of this report's unsolved problems: training loops sealed inside enclaves, end-to-end sealed tool calls, search indexes invisible to term tracking. Whoever builds any of these first sells something that cannot be commoditized by any price war. Capital chasing privacy AI should buy the gaps, not the switch.

So, trust or verify? For execution-heavy, agent-dense tasks, choose trust, because every tool call inherently hands plaintext to destinations enclaves can't seal, and frontier models deserve their price in such loops. For the advanced reasoning that distinguishes one company from rivals, choose verify. Strategy, planning, and judgment distilled from years of professional experience constitute precisely that disputed alpha. The path forward lies in fine-tuning open-source models with these proprietary insights within corporate-controlled boundaries. In domains where a company's alpha lives, expert-refined open models already defeat frontiers in accuracy and cost simultaneously, and the infrastructure to build them in private environments is arriving node by node.