Hook: The Numbers That Don’t Add Up
On paper, a $40 million Series A led by a16z at a $400 million valuation for an AI evaluation startup sounds like business as usual in the 2025 hype cycle. But the data suggests something else. Vals AI claims its revenue has already reached “eight times the full-year 2025 projection” — a statement that, upon deconstruction, reveals a temporal ambiguity common in fundraising narratives. The actual revenue figure is undisclosed, the client list unverified, and the core product, while promising, operates on a technical premise that mirrors the very flaws it aims to fix. This is not a story about a new AI model. It is a story about the architecture of trust in a trustless system — and why the crypto industry has been here before.
Context: The Rise of the Third-Party AI Auditor
Vals AI, founded by an anonymous team (the original article cites a source called “Dongcha Beating”), positions itself as an independent evaluation platform for large language models. Its innovation is not algorithmic but infrastructural: it extracts real development tasks from GitHub pull request histories, runs them against models using hidden tests, and scores performance on a per-codebase basis. Think of it as SWE-bench meets enterprise SaaS. The company claims that OpenAI, Anthropic, Google, Meta, and xAI already cite its “model cards” — a claim that forms the bedrock of its credibility. a16z’s bet is that this evaluation layer becomes the standard for AI procurement, akin to how SOC 2 audits became mandatory for cloud vendors.
But the market context is critical. Public benchmarks like GSM8K and HumanEval are increasingly contaminated — models are trained on test sets, gaming the scores. Vals AI’s pitch is a direct response to this crisis of trust. Yet, as I learned during my 2017 ICO audit framework analysis, when a project claims to solve a trust problem, the first question is: who audits the auditor? The crypto industry’s history with centralized oracles, exchange reserves, and even smart contract auditors provides a sobering precedent.
Core: Deconstructing the Myth of Independence in AI Evaluation
Vals AI’s technology is an engineering innovation, not a scientific breakthrough. The core method — extracting tasks from public GitHub PRs and running hidden tests — is a productization of existing dynamic evaluation techniques. The innovation lies in the integration: a SaaS layer that lets enterprises evaluate models on their own codebases without building their own benchmarks. This is a valid niche, but it carries at least three unaddressed risks.
First, data contamination cannot be ruled out. If the historical PRs are from public repositories, they may already be in the training data of the evaluated models. Vals claims to use “private and protected” code, but the article does not disclose how this filtering works. During my post-mortem of the LUNA collapse, I learned that algorithmic feedback loops are often invisible until they fail. Similarly, if Vals’ test set is static, model providers can eventually reverse-engineer it. The company’s defense — that they create custom evaluations per client — helps, but it introduces a scalability problem: each custom evaluation requires human annotation, especially in regulated domains like finance and law. The cost structure for this labor is not mentioned, raising questions about the sustainability of the business model.
Second, the revenue claim is structurally ambiguous. The phrase “eight times the full-year 2025 projection” is a classic fundraising warm-up. Without disclosing the base year’s projection, the multiple is meaningless. The valuation of $400 million implies a 9–10% dilution for the Series A, which is standard, but it also implies a multiple of at least 10x on undisclosed revenue. This is a bet on “category creation,” not on current financials. I have seen this pattern before: during DeFi Summer in 2020, I tracked Uniswap V2 liquidity and found that TVL spikes correlated with social sentiment, not sustainable yield. The correlation between Vals AI’s valuation and its actual traction is similarly unverified.
Third, the independence of the auditor is compromised by its capital structure. a16z is a lead investor in Vals AI and also a major investor in several AI companies that could become clients. The “third-party” label is thus a narrative convenience, not a structural guarantee. In crypto, we saw this with centralized exchange audits: the auditors were paid by the entities they audited, leading to conflicts of interest. Vals AI’s model is no different. The company’s claim that model cards from OpenAI and Anthropic cite its data is unverifiable without access to the cited cards. The article itself admits that this information comes from company self-reporting — a reliability grade of C on my empirical skepticism scale.
Contrarian: The Blind Spot — Centralized Evaluation Is Not Trustless
The contrarian angle is that Vals AI’s success could actually worsen the trust problem. By centralizing evaluation into a single, VC-backed entity, the industry substitutes one gatekeeper (model providers) for another (the evaluator). The risk is that Vals AI becomes a bottleneck, where its scoring methodology is opaque, its commercial relationships influence results, and its failure to detect a model collapse — like the Terra LUNA failure — could cascade across the entire AI procurement ecosystem.
During my five-year study of decentralized compute networks (Render, Akash) for the “Compute as the New Gold Standard” series, I observed that the most resilient systems are those that distribute verification across multiple independent nodes. The crypto industry already solved this problem with on-chain oracles (Chainlink) and zero-knowledge proofs. A truly trustless AI evaluation would allow anyone to run a test on a model’s output and cryptographically verify the result. Vals AI is a centralized intermediary in a market that needs decentralized infrastructure.
Furthermore, the article’s source is a Web3/blockchain monitoring channel, implying that the crypto community is aware of this narrative. Yet the article itself does not address the blockchain angle. The omission is telling: Vals AI is a product of the traditional VC world, with no token model, no decentralization, and no transparency beyond the standard SaaS terms. The architecture of value in a trustless system requires more than a third-party claim; it requires verifiable, immutable records. Vals AI provides the former, not the latter.
Takeaway: The Next Narrative Is Verification, Not Evaluation
The Vals AI funding round is a signal that the market recognizes the need for independent AI evaluation. But the solution on offer is a stopgap, not a paradigm shift. The real opportunity lies in building a decentralized verification layer — one where models are evaluated on-chain, results are publicly auditable, and the evaluator cannot be captured by the evaluated. The question is not whether Vals AI will succeed, but whether the industry will repeat the pattern of centralized trust that crypto was designed to eliminate. Following the code where the humans fear to tread — that is where the next narrative will emerge.