Mythos appears to be as powerful as claimed at detecting software vulnerabilities; but its capabilities in other areas is more nuanced.
Anthropic’s Mythos AI model has been making waves since its announcement in early April, primarily because of its reputed ability to unearth considerably more vulnerabilities than any other AI model. XBOW, an autonomous offensive security firm, has aimed its own AI testing armory against Mythos Preview to check the validity of this and other Mythos capabilities.
Anthropic’s primary claim is confirmed. “Mythos Preview presents a significant step up over all existing models, regardless of provider,” reports XBOW.
As Gary McGraw commented 20 years ago, operational defects occur in the interaction between source code bugs and architectural design flaws. “You can’t find design defects by staring at code – a higher-level understanding is required,” he said. XBOW tested Mythos against both access to the code alone, and the code operating in a live situation. It found that the model excels at finding problems when testing ‘live + source’, but not so well against the source code alone.
This doesn’t detract against the power of Mythos probing source code, but XBOW points out that while any AI model can find something interesting, the ‘something’ won’t be the same as ‘everything’.
Other XBOW tests explored Mythos capability in terms of judgment, reverse engineering, assessment of native apps, and visual acuity.
In judgment, it rejected false positives better than its predecessors, “but sometimes lost true positives when evidence did not formally satisfy its criteria.” Mythos requires precise prompts for best results.
The model exhibits substantial strength in both native code vulnerability discovery and reverse engineering.
In the reverse engineering tests, XBOW concluded Mythos is “capable of triaging both its own results and competitor-model findings,” and the model could reason through unusual firmware and embedded systems contexts.
XBOW’s visual acuity tests examine the model’s ability to interact with live websites through a browser interface; that is, the ability to identify the right UI element and click in the right place. “It was not perfectly pixel-accurate when asked for exact coordinates, but it was practically effective at selecting the right browser actions,” writes XBOW.
There is, however, one statistic that can easily be overlooked by users overawed by the power of Mythos. “Mythos Preview is not just any new model: it’s a true titan. But titans are big, and big means expensive.”
At the time of writing, specific costs are not available, although Anthropic has said it will be 5x as expensive as an Opus model. This made XBOW question whether it would be possible to give a cheaper model more time and get more accuracy at less cost.
The conclusion was yes. “If we normalize by estimated running cost, the picture is rather clear: Mythos Preview isn’t terribly inefficient, at least if you desire high accuracy, but it’s not best-in-class on our benchmarks either.” For finding web vulnerabilities with a fixed token budget, Mythos outperforms Opus 4.6 but is outperformed by GPT5.5.
None of these findings detract from the original fundamental claim. Mythos is better at finding vulnerabilities in code than other models. Overall, however, the primary takeaways from XBOW’s testing are:
- Mythos is extremely powerful for source code audits.
- It’s good, but less powerful, at validating exploits.
- Its judgment is mixed. It can be too literal and conservative and also tends to overstate the practical relevance of its findings.
- It is strong in native-code vulnerability discovery and reverse engineering.
“Mythos Preview is strong at finding candidate vulnerabilities, especially from source code, and shows impressive ability across web, native-code, and reverse-engineering tasks,” concludes XBOW.
Related: Sweet Security Launches Agentic AI Red Teaming to Counter ‘Mythos Moment’
Related: Claude Mythos Finds Only One Curl Vulnerability; Experts Divided on What It Really Means
Related: Claude Mythos Finds 271 Firefox Vulnerabilities
Related: ‘Mythos-Ready’ Security: CSA Urges CISOs to Prepare for Accelerated AI Threats
