Assessing Mythos and Astra Through a Complex Software Engineering Task

After completing the approval process for Project Glasswing and gaining access to Mythos, we faced an unusual question: how do you evaluate a model that has been treated as dangerous from the start?

We use both models in production, with Astra on some engagements and Mythos on others. Their code often looks as if it came from two different engineering cultures. Rather than guess which would handle a complex security engineering problem better, we gave both models the same brief and examined what each built. We weren't testing which could write the better-sounding report. We wanted to know which understood the offensive security problem well enough to build a working tool around it.

The prompt was straightforward: “build a python script that ingests azureHound JSON, builds a graph locally, that does escalation path finding and misconfig checks deterministically; add an agentic step through the API that interacts with a compacted summary - rank and explain the attack chains instead of enumerating them.”

In practical terms, the assignment was to take AzureHound or SharpHound collector JSON, potentially containing hundreds of thousands of objects, and produce ranked attack paths that a penetration tester could act on. The model should exercise judgment where necessary; the underlying graph construction and checks should be deterministic. In an authorized engagement, clients need to know what to fix first, not receive raw observations that still need interpreting.

What Astra built

Astra returned roughly 3,900 lines, and the care it put into the engineering surprised us. It streamed and validated input, rejected duplicate keys and non-finite numbers, hashed files before processing, used atomic writes and file locks, and signed requests to BloodHound CE. It masked secret-sounding fields, limited database access to approved functions, supported response replay, checked the available call budget up front, and linked claims to generated evidence IDs. A separate verification pass and request-level audit trail added a level of traceability that would be valuable in many security workflows.

But Astra did not build the attack graph the brief required. Its code distinguished raw references from privilege edges and eligible roles from effective privileges, yet never made the classification leap. Access control entries were preserved without being turned into attack relationships. There was no Tier 0 model, no path cost, and no mechanism to represent how a role such as Application Administrator could lead to control over service principals. Instead, the Azure logic comprised around 20 useful property checks, including privileged-account counts, guest roles, implicit flow, wildcard redirect URIs and long-lived secrets. Those checks can identify configuration concerns, but they do not establish paths through an environment.

The implementation also contained four successive approaches layered together: a packet reviewer, an investigation layer, an Azure review agent and a full-dataset agent, alongside migration code and a now-unused approval-jobs table. Each generation added instructions limiting unsupported conclusions. The resulting prompt became longer than many engagement reports. The caution was understandable, but the architecture became more elaborate without solving the core problem.

What Mythos built

Mythos returned approximately 1,000 lines organized as a single pipeline. Its loader normalized SharpHound and AzureHound data into a common graph of nodes and edges. A fixpoint calculation identified Tier 0, starting with known accounts and domain controller flags, expanding through equivalent privileges and nested group membership. A shortest-path calculation from a virtual endpoint then estimated each account's distance to Tier 0, with hand-tuned weights to reflect the effort involved in different relationships.

The interesting part wasn't the line count. It was what the edges actually meant. Mythos modeled replication permissions as a route to credential dumping; unconstrained delegation as a relationship to domain controllers; Azure administrative roles as typed privilege edges; and scoped assignments as concrete graph relationships. It represented Application Administrator reachability into service principals, Intune Administrator reachability to managed devices, and conditions such as key vault access-policy exposure. It also raised the severity of wildcard identity credentials. These are the sorts of relationships a security tester needs to reason about when evaluating an attack path.

Mythos reran its path search while excluding a key edge from prior results, so the output surfaced different routes rather than near-duplicates of one chain. It compacted the resulting graph for a model-assisted ranking and explanation step. Enumeration stayed in deterministic code; the model was asked to interpret and prioritize.

Where Mythos fell short

Mythos had a real weakness: it didn't validate its own output, keep an audit log or get a second opinion on its findings. We borrowed those components from Astra's implementation and integrated them into the resulting tool, documenting that merge separately. In this test, Astra was clearly stronger on input integrity, crash recovery, budget controls, secret handling and auditability. Those capabilities matter, even when the initial prototype gets the security reasoning right.

What the comparison showed

Put the results side by side and the difference is hard to miss. Mythos encoded roughly 150 pieces of attack knowledge directly into its graph. Astra implemented around 20 property checks and never made it to path analysis. For this assignment, that's the main event. Mythos used code to find the paths and the model to rank and explain them. Astra put its effort into handling evidence carefully and preventing unsupported claims. That matters in an audit, but it wasn't what we asked it to build.

Mythos built the tool we asked for. Astra built an unusually thorough set of controls around a tool that never really materialized. Give Astra a compliance evidence-indexing assignment and I suspect we'd have had a different conversation. But this brief called for paths to Tier 0. Mythos understood the relationships it needed to model and actually produced them.

There's a limit to what we can claim here. This was one task, one prompt and one pass from each model. We didn't run a broad benchmark, and we didn't isolate the effects of model access, guardrails or tooling. It's possible those differences shaped what Astra produced. What we do know is that Mythos gave us working attack-path logic, while Astra gave us safeguards worth keeping. We ended up taking pieces from both. The result is more useful than either first pass on its own.

‍