# One Graph, Two Jobs: How Our Infrastructure Graph Doubles as a Tuning Ground for AI SREs

We say modeling infrastructure as a graph is optimal for AI, and almost nobody gets why. It's not that models like graphs. It's that a graph is an interface you can simulate: every production incident Annie investigates can be rebuilt as a graph state, at scale, for nearly free. OpenAI's Sol 6.1 maxed out our old eval set, so we built a harder one from real production failures and tuned Annie's brain on half of it. Sol went from 64% to 92% on the held-out half. All made seamless thanks to our graph.

Author: Ghazi Felhi

Published: 2026-10-23

Source: https://anyshift.io/blog/graph-sim2real-ai-sre

I don't think anyone gets why we say modeling infrastructure on a graph is optimal for AI to browse it.

Fair enough: the usual arguments are weak. "Graphs are an intuitive model for infrastructure" is true, and graph-based root-cause analysis has a long research record ([Groot at eBay](https://doi.org/10.1109/ASE51524.2021.9678708), [KGroot](https://doi.org/10.1016/j.eswa.2024.124679)), but that says nothing about AI. "LLMs reason better over graphs" is not a given either: in our own internal experiments it depends heavily on how the graph is designed, and a badly designed graph can make an agent worse. **Our actual reason is more mundane and more useful: a graph is an interface you can simulate.** Annie, our AI SRE agent, sees most of production *through* the graph: it's her entry point, and it holds the information that matters most in an incident. So any production incident can be rebuilt as a graph state, plus the few other signals (like logs) that surround it, and Annie can be evaluated and improved on simulated incidents that look, from where she sits, exactly like the real thing.

This is how we've always evaluated and improved Annie. We're sharing it now to celebrate a milestone: OpenAI's Sol 6.1 just scored **100%** on our previous eval set (congrats to OpenAI 🎉). That meant building a harder one, and it made a good occasion to show the whole loop at work.

Here's the outline:

1. **The graph**: what it's built from, and how its design evolved.
2. **Simulation**: how we turn real production incidents into graph states Annie can investigate, and how that differs from public SRE benchmarks that spawn real systems.
3. **The experiment**: a harder eval built from real production failures, and Annie's brain tuned on it.
4. **Results**: Sol 6.1 went from **64% to 92%** on incidents it was never tuned on; we also check what carries over to another model.

## The graph: seeded by experts, shaped by usage

The Anyshift graph models production as resources and their relationships, plus a timeline of what changed and when. It's built from the sources an SRE would open during an incident:

- **Infrastructure as code**: Terraform files and state, which resources were declared and how.
- **Cloud APIs**: AWS, GCP and Azure, which resources actually exist right now.
- **Runtime**: Kubernetes workloads, nodes and pods, and their lifecycle events.
- **Change streams**: deploys, config edits, IaC applies and merged PRs, attached to the resources they touched.
- **Telemetry links**: logs, metrics and alerts attached to the components they describe.

(Our CEO Roxane [wrote about why topology matters](/blog/why-ai-sre-needs-topology-not-just-telemetry) for an AI SRE.)

Why so many sources: no single one tells the truth. When we measured it in 2025 on five client environments, **35% to 94% of running cloud resources had no counterpart in Terraform state**. Peer-reviewed work documents the mechanisms behind that drift ([Hassan et al., FSE 2024](https://doi.org/10.1145/3660790) catalogue state-reconciliation defects in IaC tools). And change streams matter most of all: Google's SRE book puts it at [roughly 70% of outages coming from changes to a live system](https://sre.google/sre-book/introduction/).

The schema was jump-started by our subject matter experts: what an experienced SRE checks first, which relationships matter in an incident, which changes tend to be the culprit. Since then it has been iterated on by **inferring what clients need from how Annie actually uses it**: the queries she runs, the hops she wishes she could make, and the places where she gives up or gets corrected.

## Simulating production by writing graph states

![Live-system benchmarks rebuild production; we rebuild the interface](/images/blog/graph-sim2real-two-approaches.png)

Public SRE-agent benchmarks mostly take the honest, expensive route:

- [AIOpsLab](https://arxiv.org/abs/2501.06706) (MLSys 2025), [ITBench](https://arxiv.org/abs/2502.05352) (ICML 2025) and [SREGym](https://arxiv.org/abs/2605.07161) **spawn real systems** (clusters, microservices, databases), inject faults, and let those systems produce the traces the agent investigates through real APIs.
- [OpenRCA](https://iclr.cc/virtual/2025/poster/32093) (ICLR 2025) instead ships **recorded telemetry**: 335 failures and 68 GB of logs, metrics and traces.

Both make sense when you want a test bed generic enough that results transfer to *anyone's* agent and *anyone's* stack. They also show how far from solved the problem is: the best model resolved 11.34% of OpenRCA's cases, and agents resolved 11.4% of ITBench's SRE scenarios.

We're in a different position. **At Anyshift we build and own the very interface between the AI and production.** The graph abstracts away most of the production state, so we don't need to spawn a cluster to make a node go NotReady, or run a database to make a migration fail. We write the graph state, and the logs, that such an incident *would* leave behind, and Annie investigates it through exactly the tools she uses in production.

A simulated incident is:

- **A world**: around 1,500 resources and 4,000 to 11,000 change events in the graph, plus around 100,000 log lines, for a single incident.
- **An alert** written the way it really fired.
- **A ground truth** taken from how the real incident was actually resolved.
- **A strict LLM judge** that only passes an answer naming the real root cause, not a plausible-sounding one. LLM judges reach [over 80% agreement with human raters](https://arxiv.org/abs/2306.05685) (NeurIPS 2023) when given precise criteria ([G-Eval](https://doi.org/10.18653/v1/2023.emnlp-main.153)). They also have [known biases](https://doi.org/10.18653/v1/2024.acl-long.511), which is why our criteria grade one specific failure, not overall quality.

Building a new incident is a script, not an infrastructure project, so scaling and diversifying the simulation is cheap.

This is the classic **sim-to-real** recipe ([survey](https://arxiv.org/abs/2009.13303)). Robotics learned long ago that you can train in a simulator and deploy on hardware if the simulator varies enough and the interface matches; see [domain randomization](https://arxiv.org/abs/1703.06907) and OpenAI's [Rubik's cube hand](https://arxiv.org/abs/1910.07113). Hugging Face and Pollen Robotics' [Microduck](https://pollen-robotics.com/microduck/) is the recent, very cute proof: every behavior is a policy trained in a physics simulator and deployed straight onto the robot ([TechCrunch](https://techcrunch.com/2026/08/27/hugging-face-is-selling-a-cute-399-open-source-duck-robot-microduck/)). **Our graph plays the role of the physics simulator, with one advantage over robotics: the interface Annie trains on and the one she runs on are the same, so the sim-to-real gap shrinks to how realistic the incidents are.**

The other half of the setup is what Pierre wrote about in [our brain / harness split](/blog/annie-v3-model-agnostic-harness). Annie's operational knowledge (her **brain**: method, skills, prompts) lives apart from the execution layer. That makes the brain the thing we optimize, with everything else held fixed.

## Sol 6.1 maxed out our eval. So we built a harder one.

On our previous eval set's held-out split, **OpenAI's Sol 6.1 scored 100%**, and the rest of the latest model series weren't far behind:

| Previous eval set, held-out split (3 runs per incident) | Score |
|---|---:|
| **Sol 6.1 @ low** | **100%** |
| Sol 6 @ low | 98.9% |
| Luna 6 @ high | 96.6% |
| Sonnet 5 @ high | 95.5% |
| Luna 6 @ max | 94.4% |
| Luna 6 @ medium | 93.3% |
| Sonnet 5 @ low | 92.1% |
| Luna 6 @ low | 89.9% |

Back to work for us: static benchmarks [saturate](https://arxiv.org/abs/2104.14337), and this one had. The fix is the same one Dynabench proposed: keep building new cases where today's models fail.

So we built **prod-hard**, deliberately to trip Sol 6.1: incidents modelled on real production failures, made harder until the strongest model we had stopped acing them. We started from the ones where a human engineer had to step in and correct Annie in the incident thread, then widened to hard cases across every domain we see in production. Each one stacks the traps that tripped her:

- a cause that landed days before the alert;
- a decoy change right at the onset;
- a failure that only hits part of the fleet;
- a lot of change noise. Models are [easily distracted by irrelevant context](https://arxiv.org/abs/2302.00093), and real incidents are full of it.

We split it into two halves, **train** and **test**. Near-duplicate incidents are kept on the same side, and the mix of domains and difficulty is matched.

![The tuning loop: run on train, read the failures, edit the brain, repeat; test is only ever scored](/images/blog/graph-sim2real-loop.png)

Then we tuned Annie's brain for Sol 6.1 using **only the train half**:

1. Run the train set and read every failing investigation together with the judge's reasoning.
2. Turn what we learn into generic method changes in the brain. Never scenario facts: a checker rejects any brain edit containing an identifier from any incident.
3. Run small train batches, correct, and repeat.

The failure was almost always the same, and very human. Sol would find the mechanism, and even the misconfigured value, then stop. It never asked *when that value became what it is, and what changed it*. Its change searches were bounded to the incident day, while the real culprit had landed days earlier. That's the classic latent failure that only bites under a later trigger, the pattern Richard Cook describes in [*How Complex Systems Fail*](https://how.complexsystems.fail/): systems "contain changing mixtures of failures latent within them." Five small brain revisions later (+23 lines of method), the train set was saturated.

## Results

![Before and after tuning, on the held-out test half](/images/blog/graph-sim2real-results.png)

| Held-out test half (3 runs per incident) | Before | After tuning |
|---|---:|---:|
| **Sol 6.1 @ low** | 64% | **92%** |
| **Luna 6 @ max**, same Sol-tuned prompts, never tuned on | 20% | 24% |

- **Sol on the held-out half: 64% → 92%.** The tuned brain fixed 15 incidents and broke 1. An exact McNemar test, the standard for paired classifier comparisons ([Dietterich, 1998](https://doi.org/10.1162/089976698300017197)), gives p = 0.0005.
- **No over-correction.** "Look for the older change" could easily make Annie blame a stale change when the deploy at onset really is the cause. On a separate set of incidents, mostly cases where the change at onset really is the cause, the score is unchanged: 100% before, 100% after.
- **Cost.** About 8% more per investigation and 14% more time: a few more history lookups, in exchange for 28 points.
- **Transfer to another model is partial.** OpenAI's Luna 6 at max effort with the Sol-optimized brain, untouched, moves from 20% to 24% of incidents. That's an 11-point gain in individual runs (24% → 35%), but far from Sol's jump. That's expected: the set was built around Sol's failure modes, and Luna fails differently. Per-model tuning is the obvious next run, and the same pipeline does it.

**The process was close to effortless.** Building the set, splitting it, reading the failures and revising the brain is a pipeline, not a project. It will keep running as new production failures come in, so each new class of mistake becomes simulated incidents, then a brain fix, then a held-out score. That's what makes Annie stronger over time.

## So why a graph?

Not because AIs like graphs. Maybe they do, but we don't know. It's because **the graph is the interface, and an interface you own is an interface you can simulate.** Every incident Annie sees in production can be turned into practice, at scale, without standing up a single cluster. And what she learns in the simulator transfers to production for free, because, from where she stands, the simulator and production look the same.