What 6,336 LLM-Routed Graph Queries Taught Us About Accuracy, Caching, and Cost

We were preparing to switch the model behind the Anyshift Graph API /ask endpoint. Although the code change was small, proving that the new configuration would preserve answer quality under production conditions required a larger piece of engineering.

We built an evaluation suite from 352 approved questions across two live infrastructure graphs, ran every question three times against six models, and collected 6,336 API executions. We then isolated prompt-caching behavior and instrumented a production request from the client edge down to graph execution.

The work gave us enough evidence to switch models and reinforced a design principle for AI infrastructure: use the model for the ambiguous part, then make the rest of the path deterministic and observable.

The Model Has One Bounded Job

/ask lets a user ask an infrastructure question in natural language. A request such as “Which services depend on checkout?” must become a precise graph operation before the Graph API can return evidence.

We keep the LLM's role narrow. It selects a structured route from a constrained schema. Deterministic code then normalizes that route, executes the graph query, and formats the response.

A model can recognize that “ECS service” refers to a resource type without reproducing the exact internal enum. It can understand an IAM-role investigation while omitting a default time window. The normalizer resolves aliases and applies safe defaults before execution.

The model supplies semantic intent to a controlled query catalog and cannot write an arbitrary graph query.

A Benchmark Built From Real Graphs

Synthetic questions would have missed the naming, topology, and phrasing found in an operating infrastructure graph. We generated candidates from deterministic query-catalog recipes against two live graphs, reviewed the expected evidence, and approved 352 questions across 88 test cases.

The suite includes dependencies, blast radius, datastores, cloud resources, identities, events, and operational changes. Each test has a deterministic live-graph reference. Scoring does not rely on another LLM deciding whether an answer sounds correct.

The benchmark measures natural-language routing for this Graph API. It does not measure general reasoning, coding, or agent performance. We observed latency from one laptop through a tunnel to live graph environments, making it useful for matched comparisons rather than provider-level latency claims.

Six Models, Nearly Identical Final Answers

Three models produced a correct final answer on every request: Haiku 4.5, Luna, and Terra. The remaining three landed between 99.15% and 99.72%.

All six models completed every HTTP request successfully. On this constrained workload, the larger models added cost and latency without improving the final result. This finding is specific to the routing task. Synthesis and open-ended investigation can reward the capabilities of a larger model.

Why Route Accuracy and Answer Accuracy Differ

We score the path at three layers:

• Raw model route: an exact comparison between the model's structured output and the canonical route.
• Normalized route: the same comparison after deterministic aliases and defaults are applied.
• Final answer: the evidence returned by the executed graph query compared with its deterministic reference.

Raw-route accuracy ranged from 42.71% to 52.94%. Normalization lifted every model to roughly 95%. Final-answer accuracy reached 99.15% to 100%.

One retained Luna request against our own production graph makes the distinction concrete:

The model understood the request but used a natural-language label instead of the exact internal enum, so the raw-route comparison failed. The normalizer canonicalized the label, graph resolution selected the AWS-specific resource type, and the answer matched the deterministic reference. The normalized route and final answer both passed.

The raw metric remains valuable. It tells us how much the product depends on normalization. The final-answer metric tells us whether the user received the right evidence. Keeping both prevents a strong normalizer from hiding model drift and prevents strict syntax differences from being mistaken for product failures.

Haiku and Luna Were Both Strong Choices

Haiku 4.5 delivered the best broad-run latency: a 1.30-second median and a 2.52-second p95. Its final answers were perfect. It remains an excellent model for a fast, bounded routing task.

Luna also delivered perfect final answers and cost $0.255 per 1,000 requests in the same run, about 79% less than Haiku's $1.210. Its 1.47-second median was close, while its 3.69-second p95 was slower in that run.

A later paired study, with both models using explicit caching, narrowed the latency difference further. Luna's median was 949 ms against Haiku's 981 ms; Haiku kept the lower p95 at 2,041 ms against 2,380 ms. Their normalized routes had no disagreements across 1,056 matched requests. Luna cost $0.217 per 1,000 requests, about 82% less than Haiku.

The models were competitive for different reasons. Haiku offered tight and dependable latency. Luna matched the answer quality at a substantially lower observed cost. For a high-volume endpoint with deterministic safeguards, Luna was the better production fit.

Explicit Caching Reduced Cost, Not Latency

Both providers support automatic caching for eligible prompt prefixes. The /ask router has a stable system prompt and schema, making that shared prefix an obvious optimization target. Their APIs also support explicit cache boundaries: OpenAI describes the behavior in its current model guidance, and Anthropic documents automatic and explicit prompt caching.

We compared Luna with automatic caching against Luna with an explicit cache boundary. Each configuration handled the same 352 questions three times.

Explicit caching reduced observed cost by 14.2% and cache writes by 86.4%, with no accuracy change or meaningful latency movement.

This is why we record uncached input, cache reads, cache writes, and output separately. A single “input tokens” counter would show nearly the same prompt volume while concealing the change in billed work.

We also tested Luna's fast mode. It improved median latency by 7.3% and p95 by 11.4%, with no route disagreements, while increasing cost by 109%. Standard mode was the better operating point for this endpoint.

The pricing calculations use the published standard rates available on August 10, 2026. Luna was priced at $0.20 per million input tokens, $0.02 per million cached input tokens, $0.25 per million cache writes, and $1.20 per million output tokens. Haiku 4.5 was priced at $1 per million input tokens, $0.10 per million cache reads, $1.25 per million five-minute cache writes, and $5 per million output tokens. See the current OpenAI model pricing and Anthropic pricing for updates.

Production Latency Needed One More Layer of Evidence

The local benchmark suggested that model choice influenced latency. A production run from the same laptop reported a 3.46-second end-to-end median, which was much higher than the model-router timing. We instrumented each boundary before drawing a conclusion.

The client and edge residual correlated most strongly with end-to-end latency in this run (r=0.933). Router latency had a weaker correlation (r=0.303); authentication and graph execution were small. These measurements diagnose one request path from one client location. They do not isolate Wi-Fi, Internet routing, TLS, CDN, or load-balancer time individually.

Model timing alone could not explain the latency a user felt. We now retain timings for the client, request handler, authentication, router, and graph so later regressions can be assigned to the right boundary.

The same production run completed 120 of 120 requests successfully, with 120 correct normalized routes and reference answers, no retries, and a 99.80% cache-read share. Observed Luna cost was $0.216 per 1,000 requests, an 88.8% reduction from the same traffic priced as fully uncached input.

What We Shipped

We shipped:

• A provider-agnostic router that can run the complete endpoint on an Anthropic or OpenAI model.
• A 352-question, 88-case suite grounded in deterministic evidence from live infrastructure graphs.
• Separate measurements for raw model behavior, normalized routes, and final answers.
• Explicit prompt caching with token-class and cost accounting.
• Boundary-level latency instrumentation in production.
• A production configuration using GPT-5.6 Luna in standard mode with explicit caching.

The benchmark can be reproduced as the graph evolves, while the architecture leaves room for finer-grained model selection later. For this release, one model handles the complete endpoint, which keeps operating behavior easy to reason about.

Methodology and Limitations

The two source graphs were live, so their contents can change. We protected the comparison by generating a deterministic reference for each question at run time, running models against the same snapshots as closely as practical, and repeating every case three times.

Cost is calculated from measured token classes and the published rates captured for the experiment. It excludes provider discounts, taxes, infrastructure overhead, and the cost of graph execution. Latency includes our real client path and should be read as a matched system measurement.

The public sanitized aggregate dataset includes the model totals, cache study, production timing summary, methodology, and claim boundaries used in this article. It excludes customer names, graph identifiers, questions, and resource values.

The Product Lesson

The most useful model was the one that performed its bounded role reliably inside a system with deterministic controls. Luna gave us the final-answer quality we needed at the lowest observed cost. Haiku showed that a small Anthropic model can deliver the same quality with excellent latency. The normalizer, query catalog, cache policy, and instrumentation made either model operationally credible.

That combination now powers natural-language questions in the Anyshift Graph API: model flexibility at the routing boundary, deterministic infrastructure evidence at the answer boundary.