Beyond Log Search: What We Learned Building a RAG-Based Incident Diagnosis System

If you’ve ever been paged at 2 a.m. for a service degradation you don’t immediately recognize, you already know the real bottleneck in incident response. It’s rarely the fix. It’s finding out whether anyone has seen this before.

Industry surveys indicate that 35–55% of incident-resolution time is spent on knowledge retrieval rather than remediation, involving the review of runbooks, old tickets, postmortems and internal wiki pages that often provide only partial answers. That’s not a tooling gap so much as a search problem wearing an incident-management costume. We spent the last several months building a system to close it, and the results were strong enough that we wrote them up as a peer-reviewed paper, accepted at IEEE GAISS 2026. This is the practitioner version of that work: What we built, what surprised us and what we’d tell another team considering the same thing.

Why the Usual AIOps Toolbox Doesn’t Cover This

Most AIOps tooling is good at a narrower job than people assume. Anomaly detection, log clustering, alert correlation — all of it operates on numerical or structured telemetry. Metrics, traces, log volumes: That’s useful, but it’s not where the knowledge actually lives. The thing that tells you “We saw this exact failure mode 18 months ago and the fix was a connection pool setting, not a redeploy” is sitting in a postmortem doc or a closed ticket, in plain English, not in a Prometheus series.

Large language models (LLMs) are obviously good at plain English. The problem is that a standalone LLM asked to diagnose a production incident will happily generate a plausible-sounding root cause that has nothing to do with your actual infrastructure, because it has no access to what actually happened last time. Retrieval-augmented generation exists precisely to fix this class of problem that is to ground the model’s output in real, retrieved organizational knowledge instead of letting it improvise. That’s the bet we made, and it’s what the rest of this is about.

What We Actually Built

The system is organized into five layers, and honestly, the boring parts mattered more than we expected.

• Data Sources: Incident tickets, runbooks, postmortems and live alerts, pulled from ServiceNow, PagerDuty, Confluence, GitHub and Prometheus/Dynatrace. Nothing exotic — the point was to work with what teams already have, not force a migration to some purpose-built knowledge tool nobody would adopt.
• Ingestion and Indexing: This is where most of the real engineering effort went, and where we made our first wrong assumption. We started out chunking every document the same way we fixed windows of a few hundred tokens with some overlap. It worked fine for long-form runbooks and postmortems. It was noticeably worse for incident tickets, which have their own internal structure (summary, timeline, resolution) that a fixed window chops through arbitrarily. We ended up with three separate chunking strategies depending on document type: Fixed-window for long-form docs, boundary-aware chunking that respects section structure for tickets and sentence-level chunking for short alert annotations, where overlap just adds noise. Each chunk carries provenance metadata — source document, section, timestamp and service labels — which turned out to matter more than the chunking strategy itself, for reasons I’ll get to.
• Retrieval: Chunks get embedded and stored in a FAISS index for approximate nearest-neighbor search. On an incident, we build a query from the alert context or the incident description, embed it the same way and pull back candidates. This is the part that’s easy to skip if you’re moving fast: We re-rank the candidates with a cross-encoder before anything reaches the language model. Skipping straight from vector search to generation is tempting because it’s simpler, but the re-ranking step turned out to be one of the two biggest contributors to accuracy in the whole system.
• LLM Reasoning: The retrieved context, along with incident metadata, gets assembled into a structured prompt that asks the model to reason step by step before producing a structured output: Ranked root causes, a confidence score, ordered remediation steps, an escalation flag and importantly, links back to the source documents it used. We validated the reasoning layer against GPT-4-Turbo, Claude 3 Opus and Llama 3.1 70B, and it’s model-agnostic by design. The specific model matters less than you’d think, which is a theme that shows up again below.
• Feedback Loop: Resolved incidents get annotated with confirmed root causes and fed back into the knowledge base. This is the part that keeps the system from going stale as infrastructure and failure modes evolve, and it’s also the part that’s easiest to underinvest in once the initial build is ‘done’.

The Result That Surprised Us

We evaluated the system against 2,400 annotated incident scenarios, a mix of real historical incidents from a midsized financial services organization (anonymized for the study), the public DeathStarBench microservice benchmark with injected faults and synthetic edge cases covering things such as cascading failures. Against a standalone LLM, a BM25 keyword-retrieval baseline, a supervised ML classifier and a rule-based expert system, the RAG framework came out ahead on every metric: 87.3% root-cause identification accuracy, against 71.8% for the next-best approach (BM25 plus the same LLM). Mean diagnosis time dropped 59% overall, and the improvement was largest exactly where it matters most: P1 critical incidents went from an average of 48.2 minutes of manual search down to 19.8 minutes.

None of that surprised us much, honestly. We built the thing expecting retrieval augmentation to help, and it did.

What surprised us was the ablation study.

We tested what happens when you remove individual components one at a time. Pulling out the re-ranking step cost 7.7 percentage points of accuracy. Pulling out the semantic chunking strategy cost 10.9 points — the single biggest hit of anything we tested, bigger than swapping the underlying LLM would have been. Metadata filtering and the feedback loop each cost a few points on their own. In other words, the unglamorous plumbing — how you cut up your documents and how you re-rank what comes back — mattered more to the end result than which frontier model was doing the reasoning. If you’re planning a project like this and budgeting your engineering time toward “Which LLM should we use,” that’s probably the wrong place to spend the argument.

The other number that stuck with us wasn’t an accuracy metric at all. We ran a satisfaction survey with the SREs actually using the system, and it came back at 4.6 out of 5, well above every baseline, including the plain LLM approach. The qualitative feedback pointed at one specific thing: Engineers trusted the output more when they could click through to the actual source document the model was drawing from. That’s worth sitting with if you’re designing anything AI-assisted for a high-stakes operational context — the explainability layer isn’t a nice-to-have bolted on for compliance; it’s arguably doing more work than the model’s raw accuracy in getting people to actually use the thing.

Where It’s Weaker Than the Numbers Suggest

Three caveats worth being upfront about:

• The evaluation data leans toward financial services and web application architectures; we haven’t validated this against telecom, embedded systems or mainframe environments, and I wouldn’t assume it generalizes cleanly there.
• Ground-truth root cause labels were assigned by human SREs, which means some of the ‘wrong’ answers in complex multi-root-cause incidents might be more ambiguous than the accuracy number implies.
• We didn’t independently measure the hallucination rate of the reasoning layer. The retrieval grounding clearly helps, but we can’t currently tell you that it eliminates the problem — only that provenance links let a human catch it when it happens.

There’s also an operational cost that doesn’t show up in the accuracy table: A stale knowledge base degrades quietly. New services get deployed, failure modes shift and if nobody’s auditing what’s in the index, you eventually end up confidently retrieving outdated advice. The feedback loop helps, but it’s not a substitute for someone periodically owning the question of whether the knowledge base still reflects reality.

If You’re Evaluating Something Like This

A few practical takeaways, if you’re a team sizing up whether to build in this direction rather than just watching from the sidelines:

• Don’t treat chunking strategy as an afterthought; it was the single largest lever in our results, and it’s cheap to get wrong quietly.
• Budget real engineering time for re-ranking rather than shipping naive top-k retrieval straight into your prompt; it’s a small addition with an outsized effect.
• Design for provenance from Day 1, not as a later polish pass; it’s what actually earns engineer trust, more than raw accuracy does.
• Don’t assume the choice of underlying LLM is where your competitive advantage lives; in our results, it very much wasn’t.

The latency numbers are also worth knowing if interactive incident response is the goal: Our end-to-end p95 came in at 3.5 seconds, with the LLM generation step itself as the dominant cost and vector search staying fast (142ms p95) even at more than 10,000 indexed documents. That’s compatible with a human sitting in the loop during an active incident, which was a hard requirement for us — nobody wants a diagnostic tool that’s slower than just paging a teammate.

We’re continuing to work on a few open problems from here: Pulling in real-time observability signals such as metric time series and trace graphs as additional retrieval modalities, not just text, and tightening hallucination mitigation through claim-level verification against the retrieved sources, rather than relying on provenance links as the only backstop. If you’re working on something adjacent, I’d be glad to compare notes.

A reference implementation of the framework described here — the ingestion connectors, chunking and indexing pipeline, retrieval and re-ranking logic and the evaluation harness used to produce the results above — is available on GitHub under the MIT license.