Evidence-grounded incident investigation for distributed systems.

IncidentProof turns fragmented logs, metrics, traces, and deployment events into traceable evidence, then constrains incident hypotheses to that evidence so unsupported conclusions can be rejected and uncertainty can remain explicit.

Project Overview

Production incidents rarely fail in one obvious place.

A latency spike may appear in one service while the actual problem originates several dependencies away. Deployments, database changes, retries, partial traces, noisy logs, and missing telemetry can all happen around the same time.

This makes incident investigation difficult even for experienced engineers.

It becomes even harder when AI is introduced. A model can produce a convincing root-cause explanation even when the available telemetry does not actually support it.

IncidentProof explores a different approach: evidence first, reasoning second.

The system normalizes heterogeneous telemetry into stable evidence records with provenance, reconstructs incident context deterministically, retrieves relevant evidence, and represents candidate explanations through structured hypothesis contracts.

A deterministic critic validates whether those hypotheses are actually grounded in the available evidence.

When the evidence is insufficient, ABSTAIN is a valid result.

IncidentProof deliberately does not try to manufacture certainty where the telemetry does not justify it.

The Problem

Incident response is fundamentally an evidence problem.

During an outage, engineers may need to reason across:

application logs
metrics
distributed traces
deployment events
configuration changes
database or infrastructure events
incomplete or delayed telemetry

These signals rarely arrive as one coherent story.

A deployment occurring immediately before an outage may be relevant, but temporal proximity alone does not prove causality. A slow database span may be important, but it may only be a downstream symptom. Several changes may happen within the same investigation window.

Human investigators can anchor on the first plausible explanation.

AI systems can amplify that failure mode by turning correlations into fluent causal narratives.

IncidentProof therefore separates three concepts:

Observation
Something directly present in telemetry.

Deterministic inference
A relationship mechanically derived from known evidence.

Hypothesis
A possible explanation that may still be wrong.

That separation prevents a hypothesis from quietly becoming a fact simply because it sounds convincing.

Core Design Principle

Telemetry is evidence. Deterministic relationships are inferences. Possible explanations are hypotheses. They are not interchangeable.

IncidentProof keeps those layers separate throughout the investigation pipeline.

The goal is not to automatically announce a root cause.

The goal is to make every explanation answerable to the available evidence.

How IncidentProof Works

1. Telemetry becomes normalized evidence

IncidentProof ingests synthetic:

Logs
Metrics
Distributed traces
Deployment / change events

Each physical record is normalized into a common Evidence representation containing information such as:

evidence_id
timestamp
source_type
source_name
service
environment
correlation_id
trace_id
attributes
provenance

Evidence IDs are stable, while provenance preserves where each observation physically came from.

Normalization does not convert telemetry into conclusions.

2. Incident context is reconstructed deterministically

The system uses timestamps and explicit identifiers such as correlation IDs and trace IDs to reconstruct investigation context.

This stage organizes what is known without attempting to infer a root cause.

Normalized Evidence
        ↓
Timeline reconstruction
        ↓
Explicit relationships
        ↓
Incident context

3. Relevant evidence is retrieved

Investigation evidence can be selected using deterministic criteria such as:

time window
service
source type
correlation ID
trace ID

Retrieval selects evidence.

It does not rewrite the evidence or convert it into a narrative.

4. Hypotheses must cite evidence

Candidate explanations are represented as structured hypotheses rather than unconstrained prose.

A hypothesis can explicitly contain:

Supporting evidence
Contradicting evidence
Missing evidence
Open questions

This allows the system to ask a much more useful question than:

“Does this explanation sound plausible?”

Instead:

“What evidence does this explanation depend on?”

5. A deterministic critic validates grounding

The critic verifies that evidence references actually exist and belong to the active incident.

It can reject malformed or unsupported grounding without pretending to solve general causal inference.

This distinction is central:

VALID EVIDENCE REFERENCE
!=
PROOF OF CAUSALITY

The critic proves contract and grounding integrity, not universal truth.

6. The system can abstain

Real incidents are frequently under-observed.

Instead of forcing every investigation into a confident answer, IncidentProof treats:

ABSTAIN

as a first-class outcome.

If several explanations remain plausible or the necessary evidence is missing, the system can explicitly state that the available telemetry does not justify a conclusion.

The included ambiguous incident fixture was designed specifically to test this behavior.

Architecture

For the website, I would show this compact version:

Logs      Metrics      Traces      Changes
  \          |           |            /
   └─────────┴───────────┴───────────┘
                     ↓
          Deterministic Parsers
                     ↓
           Normalized Evidence
       stable IDs + provenance
                     ↓
        Incident Reconstruction
                     ↓
          Evidence Retrieval
                     ↓
       Structured Hypotheses
       support / contradiction
        missing evidence
                     ↓
         Deterministic Critic
              ↙          ↘
          GROUNDED      REJECTED
              \          /
               \        /
                 ABSTAIN
                    ↓
             Human Engineer

Human judgment deliberately remains outside the automation boundary.

Two Incident Scenarios

Checkout Latency Incident

The first synthetic fixture represents an investigation across logs, metrics, traces, and system changes.

It exercises the full evidence pipeline:

Raw telemetry
→ normalization
→ reconstruction
→ retrieval
→ hypothesis
→ grounding validation

The purpose is to demonstrate how heterogeneous operational signals can be transformed into one traceable investigation context.

Ambiguous Catalog Latency Incident

The second fixture intentionally avoids an obvious answer.

Catalog latency increases after both:

a deployment
a database failover/change

Multiple explanations therefore remain plausible.

This scenario tests whether IncidentProof can preserve competing hypotheses, contradictory evidence, and missing information rather than simply selecting whichever event happened closest to the failure.

In some cases, the correct output is:

ABSTAIN: the evidence does not currently distinguish the competing explanations.

That behavior is intentional.

Evaluation

IncidentProof includes a fixture-based evaluation harness focused on separate engineering properties rather than a single artificial “AI accuracy” score.

Evidence-reference validity

Do hypothesis references point to real evidence belonging to the incident?

valid references
───────────────
total references

Unsupported-claim rejection

Can the evaluation path reject synthetic claims known not to be supported by the fixture?

Abstention correctness

When an incident is deliberately ambiguous or incomplete, does the system avoid an unsupported confident conclusion?

These dimensions remain separate because combining grounding, support, and abstention into one number would hide different classes of failure.

Safety and Authority Boundary

IncidentProof v0.1 is deliberately read-only.

It can:

Read telemetry
Normalize evidence
Correlate evidence
Retrieve incident context
Represent hypotheses
Validate evidence references
Reject unsupported grounding
Abstain
Evaluate synthetic incidents

It cannot:

Restart services
Execute production commands
Roll back deployments
Modify infrastructure
Change databases
Call cloud control planes
Declare an incident resolved

This is an architectural boundary, not just a UI restriction.

Any future AI reasoning layer should inherit the same default: evidence-constrained output with human-controlled operational actions.

Important Engineering Decisions

A compact website section could use these four cards:

Evidence identity before AI reasoning

Every physical telemetry record receives stable identity and provenance before it can influence a hypothesis.

Deterministic retrieval first

v0.1 intentionally uses deterministic evidence retrieval rather than embeddings or semantic search, making the investigation substrate easier to test and reason about.

Grounding is not truth

The critic validates whether a hypothesis references legitimate evidence. It deliberately does not claim that valid citations prove causality.

Abstention over manufactured certainty

When telemetry cannot distinguish between plausible explanations, the system preserves uncertainty rather than fabricating a winner.

Technology & Engineering

Area	Approach
Core	Python
Inputs	Synthetic logs, metrics, traces and change events
Evidence	Normalized records with stable IDs and provenance
Correlation	Timestamp, service, correlation ID and trace ID
Retrieval	Deterministic evidence filtering
Reasoning Contract	Structured hypotheses with support, contradiction and missing evidence
Validation	Deterministic evidence critic
Uncertainty	Explicit `ABSTAIN` outcome
Evaluation	Fixture-based grounding, unsupported-claim and abstention evaluation
Runtime	Fully offline v0.1
Operational Authority	Read-only
AI Dependency	None required in v0.1

The project intentionally establishes the evidence and control substrate before adding probabilistic reasoning.

What IncidentProof Does Not Claim

IncidentProof does not claim to:

prove root cause
solve causal inference generally
replace observability platforms
automatically remediate incidents
guarantee semantic support simply because a citation exists
achieve production incident accuracy
demonstrate production-scale telemetry ingestion
require an LLM for its core value

v0.1 is deliberately focused on building the trustworthy substrate that a future AI investigator would need to obey.

Project Artifacts

Artifact	Description	Link
Source Repository	Complete IncidentProof v0.1 implementation, fixtures, tests, evaluation assets, and documentation.	View repository
Architecture Diagram	Canonical evidence-first architecture showing telemetry ingestion, normalization, reconstruction, retrieval, hypothesis contracts, critic, abstention, and the human boundary.	View architecture diagram
System Design	Detailed architecture notes covering evidence identity, deterministic reconstruction, retrieval boundaries, hypothesis contracts, critic behavior, ambiguity handling, and human/AI responsibilities.	Read system design
Incident Model	Defines the separation between observation, deterministic inference, hypothesis, and abstention.	Read incident model
Evaluation Design	Explains evidence-reference validity, unsupported-claim rejection, abstention evaluation, fixture isolation, and why there is no composite score.	Read evaluation design
Failure Modes	Covers missing telemetry, clock skew, noisy signals, partial traces, misleading temporal ordering, and other investigation risks.	Review failure modes
Security & Authority Boundary	Documents the read-only operating model and what the deterministic layer, future AI layer, and human are each allowed to control.	Read security model

The hardest problem in AI-assisted incident investigation is not generating another explanation. It is knowing which parts of that explanation the evidence actually supports.

IncidentProof builds that boundary first.