Don’t trust your agents. Prove them.

Continuous assurance for AI SOC agents. Verify that every closure, escalation, and suppression was earned, using your own evidence.

▼ SCROLL TO TRACE

// Live agent trace

Every decision. Traced. Checked against evidence.

Your AI SOC analyst or triage agent makes hundreds of decisions per run. DetectTrace connects each trace to the security evidence the agent could see, then checks whether its claims were supported, whether required evidence was skipped, and whether its confidence was earned. A correct verdict reached for the wrong reason still gets flagged.

agent://triage-07Live
ILLUSTRATIVE CASE REVIEWPENDING
Disposition vs. adjudicated outcomeMATCH
Claims supported by evidenceFLAG: 1 unsupported
Required evidence coverageFLAG: sign-in history skipped
Confidence vs. evidenceFLAG: overconfident
Sample output for illustration. Not customer data.

// How it works

Trace. Attest. Prove.

Three steps from black-box agent to defensible decisions.

Trace

Connect your agent’s traces to the security evidence behind each case, so every decision is tied to what the agent could actually see.

Attest

Score reasoning, evidence coverage, and confidence on every case, and replay a regression suite whenever a model, prompt, or tool changes.

Prove

Get findings, trends, and a monthly assurance memo your leadership can act on, with each result linked to its evidence.

// What we catch

Right answer. Wrong reason.

Agreement with analysts is a lagging indicator. These are the failures that verdict-only testing misses.

Unsupported reasoning

The verdict was correct, but a key claim has no backing in your telemetry. It is a latent failure waiting for a different alert.

Skipped evidence

The case was closed without the sign-in history, process tree, or consent record the investigation required.

Unearned confidence

A high-confidence benign closure that turned out to be wrong, flagged before it becomes an unsafe default.

Silent regressions

A prompt or model change that quietly flips past decisions. Every analyst override becomes a replayable test.

Vendor-neutral by design. DetectTrace works alongside your security stack and your agent vendor, and does not replace either.

// Proof

Evidence, not adjectives.

See exactly what you receive. Every item below appears only when it is real.

What you receive

Monthly assurance memo · Triage agent · Impossible travelSample · illustrative data

Headline metrics

Disposition agreement87%vs. adjudicated outcome▲ 3 pts vs. last month82–91% · n=200Community
Unsafe closure rate2.0%closed benign, risk remained▼ 1.0 pt vs. last month0.8–5.0% · 4 of 200Community: count
Critical evidence omission9%share of cases▼ 4 pts vs. last month5–14% · n=200Community: checklist
High-confidence errors1.5%confident and wrong▬ flat vs. last month0.5–4.3% · 3 of 200Design-partner service
Escalation-policy conformance94%per customer policy▲ 2 pts vs. last month90–97% · n=200Design-partner service
Regression pass rate96%after last change▼ 2 pts vs. last month87–99% · 48 of 50Design-partner service

Dimension scores (0–4)

Evidence completeness3.0
Grounding2.8
Investigation depth2.5
Correlation & scope3.2
Conclusion correctness3.4
Action appropriateness3.6
Flagged case · Closed benign. Claim “usual location” has no supporting sign-in evidence. Added as regression test.
Cases correlated: 94%Marked “insufficient data”: 12Provisional scores: 6Blinded analyst agreement: 88%

All values are invented for illustration. They are not customer results or benchmarks. Tags show where each metric comes from: the open-source Community Edition, or the design-partner service. Dimension scores for grounding, correlation and conclusions are design-partner features.

Case studies

Security & assurance commitments

// Community Edition

Open source. Runs on your machine.

The Community Edition is public on GitHub under the MIT license and still in development. It reads your agent’s OpenTelemetry or Langfuse traces and your analysts’ verdicts, then writes a dashboard that opens offline. DetectTrace sends no telemetry and makes no update checks.

Community Edition

Free · self-hosted · MIT

  • Agreement with analysts, corrected for chance (Cohen’s kappa)
  • Dangerous false closes, with case IDs
  • Evidence completeness against your own playbook checklist
  • Skipped steps by prompt or model version, with weekly trend
  • 95% confidence intervals and sample sizes on every value
  • Run as files, a local browser app, or a team service

Design-partner service

Continuous assurance on your live cases

  • Claim support checked against tool results
  • Calibrated judging and confidence-error tracking
  • Regression replay with go/no-go gates and change attribution
  • Prompt-injection and adversarial testing
  • Live SIEM and case correlation, blinded analyst review
  • Monthly assurance memo and hands-on ground-truth setup

Requires traces from an agent you can instrument or export, plus analyst verdicts for the same cases. It cannot assess a closed third-party agent whose traces you cannot get. The Community Edition shows what changed, not why, and it does not fail builds.

git clone https://github.com/jamesbower/detecttrace.git cd detecttrace && uv sync uv run detecttrace demo

Requires Python 3.11 or later and uv. The demo runs on synthetic data. A PyPI release is being prepared.

Stop hoping. Start proving.

We are working with a small group of founding design partners. One agent, one alert class, built around your SOC, starting with a scoped Evaluation Sprint.