Observability

Overview

Observability measures one specific thing: when a workflow does something consequential, is there a record of it that doesn't depend on manually re-running the execution to find out? A workflow can be fully reliable — no crashes, every retry handled correctly — and still fail this check, because reliability is about the workflow behaving correctly, and observability is about whether anyone would know if it didn't.

How this affects your score

Observability is weighted 0.15. Findings range from low to high severity, scaling with how consequential the write in question is — a financial or destructive write with no audit trail is a meaningfully worse gap than a low-stakes internal write with the same gap.

Worked example: a workflow with one high-severity finding (no audit log on a financial write) scores 100 − 15 = 85 on Observability, contributing 85 × 0.15 = 12.75 points toward the total.

Findings

Missing audit log on a consequential write

Severity: medium → high. If a workflow charges a card, deletes a record, or posts something publicly, and there's no separate record of that action being taken — success or failure — the only source of truth is whatever system was written to.

Example: a workflow processes refunds through a payment API but doesn't log the refund attempt anywhere the team controls; when a customer disputes a refund, the only record is inside the payment provider's own dashboard, with no link back to which automation triggered it or why.

Fix: log the outcome of every financial, destructive, or public write — success or failure — with enough context (a record ID, an external transaction reference, a timestamp, the execution ID) to trace it back without re-running anything.

Missing execution summary on a consequential write

Severity: low → medium. A lighter-weight gap than a missing audit log: even without full audit-trail rigor, does anything capture that this execution ran and what it did? Without it, confirming a specific run happened means opening the execution list and reading through node outputs one at a time.

Example: a workflow updates inventory counts across a dozen SKUs per run but produces no summary of what changed; investigating a discrepancy means manually reading through every past execution's node outputs.

Fix: for frequent, lower-stakes writes, a lightweight summary step (what ran, what it touched, whether it succeeded) is usually enough — it doesn't need the same rigor as a full audit log.

Recommended fixes

1. Add an audit log entry for every financial, destructive, or public write, tagged with the workflow execution ID.
2. Don't rely solely on the downstream system's own logs — they rarely capture which automation triggered the action or why.
3. For high-frequency, lower-stakes writes, add a lightweight execution summary rather than skipping observability entirely.

Related categories

• Reliability — the failure-handling side of the same writes this category tracks after the fact
• Security — what's exposed before a write happens, versus what's recorded after
• AI Governance — agent-driven writes carry the same observability requirement as any other write