Confidence Scoring

Why Confidence Matters

Not all role memories are equally trustworthy. A decision documented in a formal governance record carries more institutional authority than one extracted from a meeting transcript by an AI model. A workaround confirmed by two successive role holders is more reliable than one captured from a single offhand comment. The Confidence Scoring engine makes these differences visible and actionable.

Confidence scores drive three key downstream behaviours: retrieval ranking (higher-confidence entries surface first in RAG queries), review routing (low-confidence entries are flagged for human validation), and coverage reporting (coverage metrics distinguish between high-confidence and low-confidence coverage of a domain).

Confidence Factors

The confidence score for each memory entry is calculated from a weighted combination of factors:

• Source type: Formally authored documents (architectural decision records, process guides, governance papers) carry higher base confidence than automatically extracted transcripts or inferred entries.
• Extraction method: Entries explicitly created by a role holder carry higher confidence than those generated by the AI extraction pipeline from unstructured content.
• Human validation status: Entries that have been reviewed and confirmed by a role holder or subject-matter expert receive a significant confidence uplift. Entries that have been reviewed and corrected are re-scored. Entries that have never been reviewed are marked as pending validation.
• Corroboration: An entry that is supported by multiple independent sources — two separate documents, a document and a transcript, a role holder's input and a related artefact — carries higher confidence than a single-source entry.
• Recency: Role knowledge has a temporal dimension. A workaround documented in 2019 that has never been updated may have been superseded. Recency affects confidence, with a decay function that reduces confidence for entries that have not been reviewed or updated over time.

Confidence Thresholds

Organisations configure confidence thresholds that determine how entries are treated. Entries below a low-confidence threshold are held out of the active memory store and routed to the Human Validation Loop. Entries above the high-confidence threshold are committed to the store and made available for retrieval. Entries in the middle band are available for retrieval but flagged in outputs so users know the confidence level of the information they are receiving.

Confidence in Successor Briefs

When the Successor Brief Generator draws on memory entries, it includes confidence indicators in the output. Critical operational information drawn from low-confidence entries is explicitly flagged for verification — a successor is told not just what the role memory contains, but how much to trust it and what to verify before acting on it.