Deduplication Engine

The Duplication Problem

In a multi-source, continuous capture environment, duplication is inevitable. The same decision might be referenced in a project document, a meeting transcript, and a retrospective report. The same workaround might be captured by two successive role holders independently. The same vendor relationship might be documented in multiple entries from different contexts.

Without deduplication, the memory store grows noisy. Retrieval returns redundant entries. Coverage scores are inflated. Successor briefs repeat themselves. The Deduplication Engine keeps the store clean by detecting and reconciling duplicate and near-duplicate entries without losing the distinct contextual contributions of each source.

Similarity Detection

Deduplication operates at multiple levels of similarity:

• Exact duplicates: Entries with identical or near-identical content from the same or different sources. These are flagged for automatic merging.
• Semantic near-duplicates: Entries that describe the same institutional fact with different wording, from different sources or time periods. Detected through vector similarity comparison in the embedding space — if two entries are within a configurable cosine similarity threshold and share the same classification type and domain, they are candidates for deduplication review.
• Superseded entries: Entries where a newer entry explicitly or implicitly replaces the content of an older one. For example, if a new process decision supersedes an older documented approach, the older entry should be marked as superseded rather than remaining as an active, equal-weight entry in the store.

Merge and Reconciliation

When duplicate candidates are identified, the Deduplication Engine performs a merge operation. The merge combines the content of the duplicate entries, resolves conflicts (where two entries describe the same fact differently, the higher-confidence or more recently validated version is preferred), and retains the source attribution of all contributing entries. The merged entry carries the combined provenance of its sources, which increases its confidence score.

Merges that involve substantive content differences — where the two entries are similar but not identical in meaning — are routed to the Human Validation Loop rather than being merged automatically.

Impact on Coverage and Confidence

Deduplication directly affects coverage and confidence scores. Before deduplication, a domain might appear well-covered based on entry count, while in reality the counts are inflated by duplicates. After deduplication, coverage scores reflect the true diversity of knowledge captured rather than the volume of captures. Merged entries typically carry higher confidence than either source entry individually, improving the overall quality of the memory store.

Audit Trail

Every merge and deduplication decision is recorded in the audit trail. Original entries are archived rather than deleted, so that a merge decision can be reviewed and reversed if a validation reviewer determines that the merged entries were in fact distinct and should remain separate.