Scattered postmortems stalled learning — Confluence & Box vector search with PMO sign?off

Overview

A large NGO struggled to learn from failed initiatives because postmortems and evaluations were buried across shared drives and Confluence spaces. Teams repeated mistakes, and strategy decks leaned on anecdote rather than documented lessons. Intelligex created a searchable knowledge base with vector search across Confluence and Box, wrapped in a governance flow that required Program Management Office (PMO) approvals for summaries and automatic anonymization of sensitive details. Strategy leaders could surface comparable cases quickly, reuse lessons with cited evidence, and plan with fewer blind spots and less rework.

Client Profile

• Industry: International nongovernmental organization (humanitarian and development programs)
• Company size (range): Multi-regional operations with country and technical portfolios
• Stage: Institutionalizing knowledge management and portfolio learning
• Department owner: Strategy, Analytics & Executive Leadership (Strategy Office / PMO)
• Other stakeholders: Monitoring & Evaluation (M&E), Program Directors, Grants & Compliance, Regional Leadership, Legal & Privacy, IT/Security, Communications, Learning & Development

The Challenge

Project postmortems, donor evaluations, and closeout reports lived in Confluence spaces and Box folders with inconsistent naming, tags, and templates. Some were scans; others mixed narrative with tables and attachments. Teams relied on tribal knowledge to find comparable cases, and search results were uneven. When strategy leaders asked, “where have we seen this risk and what worked next time,” staff compiled ad hoc reading lists that missed key context.

Privacy and compliance concerns slowed sharing. Reports contained beneficiary anecdotes, partner names, and sensitive incident details. Redactions were performed manually and inconsistently, and there was no lightweight path to produce an anonymized lesson summary with citations and PMO sign-off. Existing tools worked for storage and collaboration, but they did not provide a governed way to search across repositories, protect sensitive content, and promote validated lessons to leadership.

Why It Was Happening

Taxonomies and identities were fragmented. Confluence spaces used local labels, Box folders followed varied grant naming, and project IDs did not map cleanly across systems. Templates differed by funder and program, and postmortems ranged from structured forms to free text. Without a mastered tagging scheme and stitched identifiers, retrieval depended on who remembered a document existed.

Governance arrived late. Teams wrote summaries in slides without required citations, anonymization happened after distribution, and there was no approval gate to confirm that sensitive details were masked and that lessons were represented faithfully. Even when a strong case study existed, it was hard to find and harder to reuse with confidence.

The Solution

We implemented a governed knowledge layer that indexed postmortems and evaluations across Confluence and Box, enabled vector search, and routed lesson summaries through PMO approvals with automated anonymization. Confluence pages and Box documents were crawled with metadata capture; text was embedded for semantic search; and a portal returned results with facets for region, sector, donor, and risk theme. A summarization service generated draft “lessons learned” cards with inline citations and ran sensitive text through an anonymization pipeline. PMO reviewers approved or edited summaries before they appeared in leadership views. Core tools stayed in place; the orchestration added retrieval, synthesis, and governance around them.

• Confluence indexing via API with space, label, and page metadata capture (Confluence REST API)
• Box content ingestion with file metadata, versions, and permissions respected (Box Developer)
• Vector search over embedded document text and metadata, backed by Azure Cognitive Search
• Draft lesson cards generated from retrieved passages with citations to page anchors and file locations (retrieval?augmented pattern)
• Automated detection and anonymization of sensitive details in drafts using Microsoft Presidio, with human-in-the-loop correction
• Role-based access and result trimming by user group, aligned to existing identity groups (Okta Groups)
• PMO approval workflow for summaries and tags with comments, tracked edits, and rationale (Power Automate Approvals)
• Conformed metadata model in a warehouse for projects, donors, sectors, and geographies, with snapshotting and lineage
• Leadership dashboard with filters for geography, sector, donor, and risk theme; drill-through to governed source excerpts
• Audit trail for crawls, embeddings, summary edits, anonymization events, and approvals

Implementation

• Discovery: Cataloged Confluence spaces and Box folder structures; collected representative postmortems and donor reports; noted sensitive content patterns; and gathered current tags, templates, and program IDs. Reviewed recent strategy decks to identify repeated knowledge gaps and misclassifications.
• Design: Defined a lightweight taxonomy for sector, geography, donor, risk theme, and program identifiers. Authored crawl rules, metadata mappings, and permission trimming. Designed the vector search index and facets, the lesson card template with citation fields, the anonymization policies, and the PMO approval sequence with change logs.
• Build: Implemented connectors to Confluence and Box with delta crawls and metadata capture; built the embedding pipeline and vector index; developed the lesson card generator with citation injection; integrated Presidio for anonymization; configured approval flows; and published leadership views with drill paths back to governed excerpts.
• Testing and QA: Ran historical postmortems through indexing and search; compared retrieved sets against expert-curated collections; validated anonymization on sensitive narratives; exercised PMO edits and approvals; and verified that permissions and citations behaved as expected across regions and roles.
• Rollout: Launched search in read-only mode to PMO and M&E while teams validated results and tags. Enabled lesson card drafts for a subset of sectors and regions, then required PMO approvals before cards appeared in leadership dashboards. Kept an exception lane for urgent briefings with post-review documentation.
• Training and hand-off: Delivered quick guides for program teams on tagging and template use, for PMO reviewers on summary editing and anonymization review, and for strategy staff on searching and citing lessons. Assigned stewardship for taxonomy updates and connector scope with a change-control cadence.

Results

Leaders and program teams could find comparable cases quickly, read cited lessons with anonymized excerpts, and drill to governed source passages when needed. Strategy reviews drew on consistent language for risk themes and countermeasures, and PMO approvals ensured summaries reflected context without exposing sensitive details. Debates shifted from “does a lesson exist” to “how do we adapt it here.”

Rework tapered because templates, tags, and anonymization rules operated upstream. Indexing and vector search surfaced relevant content across repositories, and approvals produced a clear record of what was published, by whom, and why. Planning cycles moved with better visibility into known pitfalls and mitigations, and postmortems were referenced instead of re-created.

What Changed for the Team

• Before: Lessons were scattered in folders and pages with inconsistent tags. After: Vector search returned governed results with facets and citations.
• Before: Sensitive anecdotes were redacted manually and unevenly. After: Anonymization ran in the pipeline with PMO review before publication.
• Before: Summaries lived in slides with unclear provenance. After: Lesson cards cited exact passages and linked back to Confluence or Box.
• Before: Knowledge reuse depended on who remembered an example. After: A permissions-aware portal made comparable cases easy to find within policy.
• Before: Approvals and edits were lost in email. After: A workflow captured PMO edits, decisions, and rationale with an audit trail.

Key Takeaways

• Index and embed existing knowledge sources; retrieval quality improves when taxonomies and permissions are respected at the source.
• Require citations and anonymization by default; credible, reusable lessons depend on provenance and privacy.
• Put PMO review in the flow; small gates prevent misinterpretation and protect sensitive content without slowing teams.
• Keep Confluence and Box; add a search and governance layer rather than moving content to a new system.
• Maintain a lightweight taxonomy with stewardship; consistent tags make vector search and filtering far more effective.

FAQ

What tools did this integrate with?
We indexed Confluence via its API (Confluence REST API) and Box via its developer platform (Box Developer). Vector search ran on Azure Cognitive Search. Anonymization used Microsoft Presidio. Approvals were captured in Power Automate Approvals, and access was governed by existing identity groups such as Okta Groups.

How did you handle quality control and governance?
We defined a lightweight taxonomy for sector, geography, donor, and risk themes, and mapped project IDs across repositories. Every lesson card required inline citations to source passages. Anonymization policies flagged names and sensitive entities for masking, with a human-in-the-loop PMO review. Approvals, edits, and publication events were logged with timestamps, reviewers, and rationale.

How did you roll this out without disruption?
Search launched in read-only mode for PMO and M&E, using content as-is from Confluence and Box. After validating facets, citations, and anonymization behavior, we introduced lesson card drafts and required PMO approvals before cards appeared in leadership views. No content was moved; the new layer orchestrated indexing, retrieval, anonymization, and approvals around existing tools.

How did you protect sensitive information and consent?
Anonymization ran on draft summaries and quoted excerpts before publication, with PMO reviewers confirming that masking aligned to privacy guidance and donor expectations. Access to raw documents remained restricted by existing permissions, and the portal trimmed results by role. Sensitive full-text content was never exposed outside authorized groups.

How were tags and taxonomy enforced across repositories?
Crawlers captured existing labels and folder metadata, then applied mappings to a governed taxonomy. PMO reviewers could adjust tags on lesson cards during approval, and changes fed back into the metadata store. Templates for new postmortems included recommended tags, and stewardship owned periodic updates to definitions and mappings.