A Playbook for Capturing Enterprise Tacit Knowledge

A structural limitation of current enterprise AI deployment is the assumption that out-of-the-box frontier models will autonomously solve complex workflows. State-of-the-art models are incredibly capable, but when dropped into an enterprise environment, they hit a strict performance ceiling. They default to generic best practices because their reasoning is constrained by static documentation.

The most valuable asset in any organization isn't written in a manual. It is "tacit knowledge," comprised by the unwritten heuristics, objection-handling strategies, and situational intuition held by a handful of veteran employees. In my day-to-day work talking to business leaders and executives, this is the most explicit pain point I hear: when veteran personnel leave, that critical corpus of knowledge walks out the door, leaving new hires with impossible onboarding curves.

The true competitive advantage in enterprise AI isn't simply plugging into the largest general-purpose model. It lies in building architectures that systematically extract this tacit knowledge from your top performers and using it to train specialized, cost-effective models structurally aligned to your business. These custom systems not only outperform generic models on domain-specific tasks, but they explicitly preserve your institutional knowledge, serving as an interactive training ground to rapidly accelerate the onboarding of new hires.

The following details the design decisions we made to adapt foundation models into these complex workflows, shifting them from generic chatbots into agents capable of independently strategizing against real constraints.

1. Defining Scenario-Driven Evaluation Suites (Archetypes)

What We Did: Development began by constructing a standardized evaluation suite. We defined detailed client profiles called "archetypes." An archetype is a rigorously defined state environment with specific constraints: current technical infrastructure, strict budget parameters, historical performance metrics, and compliance requirements.

Rather than evaluating the model on generic prompts, the system was forced to generate strategic plans tailored precisely to these constrained environments to establish a performance baseline.

How You Can Do This: A robust evaluation suite requires 15 to 20 highly diverse archetypes.

• Data Extraction: To prevent bias, archetypes should be derived from empirical CRM data. Exporting historical deal data (e.g., closed-won and closed-lost accounts) and clustering by variables such as deal size, industry, and tech stack provides a ground-truth distribution.
• Synthesis Prompt: Pass the exported CSV into an LLM with code execution or data analysis capabilities, using the following prompt:

You are an expert data analyst. I have attached a CSV of our historical 12-month CRM data.
Please perform the following operations:
1. Cluster this data into the 15 most distinct customer segments based on deal size, industry, current tech stack, and win/loss reason.
2. For each cluster, synthesize a highly detailed, constrained "Archetype" client profile.
3. Ensure each Archetype explicitly lists its hard constraints: strict budgets, rigid compliance requirements, and technical debt. Do not create "happy path" profiles; force realistic constraints.

• Edge Case Injection: If empirical data is unavailable or noisy, manual construction by domain experts is required. It is critical to enforce the creation of boundary-condition scenarios (e.g., accounts with strict legal compliance or severe budget constraints) rather than ideal "happy paths." Evaluating against historical failures is an effective method to expose when a model defaults to generic advice.

2. Extracting Conditional Logic (Delta Rules)

What We Did: Once the foundation model generated baseline outputs for the evaluation suite, domain experts were utilized to critique the outputs. The logic underlying these human critiques was extracted into a structured format called "Delta Rules."

To maintain system flexibility and prevent the model from degrading into a rigid decision tree, rules were isolated into two categories:

1. Universal Rules: System-wide invariants. For example, enforcing a mandatory daily budget ratio calculation before any strategic planning occurs. If a mock client prototype fails this mathematical check, the agent pivots the strategy.
2. Conditional Rules: Trigger-based logic activated only by specific archetype constraints. For instance, if an archetype indicates a highly regulated industry, the agent fundamentally alters its targeting parameters to comply with privacy regulations.

How You Can Do This: Extracting heuristic logic requires live, conversational pressure-testing rather than asking experts to write documentation. I chose to elicit this tacit feedback verbally in a live interview because real-time critique surfaces instinctive, context-sensitive judgment that experts often omit when asked to formalize their knowledge in writing. It also creates room to probe the reasoning behind an initial reaction, clarify exceptions, and identify the subtle cues that change how a recommendation should be made.

• Live Critique: Have the domain expert review the AI's baseline output line-by-line during a recorded session. Instruct the expert to imagine they are the salesperson's manager and that the AI is a junior sales representative preparing a pitch for this specific client archetype. Ask them to critique the pitch exactly as they would in a real coaching session: what they would correct, what signals they noticed, and how they would expect the representative to adapt.
• Automated Extraction: Process the transcript through an LLM to extract the Delta Rules. The extraction prompt must enforce a strict schema:

You are an expert knowledge extractor analyzing a transcript of a human expert critiquing an AI-generated business plan.

Extract the expert's feedback into two distinct categories:

1. UNIVERSAL RULES
Capture behaviors or standards that should apply across nearly all instances of this client archetype, regardless of the specific scenario.
For each rule, provide:
- UNIVERSAL STANDARD: What must the AI consistently do or avoid?
- RATIONALE: Why should this behavior apply across nearly all instances of the archetype?

2. CONDITIONAL RULES
Capture trigger-condition-response logic that applies only when a specific constraint, cue, or scenario is present.
For each rule, provide:
- WHEN IT APPLIES: What client context or condition makes this rule relevant?
- OBSERVABLE SIGNAL: What concrete evidence in the archetype, client behavior, or available data activates the rule?
- PITCH RESPONSE: How should the AI change the pitch once the signal is observed?
- RATIONALE: Why does this adaptation improve the real-world outcome?

Do not misclassify generic advice as a universal rule. A universal rule must be consistently useful across nearly all instances of the archetype. Do not state a conditional rule without an observable activation signal and a specific change to the pitch. Preserve meaningful exceptions or uncertainty expressed by the expert.

3. Global Rule Propagation and Regression Testing

What We Did: To enforce rule consistency, the system utilized an automated propagation pipeline. When a new Delta Rule was added to the central logic repository, a global propagation event was triggered. The system executed the updated prompt architecture against all archetypes in the evaluation suite, regenerating all baselines. This ensured that an insight derived from one edge-case scenario improved the agent's baseline reasoning across the entire corpus.

How You Can Do This: Prompt architecture must be treated like software code, requiring strict regression testing.

• Token Economics & SLMs: Regenerating complex archetypes through massive models continually degrades token economics. To control compute costs at scale, Small Language Models (SLMs) can be deployed as fast evaluation gates. Their job is to quickly scan the generated baselines and detect if anything is fundamentally broken. If the SLM detects a logic break, the failure is routed back to a larger, expensive frontier model to execute the fix.
• SLM Evaluation Prompt: Implement a low-parameter model with a breakage-detection prompt similar to this:

You are an automated evaluation gate. Review the attached AI-generated strategic plan against the client's hard constraints.
Your only job is to detect if the baseline logic is broken. 
Answer YES if you detect any of the following breakages:
1. The plan violates the mathematical budget constraints.
2. The plan references non-compliant platforms.
3. The output formatting is completely broken.

If any answer is YES, output "BROKEN" and provide a repair handoff for the frontier model with:
- SPECIFIC FAILURE: Identify the exact constraint, compliance requirement, or formatting rule that was violated.
- REQUIRED CORRECTION: Instruct the frontier model how the strategic plan must change to resolve the failure.
- PRESERVE: Identify any valid parts of the plan that should remain unchanged.

The frontier model must revise the plan to satisfy the violated constraint, remove or replace any non-compliant recommendation, and repair broken formatting without unnecessarily rewriting valid strategy. It must then return the corrected plan for evaluation against the same client constraints.

If no breakages are detected, output "PASS".

• Failure State Testing: When updating the central repository, it is critical to test against failure states. If the system is not explicitly evaluated on broken data pipelines or client pushback, the model will overfit to ideal conditions.

4. Resolving Context Limits via Agent Swarms

What We Did: As the central repository of rules expanded, a single unified model became brittle, demonstrating degraded reasoning and increased hallucination rates. It could no longer balance the conflicting priorities of different workflow domains.

To resolve this, the architecture was decoupled into specialized agents. Distinct personas were built for specific subject matters (e.g., video strategy, privacy-measurement pipelines). When evaluating a complex archetype, these agents operated as a swarm.

By utilizing State-of-the-art (SOTA) models for the orchestration and reasoning layer, the agents reliably collaborated, debated trade-offs, and rigorously critiqued each other’s strategic plans before consolidating a final output.

How You Can Do This:

• Domain Splintering: Once prompt context size degrades output quality, the persona must be splintered into specific disciplines with isolated instruction sets.
• Adversarial Debate: Programming agents with conflicting objectives (e.g., Agent A maximizes volume; Agent B minimizes risk) forces them to navigate complex friction points natively, exposing edge cases that single-agent architectures miss.
• The Swarm Debate Prompt: To initiate this clash, instantiate two distinct agents with opposing operational imperatives. For the risk-mitigation persona, use a prompt like this:

You are the Compliance Persona operating within an agent swarm. Your sole functional objective is to minimize institutional risk.
Another agent (the Growth Persona) has proposed the attached strategic plan. 
Review their plan and aggressively critique it from a purely risk-averse perspective. 
Identify every edge case, compliance violation, or overly optimistic assumption. 
Force the Growth Persona to amend their plan until you determine the risk is negligible. Do not compromise.

System Outcomes and Second-Order Effects

By iterating through this calibration engine, the AI's output shifted from generic best practices to highly contextual strategic plans matching the heuristic judgment of top-performing staff.

This architecture enables two significant secondary applications:

1. Accelerated Onboarding: Junior staff can pressure-test strategies in a sandbox environment by role-playing with an agent that has absorbed senior-level heuristics.
2. Producing Gold-Layer Data: Every expert-corrected baseline captures the original prompt, the model's chain of thought, and the human-aligned golden response. This loop autonomously generates pristine, gold-layer data, which is the exact high-fidelity dataset required for downstream fine-tuning of future models.

Adapting AI to enterprise workflows requires building deterministic systems that continuously ingest, codify, and scale human expertise. Establishing these localized feedback loops is the necessary path toward pushing beyond the limitations of generic models.

Thank you Zac, Bailey, Teresa, Max, Max, Colton, and Michael for your collaboration on this project.

Originally published: 2026-07-25