---
id: "concept-ai-orchestration-layer"
type: "concept"
source_timestamps: ["§ Training for a Marathon When You Can Only See Two Feet Ahead"]
tags: ["technical-architecture", "llm-routing", "data-security"]
related: ["prereq-secure-infrastructure", "action-build-orchestration-layer", "entity-microsoft-azure"]
definition: "An internal, low-code middleware system that securely routes prompts to various third-party LLMs based on cost and capability, keeping all data within a secure corporate perimeter."
sources: ["execution"]
sourceVaultSlug: "hbr-seg-execution"
originDay: 8
articleStem: "hbr-cl-93-legacy-financial-all-in-genai"
sourceUrl: "https://hbr.org/2025/03/how-a-legacy-financial-institution-went-all-in-on-gen-ai"
sourceTitle: "How a Legacy Financial Institution Went All In on Gen AI"
---
# Secure AI Orchestration Layer

## Secure AI Orchestration Layer

To balance agility with the strict security requirements of the financial sector, [[entity-moodys|Moody's]] developed an internal **orchestration layer** that sits on top of multiple commercial foundation models — **OpenAI, Anthropic, Meta, and Google**.

This **low-code/no-code** system securely allocates user prompts to different models based on variables like **probable inference costs** and **specific model strengths**. Crucially, the architecture ensures that all AI interactions and proprietary data remain **entirely within Moody's secure infrastructure**.

The payoff: Moody's could integrate new AI capabilities — such as **PDF interrogation** or **image analysis** — onto employee desktops **within hours** of those features hitting the commercial market, without compromising customer trust.

**Definition:** An internal, low-code middleware system that securely routes prompts to various third-party LLMs based on cost and capability, keeping all data within a secure corporate perimeter.

### Connections
- Depends on [[prereq-secure-infrastructure]] and the [[entity-microsoft-azure]] partnership (secure cloud + OpenAI access).
- The build task: [[action-build-orchestration-layer]].
- Enables [[claim-proprietary-models-not-competitive-advantage]] — the layer is *how* off-the-shelf models get applied to proprietary data quickly.
- The contrarian stance it embodies: [[contrarian-off-the-shelf-over-proprietary]].

### Enrichment note
Moody's public GenAI risk-solutions materials support the general architecture (secure GenAI grounded in its data estate with enterprise controls), but the **exact routing logic** across OpenAI/Anthropic/Meta/Google is asserted only in the HBR/extraction account, not independently confirmed. This design also maps to **RAG (retrieval-augmented generation)** patterns — grounding responses in trusted proprietary corpora to reduce hallucination risk. Open question on dependence: [[question-long-term-vendor-lock-in]].