How can an organisation say yes to AI faster, and still stay in control?

Overview

As AI moves from pilots to production, the questions change. Which models are we running? Which use cases are high risk? What evidence shows they are safe? Without answers, approvals stall or, worse, nobody asks.

I define the AI governance framework across a product portfolio: use-case risk tiering, model inventory, evaluation evidence and audit trails, aligned to the NIST AI RMF, with attention to UAE IA / NESA and ADHICS where they apply. The organisation’s internal policies are confidential, so this page describes the structure of the framework and the reasoning behind it.

Problem statement

Governance is often treated as paperwork that arrives after the build. That makes it slow, and it makes it disconnected from the evaluation work that actually produces evidence.

The symptoms are familiar:

• One process for everything. A low-risk internal summariser and a customer-facing agent with tool access go through the same review, so either the summariser waits too long or the agent gets too little scrutiny.
• No inventory. Nobody can list with confidence which models are in use, where, and who owns them.
• Evidence written for auditors. Teams produce documents describing testing instead of pointing to the tests themselves, so the evidence is late and hard to verify.
• Decisions without a trail. Approvals and accepted risks are made in meetings and emails, and cannot be reconstructed later.

Engineering objectives

1. Match scrutiny to risk, so low-risk use cases move quickly and high-risk ones get proportionate review.
2. Know what is running: every model and AI use case recorded with an owner and purpose.
3. Reuse engineering evidence: evaluation, red-team and guardrail results become governance evidence directly.
4. Make decisions traceable through audit trails and risk registers.
5. Align to recognised frameworks so the approach is defensible to leadership, auditors and regulators.

Solution architecture

The framework is a lifecycle that every AI use case passes through. The weight of each step depends on the use case’s risk tier.

Its components:

• Use-case register and risk tiering: each AI use case is tiered by potential impact.
• Model inventory: every model in use, with its owner and purpose, supported by model cards.
• Evidence store: results from evaluation pipelines, red-team exercises and guardrail tests, linked to the use case they cover.
• Risk register and audit trail: identified risks, decisions, accepted residual risk and who accepted it.

The structure maps naturally onto the four functions of the NIST AI RMF: Govern (roles, policies, the register itself), Map (use-case context and tiering), Measure (evaluation evidence) and Manage (decisions, mitigations and monitoring).

Technical approach

Risk tiering by use case

Tiering is done per use case, not per model, because the same model can be low risk in one context and high risk in another. Typical factors include who is affected, whether outputs reach people outside the organisation, whether the system can take actions, what data it touches and whether a person reviews outputs before they are used. An illustrative tier definition:

These tiers are examples to show the shape, not a recommended policy.

Model inventory and model cards

The inventory answers basic questions quickly: what is this model, who owns it, which use cases depend on it, and when was it last evaluated. Model cards record intended use, limitations and evaluation results in a consistent format. Kept as structured data rather than free text, the inventory can be queried and checked automatically. An illustrative entry:

# Illustrative entry only.
use_case: support-assistant
risk_tier: high
owner: product-support
models:
  - name: general-purpose-llm
    role: generation
evidence:
  evaluation: reports/eval/latest
  red_team: reports/redteam/latest
  guardrails: reports/guardrails/latest
last_reviewed: 2024-03-15

Evidence from evaluation

The most important design choice is that governance evidence is produced by engineering work that happens anyway. Evaluation pipelines, red-team exercises and guardrail tests already generate results. The framework links those results to the use case and tier, instead of asking teams to write a separate document describing them.

Regional alignment

The framework follows the NIST AI RMF as its backbone, with attention to UAE IA / NESA for information assurance and ADHICS for healthcare where they apply. A practical approach is to map each regional control to the framework step that already produces the relevant evidence, rather than running a parallel compliance process.

Validation strategy

A governance framework is validated by whether it works in use:

• Walk-throughs with real use cases. Taking existing use cases through the lifecycle shows where tiering criteria are ambiguous or requirements are unclear.
• Inventory completeness checks. Comparing the inventory with what is actually deployed reveals models or use cases that were never registered.
• Evidence freshness. Each high-tier entry should point to recent evidence. Stale links are a signal that monitoring has lapsed.
• Audit trail reconstruction. Picking a past decision and reconstructing why it was made, from the trail alone, tests whether the trail is complete.

Metrics and measurement

Metrics commonly used to see whether governance is working include:

• Share of deployed AI use cases that are registered and tiered.
• Share of high-tier use cases with current evaluation and red-team evidence.
• Time from registration to decision, by tier. Low-tier time is a direct measure of whether governance is slowing teams down.
• Open risks in the register, by severity and age.
• Number of models without a named owner, which should be zero.

Challenges and trade-offs

• Lightweight versus thorough. Too much process and teams route around it. Too little and it provides no assurance. Tiering is how the framework tries to have both.
• Tiering disagreements. Teams naturally argue for lower tiers. Clear criteria and a named arbiter keep tiering consistent.
• Keeping the inventory current. Inventories decay quickly unless registration is part of the deployment path.
• Multiple frameworks. International and regional frameworks overlap but differ in vocabulary. A single internal structure mapped to each is easier to maintain than parallel processes.

Outcome and lessons learned

Governance becomes part of delivery rather than a gate at the end. The framework is designed so low-risk use cases move quickly, while high-risk ones get the scrutiny they need, backed by evidence the engineering teams already produce.

The lessons:

• Tier the use case, not the model.
• The best governance evidence is a test result, not a document about a test.
• Governance that is visibly faster for low-risk work earns the goodwill needed for rigour on high-risk work.

Further reading

• Governance is how you go faster
• NIST AI Risk Management Framework
• NIST AI RMF Playbook
• OWASP Top 10 for LLM Applications
• Proving a RAG system is grounded
• Red-teaming LLM and agentic applications