AI Governance Framework: Build It Before Regulators Do

1. EU AI Act: Regulation (EU) 2024/1689 entered into force 1 August 2024. High-risk AI obligations apply from 2 August 2026.
2. High-risk AI categories include systems used in employment decisions, credit scoring, insurance pricing, and critical infrastructure management.
3. Core EU AI Act governance obligations for high-risk systems: risk management systems (Article 9), technical documentation (Article 11), logging (Article 12), and human oversight procedures (Article 14).
4. ISO 42001:2023 is the first internationally recognised management system standard for artificial intelligence, providing structured implementation paths for AI governance programmes.
5. UK SM&CR creates personal accountability for AI governance failures where a senior manager cannot demonstrate adequate oversight of AI-assisted decisions.
6. National supervisory authorities under the EU AI Act have enforcement powers against non-compliant providers and deployers of high-risk AI systems.

Several regulatory developments have turned AI governance from a risk management conversation into a compliance obligation. Regulation (EU) 2024/1689, the EU AI Act, entered into force on 1 August 2024 and is applying in phases. Prohibitions on unacceptable-risk AI applied from 2 February 2025 and general-purpose AI model obligations from 2 August 2025. Requirements for high-risk AI systems, including AI used in employment decisions, credit scoring, and critical infrastructure management, apply from 2 August 2026.

The FCA has made its expectations equally clear. Its publications on AI and machine learning established that firms using AI in regulated activities are fully accountable for the outcomes those systems produce. The FCA's supervisory focus has been on model explainability, bias testing, and the adequacy of human oversight. In the US, the SEC has brought enforcement actions against firms for materially misleading statements about their AI capabilities (so-called AI washing), establishing that AI-related misrepresentation carries securities law consequences.

Despite the regulatory shift, most organisations' AI governance programmes remain reactive: centred on documenting what already exists, without the pre-deployment controls, audit trails, and accountability structures that regulators now require. The board-level risk implications of this gap are significant for regulated firms where senior manager accountability is live from day one.

Shadow AI is the most common reactive pattern: business units deploying AI tools without IT, legal, or compliance oversight. The capability accumulates; the governance infrastructure doesn't.

The other common failure is retroactive documentation. Compliance teams document AI use after the fact, producing records that describe what a model does without evidencing pre-deployment risk assessment, bias testing, or human oversight controls. These records describe the system. They don't demonstrate governance.

Both produce accountability gaps: no documented chain of responsibility when an AI-assisted decision causes harm or attracts regulatory scrutiny. When the FCA or another regulator asks who was responsible for governing a specific AI system, "we documented it afterwards" is not a sufficient answer.

The EU AI Act requires a governance structure that operates throughout the lifecycle of a high-risk system: risk management systems before deployment (Article 9), conformity assessments (Article 43), registration in the EU database (Article 49), and technical documentation maintained throughout (Article 11). Post-deployment records don't satisfy these obligations.

And for FCA-regulated firms, the Senior Managers and Certification Regime creates personal accountability for AI governance failures. Where a firm's AI system produces discriminatory outcomes or material errors, the regulator asks who was responsible for its governance. If the senior manager responsible can't demonstrate adequate oversight, enforcement exposure is direct.

A defensible AI governance framework is an operational infrastructure: controls, processes, and records that produce demonstrable evidence of governance. It has six components.

AI Inventory and Risk Classification

Before governing AI, an organisation must know what AI it's running. An AI inventory should capture every system in use, the data it processes, the decisions it influences, and the regulatory classification of those decisions. Under the EU AI Act, risk classification determines the compliance obligations that apply: prohibited, high-risk, limited-risk, or minimal-risk.

For organisations subject to the EU AI Act, Article 11 requires technical documentation for high-risk systems and Article 49 requires registration in the EU database. The inventory is the legal baseline on which every other control depends.

Pre-Deployment Risk Assessment

High-risk AI systems under the EU AI Act require a documented risk management system under Article 9 that operates throughout the whole lifecycle of the system. Risk assessment must happen before deployment. A pre-deployment assessment should address model purpose and scope, training data quality and potential bias, explainability requirements, failure modes and their operational consequences, and the human oversight controls in place.

ISO 42001:2023, the international standard for AI management systems, provides a structured framework for implementing risk assessment processes that map directly to these requirements. An organisation implementing Clause 6.1 (actions to address risks and opportunities) against its AI inventory builds precisely the evidence base that regulators expect.

Audit Trails and Decision Records

An AI decision is auditable when a contemporaneous record exists of the inputs, the model version, the output, and (where a human reviewed that output) the human decision. Without this record, an AI-assisted decision can't be reconstructed during regulatory review, litigation, or internal audit.

For AI systems influencing lending decisions, employment outcomes, or insurance pricing, decision records are both a regulatory requirement and a litigation risk management tool. EU AI Act Article 12 requires high-risk AI systems to automatically generate logs sufficient for post-hoc reconstruction of the system's operation. Implementing logging at the model level and retaining decision records in a structured, retrievable format is a technical and governance requirement simultaneously.

Model Governance: Validation, Monitoring, and Drift Detection

A model that performed accurately at deployment won't necessarily continue to perform accurately as the data environment changes. Model drift (the degradation in model performance over time as real-world patterns diverge from training data) is a live risk in any deployed AI system. Detecting it requires ongoing monitoring, periodic revalidation, and a defined process for remediation when performance falls below acceptable thresholds.

The PRA’s Model risk management principles for banks provide a validation framework applicable beyond credit risk models: validate before deployment, monitor on an ongoing basis, and revalidate when material changes occur to the data environment or business context. These principles apply to any AI system influencing consequential decisions in a regulated context.

Designing Effective Human Oversight

Both the EU AI Act and FCA expectations require human oversight of high-risk AI decisions, but oversight is frequently implemented as a nominal approval checkbox. Effective human oversight has four genuine components: the reviewer must have access to the information needed to evaluate the AI's output; they must have authority to override it; they must have the training to identify when an override is warranted; and the review must be documented.

Article 14 of the EU AI Act sets out specific requirements for high-risk systems: natural persons overseeing the system must fully understand its capabilities and limitations, detect signs of anomalous performance, and intervene. Designing oversight as a genuine control, with documented evidence of each review, is one of the areas where governance programmes most frequently fall short.

Why Accountability Structures Matter

Every AI system should have a named owner with documented responsibility for its governance. In SM&CR firms, this responsibility maps to a Senior Manager Function where the AI system's outputs fall within a regulated activity. Outside financial services, accountability structure should be captured in the AI inventory and in board-level risk reporting.

The accountability structure serves a practical purpose. When a regulator opens an investigation, the first question is who is responsible. An organisation that produces a named owner, their oversight records, and the governance documentation they maintained is in a materially different position to one that cannot.

The EU AI Act applies in phases. Understanding which obligations are already live and which are approaching is the starting point for any implementation programme.

The business case for proactive AI governance extends well beyond regulatory compliance. In financial services, firms that can demonstrate credible AI governance to the FCA or PRA (documented risk assessments, explainability reports, model validation records) can deploy AI in high-stakes decisions: credit decisions, fraud detection, customer risk classification. Firms lacking this governance infrastructure hit a ceiling: legal and compliance teams block use cases where the controls to support them are absent.

ISO 42001:2023 certification (the first internationally recognised certification for AI management systems) provides a verifiable, third-party-audited signal of governance maturity. For organisations procuring AI from vendors, ISO 42001:2023 alignment is increasingly a contract qualification criterion. For regulated entities deploying AI, it provides a structured implementation path that maps to regulatory expectations.

The practical advantage of moving now is significant. Organisations that wait until August 2026 to implement high-risk AI governance will do so under time pressure, potentially alongside competitors, and after whatever enforcement actions the European AI Office and national supervisory authorities take against early non-compliers. Building the framework today means doing it at a pace the organisation controls.

AI governance is a GRC function. It needs the same infrastructure as any other compliance programme: risk registers, control libraries, evidence collection, audit trails, and reporting. The distinction is that the subject matter is novel and the regulatory frameworks are still maturing.

An AI inventory runs as a risk register; pre-deployment assessments are templated and tracked; model monitoring outputs are ingested as control evidence; and decision records are stored against relevant control objectives. For teams already running GRC programmes, applying that infrastructure to AI governance is faster and lower-risk than building from first principles.

Gracie AI Agents with Personas and Skills automates the evidence collection and monitoring work that AI governance demands at scale. On SureCloud's compliance management platform, AI inventory entries run as risk register items, pre-deployment assessments are tracked against control objectives, and model monitoring outputs are captured as continuous control evidence, with the audit trail regulators expect built into the workflow from day one.