AI fails in production, not on paper

The same pattern appears in most large organisations. A working group forms, a responsible-AI policy is drafted and approved, a committee begins meeting each quarter, and a statement goes up on the website. Meanwhile the models already in production keep running, largely unwatched, doing things nobody has checked against the purpose they were approved for.

The policy was never the hard part. Writing down that AI should be fair, safe and accountable takes a few weeks, and most organisations manage it competently. The hard part is making a live system behave that way every day, under load, as it is retrained, extended and pointed at new data. That work begins after the document is signed, which is precisely where most programmes lose momentum. I have spent the last several years inside that gap, for banks, telecommunications operators and government agencies, and the failure modes repeat with remarkable consistency. There are four of them.

The model is governed; the use is not

Governance effort attaches naturally to the model: the artefact that was procured, tested and approved. Risk, however, attaches to the use. The same model can be low-impact in one deployment and material in another, because risk moves with purpose, affected persons, data, autonomy and downstream consequence.

One example makes the gap concrete. A customer-service assistant is approved to retrieve account and product information. Months later, frontline staff and customers have begun relying on the same assistant for suitability, credit or financial guidance. The model is unchanged. The service identity, the API and the output format are unchanged. Accuracy and safety metrics remain within tolerance, so every dashboard is green. What changed is the business purpose, the affected decision and the consequence, and no control was watching for that, because every control was pointed at the model.

Approval decays

An assessment made at a point in time does not remain valid after a model substitution, a new data source, an expanded user group, a new tool connection or a change in decision context. Yet most governance programmes treat approval as a state rather than a claim that has to keep being true. The inventory records that a use was approved in March. It records nothing about whether the conditions of that approval still hold in September.

The practical consequence is that the age of an approval becomes a measure of its unreliability. The remedy is not more frequent committee reviews, which merely shorten the interval over which the same decay occurs. The remedy is to define, in advance, which changes invalidate the approval: changes to purpose, users, data, model, tools, autonomy, workflow or downstream consequence. Those are the change triggers, and a material change should force reassessment before continued use, not after the next scheduled review.

Monitoring is represented as control

Boards are routinely told that an AI system is “monitored”, and the word carries an implication of control that it has not earned. A log produced after external effect is evidence of what happened. It is not a control over whether the thing should have happened. If a model output influences a customer, a regulated workflow or a material decision, and the organisation’s first opportunity to intervene arrives after the fact, then the organisation is not controlling that use. It is narrating it.

Control means an enforceable outcome at the moment it matters: permit, observe, escalate, deny or suspend, applied before external effect where the consequence can reasonably be held. Monitoring feeds that decision. It does not substitute for it.

Human oversight is stated, not engineered

Nearly every AI policy requires human oversight, and very few specify what would make it real. A reviewer without timely evidence, capacity, authority and a defined safe state does not constitute an effective control. A notification sent after execution is not oversight. An approval queue that a reviewer cannot keep up with is not oversight. A named individual who lacks the decision rights to stop the system is not oversight.

Engineering oversight means deciding where the human sits in the workflow, what evidence they see, how long they have to act, what happens when they do nothing, and what the system does when the question cannot be resolved in time. Unresolved high-impact events should default to a defined safe state, and that default is a design decision someone has to make and own.

What an operating layer consists of

Each of the four failures is closed by the same structure, installed use case by use case rather than declared enterprise-wide.

First, an authorised-use baseline: a versioned record of the purpose and prohibited purposes, the accountable business, technical and risk owners, the data classes and sources, the model, tools and integrations, the affected decisions, the risk limits, the human oversight arrangements, the evidence duties and the material-change triggers. The baseline is owned by the institution, not the vendor, and it is attestable.

Second, a runtime decision: material live activity is evaluated against the current baseline, testing identity, purpose, data, version, context and consequence, so that the question “is this use authorised now” has an answer at the moment it is asked rather than at the next review cycle.

Third, an enforceable control outcome: permit, observe, escalate, deny or suspend, binding where consequential effect can be held.

Fourth, operational evidence: the applicable baseline, the rule, the decision, the escalation, the owner action and the outcome are preserved as they occur. Evidence built this way costs almost nothing to capture and holds up in front of an auditor or a regulator, because the record was created as the decisions were made. Evidence reconstructed afterwards from meeting notes and chat threads is slow, expensive and never quite convincing.

Start bounded, not enterprise-wide

The most reliable way to build this layer is to refuse to build it everywhere at once. Select one consequential use. In the first thirty days, name the owners and define the baseline and the control point. In the next thirty, instrument it: bind live events to the baseline version, configure the checks, calibrate the outcomes and the reviewer workflow. In the final thirty, operate it, measure exception volume and contextual drift, and decide with evidence whether to scale, remediate or suspend. Ninety days produces a working control, a defensible evidence pack and, more valuably, an institution that now knows what this work actually requires.

The sector context makes the case for urgency. The 2026 AICB–Ecosystm benchmark of Malaysian banks and development financial institutions found 44 per cent of institutions at a developing stage of AI readiness, 33 per cent with structured AI governance and model risk management, and 27 per cent applying formal AI risk tiering, against near-universal adoption ambitions. Adoption has outrun control, and the distance between the two is not closed by another policy. It is closed by the operating layer described above, built one consequential use at a time.

Vivegavalen Vadi Valu leads the AI governance practice at Responsible AI Solutions. The practice · Contact

← All perspectives