A few years ago, the model that told your business whether to buy the second site, replace the truck fleet, or hold more cash in reserve was a spreadsheet. You could open it, read every assumption, and argue with the formula. Now an AI tool often does that same job, and nobody can open it up and check the working. The decision looks identical. The way you should govern it is not.
We call this AI model risk, and it is arriving quietly inside Australian businesses through software they already pay for. The same core question runs all the way up to the most extreme use of AI anywhere, military targeting, where firms like Palantir build their whole governance approach around it. If it matters there, it matters for your capital decisions in Perth too.
A forecasting model takes numbers in and gives a decision out. For decades that engine was statistical: a discounted cash flow, a Monte Carlo simulation, a credit scorecard sitting in a spreadsheet. Banks have governed these under a discipline called model risk management since the US Federal Reserve published its guidance, known as SR 11-7, back in 2011. The principle is plain: any method that turns data into a decision is a model, and a model needs an owner, validation, and monitoring.
Here is the part most businesses miss. When you replace that statistical engine with AI, the asset on your books has not changed. It still answers the same question. What changes is how it can fail, and the new failure modes are worse.
| The old statistical model | The AI model |
|---|---|
| You can read every assumption and formula | Often a black box, even to the people who run it |
| Validate once, trust it until you change it | Drifts as the world moves, so it needs ongoing monitoring |
| Same inputs give the same answer | Same inputs can give different answers |
| Changes only when you change it | The vendor can update the underlying model without telling you |
| Limited attack surface | New risks: data leakage, poisoned inputs, confident wrong answers |
So the governance burden goes up, not down. You traded a glass box for a black box, and the odds are good that your oversight did not change to match. That is the heart of AI model risk.
This is not a hypothetical risk. It is happening now, usually without a purchase order. Here are four everyday decisions where the model has quietly become AI.
Should we invest. Buying a second site, replacing a vehicle or machine fleet, putting solar on the roof, opening another clinic. The old model was a payback tab in Excel that the owner built. Now a chatbot or a copilot writes the business case and predicts the return. The danger is a bet-the-business number that nobody can show the working for.
Will we run short of cash. Cash-flow forecasting, and the call on whether to lock in a foreign exchange rate for an overseas supplier order. This used to be a rolling spreadsheet in Xero or MYOB. AI forecasting is now built into the accounting stack, and roughly 41% of mid-market firms already get it through tools like Copilot on Dynamics or AI on NetSuite without buying a treasury system. A confident wrong forecast can drive a drawdown or a hedge, with no explanation attached.
Should we extend this customer credit. The old model was an ageing report, a credit policy, and gut feel. Now a tool scores debtor risk and flags invoices likely to go bad. The problem comes when you cannot tell the customer, or your auditor, why the model flagged them, and when the model drifts without anyone noticing.
Replace it or run it another year. The delivery van, the CNC machine, the MRI scanner, the HVAC plant. The old model was a maintenance-cost spreadsheet. Now predictive tools recommend when to replace. You are left with an opaque call on a six-figure asset.
The sharpest part for a smaller business is this. At your size the AI model is rarely something you consciously bought and put on a register. It switched on as a feature inside a tool you already had. So you are running an AI decision model, and it never hit any register at all. That is the gap we keep finding.
Not every AI carries the same risk, and the difference is worth getting right. An AI agent takes actions in the world: it can move money, change a setting, send an email. An AI model produces an output that a person reads and then acts on. The person is the circuit breaker. A model genuinely needs lighter controls than an agent on the action side, because on its own it cannot actually do anything. For agents, the controls are mostly about permissions, what each one is allowed to touch, which is the same discipline as access management for your staff.
The trap sits in between. A human in the loop is only a real control if that human can tell when the output is wrong and is willing to overrule it. With the old spreadsheet they could interrogate the logic. With an opaque AI framing a capital call across a fleet, if the decision-maker is rubber-stamping a number they cannot challenge, then the model is making the decision and the human is decoration. An advisory model trusted like an oracle is, in risk terms, closer to an agent than the org chart admits.
To see the framework at its limit, look at where the stakes are highest. Palantir builds the software used in military targeting, the part of the process some call the kill chain. You would expect the governance to be heavy, and it is. Their stated approach rests on four controls: a mandatory human in the loop for any critical action, so the AI cannot execute a strike or a major transaction on its own; granular permissions defining who can do what; full audit trails logging every input, output, and the human who signed off; and approval checkpoints where the operator has to justify accepting the AI’s suggestion.
Read that list again. Human gate, permissions, logging, sign-off. It is the same architecture a well-run business should wrap around an AI that decides where capital goes. The only thing that differs is the materiality dial.
It is also where the framework gets tested in public. After reports of civilian casualties in strikes that used such systems, lawmakers questioned how much judgement was really being handed to the algorithm, even with a person nominally in the loop. Palantir’s line is that a human always makes the final call. The criticism is that under time pressure the checkpoint can become a rubber stamp. That is the oracle trap again, with lives attached. If a defence contractor cannot fully settle whether the human is genuinely deciding, a busy finance team approving an AI forecast at month-end should be honest about the same risk.
You might assume a law forces you to govern this. It does not. In December 2025 the National AI Plan confirmed that Australia will rely on existing laws and sector regulators rather than a standalone AI Act or mandatory guardrails, at least for now. The main reference for business is the Guidance for AI Adoption, published by the National AI Centre in October 2025, which sets out six voluntary practices: decide who is accountable, assess impact, measure and manage risk, share information, test and monitor, and keep meaningful human control.
Two things follow. First, the AI governance burden is now a choice driven by your own risk appetite, not a compliance box to tick, which is exactly why it gets skipped. Second, existing law still bites: the Privacy Act applies to the data feeding these models no matter what the voluntary guidance says. The government even publishes a free AI register template, which tells you how central the register idea has become. Your AI decision models belong on one.
This is where a managed IT partner earns its keep, and where an honest one draws a line. We build and run the technical half: access control, audit logging, knowing where your data goes and which model version produced a given result, monitoring for drift, and the register itself. What we do not do is tell you whether the AI’s number is right. Whether a capital allocation is sound is your call and your own experts’ call, not ours. We make the output accountable and traceable. You stay accountable for the decision. That split is the whole game, and it is the same work whether the AI is an advisory model or an agent, sitting alongside your managed cyber security and the strategic view a virtual CIO brings to the board table.
Find your shadow AI models. Walk through the decisions that used to run on a spreadsheet: cash flow, capital spend, credit, asset replacement. Then ask which tool makes those calls now. The AI ones that nobody formally chose are the first to worry about.
Sort them by stakes and type. For each one, ask two questions. How material is the decision, and is this a model a human acts on or an agent that acts by itself. A high-stakes model that gets trusted without challenge deserves the same scrutiny as an agent.
Put the serious ones on a register, then talk to us. Give each high-materiality model a named owner, logging, version monitoring, and a human who can genuinely override it. Book a free AI governance review with our Perth team and we will help you find what is already running and lock down the controls around it.
AI model risk is the chance that an AI system used to inform or make a decision produces a wrong, biased, or unexplainable output that leads to a poor business decision. It is the modern version of model risk management, the discipline banks have used for decades. The difference is that AI models are harder to inspect and can drift over time, so the oversight has to be ongoing rather than one-off.
Yes. Any AI that informs a real decision, such as a forecast, a credit call, or a capital spend, is a model that should have an owner and basic oversight, even in a small business. In Australia this is voluntary under the Guidance for AI Adoption, but the Privacy Act still applies to the data involved, and managing AI model risk is increasingly expected by clients, insurers, and auditors.
An AI model produces an output that a person reviews and acts on. An AI agent takes actions on its own, such as moving money or changing a system. Agents need tighter permission controls, but a model trusted without challenge can carry just as much AI model risk, because the human checkpoint stops being a real control.
No. As of the National AI Plan in December 2025, Australia relies on existing laws plus the voluntary Guidance for AI Adoption rather than a dedicated AI Act. Governing your AI is a choice, though it is becoming a benchmark for demonstrating trust to clients, insurers, and regulators.
Check the tools you already pay for. Accounting platforms, copilots, and industry software now ship AI features that forecast, score, or recommend, often switched on by default. If a tool gives you a number or a recommendation you cannot fully explain, that is an AI model worth reviewing.
Yes, for the technical half. A managed IT provider can set up access control, audit logging, data tracking, version monitoring, and the AI register. Validating whether the AI’s answer is actually correct stays with your own domain experts. The provider makes the output traceable and accountable so your people can make the call with confidence.
At Epic IT, our specialists focus on long-term solutions for your business.
Our 90-day guarantee is simple: if you’re not satisfied, we’ll refund your onboarding fee and give you $5K. Contact us today!
Call Us
1300 EPIC IT
Email Us
info@epicit.com.au
Perth HQ
HQ – Unit 2, 62 Angove Street North Perth WA
Queensland Office
12 The Cct, Brisbane Airport QLD
Sydney Office
Unit 3, 104 Anglesea Street, Bondi NSW
