What AI consulting should look like after go-live.

Most conversations with an AI consulting firm are about getting something live: the use case, the data, the build, the go-live date. Far fewer are about what happens the week after. That is a gap, because an AI system does not stay finished the way a report or a dashboard does. The data it reads shifts, the model underneath it is versioned and eventually retired by its provider, usage grows, and the rules it operates under change.

This piece sets out what ongoing support for an AI system should cover, the three ways Australian organisations usually buy it, which obligations carry on after launch, and how to tell when you should stop paying an AI consulting firm to look after it.

Why an AI system needs looking after.

Conventional software mostly does tomorrow what it did today. An AI system is different in four ways, and each one is a reason the work does not end at go-live.

• The data moves. New products, new forms, a changed process in one team: the inputs drift away from what the system was tested on, and quality drops slowly enough that nobody notices for months.
• The model is not yours. Most AI systems run on a model supplied by a vendor. Vendors release new versions and publish retirement dates for old ones, so at some point you will change model whether you planned to or not.
• Usage grows. A system that handled fifty requests a day in the pilot may handle two thousand a year later, which changes the running cost and exposes cases nobody tested.
• The rules change. Privacy obligations, sector standards and your own policies move, and the system has to move with them.

Six things ongoing support should cover.

• Evaluation against a fixed test set. A set of real, representative cases with agreed correct answers, re-run every time the model, the instructions or the data connection changes. Without it, every change is a guess.
• Quality monitoring in production. Sampled review of live outputs, plus the rate at which people override or correct the system. A rising override rate is usually the first sign of drift.
• Model and version changes. Planned upgrades, tested against the evaluation set before they go live, with a way to roll back.
• Cost and usage. A monthly view of what the system costs to run against what it does, so growth in volume is a decision rather than a surprise.
• Access review. A regular check that the system still holds only the permissions its job needs. Our piece on what an AI agent should be allowed to do covers how to set those permissions in the first place.
• Incidents. A named person to call when the system gets something wrong, an agreed response time, and a record of what happened and what changed as a result.

None of this is exotic. It is the same discipline a well-run data platform already has, applied to a system whose behaviour can change without anyone changing the code.

Three ways to buy it.

Australian organisations tend to land on one of three arrangements. Each suits a different situation, and the right one often changes over the first year.

• Run it yourselves. The firm hands over the evaluation set, the monitoring and the runbooks, and your team takes it from there. This works when you have people who can read an evaluation result and act on it. It fails when the handover is a document nobody opens.
• A retainer with the firm that built it. A fixed block of hours a month with named people who know the system. This suits the first six to twelve months, while volumes settle and your team learns the system.
• A managed service. The firm runs the system to agreed service levels. This suits systems that support an important operation and organisations that do not want to hold the skills in house. It is also the arrangement most likely to quietly become permanent.

Whichever you choose, agree up front what success looks like after launch. Our guide to measuring an AI consulting engagement covers how to set those measures before the work starts.

Obligations that do not stop at launch.

Several Australian obligations apply for as long as the system runs, not only when it is built.

• The Privacy Act. Australian Privacy Principle 11 requires reasonable steps to protect personal information on an ongoing basis. A system whose access has never been reviewed since launch is hard to describe as a reasonable step.
• Automated decisions from 10 December 2026. Privacy policies must describe certain decisions made with the help of computer programs. If the system's role changes after launch, the disclosure may need to change with it. Our piece on AI agents and the December privacy rule sets out what is covered.
• CPS 230 for APRA-regulated entities. Banks, insurers and super funds must manage operational risk and their material service providers. If an outside firm supports an AI system that a critical operation depends on, that arrangement may fall inside the standard.
• Sector rules. Registered NDIS providers, for example, must keep an incident management system. If an AI system is part of how shift notes or incidents are recorded, its failures belong in that system too. Our piece on AI consulting for NDIS and home care providers covers how that work is set up.

When to stop paying for it.

Good ongoing support should get smaller over time, not larger. Three signs it is no longer earning its cost:

• You receive invoices but no evaluation results, override rates or cost reports.
• The same problems recur month after month without a change to the system.
• Nobody in your organisation can explain how the system works or what it would take to change it.

The last one is the most important. If the support arrangement has left you unable to run, change or switch off your own system, you have bought a dependency rather than a service.

Questions to ask before you sign.

• What will you hand over at go-live, and in what form?
• Is there an evaluation set, and who owns it?
• How will we know when quality is slipping, and who looks?
• What happens when the model provider retires the version we use?
• What does a month of support include, and who are the named people?
• How do we reduce or end the arrangement, and what do we keep when we do?

A firm that answers these plainly has run AI systems in production, not just launched them. Our outline of what an AI consulting engagement looks like shows where handover sits in the whole piece of work.

Why RUBIX.

RUBIX is an independent, Australian-owned data and AI consultancy, founded in Melbourne in 2011. For 15 years our work has been the systems organisations depend on after launch: data platforms, governance and reporting, and more recently the AI agents that run on top of them. We have delivered more than 450 projects for over 115 organisations, and most of that work has had to keep running long after the project closed.

• Built to be handed over. We build the evaluation set, the monitoring and the runbooks as part of the system, so you can run it yourselves if that is the right answer.
• Senior people, embedded. Our forward deployed engineers work inside your team, so the knowledge stays with you rather than with us.
• Governance as part of the build. Our AI and data governance work sets the access rules, the audit trail and the review cycle alongside the system, not after it.

If you are planning an AI agent or any AI system, ask what the first year after go-live looks like before you agree what the build will cost. The answer tells you more about the firm than the proposal does.

Frequently asked questions.

General information only, not legal, financial or regulatory advice. Current as at November 2026.