AI and the Human in the Loop11 Min Read

AI will change how work gets done. That is no longer the question. The question for leaders is how to strike the right balance: use AI to improve speed, cost and insight while protecting judgment and preparing employees for roles that will be different from the jobs they hold today.

That balance matters because AI is not simply another software upgrade. It changes who prepares the work, who reviews it, what evidence is available and where accountability sits. Companies that treat adoption only as a technology deployment will miss the harder part. They also have to redesign the work around it.

AI can compare large volumes of information, identify patterns and prepare an analysis in seconds. Those capabilities can release capacity and improve business performance. They also create a less obvious risk. An AI-generated answer can appear polished and authoritative while relying on incomplete or inaccurate data. Organizations must prepare their people, data and processes as deliberately as they prepare the technology.

The human risk when AI looks right but is wrong

Truth bias is the psychological principle that, unless people have a reason to question an answer, they generally begin by assuming the information presented to them is credible. That tendency helps routine communication function. It becomes a problem when speed, confidence and professional formatting substitute for evidence. Under deadline pressure, an experienced reviewer can accept a plausible answer too quickly.

Let’s say A finance team uses AI to review month-end accruals. The system can see purchase orders, invoices and prior-period activity, but it cannot see a vendor agreement that was recently signed outside the normal workflow. Based on the information available, it recommends reversing the accrual. The recommendation is wrong, but it looks right. That is the point. A reviewer who checks only the calculation may approve it. A reviewer who understands the process asks what may be missing, verifies the agreement and makes the correct decision.

A human in the loop is more than a final approval. The reviewer needs access to the evidence, enough time to investigate, authority to challenge or stop the process, and clear accountability for the decision. If any one of those is missing, the human review can become ceremonial. The person is present, but the control is not working.

Four disciplines for responsible AI adoption

1. Recognize truth bias. Leaders should assume that everyone can be influenced by a fluent and confident answer. For decisions that matter, reviewers need the supporting sources and the assumptions behind the result. They also need to ask a basic question: What must be true for this answer to be correct, and what information could the system not know?

2. Put governance in place. Governance should define the tools employees may use, the data those tools may access, approved uses, process ownership and decision authority. It should also identify who can change or stop an application when results deteriorate. The exact structure will vary by company, but the responsibilities cannot be implied. They need to be assigned.

   Current guidance supports this risk-based approach. The NIST AI Resource Center maintains practical resources for testing, evaluation, verification and validation while NIST updates the AI Risk Management Framework. The Federal Reserve's revised 2026 model risk guidance, although written for banking organizations, makes the same broader point: the level of oversight should reflect how a model is used and the business consequence if it is wrong.

3. Set control points according to risk. Human review should increase with materiality, judgment and the difficulty of reversing an error. A low-risk draft may need a source check. A journal entry, payment recommendation or forecast used for a significant decision may require independent evidence, defined thresholds, documented exceptions and explicit approval.

   Organizations also need to separate different types of change. Model drift occurs when a model's performance changes over time. A model or version change is different and should trigger regression testing. Data drift occurs when the characteristics of the inputs change. Business-rule drift occurs when policies, assumptions or operating conditions change. Any one of these can make a control that worked six months ago unreliable today.

   Calibration cases provide a practical way to manage that risk. These are a controlled set of test scenarios with an expected result and a documented reason for that result. The set should include normal transactions, material exceptions, boundary conditions, missing-data scenarios and cases where the correct action is to escalate rather than answer. Run the cases before the initial release and again after a model, prompt, source, policy or material business condition changes. If the results move outside the approved range, the process should stop for review.

4. Redesign roles and build capability. Upskilling is useful only when people understand what work they are being prepared to do. As AI becomes part of the operating model, employees will spend less time preparing routine outputs and more time framing the work, testing evidence, managing exceptions and improving the process. The five work modes below provide a practical way to think about those responsibilities.

These are work modes, not five new job titles. One employee may operate in several modes, and the mix will change as an application moves from an idea into normal operations. The International Labour Organization's 2025 research reached a similar conclusion at a broader level: because human input remains necessary, most jobs exposed to generative AI are more likely to be transformed than made redundant.

The practical workforce question is how to use the capacity that AI creates. Some of it can support growth without adding staff at the same rate. Some can be redirected into analysis, customer service or process improvement. Leaders should decide where that capacity will go and train people for those responsibilities before the old tasks begin to disappear. They should also capture institutional knowledge before automating the work that carries it.

Where leaders can begin

Start with one process that matters but can be contained. A month-end analysis, reconciliation or policy inquiry usually provides enough value to test the approach without placing a major decision fully at risk. Then work through five steps:

1. Define the objective. State the business result you expect and how you will know whether the use is helping

2. Map the work. Identify the decision, the available data, the system of record and the undocumented knowledge people use today

3. Design the control. Set the risk level, assign ownership and identify where human review or escalation is required

4. Build calibration cases. Establish expected results for normal cases, exceptions, missing data and known failure conditions

5. Pilot and monitor. Test the process, track performance, rerun the calibration cases after material changes and develop employees for the work modes the process requires

The same principles apply beyond finance in operations, supply chain, human resources, IT and customer service. AI adoption will not be solved by one policy or one training course. It becomes part of the operating model when leaders decide where AI can help, what must remain human, how results will be tested and how people will be prepared for the roles that follow.

Contact Our AI Adoption Professionals