Imagine opening a support inbox on a Monday morning. A customer says a parcel has gone missing and wants it sent again. A writing assistant can produce a courteous reply in seconds. Somewhere between the greeting and the sign-off comes a consequential sentence promising another parcel. Who checked the despatch record? Did anyone arrange a replacement? The distinction between composing that sentence and making it true runs through the evidence on AI and human judgement at work.
What changed on a real support desk
Economists Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied a software business as it introduced an assistant to its customer support agents in stages. The system watched chats and suggested replies. Staff could edit each suggestion, use it or carry on without it. The researchers could compare agents before and after access and with colleagues still waiting for the tool. Their published study of 5,172 agents found that access increased the number of issues resolved per hour by 15 per cent on average.
Experience changed the result. Newer and less skilled agents improved both speed and quality. The most experienced and highly skilled saw small gains in speed alongside a small decline in quality. The largest gains came on moderately rare problems. Staff had encountered those problems too infrequently to acquire much practice, yet enough examples existed for the software to offer useful suggestions.
Anyone who has searched a shared drive for an answer can recognise the opportunity. An experienced colleague may know which fix to try when a particular error follows a software update. Getting that suggestion into the live conversation can spare a new recruit several minutes and a customer another explanation. The colleague who discovered the fix still has another job when the next update produces an unfamiliar error.
When a better presentation concealed a worse decision
A randomised experiment involving 758 consultants asked participants to tackle assignments with or without access to GPT-4. One exercise involved developing a new shoe, identifying possible buyers and writing material for a launch. Across the set of 18 tasks, people with AI completed 12.2 per cent more work items on average. Their submissions also received higher quality ratings. Generating several ideas for a shoe gave judges something tangible to compare.
In a different exercise, consultants had to recommend which brand a business should invest in, drawing on financial figures and staff interviews. The researchers had constructed a case with a correct answer. People using AI were about 19 percentage points less likely to identify it than those working without the tool. Even incorrect answers from the assisted group earned higher marks for coherence. A supervisor reading for clarity could have approved precisely the wrong recommendation.
The abilities involved sit side by side in a working day. A buyer might ask for three ways to explain a price change to a supplier, then ask whether the change is justified. The second request calls for checking the contract, the quantities and the dates. A polished answer to the first question gives the buyer little evidence about how well the system will handle the second.
A test built around one missing parcel
Return to the hypothetical parcel. Give two colleagues the same fictional case. The retailer’s system records a parcel as delivered on Friday. A carrier’s note says it was handed to reception. The customer works in a building with two reception desks and has checked one. Under the retailer’s imaginary procedure, a replacement needs a supervisor’s approval. Ask each colleague to prepare a reply and record the next action, allowing only one to use an assistant.
A good reply acknowledges the search the customer has already made. It asks which reception desk they checked, offers to contact the carrier and gives a time for an update. The case record should assign that follow-up to somebody. Check whether the writer quietly upgraded the carrier’s note into evidence that the customer received the parcel. Check, too, whether the promised update could actually be delivered before the carrier’s office opens.
Time both colleagues until the case record and reply are ready to use. Include any time spent correcting errors. Count an unauthorised replacement or a missing follow-up separately from minor wording edits. Then switch the tool between colleagues for a different case. This small exercise can expose a weak handover or an ambiguous procedure. A larger trial with balanced assignments would be needed to estimate typical time savings across the desk.
What a UK trial could measure
The Government Digital Service ran a Microsoft 365 Copilot trial involving 20,000 government employees in late 2024. Its report put estimated savings at 26 minutes per person per day. In a letter to the Public Accounts Committee, the department explained that participants reported the time themselves and chose how to use the tool. That figure captures people’s experience of their day. Calculating how many extra cases a department could finish requires observations of actual workloads and outcomes.
Minutes are particularly slippery when one person’s shortcut creates another person’s check. If an assistant drafts minutes of a meeting, somebody must establish whether a proposed deadline became an agreed deadline. The test is easy to run using meetings for which a reliable record exists. Ask reviewers who have never seen the draft to identify the decisions from the original discussion. Compare their findings with the generated minutes, including any dissent that disappeared in the summary.
Suppose a team drafts six letters in a morning and saves eight minutes on each first draft. That looks like 48 minutes gained. If checking the facts takes an extra five minutes per letter, the saving across the team falls to 18 minutes. The arithmetic is illustrative, but the choice of clock is real. A manager timing the first writer would reach a different conclusion from a manager timing both colleagues, even if the final letters were identical.
Who gets to learn the difficult part?
For a junior employee, an instant suggestion can make a difficult shift manageable. It can also shorten the moment in which they would have worked out why a suggested action fits. A supervisor could occasionally ask the employee to explain what evidence would make them choose a different answer. In the parcel case, a signed collection record or a second customer reporting the same problem would change the line of enquiry. Being able to say why is a skill worth practising.
For employers considering staffing changes, the studies leave an unanswered question. Neither a completed consulting exercise nor an extra resolved chat establishes how many people a UK team will need next year. Demand may grow, old cases may reopen and staff may spend the time saved on conversations that previously went unfinished. Managers can see which is happening by following cases through to resolution and asking customers whether the promised action took place.
Before sending the reply about the missing parcel, a colleague should be able to point to the delivery note, name the person calling the carrier and say when the customer will hear back. Once those details are settled, an assistant can help phrase a difficult message kindly. The customer will judge the service when the phone rings or the parcel turns up.