🔍 Read the full analysis: ChatGPT Helped With Data Science, But Consultants’ Skills Didn’t Stick on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A randomized trial involving Boston Consulting Group consultants found that workers using GPT-4 scored higher than a comparison group on coding, statistics and prediction tasks. But on a follow-up technical test without ChatGPT, researchers found no evidence that the AI group had learned more.
A randomized trial at Boston Consulting Group (BCG) found that consultants using GPT-4 performed better than a comparison group on data-science tasks, but researchers found no evidence of a similar advantage when participants later answered technical questions without the tool. The study, published in Nature Human Behaviour on October 26, 2026, suggests that AI assistance can expand what workers accomplish in the moment without necessarily teaching them to do the work independently.
The trial allocated 986 BCG consultants to a ChatGPT group or a control group that received access to Google and sites including Stack Overflow and Khan Academy. After brief training, participants completed two of three 90-minute assignments: Python coding, statistical analysis or prediction. The main analysis included 487 consultants who submitted work on both tasks. Most participants, according to the authors, reported no coding experience.
Compared with the control group, the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction. The researchers also report that the AI group completed coding and prediction faster on average, by about nine and 14 minutes respectively. Time spent on the statistics task did not differ significantly.
The researchers compared results with those of 44 BCG data scientists who completed the tasks without AI. Consultants using ChatGPT remained below that expert benchmark on average across all three tasks. Afterward, participants answered five technical questions without ChatGPT. The authors report no evidence that the treatment group performed better than the control group on this follow-up test.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
AI Assistance Is Not Skill Retention
The results distinguish between task performance with AI and what workers can do after the tool is removed. For employers, a strong deliverable produced with a chatbot may reflect effective use of assistance rather than a lasting increase in the employee’s independent expertise.
That difference matters for training, staffing and quality control. Companies may need to assess both the work people produce with AI and whether they can explain or reproduce key steps without it. The study does not show that AI cannot support learning in other settings; it finds no evidence of an advantage on this particular tool-free test after these tasks.
The performance gains also came with a potential downside. The authors report that participants using ChatGPT became more confident that the system could solve problems it could not. They describe that analysis as exploratory, so it should not be treated as a definitive finding about every user or AI system.
Products Worth Considering
Amazon Kindle Paperwhite (12th Generation, 2024), 16GB, 7-Inch Display, Raspberry
Bose QuietComfort Wireless Noise-Cancelling Headphones, White Smoke
DJI Osmo Mobile 8 Phone Gimbal with Mic Mini Transmitter, Infinity Black
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Designed
The experiment took place in late March and early April 2024 and was preregistered before data collection, on March 13, 2024. The AI group received 15 to 20 minutes of instruction on prompting ChatGPT for technical problems and then used GPT-4. The control group received training on searching for information online and using reference sites.
Participants worked on tasks involving data cleaning and merging in Python, identifying problems in a statistical model and interpreting residual plots, or building a model of the predictability of international football matches. The tasks were designed by BCG data scientists together with OpenAI’s Economic Impacts Research team. According to the study description, ChatGPT could guide participants but could not correctly answer the questions on its own.
The expert benchmark provides a comparison with experienced data scientists, while the control group helps estimate the effect of access to ChatGPT relative to common online resources. The paper’s title describes the performance gains as “temporary upskilling”: the observed benefit during assisted work was not matched by an advantage on the later test without the tool.
““We find no evidence” that the ChatGPT group performed better on the technical questions answered without the tool.”
— The study authors
What the Trial Cannot Establish
The findings come from one consulting firm, a specific set of data-science tasks and a brief GPT-4 training session. They do not establish whether longer instruction, repeated practice, different AI tools or other kinds of work would produce lasting learning. The study also tested retention through five technical questions, not through a long-term assessment of job performance.
Participation declined between allocation and submitted work: 573 consultants began the survey and 487 submitted work on both tasks. The supplied study summary does not explain in detail how that attrition affected the results. It also does not establish whether the reported confidence effect would appear in other workplaces; the authors label that analysis exploratory.
The authors’ conclusion is limited to the evidence reported: they found no evidence of better performance by the AI group on the immediate tool-free test. That is not the same as proving that no participant learned anything, or that ChatGPT cannot contribute to learning under different conditions.
How Employers May Apply the Findings
The published study does not announce a follow-up trial or a specific next milestone. For employers, the immediate implication is to distinguish AI-assisted output from demonstrated independent skill when evaluating training and work quality. Assessments that include explanation, verification and tool-free tasks could help make that distinction, though the study does not test a particular workplace policy.
Further research would be needed to determine whether different training designs help workers retain skills, whether effects last beyond the study’s follow-up test and how the results transfer to other jobs and AI systems. Until then, the clearest finding remains that ChatGPT improved task scores in this trial, while the researchers found no detectable group advantage on the subsequent technical questions without the tool.
Key Questions
What did ChatGPT change in the BCG trial?
Consultants using GPT-4 scored higher than the control group on coding, statistics and prediction assignments. The reported differences were 49, 20 and 17 percentage points, respectively.
Did consultants perform as well as BCG data scientists?
Not on average. The ChatGPT group remained below the 44 data scientists who completed the tasks without AI, although consultants with coding experience came closer on coding and statistics.
Did participants retain the skills after ChatGPT was removed?
On a follow-up test of five technical questions completed without ChatGPT, the authors found no evidence that the AI group performed better than the control group. The test does not establish that no individual learned anything.
Does the study prove AI cannot teach people data science?
No. It examined one firm, a brief training session, specific tasks and a particular follow-up test. The findings do not settle whether other teaching methods, longer practice or different tools could support lasting learning.
Source: ThorstenMeyerAI.com
