Researchers investigating the effects of AI assistance on human skill development have found that while AI can improve immediate task performance, it may compromise long-term learning. The study, published on arXiv, used a controlled experiment involving logic puzzles to observe how on-demand AI assistance influenced participants' abilities before, during, and after AI access. A key finding indicates that lower costs for AI assistance led to more frequent use, which in turn correlated with poorer performance once the AI was no longer available.
The experiment was structured in three phases. In the initial phase, participants completed logic puzzles without any AI assistance to establish a baseline for their abilities. The second phase introduced on-demand AI assistance, with varying costs for requesting help. This manipulation of cost aimed to induce different levels of AI engagement among participants. The final phase again removed AI assistance, allowing researchers to evaluate the long-term impact on skill retention.
A central conclusion from the study is that participants who relied more heavily on AI assistance during the access phase showed a decline in their unassisted performance later. Furthermore, their performance with AI assistance tended to overestimate their actual unassisted ability. The researchers employed a Bayesian latent ability model to differentiate between initial skill levels, post-AI skill levels, and individual changes in skill over time. This model helped to assess the relationship between independent reasoning during the AI-access phase and overall skill development.
The findings suggest that when AI assistance acts as a substitute for independent reasoning, skill development is weaker. Greater effort in independent problem-solving was associated with more substantial gains in latent ability. This aligns with concerns that easily accessible AI might reduce the necessity for sustained independent cognitive effort.
Another related study, also published on arXiv, explored how both AI usage and the informativeness of the assistance shape learning in logical reasoning tasks. This research similarly concluded that greater AI usage is linked to weaker skill development. Heavy users of AI performed worse compared to peers with similar initial abilities, while light users performed comparably to those who did not use AI. This effect was mediated by the informativeness of the AI. Low-information AI did not improve immediate performance or preserve post-AI performance and was associated with weaker overall learning. In contrast, high-information AI improved short-term performance without, on average, reducing outcomes after AI assistance was removed, though with varied individual effects. These studies collectively suggest that AI can either complement human skill by enhancing independent reasoning or act as a substitute that undermines it, highlighting the need to regulate AI access and usage to promote skill development.
The implications extend beyond logic puzzles, touching upon various domains where AI is increasingly integrated, such as education, software development, and medical diagnosis. The tension between immediate performance gains and long-term skill development is a recurring theme in research on human-AI interaction. For instance, research on AI assistance in software development found that AI use impaired conceptual understanding, code reading, and debugging abilities among novice workers, without providing significant efficiency gains. Participants who fully delegated coding tasks to AI showed some productivity improvements but at the cost of learning the underlying library.
Another study on mathematical reasoning and reading comprehension tasks demonstrated that even brief interactions with AI assistance (around 10 minutes) could reduce persistence and impair unassisted performance. Participants in the AI-assisted group had a lower solve rate and higher skip rate on subsequent unassisted problems compared to a control group. This suggests that AI might condition users to expect immediate answers, thereby diminishing their experience in working through challenges independently, which is crucial for skill acquisition.
These findings collectively underscore a trade-off in AI-assisted problem-solving. While AI can offer short-term performance boosts, excessive reliance may reduce independent effort and hinder learning outcomes. The manner in which AI is used, rather than merely its availability, appears to be a critical factor in determining its impact on human learning. Future research will need to examine these dynamics in more specialized and high-stakes contexts to understand their generalizability.