Over the past few years, artificial intelligence (AI) has rapidly gained popularity, progressing at a surprising pace. We imagine humans and AI working side-by-side, with human creativity and AI’s analytical power combining to make strategic decisions and solve all types of problems.
According to a new study from the MIT Center for Collective Intelligence (CCI), there are times when we work well with artificial intelligence, and other times when we perform better on our own or AI works better alone.
Michelle Vaccaro, an MIT doctoral student and CCI affiliate, along with MIT Sloan School of Management professors Abdullah Almaatouq and Thomas Malone, published their study and findings in the peer-reviewed journal Nature Human Behavior (citation below).
They say that theirs is the first large-scale *meta-analysis conducted to gain a better understanding of human-AI combinations in task completion. Specifically, when they are useful and when they are not.
* A Meta-Analysis examines many research studies to draw broader conclusions on a specific topic. It is a study of multiple studies.
The researchers found that combining humans and AI for creative tasks showed great potential. However, human-AI teams did not do so well when it came to decision-making tasks.
Humans and AI Working Together Effectively
The study results come at a time when we are both excited and apprehensive about artificial intelligence’s impact on the workforce.
We are excited because of the improvements AI can bring to productivity, efficiency, and wealth creation. However, we also worry that smart machines may eventually replace us in the workplace.
Rather than focusing on which jobs AI may take from us, Malone explains that he and the team chose to explore how effective humans and AI can be when working together.
They also aimed to determine how organizations could establish guidelines and guardrails to ensure the success of these partnerships.
A Comprehensive Meta-Analysis
The researchers carried out a meta-analysis of 370 study findings on AI and human combinations in a wide range of tasks from 106 different experiments. Their source materials included conference proceedings and academic journals from January 2020 to June 2023.
The studies evaluated three approaches to task performance:
- Human-only systems.
- AI-only systems.
- Human-AI collaborations.
The primary objective of the meta-analysis was to uncover the underlying trends revealed by the combination of these studies.
Test Outcomes
The analysis revealed that human-AI teams, on average, performed better than humans who worked on their own. However, they did not exceed the capabilities of AI systems operating without humans.
Importantly, they did not observe “human-AI synergy,” indicating that, on average, human-AI systems underperformed compared to the best results achieved by either humans or AI systems operating independently.
The findings suggest that AI systems or humans are more effective on their own than in combination.
Vaccaro said:
“There’s a prevailing assumption that integrating AI into a process will always help performance — but we show that that isn’t true. In some cases, it’s beneficial to leave some tasks solely for humans, and some tasks solely for AI.”
Factors Affecting AI-Human Synergy
The researchers added that there are factors that can affect how well AI and humans work together.
-
Making Decisions
For decision-making tasks, for example, such as deep fakes, diagnosing medical cases, and forecasting demand, AI alone performed better than human-AI teams.
-
Creative Tasks
However, humans working in combination with AI on creative tasks performed better than humans or AI on their own.
Examples of creative tasks include generating new content and imagery, answering questions in a chat, designing marketing campaigns, composing music, developing product ideas, or summarizing social media posts.
Malone said:
“Even though AI in recent years has mostly been used to support decision-making by analyzing large amounts of data, some of the most promising opportunities for human-AI combinations now are in supporting the creation of new content, such as text, images, music, and video.”
The team suggested that this advantage in creative tasks arises from their dual characteristics: they require human abilities such as creativity, knowledge, and insight, while also involving repetitive elements where AI performs exceptionally well.
When we are designing an image, for instance, we need artistic inspiration, which we are good at, and detailed execution, in which AI excels.
Similarly, creating text documents requires human knowledge and insight, while also incorporating repetitive and automated tasks like generating standardized content.
Vaccaro said:
“There is a lot of potential in combining humans and AI, but we need to think more critically about it. The effectiveness is not necessarily about the baseline performance of either of them, but about how they work together and complement each other.”
Optimizing AI-Human Collaboration
The authors believe that their findings provide guidance and lessons for employers who are looking to maximize AI’s effectiveness in the workplace.
Vaccaro stressed the importance of determining whether an AI-Human combination is really performing better than either one working on their own.
Vaccaro added:
“Many organizations may be overestimating the effectiveness of their current systems. They need to get a pulse on how well they’re working.”
They need to evaluate exactly where AI can help humans in the workplace. According to the study, AI can be very helpful in creative tasks.
Employers should, therefore, explore what kind of creative tasks would benefit the most from the insertion of AI.
Organizations must also implement clear guidelines and strong safeguards for AI use. For example, they could design workflows that harness the unique strengths of both humans and AI in a complementary manner.
Malone suggested:
“Let AI handle the background research, pattern recognition, predictions, and data analysis, while harnessing human skills to spot nuances and apply contextual understanding. Let humans do what they do best.”
“As we continue to explore the potential of these collaborations, it’s clear that the future lies not just in replacing humans with AI, but also in finding innovative ways for them to work together effectively.”
Citation
Vaccaro, M., Almaatouq, A. & Malone, T. When combinations of humans and AI are useful: A systematic review and meta-analysis. Nat Hum Behav (2024). https://doi.org/10.1038/s41562-024-02024-1