Machine learning refers to an AI’s ability to learn and act like human beings do. One of the features of AI is machine learning; it learns, like we do, as it goes along. A growing number of organizations seek to benefit from the AI revolution. However, a team of researchers warns that they should be careful. They should not put all their eggs in one basket.
The researchers, from the University of Waterloo, the Institute of Mathematics of the Academy of Sciences of the Czech Republic, Princeton University, Tel Aviv University, and Technion-IIT wrote about their study in the journal Nature Machine Intelligence (citation below). The authors were Shai Ben-David, Pavel Hrubeš, Shay Moran, Amir Shpilka, and Amir Yehudayoff.
Measuring performance of machine learning tools
The authors found that contrary to conventional wisdom, no exact method exists for determining whether machine learning tools have solved a given problem.
Lead author, Shai Ben-David, a Professor in Waterloo’s School of Computer Science, said:
“We have to proceed with caution. There is a big trend of tools that are very successful, but nobody understands why they are successful, and nobody can provide guarantees that they will continue to be successful.”
“In situations where just a yes or no answer is required, we know exactly what can or cannot be done by machine learning algorithms. However, when it comes to more general setups, we can’t distinguish learnable from un-learnable tasks.”
Impossible to predict AI-based tool performance
In this study, the researchers looked at EMX (estimating the maximum), a learning model. EMX captures several common machine learning tasks. It captures, for example, the task of identifying the best place to find a set of distribution facilities to optimize their accessibility for anticipated future consumers.
The study found that no mathematical method existed for determining how well the AI-based tool performed. In other words, it was not possible to tell whether the tool could handle a given task.
Prof. Ben-David said:
“This finding comes as a surprise to the research community since it has long been believed that once a precise description of a task is provided, it can then be determined whether machine learning algorithms will be able to learn and carry out that task.”
Citation
“Learnability can be undecidable,” Shai Ben-David, Pavel Hrubeš, Shay Moran, Amir Shpilka, and Amir Yehudayoff. Nature Machine Intelligence, Volume 1, Pages 44–48 (2019). DOI: https://doi.org/10.1038/s42256-018-0002-3.
