Ryan Burnell, “A Cognitive Approach to the Evaluation of AI Systems”

Abstract:

The capabilities of AI systems are improving rapidly, and these systems are being deployed in increasingly complex and high-stakes contexts, from self-driving cars to the detection of medical conditions. As the importance of AI grows, so too does the need for robust evaluation. If we want to determine the extent to which systems are safe, effective, and unbiased, it is vital that we understand the cognitive capabilities of those systems. In this endeavour, psychological science has a lot to offer—scientists from cognitive, developmental, and comparative psychology have spent many decades developing theories and paradigms to understand the cognitive capabilities of adults, children, and animals. Drawing on these theories and paradigms, we are working to build a framework for evaluating the cognitive capabilities of AI systems that we hope can be used to better track and regulate AI progress. I will present an initial version of the framework and discuss the open questions and challenges of applying cognitive science to AI evaluation.

This talk will take place in person at SCIoI.

Photo by Michael Dziedzic on Unsplash.