Stanford HAI’s mission is to advance AI research, education, policy and practice to improve the human condition. It serves as one of the most credible and authoritative sources for data and insights about AI to provide policymakers, researchers, journalists, executives, and the general public a deeper understanding of the field.

The AI Index is an independent initiative at Stanford HAI that is led by an interdisciplinary steering committee of experts from across academia and industry. Because benchmarks guide the direction of AI development, they must align more with how we, as humans and as a society, want to interact with these tools. It considers not only accuracy, but also fairness, toxicity, efficiency, robustness, and more. HELM, developed by scholars at CRFM, looks across multiple scenarios and multiple tasks and is more comprehensive than benchmarks we have seen in the past. I think this is an opportunity to reevaluate what we want from these tools.Īre researchers already beginning to build better benchmarks?īeing at Stanford HAI, home to the Center for Research on Foundation Models, I can point to HELM. But as we move toward AI tools that incorporate vision, language, and more, do we need benchmarks that help us understand the tradeoffs between accuracy and bias or toxicity, for example? Can we consider more social factors? A lot cannot be measured through quantitative benchmarks.

The question becomes, if benchmarks are supposed to help us reach a goal, what is this goal? How do we want to work with AI and how do we want AI to work with us? Perhaps we need more comprehensive benchmarks – right now, benchmarks mostly test against a single goal. Another way that I think of it is this: Our AI tools right now are not exactly as we would want them to be – they give wrong information, they create sexist imagery. The takeaway for me is that perhaps we need newer and more comprehensive benchmarks to evaluate against. This year, the top performing model hit 84.3% accuracy. For example, the Visual Question Answering Challenge tests AI systems with open-ended textual questions about images. So we’re seeing a saturation among these benchmarks – there just isn’t really any improvement to be made.Īdditionally, while some benchmarks are not hitting the 90% accuracy range, they are beating the human baseline. For example, the best image classification system on ImageNet in 2021 had an accuracy rate of 91% 2022 saw only a 0.1 percentage point improvement. This year across the majority of the benchmarks, we saw minimal progress to the point we decided not to include some in the report. In earlier years, people were improving significantly on the past year’s state of the art or best performance. So, for each benchmark, were researchers able to beat the score from last year? Did they meet it? Or was there no progress at all? We looked at ImageNet, a language benchmark called SUPERGlue, a hardware benchmark called MLPerf, and more some 50 were analyzed and over 20 made it into the report.

– and evaluated the state-of-the-art result in each benchmark year over a year. We looked across multiple technical benchmarks that have been created over the past dozen years – around vision, around language, etc. What did the AI Index study find regarding these benchmarks? The goal is to correctly identify as many of the images as possible. Researchers run their image classification algorithms on ImageNet as a way to test their system. One example is HAI Co-Director Fei-Fei Li’s ImageNet, a dataset of over 14 million images. It’s a way of defining what you want your tool to do, and then working toward that goal. In this conversation, Parli explains more about the benchmarking trends she sees from the AI Index.Ī benchmark is essentially a goal for the AI system to hit. “We really need to be thinking about how we, as humans and society, want to interact with AI, and develop new benchmarks from there.” “Most of the benchmarks are hitting a point where we cannot do much better, 80-90% accuracy, ” she said. In the newest AI Index, published on April 3, a team of independent researchers analyzed over 50 benchmarks in vision, language, speech, and more to find out that AI tools are able to score extremely high on many of these evaluations. 'But when you actually use the tool, it gives incorrect answers, says thing we don’t want it to say, and is still difficult to interact with. “There’s been a lot of excitement, and it meets some of these benchmarks quite well,' she said. She cites the current popular example of ChatGPT.

But that doesn’t mean most artificial intelligence tools work the way we want them to, says Vanessa Parli, associate director of research programs at the Stanford Institute for Human-Centered AI and a member of the AI Index steering committee. How good is AI? According to most of the technical performance benchmarks we have today, it’s nearly perfect.

Human benchmark meaning