👀 Which AI models Stanford researchers believe are most transparent
Among Google, Meta, and OpenAI, here's who got the highest marks.
Which companies are doing the best work providing transparency into how their AI models work? That’s the question researchers at Stanford’s Center for Research on Foundation Models sought to answer with their newly released Foundation Model Transparency Index, which ranked some of the best-known names in AI according to 100 different indicators.
The best performer scored a 54%.
Specifically, the team behind the new index looked at foundation models, which are large AI language models intended to be used for a wide variety of use cases. In evaluating each of them, the FMTI authors asked questions such as which resources were used to build each model, how human labor has been used, and what types of feedback mechanisms are in place.
Coming in at No. 1 on the list of models was Meta’s Llama 2, which outperformed everyone else but still scored only 54%.
Coming in next were BLOOMZ with 53% and GPT-4 with 48%.
The motivation behind publishing the index seems simple. "As the impact of this technology is going up, the transparency is going down," Rishi Bommasani, who contributed to the project, told The New York Times.
In the wake of social media’s rise and increasing public debate over how user data should be used and how permissions should be properly obtained, this group wants to track how norms and practices are emerging in AI.
"Hopefully once this is out, some people inside these companies will go hey, we really should be doing this because all of our competitors are — I hope this will become a basic thing that people come to expect," Bommasani explained to VentureBeat.
Side work
I’ve got a new collaborative AI-related project coming myself: Check out aiartifacts.net to be the first to hear the podcast later this month.
Over on the blog, I got into why Microsoft spent so much time and money to finally close the Activision Blizzard acquisition.
Side reads and watches
Season 2 of "Loki," which I’m admittedly a little cool to so far, but am enjoying for the Ke Huy Quan scenes
Walter Isaacson’s "Elon Musk," the biography I wasn’t sure I really needed but do appreciate for the writing and vivid scene-setting that Isaacson dependably renders (as you would expect from him)
Side shoutouts and links to friends
Alex Segura, my old pal and a prolific author (in crime fiction and comics) who has an outstanding Substack newsletter to subscribe to if you love either of those topics
Kiel Phegley, who has a new video trailer out for his hockey graphic novel "Strikers," which I mentioned here previously
Melinda Taub, who in addition to being hilarious professionally and personally has a new book, "The Scandalous Confessions of Lydia Bennet, Witch," reimagining Jane Austen’s "Pride and Prejudice"