This week I explored the digital art history tool imgs.ai and got some interesting results. imgs.ai provides “a Web-based interface, and machine-learning-based backend that allows the end user to both descriptively and visually search arbitrary image dataset.”¹ More simply put, it allows a human to guide an application using computer vision to find images in selected museum databases that are the most similar to an uploaded image, text prompt, or both. The idea being that the user will then continue to engage with more images by selecting from the suggested images that are “positive” or “negative,” furthering the search with this new information.²

The ability to mark negative results is where I hit my first issue: it wasn’t an option for me. This is likely because I was using the site without a login (getting one takes days for approval). While I understand the necessity of gatekeeping databases in the age of AI scraping and how intentionally using AI is fundamentally different from random bots shutting down your site, the irony of the website being protected being built on AI scraping images was palpable. How many of the models imgs.ai uses were built using the same tactics the required login was keeping out? Even if copyright was the driving force instead of defense against bots, how many artworks outside of these museum collections had their copyright violated to train the models to “see”? I am not anti-AI, but I think they are questions worth asking. A team that calls out “Transparency” as one of five “essential criteria”³ should say more about the black box components at work beyond naming the databases or models used.⁴ Minimally, it should give an explanation of why options are locked behind sign-up instead of forcing the user to guess.

I also wouldn’t be as harsh in my critique if imgs.ai as a tool had provided me an experience that felt more like what the article outlined. To show where imgs.ai gave me interesting results and where it fell short, let me walk you through my experimental searches.

I started with the image from Vermis I: Lost Dungeons and Forbidden Woods I digitized in my last post.⁵

I knew this wouldn’t necessarily be an easy image with its dithered graphics, but I thought it was fair to work with an image I am actively using in my art history study. While imgs.ai can only show results from the databases it has access to, unlike some other digital art history projects, it does not limit itself to a specific time period or type of art. So I didn’t expect it to return other video game related results necessarily, but I did expect it to be able to get an accurate read on the image’s general contents since testing it with an AI alt text tool had produced flawed but not totally off-base results.

imgs.ai has four feature extractors to choose from for each search, each with a different focus: “stylistic similarity,” pose, “raw color data,” and CLIP which allows combining an image with a text prompt.⁶

Results were mostly okay if not super useful until I switched to the pose extractor.

Well that’s a choice. There weren’t only but a variety of objects that had no relation to the pose regardless of the database used. The chairs would take on an added strangeness with my next search though.

I decided to try another image. One I took directly from the Smithsonian collection.

The results were better but still not great. I only got results that were clearly connected to the search image but still novel enough to be useful when I added “videogame” to my CLIP search.

All was well until I tried the pose feature extractor again.

Oh no.

Ohhhh nooooooooo.

I wracked my brain for the cause of this seating-based poltergeist haunting my results. Was it a similarity of a curve in the chair to the curve in the image? Then why were some of the EXACT SAME CHAIRS in the results from both starting images? Other more obvious connection images were not shared between results.

Away from the chairs, most results made sense as to why they were selected. They just didn’t provide anything useful beyond confirmation that yes, it has that color or a frame or someone in armor. Some images I found really intriguing, and they made me think of my image in a different way. It was most successful again when CLIP added texted to the search, in this case, “Skeleton.”

That was when I realized something important. While the point of imgs.ai is not to have an autonomous AI match images, but rather to include the human user in the research loop actively,⁷ I was doing the heavy lifting for the AI instead of with it. I was taking an image that seemed random to me and taking on the task of finding the connections and assigning meaning. In some ways, this is great. The human should be the one doing those tasks. But I was also struck that I could have done the same cognitive work and gotten the same benefit if the AI had served me completely random images from the collection and told me they were related to my image.

Two more images gave me additional insight into a major issue. Olivia Butler’s typewriter which appeared for both images…

…and the immense amount of beads that showed up search Atari 2600.

The computer wasn’t “seeing” the artworks as objects but as the images themselves. So not only was the background included but it was given as much consideration as the art. The tool appears to search hierarchically level parts, not the whole.

Perhaps if I had access to the GRAD-CAM to show me where the AI was seeing similarity,⁸ I would have had a richer experience, but either I could not find that feature or it was behind the login gate. Ultimately, the need driving the creation of the tool seems valid. But until OpenAI understands (not just consumes) Emma Stanford’s “A Field Guide to Digital Surrogates”,¹⁰ imgs.ai will be like building a chair with a staple gun: some chance of surprise success, maybe worth the experience, but you can’t make an Eames with it.