Reviving the past with artificial intelligence

While studying John Singer Sargent’s paintings of wealthy women in 19th-century society, Jessica Helfand, a former Caltech artist in residence, had an idea: to search census records to find the identities of those women’s servants. “I thought, ‘What happens if I paint these women in the style of John Singer Sargent?’ It’s a sort of cultural restitution,” Helfand explained, “reverse engineering the narrative by reclaiming a kind of beauty, style, and majesty.”

To recreate a style from history, she turned to technology that, increasingly, is driving the future. “Could AI help me figure out how to paint, say, lace or linen, to capture the folds of clothing in daylight?” Helfand discussed her process in a seminar and discussion moderated by Hillary Mushkin, research professor of art and design in engineering and applied science and the humanities and social sciences.

The event, part of Caltech’s Visual Culture program, also featured Joanne Jang, product lead at DALL-E, an AI system that generates images based on user-supplied prompts.

While DALL-E has a number of practical applications from urban planning, to clothing design, to cooking, the technology also raises new questions. Helfand and Jang spoke about recent advancements in generative AI, ethical considerations when using such tools, and the distinction between artistic intelligence and artificial intelligence.

Artistic intelligence versus artificial intelligence

Jang and Helfand identified three components to art. Machines, they said, are good at two of them: the ability to perceive data and the ability to make connections. The third component, which machines cannot be taught, is the subjective human experience.

The capabilities and limitations of generative technologies such as DALL-E are based in how they work. DALL-E 1 was a generative pre-training (or GPT) model, which is essentially an autocomplete technology. Its successor, DALL-E 2, is a diffusion model that starts with a random pattern of dots that it slowly arranges to generate the prominent features of an image.

For Helfand, the tool also offers a way to explore new realms of artistic possibility. In the 19th century, eugenicist Francis Galton produced flattened composite portraits of “criminal types” by combining police images of multiple people. Helfand used generative AI to deconstruct the composites and recreate each individual’s true likeness.

“What happens when you actively [give dimension to] somebody through a story that you know, is that you can begin to build a different story, one that moves closer to a kind of fidelity to that person.” In this way, generative AI enables artists to do the opposite of what Galton did: accruing details rather than averaging them—in an effort to restore dignity and humanity in a portrait, Helfand said.

Art and the ethics of AI

Helfand and Jang also addressed the ethical questions that arise as AI technologies become increasingly powerful. For example: How does DALL-E make decisions about what a portrait of a CEO looks like? Will each variation be a white male? While that may be statistically accurate, is it ethical?

Another issue surfaced when scientists decided to prune DALL-E’s training dataset to remove all sexual content: they saw a 25 percent drop in representation of women. That meant the model would be 25 percent less informed when creating images of women due to having less training data. So, researchers are faced with questions of how to re-weight images of women.

Such questions around generative AI systems such as DALL-E and ChatGPT, and their implications for humanity are being explored in the Caltech community. “There are many people in our community who are excited about machine learning. Conversations like this are opportunities for people to consider what problems technology can solve and what is the responsibility of humans,” Mushkin said. “There is an appetite at Caltech to talk to people working outside of the sciences about these issues. When we offered researchers a chance to meet individually with the speakers, sign-ups were full within an hour. They really want to bat around ideas with people in related fields who can bring a new perspective to what they are doing.”