Can AI really give us insight into lost masterpieces?
In 1945, the fire claimed three of Gustav Klimt’s most controversial paintings. Commissioned in 1894 for the University of Vienna, the “faculty paintings” – as they were called – bore no resemblance to any of the Austrian Symbolist’s earlier work. As soon as he introduced them, critics were in turmoil over their dramatic break with the aesthetic of the time. University professors immediately rejected them and Klimt withdrew from the project. Shortly after, the works found their place in other collections. During WWII they were placed in a castle north of Vienna, but the castle burned down and the paintings probably accompanied it. Only black and white photographs and writings from the time remain today. Still, I look them straight in the eye.
Well, not the paintings themselves. Franz Smola, a Klimt expert, and Emil Wallner, a machine learning researcher, spent six months combining their expertise to revive Klimt’s lost work. It was a painstaking process, which started with these black and white photos, then incorporated artificial intelligence and dozens of information about the painter’s art, in an effort to recreate what these lost paintings could have done. look like. The results are what Smola and Wallner show me – and even they are amazed by the captivating technicolor images produced by the AI.
Let’s be clear: no one is saying this AI brings back Klimt’s original works. “It’s not a process of recreating real colors, it’s about recoloring the photographs,” notes Smola. “The medium of photography is already an abstraction from real works. Machine learning provides a glimpse into something thought to be lost for decades.
Smola and Wallner find it delicious, but not everyone supports AI to fill those voids. The idea of machine learning recreating lost or destroyed works is, like the faculty paintings themselves, controversial. “My main concern is with the ethical dimension of using machine learning in the context of conservation,” says art restorer Ben Fino-Radin, “due to the sheer volume of ethical and moral issues that have plagued the field of machine learning “.
Certainly, the use of technology to revitalize works of human art is fraught with thorny questions. Even if there was a perfect AI capable of determining which colors or brushstrokes Klimt could have used, no algorithm can generate an author’s intention. Debates on this subject have raged for centuries. In 1936, before Klimt’s paintings were destroyed, essayist Walter Benjamin opposed mechanical reproduction, even in photographs, claiming that “even the most perfect reproduction of a work of art is lacking. ‘one element: its presence in time and space, its uniqueness. existence where it is. This, writes Benjamin in The work of art in the age of mechanical reproduction, is what he calls the “aura” of a work. For many art lovers, the idea of a computer reproducing this intangible element is absurd, if not downright impossible.
And yet, there is still a lot to learn from what AI can do. The faculty’s paintings played a central role in Klimt’s development as an artist, a crucial bridge between his earlier more traditional paintings and his more recent, more radical works. But what they looked like in color has remained shrouded in mystery. This is the puzzle that Smola and Wellner were trying to solve. Their project, organized via Google Arts and Culture, was not about perfect reproductions; it was to give a glimpse of what is missing.
To do this, Wallner developed and trained a three-part algorithm. First, the algorithm was fed by a few hundred thousand art images from the Google Arts and Culture database. It helped him understand objects, artwork, and composition. Then it was taught specifically in Klimt’s paintings. “This creates a bias towards its colors and patterns over the period,” says Wallner. And finally, the AI received color clues for specific parts of the paintings. But without color references to paintings, where do these clues come from? Even Klimt Smola’s expert was surprised at the amount of detail revealed by the writings of the time. Because the paintings had been seen as so sordid and strange, critics tended to describe them in detail, right down to the artist’s color choices, he says. “You can call it an irony of history,” says Simon Rein, the project’s program manager. “The fact that the paintings caused a scandal and were rejected puts us in a better position to restore them because there was so much documentation. And those types of data points, if built into the algorithm, create a more accurate version of what these paintings likely looked like at the time.“
The key to this precision lies in the association of the algorithm with the expertise of Smola. His research revealed that Klimt’s work during this period tended to have strong patterns and consistency. The study of paintings existing before and after the Faculty of Paintings provided clues to the recurring colors and patterns in his work at this time. Even the surprises Smola and Wallner encountered are corroborated by historical evidence. When Klimt first showed his paintings, critics noted his use of a red that was, at the time, rare in the artist’s palette. Corn The three ages of women, painted shortly after the faculty paintings, boldly uses a red, a color Smola believes to be the same color that caused an uproar when she was first seen in the faculty paintings. The writings of the time also raise a tone and a cry about the shocking green sky in another painting of the faculty. The association of these writings with Smola’s knowledge of Klimt’s particular palette of greens, when fed into the algorithm, is what produced one of the first surprising images of AI.