When A Machine Learning Algorithm Studied Fine Art Paintings, It Saw Things Art Historians Had Never Noticed
Artificial intelligence reveals previously unrecognised influences between great artists
The task of classifying pieces of fine art is hugely complex. When examining a painting, an art expert can usually determine its style, its genre, the artist and the period to which it belongs. Art historians often go further by looking for the influences and connections between artists, a task that is even trickier.
Bacon
So the possibility that a computer might be able to classify paintings and find connections between them at first glance seems laughable. And yet, that is exactly what Babak Saleh and pals have done at Rutgers University in New Jersey.
These guys have used some of the latest image processing and classifying techniques to automate the process of discovering how great artists have influenced each other. They have even been able to uncover influences between artists that art historians have never recognised until now.
The way art experts approach this problem is by comparing artworks according to a number of high-level concepts such as the artist’s use of space, texture, form, shape, colour and so on. Experts may also consider the way the artist uses movement in the picture, harmony, variety, balance, contrast, proportion and pattern. Other important elements can include the subject matter, brushstrokes, meaning, historical context and so on. Clearly, this is a complex business.
So it is easy to imagine that the limited ability computers have for analysing two-dimensional images would make this process more or less impossible to automate. But Salah and co show how it can be done.
At the heart of their method, is a new technique developed at Dartmouth College in New Hampshire and Microsoft research in Cambridge, UK, for classifying pictures according to the visual concepts that they contain. These concepts are called classemes and include everything from simple object description such as duck, frisbee, man, wheelbarrow to shades of colour to higher-level descriptions such as dead body, body of water, walking and so on.
Comparing images is then a process of comparing the words that describe them, for which there are a number of well-established techniques.
Salah and co apply this approach to over 1700 paintings by 66 artists working in 13 different styles. Together, these artists cover the time period from the early 15th century to the late 20th century. To create a ground truth against which to measure their results, they also collate expert opinions on which of these artists have influenced the others.
For each painting, they limit the number of concepts and points of interest generated by their method to 3000 in the interests of efficient computation. This process generates a list of describing words that can be thought of as a kind of vector. The task is then to look for similar vectors using natural language techniques and a machine learning algorithm.
Determining influence is harder though since influence is itself a difficult concept to define. Should one artist be deemed to influence another if one painting has a strong similarity to another? Or should there be a number of similar paintings and if so how many?
So Saleh and co experiment with a number of different metrics. They end up creating two-dimensional graphs with metrics of different kinds on each axis and then plotting the position of all of the artists in this space to see how they are clustered.
The results are interesting. In many cases, their algorithm clearly identifies influences that art experts have already found. For example, the graphs show that the Austrian painter Klimt is close to Picasso and Braque and indeed experts are well acquainted with the idea that Klimt was influenced by both these artists. The algorithm also identifies the influence of the French romantic Delacroix on the French impressionist Bazille, the Norwegian painter Munch’s influence on the German painter Beckmann and Degas’ influence on Caillebotte.
The algorithm is also able to identify individual paintings that have influenced others. It picked out Georges Braque’s Man with a Violin and Pablo Picasso’s Spanish Still Life: Sun and Shadow, both painted in 1912 with a well-known connection as pictures that helped found the Cubist movement.
It also linked (above left) Vincent van Gogh’s Old Vineyard with Peasant Woman (1890) and Joan Miro’s The Farm (1922), which contain similar objects and scenery but have very different moods and style.
Most impressive of all is the link the algorithm makes between (below left) Frederic Bazille’s Studio 9 Rue de la Condamine (1870) and Norman Rockwell’s Shuffleton’s Barber Shop (1950). “After browsing through many publications and websites, we concluded, to the best of our knowledge, that this comparison has not been made by an art historian before,” say Saleh and co.
And yet a visual inspection shows a clear link. The yellow circles in the images below show similar objects, the red lines show composition and the blue square shows a similar structural element, say Saleh and co.
That is interesting stuff. Of course, Saleh and co do not claim that this kind of algorithm can take the place of an art historian. After all, the discovery of a link between paintings in this way is just the starting point for further research about an artist’s life and work.
But it is a fascinating insight into the way that machine learning techniques can throw new light on a subject as grand and well studied as the history of art.
Ref: arxiv.org/abs/1408.3218 : Toward Automated Discovery of Artistic Influence
Follow the Physics arXiv Blog on Twitter at @arxivblog, on Facebook and by hitting the Follow button below
