Recognizing situations
Several recent news photographs were uploaded to a system called imSitu, which tried to summarize the situation depicted in each image.
The system generates a list of possible verbs for each photo. Each verb is shown with a table of recognized objects and their roles in the image:
A number at right indicates how confident the algorithm is in each prediction.
The Times used the top four predictions for each image to write simple sentences.
Predicted:
Subject
Verb
Other objects
A helicopter in the air dropping an unknown item from the helicopter to the land. (Confidence: 93.8%)
A person hoisting a flag from the land using a rope outdoors. (1.2%)
Smoke cresting outdoors. (0.6%)
A tree branching in a painting. (0.5%)
A soccer player scoring on a playing field. (Confidence: 57.3%)
A soccer player tripping over a foot on a playing field. (7.4%)
A man hitting a ball with a bat in a stadium. (6.1%)
A basketball player blocking another basketball player with a hand on a basketball court. (5.0%)
A woman plunging her body into water in a swimming pool. (Confidence: 73.8%)
A woman swimming in a swimming pool. (20.0%)
A woman submerging her body in water in a swimming pool. (1.5%)
A man releasing a fish in water. (1.4%)
A woman crowning a woman with a tiara. (Confidence: 97.8%)
A woman shopping for apparel in a shop. (0.4%)
A mime miming inside. (0.2%)
People singing. (0.1%)
People running on a racetrack. (Confidence: 48.2%)
A runner competing in track on a racetrack. (12.8%)
A runner tripping over a hurdle on a playing field. (10.9%)
Deer chasing deer on a field. (4.7%)
A person knocking on a door. (Confidence: 29.5%)
A person in a room opening a door with a hand. (13.7%)
A man kicking a door in a room. (8.6%)
A person inside locking a door with a key. (5.8%)
A horse racing in a field. (Confidence: 43.5%)
A horse jumping from land to land over a hurdle on a racetrack. (38.2%)
A woman riding a horse in a stadium. (12.3%)
A jockey competing in a horse race on a path. (2.9%)
Fire destroying a building with fire in the street. (Confidence: 85.2%)
An unknown object flaming outdoors. (7.7%)
A chef burning food in a kitchen. (5.1%)
Lava erupting outdoors. (0.9%)
A woman curtsying outdoors. (Confidence: 41.4%)
A couple dancing on a stage. (10.1%)
A soldier saluting outside. (7.6%)
A man waving his arm outdoors. (5.1%)
A soldier saluting the American flag outside. (Confidence: 92.0%)
A flag flapping outdoors. (1.3%)
A man hitchhiking on the highway. (0.9%)
A soldier parading in the street. (0.8%)
A man barbecuing meat outdoors. (Confidence: 29.2%)
A man pushing a baby buggy with his hand in a hospital. (10.1%)
A man wheeling baggage outdoors. (9.4%)
A man bowing outside. (7.7%)
A man cleaning with a towel in a kitchen. (Confidence: 73.5%)
A person dusting the floor with a rag. (3.6%)
A person vacuuming the floor with a vacuum. (2.4%)
A man peeing on the floor in a room. (0.8%)
A soldier parading in the street. (Confidence: 76.1%)
A marching band marching in the street. (15.3%)
A policeman confronting people on a road. (2.5%)
People protesting on a road. (1.2%)
A child sliding on sand to the bottom of a hill. (Confidence: 88.3%)
A man painting a wall with a paint roller in a room. (3.6%)
A person descending a slope with a ski. (2.7%)
A skier skiing on a slope. (0.8%)
A fireman dousing a fire with water outdoors. (Confidence: 39.0%)
A fire destroying a forest with fire outdoors. (28.8%)
A fireman extinguishing a forest outdoors. (12.6%)
A firework glowing outdoors. (2.5%)
Try your own photos
The examples above hint at the evolving state of the technology. imSitu can currently recognize more than 500 verbs, but may falter when trying to describe less familiar actions or scenes.
Upload and test your own images at imSitu.org/demo.
