The time is near for updates to all of Apple’s operating systems, and The Rocket Yard is here to give you the knowledge you need to upgrade with confidence. In the recent Road to macOS 12 Monterey series, we gave you a look at six of the marquee features of the Mac operating system upgrade. In this Get Ready for iPadOS 15 series, we’ll focus on the improvements being made to the iPad’s operating system. Today we’ll look at Visual Look Up, a powerful search tool that is part of iPadOS 15.
What Is Visual Look Up?
Let’s say that you’re looking at a picture. It doesn’t matter where you’re looking at the image; it could be something sent to you in Mail or Messages, a photo on a web page, or perhaps a picture in your Photo Library. That picture shows an object of some sort, and you want to know what it is.
Visual Look Up analyzes each photo to “see” what is in it. Through machine learning magic, Visual Look Up identifies what is in the picture, then does a quick search to provide information about the object.
The feature is new and I’ve found that in the beta versions of iPadOS 15, some objects are either not recognized or they are labeled incorrectly. I have photos of bison, deer, and moose in which the animal is obviously the focus of the picture, yet none of those large mammals was recognized. A photo of a nearby rock formation in the South Denver foothills was identified as Devils Tower in Wyoming. Oddly enough, several photos of Devils Tower were not recognized while others were.
My guess is that as Visual Look Up is used by more people, the accuracy should increase as the visual “database” is refined.
Examples of Visual Look Up in action
A few examples are in order to show how this new iPadOS 15 tool works. To begin with, I went through my Photo Library and tapped on a picture of a flower:
On the left, you see the image of a peony, and the usual “Info” button ( ℹ︎ ) looks different. There are two small “stars” to acknowledge that Visual Look Up recognizes something in the image. Tapping the Info button on the right highlights it, and two other indicators appear. There’s a tiny leaf icon on the flower, indicating that Visual Look Up recognizes the object as a flower. There’s also a Look Up link (in the red rectangle) to tap for more info.
When the link is tapped, here’s what Visual Look Up provides courtesy of Siri:
It’s quite impressive. Visual Look Up seems to do very well with plants like flowers, shrubs, and trees.
The feature also works smoothly with many landmarks. It recognized One World Trade Center in New York City, The Space Needle in Seattle, Mount Rushmore in Wyoming, and more. On the other hand, it failed to recognize many natural landmarks. The Grand Tetons, Niagara Falls, and Yellowstone’s Old Faithful Geyser failed the Visual Look Up test.
It recognizes dogs and cats, probably the most photographed animals in the world. The feature even tries to assign a breed to our feline and canine friends. But birds? Sharp, clear photos of bald eagles, owls, gulls, crows, turkeys, emus, ostriches, hummingbirds, and even chickens failed — although they were identified as birds by a Photo Library search of the keyword “birds”, no more detail was available.
Visual Look Up in Safari and other iPad apps
As mentioned, Visual Look Up works in other apps including Safari. However, there’s no way to tell if it “recognizes” an image other than to tap and hold on it. If the Look Up item shows up in the contextual menu, you can get detailed info with a tap.
I was able to successfully use Visual Look Up in Safari… sometimes. For example, I could pull up an image of a known landmark, tap and hold, and see the details of the item. The same thing worked with famous paintings. Once again, dog and cat pictures were immediate Visual Look Up hits. Snakes? Nope. Ducks? No. Famous people? Nada.
In the Mail app, the same tap-and-hold gesture brought up a Look Up menu item, but only on specific types of photos. The same photo embedded in a Note didn’t produce the expected Look Up item. That indicates that the feature isn’t yet “live” in apps other than Photos, Safari, and Mail. Will it be available in all apps that can display images in the future? Who knows…
Also available in a convenient pocket size
Visual Look Up is also available in iOS 15, which should be ready for upgrades around the same time as iPadOS 15. You’ll be able to use the feature with other new abilities coming in macOS 12 Monterey and iOS 15 like Live Text.
Stay tuned for more articles about the new features of iPadOS 15 in the upcoming days.
Any news or comments regarding a transcription app for the iPhone/iPad voice memos?