What is Picture Smart?

Picture Smart is a feature built into JAWS (starting with the 2019 version) that allows users to submit images for analysis. Why would you want to analyze a photo? To get a textual description and info of the picture. Below are the keyboard commands to use Picture Smart. As you can see, there are different keystroke combinations contexts we may encounter images.

Describe a photo acquired from the Pearl Camera or a flatbed scanner | Insert + Spacebar, then P, then A

Describe a selected image in Windows File Explorer | Insert + Spacebar, then P, then F

Describe the current control | Insert + Spacebar, then P, then C

Describe an image on the Windows Clipboard | Insert + Spacebar, then P, then B

Example of Using Picture Smart: Describing Current Control

We’ll try it out. Below is a picture of me. The photo is a selfie of me giving a thumbs up to the camera.

We’ll start out as if we had this image open in Photos app in Windows 10 (that is a default photo viewer in Windows 10, so this would be what we would likely get if we opened the image file). So, for this one, I am going to use the command to get a description of the current control (Insert + Spacebar, then P, then C). When I do this, JAWS will say “Picture Smart in Progress”. Then, a new window opens that contains the textual analysis of my image. This window is titled Picture Smart Results. You can Alt + Tab away from the Window and return to it. Below is what the results window tells me about my image:

Caption is a man in glasses looking at the camera.

These tags describe the photo

human face, person, smile, text.

These tags probably describe the photo

glasses, indoor.

This tag possibly describes the photo

man.

So, how did it do? Pretty well. The picture is me smiling and giving thumbs up. From the results, we got the following info pointing to me: human face, person, smile, glasses, and man. We also learned the image is indoors. The results also said text, which may lead you to believe that I might have added some text to this photo. I did not. What that seems to be pointing to is the text in the Photos app interface. We can see this for ourselves in a moment.

At the bottom of the picture Smart Results Window, there is a link labeled More results. If we activate this link, we’ll hear JAWS once again say “Picture Smart is in Progress”. Then our Picture Smart Results Window gets updated with expanded results. Below are those expanded results.

Caption is a man in glasses looking at the camera.

Total number of faces in this photo is 1.

This text appears in the photo

Photos – IMG 6158.JPG
A See all photos

• Add to
  A Edit & Create v
  IA Share.

These objects appear in the photo

Ceiling fan, Glasses.

This object probably appears in the photo

Person.

These tags describe the photo

Ceiling fan, Eye, Eyebrow, Glasses, Hand, human face, person, Smile, text, Vision care.

These tags probably describe the photo

Gesture, Happy, indoor, Jaw.

This tag possibly describes the photo

man.

As you can see, the More results link gave us a bit more information on the image. Specifically, we learned what that text result was pointing to: the text in the Photos app interface (i.e. Add to, A Edit & Create v, IA Share). We also got some info about the background (Ceiling fan). It wasn’t able to tell us that I was making a thumbs up, but it did include gesture as a possible tag.

Example of Using Picture Smart: Describing Selected Image in Windows File Explorer

Lets try again, but this time we will use the command to launch picture smart from Windows File Explorer (Insert + Spacebar, then P, then F). Very similar as our last command, we get a Smart Results Window. Below is the information it gives us:

Caption is a man wearing glasses and smiling at the camera.

These tags describe the photo

glasses, human face, man, person.

These tags probably describe the photo

indoor, selfie, smile.

Did we learn anything new? Not really, pretty similar to when we used the control command from the Photos app. However, this time we did get the word Selfie, which was missing from our first attempt with the control command. What happens when we select more results this time? Take a look below.

Caption is a man wearing glasses and smiling at the camera.

Total number of faces in this photo is 1.

These objects probably appear in the photo

Ceiling fan, Glasses, Person.

These tags describe the photo

glasses, Hand, human face, Light, man, Organ, person, Photograph, Smile, Thumbs signal, Vision care.

These tags probably describe the photo

Ceiling fan, Gesture, indoor, selfie.

Again, we have pretty similar results as we did with the control command. However, this time we did get the tag thumbs signal, which I have to interpret as meaning thumbs up.

So, that is the JAWS Picture Smart. But what did they add in 2021?

JAWS 2021 Picture Smart Improvements

Below are the features added to Picture Smart in JAWS 2021.

Describing images on web pages

From Freedom Scientific: “If focused on an image that is part of a web page, such as a photo on Facebook, pressing INSERT+SPACEBAR, P followed by C now describes the photo.”

Let’s try this out. I went to my Facebook page and found a picture I posted of my three sons (5, 3, and 1 at the time) sitting on a couch watching a movie. Below is the actual Image:

Now, before we use Picture Smart, lets see what Facebook’s automatic Alt Text describes this picture as: May be a picture of one person and baby. Not very descriptive or entirely correct. Now, lets try it with Picture Smart.

Caption is a baby sitting on a couch.

These tags describe the photo

baby, boy, child, clothing, human face, indoor, person, sofa.

These tags probably describe the photo

infant, smile.

These tags possibly describe the photo

newborn, seat.

This tag vaguely describes the photo

toddler.

As you can see, the Picture Smart gave us a little more info than Facebook. Now, lets check out the more info link:

Caption is a baby sitting on a couch.

Total number of faces in this photo is 3.

These objects probably appear in the photo

Person, Top.

This object possibly appears in the photo

Clothing.

These tags describe the photo

baby, boy, child, clothing, Comfort, Couch, Eye, Facial expression, Furniture, Head, human face, indoor, Organ, person, Smile, sofa.

These tags probably describe the photo

infant, Living room, Sharing.

These tags possibly describe the photo

newborn, seat.

This tag vaguely describes the photo

toddler.

Like before, More results gives us the most complete description of the image. Without More results, you’d likely be unsure how many people the image contained. Note that you need to navigate to the image on the web for the command to work correctly. If you try it in the wrong place, JAWS will tell you that the control is not an image.

Submitting images to multiple services to help improve accuracy 

From Freedom Scientific: “By default, images are submitted to Microsoft for analyzing. However, the Results Viewer now contains a More Results link which submits the image again to additional services for analyzing and displays an updated description. You can also add SHIFT to a Picture Smart command to use multiple services. For example, INSERT+SPACEBAR, P followed by SHIFT+F, SHIFT+C, or SHIFT+B.”

We’ve already seen what the More results link can bring us. From the two images we’ve looked at, it seems like More results gives us a better idea of the image than the standard results. Using the shift key as outlined above, you might save yourself some time and jump straight to more results.

Using Picture Smart in multiple languages

From Freedom Scientific: “If you are using JAWS or Fusion in a language other than English and you attempt to use Picture Smart, JAWS and Fusion will use machine translation to display descriptions in the particular language. You can also manually choose from 38 languages for displaying results, configurable using the new Picture Smart Language option in Settings Center.”

More languages are always a great feature.

Update: Use Picture Smart from File Explorer

You can perform Picture Smart scanning without having to remember the keyboard command! How? If you have a file in Windows File Explorer, simply navigate to it and open the context menu (Shift + F10). In the menu, you’ll see an option Picture Smart with JAWS. Simply activate this option and JAWS will perform Picture Smart on your image file.