How to Use Your Smartphone to Cope With Vision Loss

To make similar changes to Google Assistant go to Settings > Google > Settings for Google Apps > Search, Assistant and Voice, and choose Google Assistant. You may want to tap Lock Screen and toggle on Assistant Reponses on Lock Screen. If you scroll down, you can also adjust the sensitivity, toggle on Continued Conversation, and choose which Notifications you want Google Assistant to give you.

First launched in 2019, the Lookout app for Android enables you to point your camera at an object to find out what it is. This clever app can help you to sort mail, identify groceries, count money, read food labels, and perform many other tasks. The app features various modes for specific scenarios:

Text mode is for signs or mail (short text).

Documents mode can read a whole handwritten letter to you or a full page of text.

Images mode employs Google’s latest machine-learning model to give you an audio description of an image.

Food Label mode can scan barcodes and recognize foodstuffs.

Currency mode identifies denominations for various currencies.

Explore mode will highlight objects and text around you as you move your camera.

The AI-enabled features work offline, without Wi-Fi or data connections, and the app supports several languages.

Apple has something similar built into its Magnifier app. But it relies on a combination of the camera, on-device machine learning, and lidar. Unfortunately, lidar is only available on Pro model iPhones (12 or later), iPad Pro 12.9‑inch (4th generation or later), and iPad Pro 11‑inch (2nd generation or later). If you have one, open the app, tap the gear icon, and choose Settings to add Detection Mode to your controls. There are three options:

People Detection will alert you to people nearby and can tell you how far away they are.

Door Detection can do the same thing for doors, but can also add an outline in your preferred color, provide information about the door color, material, and shape, and describe decorations, signs, or text (such as opening hours). This video shows a number of Apple’s accessibility features, including Door Detection, in action. 

Image Descriptions can identify many of the objects around you with onscreen text, speech, or both. If you are using speech, you can also go to Settings > Accessibility > VoiceOver > VoiceOver Recognition > Image Descriptions and toggle it on to enable detection mode to describe what is depicted in images you point your iPhone at, such as paintings.

You don’t need a Wi-Fi or data connection to use these features. You can configure things like distances, whether you want sound, haptics, speech feedback, and more via the Detectors section at the bottom of Settings in the Magnifier app.

Guided Frame is a brand-new feature that works with TalkBack, but it’s currently available only on the Google Pixel 7 or 7 Pro. People who are blind or low-vision can capture the perfect selfie with a combination of precise audio guidance (moving right, left, up, down, to the front, or the back), high-contrast visual animations, and haptic feedback (different vibration combinations). The feature tells you how many people are in the frame, and when you hit that “sweet spot” (which the team used machine learning to find), it counts down before taking the photo.

The Buddy Controller feature on iPhone (iOS 16 and later) allows you to play along with someone in a single-player game with two controllers. You can potentially help friends or family with vision impairment when they get stuck in a game (make sure you ask first). To turn this feature on, connect two controllers and go to Settings > General > Game Controller > Buddy Controller.

While this guide cannot cover every feature that might help with vision impairment, here are a few final tips that might be handy.

You can get spoken directions when you are out and about on an Android phone or iPhone, and they should be on by default. If you use Google Maps, tap your profile picture at the top right, choose Settings > Navigation Settings, and select your preferred Guidance Volume.

Both Google Maps and Apple Maps offer a feature where you can get a live view of your directions superimposed on your surroundings by simply raising your phone. For Apple Maps, check in Settings > Maps > Walking (under Directions) and make sure Raise to View is toggled on. For Google Maps, go to Settings > Navigation Settings, and scroll down to make sure Live View under Walking Options is toggled on.

If you are browsing the web on an Android device, you can always ask Google Assistant to read the web page by saying, “Hey Google, read it.”

You can find more useful advice on how technology can support people with vision loss at the Royal National Institute of Blind People (RNIB). To find video tutorials for some of the features we have discussed, we recommend visiting the Hadley website and trying the workshops (you will need to sign up).