LookTel Recognizer Documentation
July 30, 2012 – Version 6.1
- Requirements
- Using Recognizer
- Storing and Naming Images
- Recognizing Items
- Getting the Best Images
- Useful Tips and Tricks
- Warnings
- Barcode Scanning
- Barcode Scanning Tips
- Backing Up and Restoring the Library
- Exporting Your Library
- Importing Items from Email
- Other Considerations
- Recognizer Privacy
Welcome to Recognizer
The LookTel Recognizer App is intended for persons who are
blind, or who have sufficiently impaired vision to need special
labeling to be able to identify cans, packages, ID or credit
cards, DVDs, and similar items. It permits a person to store
images of objects in a library or database, and have the IOS
device "recognize" these items later, by scanning the items
with the camera. A barcode scanner is also included to provide
additional labeling help.
The app also permits the recognition of signage, and may be
used to identify commercial establishments, landmarks, and
other features of the environment to confirm a destination, or
point of interest. The use as an orientation aid is an evolving
capability. At present, it is best used in semi-familiar and/or
familiar areas where a confirmation or verification is
helpful.
The Recognizer App is an app for IOS devices made by Apple that have a camera, and are running IOS 4.3 or greater. These include the iPhone 3GS, iPhone 4, iPhone 4S, the iPad 2, the New iPad, and iPod Touch 3 and 4, as of spring, 2012.
Once the app is downloaded on to your IOS device, you must
create an image library. Recognizer does not at present, have
access to a pre-built library. You will probably need sighted
assistance in creating the image library if you have a severe
visual impairment, or are blind.
Creating the library is easy, and can be fun, especially if you
have a friend or family member who is a shutterbug. While it
may seem to take a lot of time, it takes far less than most
people imagine to create an excellent library of images. Once
the library exists, less and less time is required of a sighted
person, as you will find Recognizer can already identify most
items that are used daily reliably and accurately every
time.
When you start the app, you will see that the main screen is
simple. It works well with VoiceOver, and is intended to be
used with VoiceOver most of the time. Some sighted assistants
may prefer to turn off VoiceOver while capturing images for
your library. The app works as well without VoiceOver turned
on.
The main screen has 4 buttons. These are:
* Settings
* Data
* Capture Image
* Start/Stop barcode scanning
The Settings button opens a menu of items:
* The number of elements or items in your Library,
* A setting for Recognition On/Off. This setting permits you to
turn off the recognition of items while you are attempting to
capture images for your library of items.
* Enter description using audio – tab,
- * Enter description using text – tab.
These tabs let you set Recognizer to use either audio labels on your items, or text labels. The use of audio labels is convenient and fast. The use of text labels permits using Recognizer with a braille output device. Text labels can be entered when working in environments with a high ambient noise, and will be read clearly later.
* Order by date/description
This is a toggle, with a default of date. Leaving this
set to date will keep the familiar organization within your
library. Toggling this setting to
“description”, will cause the items in your library
to be displayed by either a text or audio description.
You will have to double or split tap to open the detail screen
to make changes, or to see the date of image capture
reliably.
* Documentation - this manual.
* The LookTel Recognizer Privacy Statement,
* An Opt-In On/Off switch. This permits you to share in the
improvement and development of Recognizer by turning this
option to "On". Anonymous information from Recognizer is shared
with LookTel that assists the team in diagnosing problems,
assessing improvements, and implementing new features. This is
the same Opt-In switch and Privacy statement you will see when
you first open the Recognizer app after obtaining Recognizer.
You will see it again if you reinstall Recognizer. If you
skipped it then, you can now choose to Opt-In and participate
in the development and improvement of Recognizer.
* The version number for Recognizer.
The data button opens the data library. This is where captured
images are stored. When an image is captured, it is placed in
this library. It is possible to open the library, locate an
image, delete it, or edit either the text or audio tag on the
image, should you wish to do so. The screen allows listening to
the audio and text labels. Images are stored with their
names being the dates they were added to the library, unless
you have updated those images with text tags. Then, the
text labels will be the names of the items.
Since items in your library are listed in this way, by dates,
from the most recent, which are located at the top of the
screen, to the oldest items, which are located at the end of
the library, it is very easy to remove or edit the last entry
in your library should you choose to do so.
To remove the last image you scanned:
Since items in the library are listed from the oldest, to the newest, the most recent item captured will always be at the top of the first page in the library. To delete the last image captured, open the database by using a double or split tap on the data button of the main screen. Flick to the first item, and listen to the audio description. If this is the item you wish to remove, you can then do so through a few simple steps. The steps vary according to your settings.
If you chose to retain the display of items by date, you can easily flick or touch the screen to find the first item in the library, and see that it is the item you just captured. You can play the audio description, or obtain more details and delete the audio description, or the item itself.
If you chose to make a setting change to display all your items by description, the dates will not appear immediately as you flick through your library, or touch the screen. You will hear the text or audio description instead. To see the date, you must double or split tap on a description. You will then have a new screen that permits you to delete an item description and provide a new one, or delete the item entirely.
A Last Recognized Image button is available in the upper right
of the Data screen immediately after an image has been scanned.
This will permit you to delete or edit the last recognized item
in your library, without the need for you to go searching
through your library for it yourself. If you have just used
Recognizer to identify a can of beans, which you added to your
library a week earlier, you could use this feature to bring up
that item and easily change its audio description or even
delete the item entirely if you choose.
When you open the detail screen, or split/double tap on a description to obtain the detail page, you will notice an option to email that image and description. Individual items can be sent through email with their descriptions. This same screen permits you to provide a text description, and replace the audio descriptions in your library, should you wish to do so. This may be desirable if the initial audio description was difficult to understand, or you may wish to share that image with a person who cannot hear the audio description.
There is an additional button at the bottom of the screen that
permits exporting of the library through email. This is
discussed in a later section devoted specifically to this
capability. The button is labeled as "Share Items".
The capture image button starts the sequence for photographing
a given item. This process will be described in more detail in
the next section.
Begin by choosing the items you wish to recognize and
placing them on a neutral background--one with no pattern or
unusual visual characteristics. Have the sighted person assist
in making the photos, as the best photos are ones in which the
object fills the iPhone view field, but in which the entire
image is shown. Lighting is best if it is "normal"--as in good
for a sighted person to read or handle the items in an ordinary
way. Focusing on the unique visual qualities of an item,
minimizing background, and using good lighting, will produce
high quality images for reliable and fast recognition.
While Recognizer does have a capability to adjust lighting by
turning on the torch when lighting is deficient, this
capability is best used when you cannot know the lighting
around you, and the work area. Like MoneyReader, this
capability is automatic, and will only let you know it is
working when lighting is less than adequate.
Double-tap the Capture Image button to take a photo. Note that
a double-tap is required even if VoiceOver is off. Once an
image is photographed, the app will ask you to record a name
for the image. You have up to 60 seconds to name, or describe
the item, which tells what the object is, i.e., "Diced
Tomatoes", or "green beans". The recording will cut off before
60 seconds have passed if you stop speaking. You may also stop
the recording by double or split tapping on the
screen.
Alternatively, you may choose to enter a text label for the item description. You can set your preference in Settings, and Recognizer will then prompt for either audio or test, depending on your choice in settings.
If you choose to use an audio description, an additional method
for stopping the recording has been added. You may shake your
iPhone, to stop the recording of the description.
These methods are helpful if there is background noise. It is
best to record in a quiet place to minimize background sounds
in your recordings, or use a text label, which will not reflect
background sounds.
For those using audio descriptions, the longer interval has
been suggested by persons using Recognizer, in order to give
time for recording cooking directions, nutritional information,
or a reference to other relevant information.
Examples might include: Frozen Mixed Vegetables - Cook in a
microwave at full power for 7 minutes in a closed dish with no
water added. Add salt and butter to taste.
Salt Free Tomato Paste - Save for Mom’s birthday party,
to make a recipe that Mom can eat.
Fresh Pear Halves - Save for food drive. This is preserved in
high fructose corn syrup, and has too many calories for my
diet.
The app will play back the name--audio label--that you gave the
image. You can then return to the main screen of the app, or go
to the Data screen, where you can edit the name by re-recording
it, or delete the image.
After images are stored and named in your device, you are
set to recognize them. This is as simple as holding the device
so that the object is in the view of the camera, and letting
the app name the image for you. It will play either your voice
recording or the text tag, when it recognizes the image.
If the audio description is too long for a quick
identification, you can touch another item to stop the playing
of the description.
Recognition generally happens quickly, with larger objects
being recognized more quickly than very small objects, and
feature-rich images being recognized more quickly than less
feature-rich images.
There are several basic concepts that can help to insure
that you have high quality images in your library. With a
little preparation, you can easily create excellent images for
later recognition, saving time and effort daily.
* Create a good work space for capturing images
This is as simple as having a clear area on which you might
place a neutral or all white poster board, drop cloth, or
sheet, so that you have a plain and featureless background for
all images.
* Have normal lighting. Excessive or too little lighting can
also work, but probably not as reliably. To insure that you
have the best quality of images, use normal lighting in which a
sighted person can easily read all items, even small
print.
Recognizer includes an automatic adjustment capability. It will
try to compensate for low light levels, and possibly alert you
if it cannot do so.
* When images are being made, focus the camera so that the
maximum amount of the image is in view, with little or no
background showing. This will prevent the background from later
being identified as an image.
Example: If a wood grain table is used as a background, and a
jar of jelly is on the surface, with the table captured as
background in the image, the table itself may later be
identified as the jelly. When the camera is focused on the
table, it Recognizer calls it "grape jelly", this is usually
what has happened.
Using a plain background, focusing on the jelly jar itself,
with no background, or a minimal background showing, will
produce better results.
* When capturing images of similar items, focus on the
difference among the items--leaving out logos, and brand names
that are alike.
There are several useful tips that we can pass on that will
enrich your experience with this app. The first is to
photograph each object with the maximum amount of the image in
the view field. This gives the app the greatest amount of
material from which to perform a recognition later. If this is
not possible, then choose the most feature-rich area of the
image to include. You will want to know something of the
orientation of the object, as you will need that to get the
best recognition later. It only makes sense that you have to
have the same portion of the object in view to recognize it.
Photographing one side, then viewing the opposite side will not
provide any recognition. This leads us to the next useful
tip.
If you choose to include canned items in your library, and many
of us will, here is a useful tip. On many, American canned
goods, the labels are rolled on in a clockwise direction,
assuming the top of the can to be up. This places the seam
always in one orientation with respect to the can, and even
more helpfully, tells you on which side of the can the image
will be in many instances. If you set the can on a counter top,
then orient it to have the seam at a 9:00 position viewing the
can from the top, the front image of the can will be at the
6:00 position--facing you! If you photograph the can in this
orientation, you only need one photo per can, and can orient it
quickly and easily for later recognition. You need not
photograph all sides of the can or bottle.
On some cans where the label is rolled on in the reverse
direction, the image is opposite. The seam will need to be at
3:00, for the image to be at 6:00. Walmart seems to have many
items of this kind.
For packaged goods, large diameter objects with no obvious seam
to orient them, you will want to make more than one image of
the object. You can name them as front panel, or back panel of
item.
This app can recognize frozen packaged goods. We did do tests
with packages of frozen vegetables, and we find it can
recognize them, though often more slowly, and with some
difficulty--still, it can do it with only a little
patience.
When doing similar items, you may need to do all items of a
group. For example, if you want to know the difference between
tomato paste, stewed tomatoes, diced tomatoes, and whole
tomatoes, you will need to do all of the cans. Doing one kind,
and later trying to find it in your cupboard can result in
errors, as others that were not "learned", can be mistaken as
the one you want.
Example: Scanning only diced tomatoes and not the others, can
mean that when you are looking in the cupboard, and choose a
can of whole tomatoes, it may be recognized as diced, when it
is not. This assumes the same brand applies to all of the cans.
Different brands may assist in having a more accurate
recognition.
This leads to the next tip:
When capturing images of items that all have the same brand
name, logo, and other graphic similarities, it is best to focus
the camera on the aspects of the images that are different
between otherwise like items. Try to leave out of the image,
the brand name, the logo, banner, background, or borders,
focusing instead on a photo of the contents, or a symbolic
representation that is unique to that can, bottle, package, or
container. This will produce faster and more accurate
recognition later.
Naturally, we include warnings with this app. This app
should never be used as the sole means to recognize
medications, nutritional supplements, or hazardous and toxic
substances. No matter how well it performs for you, please do
not use this app as your only means to determine what
medication you are taking, or what cleaning fluid you might
wish to use, bug sprays, paints, etc.
Our tests have demonstrated that smaller items, such as
medicine bottles, eye drop bottles, etc. are not recognized
with reliability or accuracy sufficient to making this a safe
practice. Yes, we can do it, but a backup method is always
recommended to insure your safety.
For your convenience, a barcode scanner is part of
Recognizer. There are times when no sighted person is available
to help make an image for your library, and you will wish to
identify that can that might be soup, or it might be fruit
cocktail. Recognizer includes a barcode scanner that may help
you to determine which it is before you open the can. Opening
the soup, when you intended to make a fruit salad, is not
helpful.
To use the barcode scanner, activate by double or split tapping
the button. Holding the iPhone 4-8 inches, 10-20 cm, from the
item, patiently scan that item to find the barcode. Recognizer
will signal you with a tone when it captures the barcode. It
will then go online, look up the item, or if it cannot find it,
go to a screen containing the barcode translation, and look it
up through Google for you. This may provide you additional
information, or it may not.
This capability is updated to permit looking in other databases
for the item. This should prove to be more satisfactory.
The database will continue to be upgraded to improve barcode
results, and offer additional capabilities.
It is essential to *SLOWLY* scan with the camera over the
item for a barcode. Moving too fast will cause Recognizer to
miss the barcode. Some barcodes may be present, but printed in
a way that is hard for Recognizer to capture. The barcode
capability is far from perfect. You will want to put items into
the Recognizer library and save yourself time and effort with
fast and reliable identification when you encounter that item
again.
The database used to look up barcodes has been augmented. It
includes the look up of more than 10 million barcodes. If you
have been using Recognizer before June of 2012, you may notice
a difference in the barcode scanning performance.
Support for Bluetooth audio devices is available in Recognizer.
This permits greater privacy when scanning items, and using
Recognizer. Or conversely, it permits an instructor to use a
Bluetooth speaker to conduct a demonstration or a
class.
Possibly more important is the capability to use a Bluetooth braille device, which may permit many persons who are deaf and blind to use Recognizer. If you find this to be helpful, we hope that you will let us know.
Because barcode scanning is an art, and not a science, we
recommend that you practice on known items, where you may
already know there is a barcode, and develop your techniques
for scanning barcodes as reliably as scanning without vision
permits. If you are frustrated with your progress, try making a
game of it. Get sighted friends and family to participate by
agreeing to keep their eyes closed while they do it. You will
soon realize that you are better than you knew, and that your
family and friends are far more willing to help you acquire
images for your Recognizer library.
This process has inherent limitations for a person with limited
or no vision. It is what engineers call "open loop control".
This means that the feedback loop of vision to guide your
scanning is missing, so that your control over the camera and
scanning process is one of seek and hope. Sighted persons can
use closed loop control, as they can see the barcode, and
quickly scan it by moving the camera right over it, and better
yet, they can see the barcode on the screen, so they know the
app is ready to read it. They can tell it is focus, so they
hold the camera at the right distance from the barcode. These
are skills that person with limited or no vision must learn. We
find that many times, errors or frustration occur because the
camera is being held too close to the item. The optimal
distance is approximately 4-8 inches, 10-20 cm, in most
instances.
Backing Up and Restoring the Library
Creating a library of images involves spending valuable
time, and often retaining the assistance of friends or family
members to do it. You will never wish to lose your library of
images once you have it. While Recognizer is unlikely to ever
lose the library, as it does keep a backup copy in your phone,
the loss of your phone, or damage to the phone, could cause the
loss of that library--UNLESS, you have taken precautions to
save your library!
The library can be saved and emailed to yourself, for later
restoration, should it be necessary. This is done through the
Data button on the main screen. Opening the data library will
permit you to find the button for exporting items from the
library by email.
Once in your data library, you will find a button entitled
‘Share Items’ which is located at the very bottom
of the screen. A four-finger single tap close to the bottom of
the screen--with Voiceover running--will allow you to access
this button quickly and easily.
Double-tapping the ‘Share Items’ button will bring
up a simple email composition page just like you would find in
your native iPhone Mail app. On this screen you can compose a
message to yourself or simply use the message which is already
provided for you.
To send this message with your attached library, simply enter
an email address in the ‘To’ field of the message
and double tap the Send button in the top right corner of the
screen, or choose an email address from those in your Contacts
and as before, double-tap the Send button to send your message.
It is as simple as that.
Because mail systems often limit message size, the library will be broken into 100 image chunks. Each 100 image message will have to be sent as a separate email, which for larger libraries, may be annoying. We apologize for the inconvenience. We continue to see better ways to share library images.
Once you have your message addressed, you will see a dialogue
letting you know that the message is being sent and once you
double-tap the OK button to exit the dialogue, you will be
returned to the Data screen.
When you would like to import items that you have sent
yourself via email, simply open the email message which
contains the attachment, using the Mail app on your phone, and
scroll down to the bottom of the message. you will see the
attachment as a button which you can double tap. When you do
this, you will be given the choice to open the attached file in
the Looktel Recognizer app. Double-tap the appropriate button
to choose Recognizer and the app will come up for you and open
the attached file. you will be shown a screen with a progress
indicator which will announce Recognizer’s progress--in
ten-percent increments, as it loads your data for you.
In this way you can either back up your entire library of
Recognized items or a portion of it. Once the items are
imported from the email attachment, they become part of your
entire active Recognizer library, which you can access by
double-tapping the Data button on the main screen, as you
normally would.
You should back up your library whenever you have added a
significant number of images to it. You will want to back it up
if you upgrade your iPhone or IOS device to a new device, as it
will not transfer otherwise.
Example: You have an iPhone 4, and are upgrading to a new
iPhone 4S. It will be essential to back up the library by
emailing it to yourself, then opening that email on the new 4S
iPhone, so that you can easily restore the library on that
iPhone.
The libraries are additive, and Recognizer is able to
automatically remove identical items. If you load your library
twice, as you had 2 different versions of it, identical or
redundant items will be removed, so that an item only appears
once. This offers some advantages to you.
If you have family members or friends, who have iPhones, and
are willing to assist, it is possible for them to have their
own copy of Recognizer that they get from the app store, and
load your library when you email it to them. They can add items
for you, and mail the library back. They can tell what you
already have in the library, as they have your copy. This way,
they can add to your library more productively. Your new
library can be emailed back to you, and you can add it to your
current library, knowing that the redundant images will only
appear once.
There are limitations! Email services are often limited in the
number of megabytes they will accept in one email. Fortunately,
a free Google email account will permit up to 20 megabytes in a
single email. This is enough for a library of approximately 100
images. Some services are limited to 5 megabytes, so if you
seem to have problems with email and your library, this may be
why. In future, other methods for sharing library files will be
possible.
For now, Recognizer will break up a larger library into 100
image segments, and permit you to email each one separately.
This will assist with handling larger libraries of up to 500
images or so. We recognize that for larger libraries than this,
it gets very tedious to send all those email messages. As of
this writing, very few people have libraries of 300 images or
more.
LookTel values your privacy. ¬ We thank you for purchasing and using LookTel Recognizer. All anonymized information collected is aggregated and used in its aggregate form. Anonymized information does have a city-level resolution. Used in aggregate, this information eliminates the city-level location information. No other personally identifiable information, PII, is collected by your purchase and/or use of this app.
An entire paragraph was deleted here.
In the Recognizer Settings is a button that permits you to Opt-In, or volunteer to assist with the further development of Recognizer. This button is located at the end of the Settings options. Please select "Yes" to Opt-In if you wish to participate in improving LookTel Recognizer. Selecting "Yes" permits LookTel to collect anonymous data, used in aggregate form, to better determine future upgrades, analyze and understand bug reports, and meet customer requests for new features. All location information is anonymized at a city-level resolution, and used in aggregate form so as to eliminate location information. No other personally identifiable information, PII, is collected by your selecting to provide us additional data.
An entire paragraph was deleted here.
A paragraph was deleted here.
What is collected:
When you select "Yes" to Opt-In the data obtained from
recognizer includes:
The length of time the app is run, the status of the app
when closing, the time the app was in the
foreground/background, battery levels (energy use), IOS
version, and IOS device, are aggregated and provided to
LookTel, as essential data for the further maintenance and
development of the app.
Other data collected includes:
category: Error Notifications:
Crash Notifications
Problem deleting an item from the library
Problem with app initialization of the library
When database maintenance is required of the app
category: Recognition Performance
Barcode scan successfully answered
Barcode scan unanswered
QR code successfully scanned
Item successfully recognized
category: General Usage
Item Added
Item Deleted
Library Shared
Order of Library Date/Description
Explanations of Anonymous Data From Recognizer
The following explanations of the Opt-In data are provided so
that no one need have any concern as to specifically what is
collected. This list is the entire list collected from choosing
to share the Opt-In data.
The length of time the app is run,
the status of the app when closing,
the time the app was in the foreground/background,
battery levels (energy use),
IOS version, and IOS device, are aggregated and provided to
LookTel, as essential data for the further maintenance and
development of the app.
Crash Notifications
Recognizer almost never crashes, but should it do so for any
reason, LookTel would like to know, in order to correct any
problem arising from IOS updates, or from any unforeseen
circumstance.
Problem deleting an item from the library
If Recognizer fails to delete an item from the library
completely, it is helpful for us to know when the item was
added--was it just added? Or, is it an older item that has been
in the library for some time? These errors are not normal for
Recognizer, and LookTel wants to know when they occur, in order
to preserve the integrity of every Recognizer library.
Problem with app initialization of the library
Sometimes large Recognizer libraries will fail to initialize.
While we believe Recognizer is more stable in this regard than
in prior versions, it is important for us to know how often
this occurs, and with what size libraries.
When database maintenance is required of the app
Hopefully, no customer ever learns that a library has become
corrupted! Software being what it is, this is possibly too
optimistic. We want to know the bad news as much as the good.
This item provides us the number of images in the library at
the time it became corrupt. Recognizer will try to recover the
library. We expect that if such a malfunction ever occurs, that
the backup is recovered successfully, and that you are not
inconvenienced too greatly.
Barcode found,
We will always be working to improve the quality of Recognizer,
and we wish to track how often it is successful in finding a
barcode in the database for you. Ideally, it should be at least
98%. This is a work in progress. We intend to be better than we
are.
Barcode scan unanswered
For those times when you are disappointed with another failure
to find a barcode, we want you to share with us what it is you
were hoping to find. We keep score on ourselves, and intend to
do better next time.
QR code successfully scanned
QR codes are becoming ever more common. We want to know how
those using Recognizer are managing with them, and how often
Recognizer is helpful at reading QR codes.
Item successfully recognized
We like to know when Recognizer performed well! We need a
context for problems with respect to successes. Recognizer will
send the number of successful scans with an anonymous event
number.
Item Added
When you add items to the library, we want a count of the
additions, as a context for understanding deletions, failures,
or other Library issues. No images or image information is
sent--only the notice that an item is added.
Item Deleted
We want a count of items you delete from the Library, as it
helps us to understand any issues with adding images to the
Library. No image information is sent--only the occurrence when
an item is deleted.
Library Shared
This reports the sharing of the Library only. No data as to
whom and where it was shared are collected. We expect to
increase sharing options in future versions. This helps us to
understand the frequency of sharing, and managing the
Library.
