Working with Optical Character Recognition

When you want to find text on your system under test (SUT), and capturing an image of the text is not practical, you can rely on the optical character recognition (OCR) capabilities in Eggplant Functional and SenseTalk. OCR is most useful when searching for dynamic text, although you might find many other practical uses. You should spend time to understand the OCR properties discussed here so that you can make appropriate adjustments and achieve better search results in your test environment.

Eggplant Functional is set by default to use the Generic OCR platform. The Generic OCR platform includes a Text Engine, and any number of predefined text styles. Generally, you should not need to change this setting. For information about other text preference settings, see Text Preferences.

Using OCR

You can use OCR almost any time you need to search for text on the SUT. This functionality is especially well-suited to the following situations:

  • You do not know the text you are looking for ahead of time, but the script will have access to that information by the time the search is conducted.
  • The font, size, or color of the text can vary.
  • Your testing includes a variety of browsers, and different browsers can render fonts differently.
  • You need to read a text value from the SUT and bring that information back into your script for further testing or to record in an external data file.

OCR text searches are similar to image searches, and you typically use the same commands or functions. Instead of specifying the image name, you specify the text string you want to find, in quotes, preceded by text:.

Example:

Click (text:"Eggplant") // Finds the first instance of the text string Eggplant(with any formatting), then clicks it.

If you want to find all of the locations of a text string, use the EveryImageLocation() function:

Example:

log EveryImageLocation(text:"OCR") // Searches for the text string OCR and logs the screen coordinates for every instance found.

How the OCR Engine Works

When you search for text through a generic text property list, the OCR engine finds each instance of text on the SUT, and reads the text values to find the string you are looking for. You do not have to describe the appearance of your text, because text formatting is not considered.

When the OCR engine does not find your text string, Eggplant Functional throws an exception, which can cause your script to fail. However, if you enable the OCR Update panel, the OCR Update functionality attempts to apply different properties to the search to see if it can make a match. You can set this functionality to Auto Update, so that it performs without user interaction, or Show Panel, which opens the OCR Update panel on a failed OCR search.

For detailed information about using this feature, see The OCR Update Panel.

The properties that you can adjust using the OCR Update panel are also available by using the OCR Tuner. The OCR Tuner lets you adjust these OCR properties while showing you the text being read in the Capture Area. You can use the OCR Tuner to improve your OCR searches.

For detailed information about using this feature, see Using the OCR Tuner.

Using OCR Properties in Searches

To improve the reliability of OCR searches, SenseTalk includes text properties that you can use to help the OCR engine recognize text correctly in your environment. Each of the following methods might be used to improve OCR matches or speed searches.

Set a Search Rectangle

It is almost always helpful to add a search rectangle, which limits what part of the screen the search takes place in. When OCR searches the entire screen, it is not only slower, but it can also be less accurate because it is likely to come up with extra possible matches. It might also result in no matches at all.

If your OCR search doesn't find text that you expect should be found, you might try expanding the size of your search rectangle to provide the OCR engine a little more text context, which can sometimes result in better matches.

You can specify a search rectangle with exact coordinates, but it is more common in Eggplant Functional to use images to define the rectangle. Using images works better in case the location of the text is not always the same. This situation might happen, for instance, if the window the text is doesn’t appear in the same location on the screen every time the test is run.

When using images to define the boundaries of the search rectangle, keep in mind that the points used to set the rectangle are based on the hot spot, which can be moved outside of the image.

Example: Reading Dynamic Text from a Website

You might have a test that navigates to the Google Finance page, search for a specific company, and then read the stock price. To make sure that OCR reads the value of the price reliably, and nothing else, you can define a search rectangle using images. The code used could be as simple as this:

Log ReadText ("TLImage","BRImage")

You need to capture two images for the above code to work. In this example, you used images TLImage and BRImage to define the top left and bottom right corners of the search rectangle. To capture these images, choose one or more elements of the screen that are stable in relation to the text OCR reads.

This example uses a single element of the screen, the "Company" label.

TLImage.png

TLImage is an image of the "Company" label with the hot spot moved to the upper left corner of the area where the stock price is displayed.

BRImage.png

Without even moving the capture area, capture the BRImage with the hot spot moved to the bottom right corner of the area where the stock price displays.

After capturing the images and writing the SenseTalk code, the above example is able to successfully read the stock price for any company in Google Finance.

SuccessfulReadText.png

"TLImage" and "BRImage" are used to set the search rectangle for a successful ReadText() search using OCR .

If it is not possible to use images to set the search rectangle, you can use screen coordinates. The Cursor Location toolbar icon is helpful in this endeavor because it shows the current location of the mouse on the SUT. See Contrast for instructions on adding Cursor Location to the toolbar.

Use Contrast

If the background color of the sought text is known, use the Contrast property to set the contrast.

This property causes Eggplant Functional to see in black and white only. Whatever color is being used as the ContrastColor (the background color) turns white, and everything else turns black. For black text on a white background, the code looks like this:

Click (Text: "hello", Contrast:On, ContrastColor: White, ValidCharacters: "hello", Searchrectangle: ("UpperLeftImage", "LowerRightImage"))

In this situation, every pixel close to white (within the tolerance range of 45 on either side) is turned white, and everything else is read as black. This can help OCR to more clearly read the text.

What it actually sees is this image, free of any anti-aliasing:

HelloOCRImage.png

What OCR Sees with the Contrast On

When looking for text on a gray background (or another color of medium value), it can get a little more complex. It is good to set the ContrastTolerance a little lower (down to 20 or so), which narrows the number of pixels that OCR might try to turn white. Notice in the image above, a pixel between the “h” and the “e” in “hello” turned black, joining the two letters together. In this instance, OCR was still able to read the letters, but in some other circumstances, this could make the letters even more difficult for OCR to process. This is why it is always a good idea to try things out and see how they work before running a full script.

Click (Text:"hello", Contrast:On, ContrastColor: White, ContrastTolerance: 20)

Determining the Background Color

If the background/contrast color is not known, there are two ways to find the RGB values of the background color in a specific location on any platform.

The first method is to use the color picker. This can be found by clicking the Find Text icon in the toolbar of the remote screen window, and selecting the Contrast checkbox. Then enable color and click on the color box to the right to choose a specific color. When this window comes up, click on the magnifying glass, and sample some of the background color next to the text (be sure not to capture the anti-aliasing).

Finding the contrast color on a Mac

Finding the Contrast Color on Mac, Using the Magnifying Glass

The second method is to use the ColorAtLocation() function. To use this function, bring up the remote screen window in Live Mode.

If the Cursor Location toolbar icon is not already on the toolbar, it can be added by doing the following:

  1. Right-click within the toolbar.
  2. Select Customize Toolbar.
  3. Drag and drop the Cursor Location icon onto the toolbar.
  4. Click Done to close the customization dashboard.

Now do the following to use the ColorAtLocation() function.

  1. Move the mouse over the background of which the RGB color value is desired.
  2. Using the coordinates of this mouse location (shown in the field of the Cursor Location toolbar icon), run this line of code in the ad hoc do box (AHDB) found at the bottom of the run window, or in your script:

    put colorAtLocation(x,y) // where (x,y) refers to the coordinates found in the remote screen window

    Running this line of code returns the RGB value for the color at the location specified.

Set Valid Characters

ValidCharacters tells OCR which characters to take into account, and which characters to ignore. For example, this can be used to prevent OCR from misreading “O” as “0″ or vice versa.

This can be done manually for optimal control of your script, or by using validCharacters:"*" to automatically set the validCharacters to the text being searched for.

To continue with the example above searching for “Brown”, the following code might be used to set the validCharacters manually:

Click (Text: "Brown", ValidCharacters: "Brown", SearchRectangle: ("UpperLeftImage","LowerRightImage"))

It is also possible to use ValidCharacters with a variable like in this example, where the desired characters are first stored into the MyText variable :

Put "Brown" into MyText

Click (Text: MyText, ValidCharacters: MyText)

To set the validCharacters to the text being searched for, the following code would be used:

Click (Text:"Brown", ValidCharacters:"*", SearchRectangle:("UpperLeftImage","LowerRightImage"))

Ignore Spaces

As a property in a text search, IgnoreSpaces can be used to do exactly that—ignore any spaces within the search. This approach can be helpful because spacing between characters is not consistent. Sometimes OCR sees spaces where there are not any, or ignores spaces where they exist. Setting the IgnoreSpaces property to ON causes OCR to match “flowerpot” with a search for “flower pot” and vice versa. When this property is used, OCR strips spaces from both the string for which it is searching and the string that it finds to come up with a match.

Click (Text:"flower pot", validCharacters:"*", ignoreSpaces:ON, searchRectangle:("UpperLeftImage","LowerRightImage"))

Ignore Underscores

The ignoreUnderscores property causes OCR text searches to treat underscores as spaces during searches. For example, the "My_Computer" string would match My_Computer or My Computer. The ignoreUnderscores property is on by default because the OCR engine sometimes fails to recognize underscores.

Click (Text:"My_Computer", validCharacters:"*", ignoreUnderscores:ON, searchRectangle:("UpperLeftImage","LowerRightImage"))

Specify Languages

As a property in a text search, Language can be used to search for text of the specified language or languages. For a list of supported languages, see OCR Language Support.

For example, you can specify more than one language when you have characters of different languages in the same sentence. In the case of a line that reads as follows:

料理4出6「です。")

Use the following SenseTalk code to read the multi-Language characters included on the line: 

ReadText((475,179,608,212), Language:"Japanese,English")

Customize the OCR Engine Dictionary

When you use text-reading functions, you can customize the OCR dictionary to help the OCR engine find the text you’re interested in. SenseTalk includes three properties that you can use in a text property list or as options on a readText() function call to add or remove words from the OCR dictionary:

  • ExtraWords: Set this property to a list of words to supplement the built-in dictionary for the current language.
  • PreferDictionaryWords: Enable this property to tell the OCR engine to read each word as a word defined in its dictionaries, if possible. If no suitable dictionary words are found, the engine returns a nondictionary word, using its best interpretation of each character.
  • ProhibitedWords: Set this property to a list of words to exclude from recognition. If the engine sees something that looks like one of these words, it falls back to a different interpretation of the characters to get a different word instead.

You can set these properties for the whole script or for specific commands:

// Set a list of words for the whole script

set the readTextSettings to (extraWords:"ean carner")

put readText(rect, PreferDictionaryWords:yes, prohibitedWords:"can carrier tap")

 

// Set a list of words in a single command

set rect to the RemoteScreenRectangle

put readText(rect, extraWords:"ean carner", PreferDictionaryWords:yes, prohibitedWords:"can carrier")

Both versions of the code above set the words can and carrier as prohibited words, effectively removing them from the dictionary. The examples also add ean and carner as new dictionary words. This example forces the OCR engine to ignore its own best interpretation of the individual letters on the screen and fall back on a less-likely interpretation of the individual letters in order to make dictionary words out of what it sees.

The extraWords property creates a custom OCR dictionary, which can be provided command by command or for an entire script, as shown in the examples above. You can also set the extraWords dictionary for the whole script by using an external file that lists the desired words:

set the ReadTextSettings to {ExtraWords:file ResourcePath("OCR-Dictionary.txt")} // Applies the extraWords dictionary to any instance of ReadText() in the script

 

put readText((image1,image2), extraWords:file ResourcePath("OCR-Dictionary.txt")) // Uses an external file to specify the word list for an individual command

If you use an external data file, the file must be formatted with a separator between individual words, which can be either a space (white space) character or a newline character.

Tips for Working with OCR

  • When OCR doesn't find text that you think it should, try running the readText() function to see what it is seeing. Once you know what it is seeing, you are better-equipped to fix the problem. For instance, if you discover that the readText() function is returning “hello” when the text you want is a password that actually reads “hell0″, specifying validCharacters:”hell0″ might improve your results.
  • The Ad Hoc Do Box (AHDB) can be used to verify your script and make sure it is doing what it is supposed to be doing. Other helpful information can also be found by using the AHDB. A good example of this is prompting it to return a list of the colors that Eggplant Functional knows, along with their RGB values. To get this list, go to the AHDB and type put the namedColors.

    AdHocDoBox.png

    What OCR Sees with Contrast On

Improving the Speed of OCR Searches

The flexibility of OCR text searches comes with a trade-off: OCR text searches are not as fast as image searches. To keep your scripts running as efficiently as possible, follow these best practices:

  • Use the search rectangle. If you can narrow the location of the text you are looking for (e.g. in the taskbar, or within a particular window), you can set a search rectangle to limit your search to that location. For more information, see Text Preferences or Searching Part of the Screen.
  • Minimize clutter on the SUT. It never hurts to keep a neat SUT (i.e., close unnecessary windows, use a simple desktop background). When your script includes OCR text searches, a neat desktop can save you valuable seconds or even minutes of execution time.

 

This topic was last updated on October 11, 2019, at 11:18:56 AM.

Eggplant icon Eggplantsoftware.com | Documentation Home | User Forums | Support | Copyright © 2019 Eggplant