Locating and targeting user interface elements in Visual Studio Code

Browser locators

Browser locators enable you to create and store browser interface locators so that they can be shared and reused. Let's create a named locator for the search field on google.com.

Open the UI locator interface

  • Click the Robocorp icon in the Activity Bar.
  • Select a robot in the Robots panel.
  • From the Locators panel, select Add Browser Locator.

Add a new locator

A UI Inspector window and a new browser window will appear:

Browser locator modal

Navigate to www.google.com in the new browser window and select Pick in the UI Inspector window:

Pick the element to determine the locators

Click on the search text box. The UI Inspector window will now show a list of available locators you to choose from:

Getting the locator options

Click the first option name

Select the name locator option

In the browser the search box should now be outlined in purple. This outline indicates what the locator will select when used in a robot. Now that you are happy with your selection select Save as locator:

Verifying the locator

The locator requires a name. The suggested format is Website.ElementName. In our case, let's use Google.SearchField, where Google is the group name and SearchField is the element name (you could later add Google.SearchButton, Google.LoginLink, etc.).:

Naming the browser locator

Use the stored locator

To use the stored locator in your robot, copy the locator alias:

Copy the browser locator

Paste the UI locator alias:

*** Settings ***
Documentation     Locator UI demonstration.
Library           RPA.Browser.Selenium

*** Tasks ***
Search for "dinosaur" on google.com
    Open Available Browser    https://www.google.com/
    Input Text When Element Is Visible    alias:Google.SearchField    dinosaur

NOTE: browser based alias locators only work in Selenium

Windows Based Locators (Calculator, Windows 10)

Your robot needs to locate user interface elements, such as buttons. For web-based applications, when the HTML markup provides identifiers for the elements, locating them is straightforward. With desktop applications, locating and targeting the elements might be more difficult. The following example demonstrates finding locators using the Add Windows Locator option in the Robocorp icon in the Activity Bar.

Automated Windows Calculator

This feature is currently in Beta so please test it out and provide feedback!

Image template matching (Paint, Windows 10)

Automated Windows Paint

Some applications do not expose their internals in a way that can be targeted using textual locators. In these situations, you can try image template matching. This example demonstrates the use of image templates with the Paint application in Windows 10.

The RPA.Desktop library is a cross-platform solution for navigating and interacting with desktop environments (Windows, Linux, macOS!). It can be used to automate applications through the same interfaces that are available to human users.

Add the rpaframework-recognition package as a dependency in the conda.yaml file to enable image template matching:

channels:
  - conda-forge
dependencies:
  - python=3.9.13
  - pip=22.1.2
  - pip:
      - rpaframework==16.0.0
      - rpaframework-recognition==5.0.0

The robot:

  • opens the Paint application
  • selects the Text tool
  • uses the Text tool to type on the canvas
*** Settings ***
Library           RPA.Desktop

*** Keywords ***
Open Paint
    Open Application    mspaint.exe

*** Keywords ***
Write on canvas
    Wait For Element    # Insert UI locator alias
    Click    # Insert UI locator alias
    Move Mouse    # Insert UI locator alias
    Click    offset:0,100
    Type Text    Paint
    Type Text    ${SPACE}
    Type Text    >
    Type Text    ${SPACE}
    Type Text    insert-your-favorite-image-editing-software-here

*** Tasks ***
Do a paint operation
    Open Paint
    Write on canvas

We are missing the UI locator aliases for the Text tool and the canvas in the script:

Wait For Element    # Insert UI locator alias
Click    # Insert UI locator alias
Move Mouse    # Insert UI locator alias
Click    offset:0,100

Open the Image locator interface

  • Click the Robocorp icon in the Activity Bar.
  • Select a robot in the Robots panel.
  • From the Locators panel, select Add Image Locator.

Add a new locator

Press the Pick area button to add a new locator:

Add a new image baed locator

macOS: VS Code will ask for permissions. Go to System Preferences -> Security & Privacy and check VS Code in the Screen Recording section.

The mouse cursor changes into a crosshair. With the crosshair, drag a rectangle around the Text button on the Paint user interface. Select only the element that you are targeting:

VS Code: Image locator selector tool

When running the robot, the mouse click will target the center of the selected image.

VS Code saves a screenshot of the selection. Select Save as locator:

VS Code: UI locators: Image Captured

Provide a name for the locator (Paint.TextTool). Click Save:

VS Code: UI locators: Name

The locators are stored in the locators.json file at the root of your robot. The images are stored in the .images directory. The files can be shared with other developers. This way, you can build locator libraries for your applications and reuse them over and over again!

Copy the UI locator alias to the clipboard using the copy button:

Robocorp Lab: UI locators: Copy alias

Paste the UI locator alias into the robot script:

Wait For Element    alias:Paint.TextTool
Click    alias:Paint.TextTool

Now that we have the Text tool located, the remaining locator should help the robot to find the canvas to type on. Repeat the above steps, this time selecting the Tools text:

Robocorp Lab: Image locator selector tool

Store the locator using Paint.ToolsLabel as the name. Copy the locator alias and paste it into the robot script:

Move Mouse    alias:Paint.ToolsLabel
Click    offset:0, 100

Why did we select that particular element? One option for finding the canvas is to use a reference point on the user interface and then click somewhere under that point. In this case, we decided the Tools text is pretty stable (it will stay there even if the window size is bigger than expected). Using the reference point, we can instruct the robot to find that point and then click 100 pixels under that point (offset:0, 100).

The finalized Write on canvas keyword implementation looks like this:

*** Keywords ***
Write on canvas
    Wait For Element    alias:Paint.TextTool
    Click    alias:Paint.TextTool
    Move Mouse    alias:Paint.ToolsLabel
    Click    offset:0, 100
    Type Text    Paint
    Type Text    space
    Type Text    >
    Type Text    space
    Type Text    insert-your-favorite-image-editing-software-here

Performance considerations

The image template matching takes a screenshot of the entire screen for every match operation. This can add a 500-1500ms delay depending on the operating system, screen resolution, CPU, and other factors.

Conclusion

Robocorp assists you with finding the user interface elements for your robot.

September 2, 2022