Locating and targeting user interface elements in Robocorp Lab

Your robot needs to locate user interface elements, such as buttons. For web-based applications, when the HTML markup provides identifiers for the elements, locating them is straightforward. With desktop applications, locating and targeting the elements might be more difficult. The following examples demonstrate locator auto-completion in different types of applications.

Important: The locator auto-completion works only in the Notebook mode.

Locator auto-completion (Calculator, Windows 10)

This example demonstrates the interactive element locator using the Calculator application in Windows 10. The robot:

• opens the Calculator application
• adds two numbers together, using the user interface buttons
• logs the result

*** Settings ***
Library           RPA.Desktop.Windows
Library           String

*** Keywords ***
Open the Calculator
    Open Executable    calc.exe    Calculator

*** Keywords ***
Add two numbers
    [Arguments]    ${first}    ${second}
    Mouse Click    id:clearEntryButton
    Mouse Click    id:num${first}Button
    Mouse Click    # Need to find the plus button locator
    Mouse Click    id:num${second}Button
    Mouse Click    id:equalButton

*** Keywords ***
Read the result
    ${result}=    Get Element Rich Text    id:CalculatorResults
    ${_}    ${result}=    Split String From Right    ${result}    max_split=1
    [Return]    ${result}

*** Tasks ***
Calculate and log the result
    Open the Calculator
    Add two numbers    5    7
    ${result}=    Read the result
    Log    ${result}

We are missing a locator for the plus button in the script:

Mouse Click    # Need to find the plus button locator

In Robocorp Lab, with the Calculator application visible so that you can click on it, type in the locator prefix uia:. This prefix is used to trigger the locator auto-completion:

Mouse Click    uia:

With the cursor placed after the prefix (uia:), hit the TAB key on your keyboard. The mouse cursor changes into a crosshair. With the crosshair, click on the plus button on the Calculator user interface. You will see a list of available locators for that element:

Click on the automation_id:plusButton locator to insert it into the script.

Image-based locator auto-completion (Paint, Windows 10)

Some applications do not expose their internals in a way that can be targeted using textual locators. In these situations, you can try image-based targeting. This example demonstrates the interactive element locator using the Paint application in Windows 10.

Add the optional opencv-python package as a dependency in the conda.yaml file to improve the accuracy of the image-based locators:

channels:
  - defaults
  - conda-forge
dependencies:
  - python=3.7.5
  - pip=20.1
  - pip:
      - rpaframework[cv]==4.1.*

The robot:

• opens the Paint application
• selects the Text tool
• uses the Text tool to type on the canvas

*** Settings ***
Library           RPA.Desktop.Windows
Library           String

*** Keywords ***
Open Paint
    ${pid}=    Open Executable    mspaint.exe    Untitled - Paint
    Switch To Application    ${pid}

*** Keywords ***
Write on canvas
    Mouse Click Image    template=#Add image template here
    Mouse Click Image    template=#Add image template here    off_y=100
    Send Keys    Paint{SPACE}>{SPACE}insert-your-favorite-image-editing-software-here

*** Tasks ***
Do a paint operation
    Open Paint
    Write on canvas

We are missing locators for the Text tool and the canvas in the script:

Mouse Click Image    template=#Add image template here
Mouse Click Image    template=#Add image template here    off_y=100

In Robocorp Lab, with the Paint application visible so that you can click on it, place the cursor after the first template= argument. This prefix is used to trigger the locator auto-completion:

Mouse Click Image    template=

With the cursor placed after the template argument (template=), hit the TAB key on your keyboard. The mouse cursor changes into a crosshair. With the crosshair, drag a rectangle around the Text button on the Paint user interface. Select only the element that you are targeting:

When running the robot, the mouse click will target the center of the selected image.

Robocorp Lab will store a screenshot of the selected area and insert an image locator for you:

Mouse Click Image    template=${EXECDIR}\\1588235551.bmp

The locator screenshot is stored in the tasks directory. You can open the stored image and verify that it looks fine:

Now that we have the Text tool located, the remaining locator should help the robot to find the canvas to type on. Repeat the above steps, this time selecting the Tools text:

Robocorp Lab will store a screenshot of the selected area and insert an image locator for you:

Mouse Click Image    template=${EXECDIR}\\1588236522.bmp    off_y=100

Why did we select that particular element? One option for finding the canvas is to use a reference point on the user interface and then click somewhere under that point. In this case, we decided the Tools text is pretty stable (it will stay there even if the window size is bigger than expected). Using the reference point, we can instruct the robot to find that point and then click 100 pixels under that point (off_y=100).

The finalized Write on canvas keyword implementation might look similar to this:

*** Keywords ***
Write on canvas
    Mouse Click Image    template=${EXECDIR}\\1588235551.bmp
    Mouse Click Image    template=${EXECDIR}\\1588236522.bmp    off_y=100
    Send Keys    Paint{SPACE}>{SPACE}insert-your-favorite-image-editing-software-here

Locator auto-completion for browsers

Robocorp Lab assists in selecting elements when using a web browser.

Important: The locator auto-completion works only when the robot script is executed in the Notebook mode.

Run the following Robot script once in Robocorp Lab to open a browser. The Lab keeps track of the opened browser so that you can interact with it using locators. The script will complain about invalid selectors. No worries, you are going to complete the selectors soon!

Note that Robocorp Lab can communicate only with browsers that it has opened itself (for example, using the Open Available Browser keyword from the RPA.Browser library). You will not be able to interact with a browser that you opened manually.

*** Settings ***
Library           RPA.Browser

*** Tasks ***
Click on elements on a website
    Open Available Browser    https://google.com/
    Click Element    css:
    Click Element    id:

The CSS locator

Let's locate the Google Search button at https://google.com/. Here is the HTML markup for the button:

<input
  class="gNO89b"
  value="Google Search"
  aria-label="Google Search"
  name="btnK"
  type="submit"
  data-ved="0ahUKEwiD_ta6lL3rAhWF_KQKHXkXB70Q4dUDCAs"
/>

With the cursor placed after the CSS locator prefix (css:), hit the TAB key on your keyboard. In the browser, click on the Google Search button. You will see a list of locators. Click on the name:btnK locator to insert it into the script:

The ID locator

With the cursor placed after the ID locator prefix (id:), hit the TAB key on your keyboard. You will see a list of HTML id attribute locators:

Other locators

Use the link: prefix to locate links and the name: prefix to locate elements having a name attribute.

UI locators

UI locators enable you to create and store user interface locators so that they can be shared and reused. Let's create a named locator for the search field on google.com.

Open the UI locator interface

Press Ctrl+Shift+U (Windows) or Shift-Command-U (macOS) to open the UI locators interface.

Add a new locator

Press the + button to add a new locator:

The locator requires a name. The suggested format is Website.ElementName. In our case, let's use Google.SearchField, where Google is the group name and SearchField is the element name (you could later add Google.SearchButton, Google.LoginLink, etc.).:

Locate the element

Press Start to launch a new browser. Navigate to google.com. Press Locate and click on the search field. The locator is detected. The strategy, value, and the source of the locator are displayed:

Store the locator

Click Add to store the locator. You will see useful information, such as the name, type, and even a screenshot of the locator:

Use the stored locator

To use the stored locator in your robot, copy the locator name:

Use the alias: prefix and paste the locator name:

*** Settings ***
Documentation     Locator UI demonstration.
Library           RPA.Browser

*** Tasks ***
Search for "dinosaur" on google.com
    Open Available Browser    https://www.google.com/
    Input Text When Element Is Visible    alias:Google.SearchField    dinosaur

The locators are stored in the locators.json file at the root of your robot. The file can be shared with other developers. This way, you can build locator libraries for your applications and reuse them over and over again!

The UI locators feature is still very new. Please try it out and give feedback!

Conclusion

Robocorp Lab assists you with finding the user interface elements for your robot.