Locating and targeting user interface elements in Robocorp Lab
Your robot needs to locate user interface elements, such as buttons. For web-based applications, when the HTML markup provides identifiers for the elements, locating them is straightforward. With desktop applications, locating and targeting the elements might be more difficult. The following examples demonstrate locator auto-completion in different types of applications.
Important: The locator auto-completion works only in the Notebook mode.
Locator auto-completion (Calculator, Windows 10)
This example demonstrates the interactive element locator using the Calculator application in Windows 10. The robot:
- opens the Calculator application
- adds two numbers together, using the user interface buttons
- logs the result
*** Settings ***
Library RPA.Desktop.Windows
Library String
*** Keywords ***
Open the Calculator
Open Executable calc.exe Calculator
*** Keywords ***
Add two numbers
[Arguments] ${first} ${second}
Mouse Click id:clearEntryButton
Mouse Click id:num${first}Button
Mouse Click # Need to find the plus button locator
Mouse Click id:num${second}Button
Mouse Click id:equalButton
*** Keywords ***
Read the result
${result}= Get Element Rich Text id:CalculatorResults
${_} ${result}= Split String From Right ${result} max_split=1
[Return] ${result}
*** Tasks ***
Calculate and log the result
Open the Calculator
Add two numbers 5 7
${result}= Read the result
Log ${result}
We are missing a locator for the plus button in the script:
Mouse Click # Need to find the plus button locator
In Robocorp Lab, with the Calculator application visible so that you can click on it, type in the locator prefix uia:
. This prefix is used to trigger the locator auto-completion:
Mouse Click uia:
With the cursor placed after the prefix (uia:
), press Ctrl+Space
on your keyboard. The mouse cursor changes into a crosshair. With the crosshair, click on the plus button on the Calculator user interface. You will see a list of available locators for that element:
Click on the automation_id:plusButton
locator to insert it into the script.
Image template matching (Paint, Windows 10)
Some applications do not expose their internals in a way that can be targeted using textual locators. In these situations, you can try image template matching. This example demonstrates the use of image templates with the Paint application in Windows 10.
The RPA.Desktop
library is a cross-platform solution for navigating and interacting with desktop environments (Windows, Linux, macOS!). It can be used to automate applications through the same interfaces that are available to human users.
Add the
rpaframework-recognition
package as a dependency in theconda.yaml
file to enable image template matching:
channels:
- defaults
- conda-forge
dependencies:
- python=3.7.5
- pip=20.1
- pip:
- rpaframework==6.*
- rpaframework-recognition
The robot:
- opens the Paint application
- selects the Text tool
- uses the Text tool to type on the canvas
*** Settings ***
Library RPA.Desktop
*** Keywords ***
Open Paint
Open Application mspaint.exe
*** Keywords ***
Write on canvas
Wait For Element # Insert UI locator alias
Click # Insert UI locator alias
Move Mouse # Insert UI locator alias
Click offset:0, 100
Type Text Paint
Type Text space
Type Text >
Type Text space
Type Text insert-your-favorite-image-editing-software-here
*** Tasks ***
Do a paint operation
Open Paint
Write on canvas
We are missing the UI locator aliases for the Text tool and the canvas in the script:
Wait For Element # Insert UI locator alias
Click # Insert UI locator alias
Move Mouse # Insert UI locator alias
Click offset:0, 100
Open the UI locator interface
Press Ctrl+Shift+U
(Windows / Linux) or Shift-Command-U
(macOS) to open the UI locators interface.
Add a new locator
Press the +
button to add a new locator:
Select Image Based
and click Pick UI Area
:
macOS: Robocorp Lab will ask for permissions. Go to
System Preferences
->Security & Privacy
and checkRobocorp Lab
in theAccessibility
andScreen Recording
sections.
The mouse cursor changes into a crosshair. With the crosshair, drag a rectangle around the Text button on the Paint user interface. Select only the element that you are targeting:
When running the robot, the mouse click will target the center of the selected image.
Robocorp Lab saves a screenshot of the selection. Provide a name for the locator (Paint.TextTool
). Click Add
:
The locators are stored in the
locators.json
file at the root of your robot. The images are stored in the.images
directory. The files can be shared with other developers. This way, you can build locator libraries for your applications and reuse them over and over again!
Copy the UI locator alias to the clipboard using the copy button:
Paste the UI locator alias into the robot script:
Wait For Element alias:Paint.TextTool
Click alias:Paint.TextTool
Now that we have the Text tool located, the remaining locator should help the robot to find the canvas to type on. Repeat the above steps, this time selecting the Tools text:
Store the locator using Paint.ToolsLabel
as the name. Copy the locator alias and paste it into the robot script:
Move Mouse alias:Paint.ToolsLabel
Click offset:0, 100
Why did we select that particular element? One option for finding the canvas is to use a reference point on the user interface and then click somewhere under that point. In this case, we decided the Tools text is pretty stable (it will stay there even if the window size is bigger than expected). Using the reference point, we can instruct the robot to find that point and then click 100 pixels under that point (offset:0, 100
).
The finalized Write on canvas
keyword implementation looks like this:
*** Keywords ***
Write on canvas
Wait For Element alias:Paint.TextTool
Click alias:Paint.TextTool
Move Mouse alias:Paint.ToolsLabel
Click offset:0, 100
Type Text Paint
Type Text space
Type Text >
Type Text space
Type Text insert-your-favorite-image-editing-software-here
Performance considerations
The image template matching takes a screenshot of the entire screen for every match operation. This can add a 500-1500ms delay depending on the operating system, screen resolution, CPU, and other factors.
Locator auto-completion for browsers
Robocorp Lab assists in selecting elements when using a web browser.
Important: The locator auto-completion works only when the robot script is executed in the Notebook mode.
Run the following Robot script once in Robocorp Lab to open a browser. The Lab keeps track of the opened browser so that you can interact with it using locators. The script will complain about invalid selectors. No worries, you are going to complete the selectors soon!
Note that Robocorp Lab can communicate only with browsers that it has opened itself (for example, using the
Open Available Browser
keyword from theRPA.Browser
library). You will not be able to interact with a browser that you opened manually.
*** Settings ***
Library RPA.Browser
*** Tasks ***
Click on elements on a website
Open Available Browser https://google.com/
Click Element css:
Click Element id:
The CSS locator
Let's locate the Google Search
button at https://google.com/. Here is the HTML markup for the button:
<input
class="gNO89b"
value="Google Search"
aria-label="Google Search"
name="btnK"
type="submit"
data-ved="0ahUKEwiD_ta6lL3rAhWF_KQKHXkXB70Q4dUDCAs"
/>
With the cursor placed after the CSS locator prefix (css:
), press Ctrl+Space
on your keyboard. In the browser, click on the Google Search
button. You will see a list of locators. Click on the name:btnK
locator to insert it into the script:
The ID locator
With the cursor placed after the ID locator prefix (id:
), press Ctrl+Space
on your keyboard. You will see a list of HTML id
attribute locators:
Other locators
Use the link:
prefix to locate links and the name:
prefix to locate elements having a name
attribute.
UI locators
UI locators enable you to create and store user interface locators so that they can be shared and reused. Let's create a named locator for the search field on google.com.
Open the UI locator interface
Press Ctrl+Shift+U
(Windows) or Shift-Command-U
(macOS) to open the UI locators interface.
Add a new locator
Press the +
button to add a new locator:
The locator requires a name. The suggested format is Website.ElementName
. In our case, let's use Google.SearchField
, where Google
is the group name and SearchField
is the element name (you could later add Google.SearchButton
, Google.LoginLink
, etc.).:
Locate the element
Press Start
to launch a new browser. Navigate to google.com. Press Locate
and click on the search field. The locator is detected. The strategy, value, and the source of the locator are displayed:
Store the locator
Click Add
to store the locator. You will see useful information, such as the name, type, and even a screenshot of the locator:
Use the stored locator
To use the stored locator in your robot, copy the locator alias:
Paste the UI locator alias:
*** Settings ***
Documentation Locator UI demonstration.
Library RPA.Browser
*** Tasks ***
Search for "dinosaur" on google.com
Open Available Browser https://www.google.com/
Input Text When Element Is Visible alias:Google.SearchField dinosaur
The UI locators feature is still very new. Please try it out and give feedback!
Conclusion
Robocorp Lab assists you with finding the user interface elements for your robot.