How to find user interface elements using locators and keyboard shortcuts in Windows applications
Available tools for inspecting Windows applications
One way to automate Windows applications is to target UI components with their identifiers. Microsoft recommends Accessibility Insights for viewing the UI automation properties. Legacy tools such as Inspect.exe can also be used.
Inspecting Windows applications with Accessibility Insights
After installing and launching Accessibility Insights for Windows, inspecting Windows applications is straight-forward. Using the Windows Calculator as an example, hovering over the application displays the properties of the UI components.
By default, Accessibility Insights displays only a few properties, including the accessible Name
of the UI component in the DETAILS
pane. In this case, the name of the button is Five
. Using localized names for automation is not the most robust option since the labels change based on Windows language settings.
To see more properties, click on the settings icon and select Include all properties that have values
:
This will include the AutomationId
property. In this case, the value of that property is num5Button
:
You can use the value of the AutomationId
property in your robot script. Here we are using the Click
keyword from the RPA.Windows
library, prefixing the automation ID with id:
:
*** Settings ***
Library RPA.Windows
*** Keywords ***
Open The Calculator
Windows Run calc.exe
Control Window name:Calculator
*** Keywords ***
Click The Five Button Using The AutomationId Property Value
Click id:num5Button
*** Tasks ***
Automate The Calculator
Open The Calculator
Click The Five Button Using The AutomationId Property Value
Accessing UI components and functionality using keyboard shortcuts
If the application supports keyboard shortcuts (see Windows desktop application robot), it is recommended to use those whenever possible.
This is an example of a robot that controls a browser using only the RPA.Windows
library and available keyboard shortcuts.
Robocorp also supports browser automation solutions such as Selenium and Playwright.
*** Settings ***
Library RPA.Windows
*** Tasks ***
Automate desktop web browser
Open the browser
Open new private window
Focus address bar
Type Robot Framework URL and press enter
Open new tab
Type Robocorp docs URL and press enter
Search Within Page Search Input Field
Open Link To Library Documentation
Zoom in
Log Done.
*** Keywords ***
Open the browser
Windows Search Firefox
Control Window name:"Mozilla Firefox"
Open new private window
Send Keys keys={Ctrl}({Shift}p)
Focus address bar
Send Keys keys={Ctrl}L
Type Robocorp docs URL and press enter
Send Keys keys=https://robocorp.com/docs{Enter}
Open new tab
Send Keys keys={Ctrl}T
Type Robot Framework URL and press enter
Send Keys keys=https://robotframework.org{Enter}
Zoom in
Send Keys keys={Ctrl}{Add}{Ctrl}{Add}{Ctrl}{Add}
Search Within Page Search Input Field
Control Window subname:"RPA Documentation"
Send Keys locator=name:Search type:Edit depth:16 keys=rpa.windows{Enter}
Open Link To Library Documentation
Sleep 2s
Control Window subname:"RPA Documentation"
Click name:"RPA.Windows" type:Hyperlink depth:16
See the robot in action:
When all else fails: Image-based locators
Sometimes UI components in windows applications do not have an ID that could be used to target them, or they can not be accessed by keyboard shortcuts. In these cases it is still possible to locate them using image-based locators.
Conclusion
Windows desktop applications can be automated using UI component IDs, keyboard shortcuts, or image-based locators.