How to find user interface elements using locators and keyboard shortcuts in Windows applications

Available tools for inspecting Windows applications

One way to automate Windows applications is to target UI components with their identifiers. Microsoft recommends Accessibility Insights for viewing the UI automation properties. Legacy tools such as Inspect.exe can also be used. Robocorp Lab supports simple interactive inspection of applications, relying on the Inspect.exe implementation for Windows applications at the time of writing.

Inspecting Windows applications with Accessibility Insights

After installing and launching Accessibility Insights for Windows, inspecting Windows applications is straight-forward. Using the Windows Calculator as an example, hovering over the application displays the properties of the UI components.

Inspecting Windows Calculator with Accessibility Insights

By default, Accessibility Insights displays only a few properties, including the accessible Name of the UI component in the DETAILS pane. In this case, the name of the button is Five. Using localized names for automation is not the most robust option since the labels change based on Windows language settings.

To see more properties, click on the settings icon and select Include all properties that have values:

This will include the AutomationId property. In this case, the value of that property is num5Button:

You can use the value of the AutomationId property in your robot script. Here we are using the Mouse Click keyword from the RPA.Desktop.Windows library, prefixing the automation ID with id::

*** Settings ***
Library   RPA.Desktop.Windows

*** Keywords ***
Open The Calculator
    Open Executable    calc.exe    Calculator

*** Keywords ***
Click The Five Button Using The AutomationId Property Value
    Mouse Click    id:num5Button

*** Tasks ***
Automate The Calculator
    Open The Calculator
    Click The Five Button Using The AutomationId Property Value

Accessing UI components and functionality using keyboard shortcuts

If the application supports keyboard shortcuts (see Windows desktop application robot), it is recommended to use those whenever possible.

This is an example of a robot that controls a browser using only the RPA.Desktop.Windows library and the available keyboard shortcuts:

Note that using this method, Selenium is not needed at all!

*** Settings ***
Library   RPA.Desktop.Windows

*** Keywords ***
Open The Browser
    Open From Search    Firefox    Firefox

*** Keywords ***
Open New Private Window
    Send Keys To Input    ^+P

*** Keywords ***
Focus Address Bar
    Send Keys To Input    ^L

*** Keywords ***
Type Robohub URL And Press Enter
    Send Keys To Input    https://hub.robocorp.com

*** Keywords ***
Open New Tab
    Send Keys To Input    ^T

*** Keywords ***
Type Robot Framework URL And Press Enter
    Send Keys To Input    https://robotframework.org

*** Keywords ***
Open Search Dialog And Search For Robohub
    Send Keys To Input    ^Frobohub

*** Keywords ***
Zoom In
    Send Keys To Input    ^{VK_ADD}^{VK_ADD}^{VK_ADD}

*** Tasks ***
Automate Desktop Web Browser
    Open The Browser
    Open New Private Window
    Focus Address Bar
    Type Robohub URL And Press Enter
    Open New Tab
    Type Robot Framework URL And Press Enter
    Open Search Dialog And Search For Robohub
    Zoom In

See the robot in action: Automating desktop web browser

When all else fails: Image-based locators

Sometimes UI components in windows applications do not have an ID that could be used to target them, or they can not be accessed by keyboard shortcuts. In these cases it is still possible to locate them using image-based locators.

Conclusion

Windows desktop applications can be automated using UI component IDs, keyboard shortcuts, or image-based locators.