The Windows is a library that can be used for Windows desktop automation.

Library is included in the rpaframework package by default, but as shown in the below example library can be also installed separately without rpaframework.

channels:
  - conda-forge
dependencies:
  - python=3.9.13
  - pip=22.1.2
  - pip:
    - rpaframework-windows==7.0.2 # standalone Windows library (rpaframework includes this library)

About terminology

ControlType is a value referred to by locator keys type: or control. Represents type of application object, which can be e.g. Window, Button or ListItem.

Element is an entity of an application structure (e.g. certain button in a window), which can be identified by a locator. (also referred as Control)

WindowsElement is an library container object for the Element. All the keywords returning elements, will in fact return WindowsElement`s. The ones accepting `locator or root_element as arguments, will accept WindowsElement as an argument value. (locator accepts strings as well)

Structure of the WindowsElement

class WindowsElement:
    item: Control        # `item` contains object instance of the element
    locator: str         # `locator` that found this element
    name: str            # `Name` attribute of the element
    automation_id: str   # `AutomationId` attribute of the element
    control_type: str    # `ControlTypeName` attribute of the element
    class_name: str      # `ClassName` attribute of the element
    left: int            # element's rectangle left coordinate
    right: int           # element's rectangle right coordinate
    top: int             # element's rectangle top coordinate
    bottom: int          # element's rectangle bottom coordinate
    width: int           # element's rectangle horizontal width
    height: int          # element's rectangle vertical height
    xcenter: int         # element's rectangle center point x coordinate
    ycenter: int         # element's rectangle center point y coordinate

Example of the WindowsElement usage

${rows}=    Get Elements    class:DataGridRow
# ${rows} is a list of `WindowsElement`s
FOR    ${row}    IN    @{rows}
    Log To Console   ${row.name}                # access `WindowsElement`
    Log To Console   ${row.item.AutomationId}   # access `WindowsElement.item` directly
    Log To Console   ${row.item.Name}           # same as `${row.name}`
END

Locators

Locators are based on different strategies that can used identify Control object.

Available strategies that can be used for the locators:

Key Maps to search property
name Name
class ClassName
type ControlType
control ControlType
id AutomationId
automationid AutomationId
regex RegexName
subname SubName
index foundIndex (int)
offset offset coordinates (x (int), y (int)) from control center
executable target window by its executable name
handle target window handle (int)
desktop SPECIAL target desktop, no value for the key e.g. desktop:desktop and name:Calculator
process NOT YET SUPPORTED target window by its executable's process id
depth searchDepth (int) for finding Control (default 8)
path target element by its index-based path traversal (e.g. path:2|3|8|2)

About root element on locators

Locators work on currently active root element. At the start root element is the whole desktop. There are different ways on changing this root element.

Keyword Control Window is the most common method of setting certain system window as a root element for further actions using locators. In the absence of a provided root_element parameter, here's how you can control the default root element resolving:

  • Set Anchor: Sets the active anchor window from which the search begins.
  • Control Window: Controls and focuses on a window and marks it as the current active window, from which all the subsequent searches will start from in the absence of a set anchor.
  • If there's no set anchor nor active window, then the last resort will be the "Desktop" element itself.

Locators themselves support cascading syntax (denoted by character > in the locator string), which can denote root element in "parent (root) & child" terms.

For example.

Click  id:controls > id:activate

On the above example the left side of the > character, id:controls, represents the root element (can be called as "parent element" in this case). Right side of the locator string, id:activate, represents "child" element and it will be searched under the "parent element".

This way element search is more efficient, because search are restricted to certain section of element tree which can be quite huge especially on the desktop level and in certain applications.

Keyword examples:

Control Window    name:Calculator
Control Window    Calculator  # will execute search by 'name:Calculator'
Control Window    executable:Spotify.exe

some example locators, and can be omitted ie. space between locator keys means the same thing as and:

id:clearButton
type:Group and name:"Number pad" > type:Button and index:4
type:Group and name:"Number pad" > control:Button index:5
id:Units1 > name:${unit}
class:Button offset:370,0

About locator restrictions

Visual locators are not supported in this library and they can't be used in the same chain with these Windows locators. Visual locators are supported by the RPA.Desktop library. Locator chaining (image and Windows locators) support will be added in the future.

Locator syntax does not yet support OR operation (only AND operations).

About search depth

The library does element search depth by default to the level of 8. This means that locator will look into 8 levels of elements under element tree of the root element. This can lead into situation where element can't be found. To fix this it is recommended to set root element which can be found within 8 levels OR defining depth in the locator string to a bigger value, e.g. id:deeplyNestedButton depth:16. Useful keywords for setting root element are Control Window, Set Anchor and Get Element.

About the path strategy

When automation IDs and names aren't enough (or not reliable), then you can fallback to the positions of elements in a tree. This can be achieved using the path: strategy to specify a list of element positions which indicates how to traverse the tree from parent to child beginning with the resolved root.

Example: Calculator > path:2|3|2|8|2 - this locator looks for the "Calculator" window, then it looks for the 2nd direct child and then it looks for the 3rd one of the previous child and so on until it consumes the path completely. (indexes start with 1)

An alternative way to get the whole tree to explore it yourself would be to use the Print Tree keyword.

Keyboard and mouse

Keys for the keyboard actions are given using uiautomation specification.

Special keys which are given within {} syntax.

Key Maps to action
LBUTTON Left mouse button
RBUTTON Right mouse button
CANCEL Control-break processing
MBUTTON Middle mouse button (three-button mouse)
XBUTTON1 X1 mouse button
XBUTTON2 X2 mouse button
BACK BACKSPACE key
TAB TAB key
CLEAR CLEAR key
RETURN ENTER key
ENTER ENTER key
SHIFT SHIFT key
CTRL CTRL key
CONTROL CTRL key
ALT ALT key
PAUSE PAUSE key
CAPITAL CAPS LOCK key
KANA IME Kana mode
HANGUEL IME Hanguel mode (maintained for compatibility; use VK_HANGUL)
HANGUL IME Hangul mode
JUNJA IME Junja mode
FINAL IME final mode
HANJA IME Hanja mode
KANJI IME Kanji mode
ESC ESC key
ESCAPE ESC key
CONVERT IME convert
NONCONVERT IME nonconvert
ACCEPT IME accept
MODECHANGE IME mode change request
SPACE SPACEBAR
PRIOR PAGE UP key
PAGEUP PAGE UP key
NEXT PAGE DOWN key
PAGEDOWN PAGE DOWN key
END END key
HOME HOME key
LEFT LEFT ARROW key
UP UP ARROW key
RIGHT RIGHT ARROW key
DOWN DOWN ARROW key
SELECT SELECT key
PRINT PRINT key
EXECUTE EXECUTE key
SNAPSHOT PRINT SCREEN key
PRINTSCREEN PRINT SCREEN key
INSERT INS key
INS INS key
DELETE DEL key
DEL DEL key
HELP HELP key
WIN Left Windows key (Natural keyboard)
LWIN Left Windows key (Natural keyboard)
RWIN Right Windows key (Natural keyboard)
APPS Applications key (Natural keyboard)
SLEEP Computer Sleep key
NUMPAD0 Numeric keypad 0 key
NUMPAD1 Numeric keypad 1 key
NUMPAD2 Numeric keypad 2 key
NUMPAD3 Numeric keypad 3 key
NUMPAD4 Numeric keypad 4 key
NUMPAD5 Numeric keypad 5 key
NUMPAD6 Numeric keypad 6 key
NUMPAD7 Numeric keypad 7 key
NUMPAD8 Numeric keypad 8 key
NUMPAD9 Numeric keypad 9 key
MULTIPLY Multiply key
ADD Add key
SEPARATOR Separator key
SUBTRACT Subtract key
DECIMAL Decimal key
DIVIDE Divide key
F1 F1 key
F2 F2 key
F3 F3 key
F4 F4 key
F5 F5 key
F6 F6 key
F7 F7 key
F8 F8 key
F9 F9 key
F10 F10 key
F11 F11 key
F12 F12 key
F13 F13 key
F14 F14 key
F15 F15 key
F16 F16 key
F17 F17 key
F18 F18 key
F19 F19 key
F20 F20 key
F21 F21 key
F22 F22 key
F23 F23 key
F24 F24 key
NUMLOCK NUM LOCK key
SCROLL SCROLL LOCK key
LSHIFT Left SHIFT key
RSHIFT Right SHIFT key
LCONTROL Left CONTROL key
LCTRL Left CONTROL key
RCONTROL Right CONTROL key
RCTRL Right CONTROL key
LALT Left MENU key
RALT Right MENU key
BROWSER_BACK Browser Back key
BROWSER_FORWARD Browser Forward key
BROWSER_REFRESH Browser Refresh key
BROWSER_STOP Browser Stop key
BROWSER_SEARCH Browser Search key
BROWSER_FAVORITES Browser Favorites key
BROWSER_HOME Browser Start and Home key
VOLUME_MUTE Volume Mute key
VOLUME_DOWN Volume Down key
VOLUME_UP Volume Up key
MEDIA_NEXT_TRACK Next Track key
MEDIA_PREV_TRACK Previous Track key
MEDIA_STOP Stop Media key
MEDIA_PLAY_PAUSE Play/Pause Media key
LAUNCH_MAIL Start Mail key
LAUNCH_MEDIA_SELECT Select Media key
LAUNCH_APP1 Start Application 1 key
LAUNCH_APP2 Start Application 2 key
OEM_1 Used for miscellaneous characters; it can vary by keyboard.For the US standard keyboard, the ';:' key
OEM_PLUS For any country/region, the '+' key
OEM_COMMA For any country/region, the ',' key
OEM_MINUS For any country/region, the '-' key
OEM_PERIOD For any country/region, the '.' key
OEM_2 Used for miscellaneous characters; it can vary by keyboard.
OEM_3 Used for miscellaneous characters; it can vary by keyboard.
OEM_4 Used for miscellaneous characters; it can vary by keyboard.
OEM_5 Used for miscellaneous characters; it can vary by keyboard.
OEM_6 Used for miscellaneous characters; it can vary by keyboard.
OEM_7 Used for miscellaneous characters; it can vary by keyboard.
OEM_8 Used for miscellaneous characters; it can vary by keyboard.
OEM_102 Either the angle bracket key or the backslash key on the RT 102-key keyboard
PROCESSKEY IME PROCESS key
PACKET Used to pass Unicode characters as if they were keystrokes. The VK_PACKET key is the low word of a 32-bit Virtual Key value used for non-keyboard input methods. For more information, see Remark in KEYBDINPUT, SendInput, WM_KEYDOWN, and WM_KeyUp
ATTN Attn key
CRSEL CrSel key
EXSEL ExSel key
EREOF Erase EOF key
PLAY Play key
ZOOM Zoom key
NONAME Reserved
PA1 PA1 key
OEM_CLEAR Clear key

Examples.

lib = Windows()
# {Ctrl}, {Delete} ... are special keys' name in SpecialKeyNames.
lib.send_keys('{Ctrl}a{Delete}{Ctrl}v{Ctrl}s{Ctrl}{Shift}s{Win}e{PageDown}') #press Ctrl+a, Delete, Ctrl+v, Ctrl+s, Ctrl+Shift+s, Win+e, PageDown
lib.send_keys('{Ctrl}(AB)({Shift}(123))') #press Ctrl+A+B, type '(', press Shift+1+2+3, type ')', if '()' follows a hold key, hold key won't release util ')'
lib.send_keys('{Ctrl}{a 3}') #press Ctrl+a at the same time, release Ctrl+a, then type 'a' 2 times
lib.send_keys('{a 3}{B 5}') #type 'a' 3 times, type 'B' 5 times
lib.send_keys('{{}Hello{}}abc {a}{b}{c} test{} 3}{!}{a} (){(}{)}') #type: '{Hello}abc abc test}}}!a ()()'
lib.send_keys('0123456789{Enter}')
lib.send_keys('ABCDEFGHIJKLMNOPQRSTUVWXYZ{Enter}')
lib.send_keys('abcdefghijklmnopqrstuvwxyz{Enter}')
lib.send_keys('`~!@#$%^&*()-_=+{Enter}')
lib.send_keys('[]{{}{}}\|;:'",<.>/?{Enter}')

Using access key of the element (element property -> AccessKey 'alt+s'). The (+s) means that previous special key is kept down until closing parenthesis is reached.

On the below example this means that 'ALT' key is pressed down, then '+' and 's' keys are pressed down before they are all released up.

Send Keys   keys={Alt}(+s)

Mouse clicks can be executed with keywords specific for a type of a click, e.g. Click (normal click), Double Click and Right Click.

How to inspect

Most common, and recommended by Microsoft, inspector tool for Windows is Accessibility Insights that can be installed separately. Other options are tools Inspect Object and UI Automation Verify, which can be accessed by installing Windows SDK.

A more programmatic approach is to run Print Tree    log_as_warnings=${True} keyword and then observe in the logs the found elements structure starting from Desktop as root. (refer to keyword's documentation for more details)

Recording

The package provides some rudimentary inspecting and recording via script windows-record, which can be started in the command line (in a environment containing rpaframework-windows installation).

Recording inspects elements on mouse click and can be stopped with keyboard ESC. Expected console output.

Caveats

  • Make sure your display scaling is set to 100%, otherwise you might encounter issues when clicking or interacting with elements. (since offsets and coordinates get distorted)
C:\Users\User\robots\>windows-record  # or >python -m RPA.Windows
keyboard and mouse listeners started

--------------------------------------------------------------------------------
COPY & PASTE BELOW CODE INTO *** Tasks *** or *** Keywords ***
--------------------------------------------------------------------------------

Control Window    Taskbar  # Handle: 131380
Click   name:"Type here to search"
Control Window    Calculator  # Handle: 3411840
Click   name:Five
Click   name:Eight
Click   name:Five

--------------------------------------------------------------------------------

Examples

Both Robot Framework and Python examples follow.

The library must be imported first.

*** Settings ***
Library    RPA.Windows

Windows Calculator task

*** Tasks ***
Do some calculations
    [Setup]  Windows Run   calc.exe
    Control Window    name:Calculator
    Click    id:clearButton
    Send Keys   keys=96+4=
    ${result}=    Get Attribute    id:CalculatorResults    Name
    Log To Console    ${result}
    ${buttons}=  Get Elements  type:Group and name:"Number pad" > type:Button
    FOR  ${button}  IN  @{buttons}
        Log To Console   ${button}
    END
    [Teardown]   Close Current Window

Python example

from RPA.Windows import Windows

library = Windows()

def test_do_some_calculations():
    try:
        library.windows_run("calc.exe")
        library.control_window("name:Calculator")
        library.click("id:clearButton")
        library.send_keys(keys="96+4=")
        result = library.get_attribute("id:CalculatorResults", "Name")
        print(result)
        buttons = library.get_elements('type:Group and name:"Number pad" > type:Button')
        for button in buttons:
            print(button)
    finally:
        library.close_current_window()

Importing

Initialize self. See help(type(self)) for accurate signature.