Java application UI automation library using Java Access Bridge technology.
The library utilizes java-access-bridge-wrapper package to interact with Java UI. Currently only the 64-bit Windows OS is supported.
Inspecting elements
The recommended tool for inspecting Java application is:
Google's Access Bridge Explorer
The Accessibility Insights for Windows can show element properties if application framework supports Windows UI Automation (UIA), see more at using Accessibility Insights. Then recommended library would be RPA.Windows library.
Steps to enable
- Enable the Java Access Bridge in Windows
- Set environment variable RC_JAVA_ACCESS_BRIDGE_DLL as an absolute path to WindowsAccessBridge-64.dll. It is also possible to give DLL location as library initialization parameter access_bridge_path.
C:\path\to\java\bin\jabswitch -enable set RC_JAVA_ACCESS_BRIDGE_DLL=C:\path\to\Java\bin\WindowsAccessBridge-64.dll
*** Settings *** Library RPA.JavaAccessBridge access_bridge_path=C:\path\to\Java\bin\WindowsAccessBridge-64.dll
About Java wrapper callbacks and actions
There might be a compability issue with callbacks and actions on target Java application. Possible reasons:
- target application is executed with 32-bit Java
- target application does not support callbacks and/or actions
Workaround for this situation is to initialize JavaAccessBridge library with parameter ignore_callbacks=True. Then application's element information is still accessible and any actions on those elements can be performed with RPA.Desktop library.
Note. There are still keywords, for example. Call Element Action, which will cause error if used in this situation. To be fixed in future release.
*** Settings *** Library RPA.JavaAccessBridge ignore_callbacks=True
Locating elements
To automate actions on the Java application, the robot needs locations to various elements using a feature called locators. Locator describes properties of an element.
At the moment library contains basic level support for locators.
The common locator types are name and role.
To identify element with more than one property and can be used, for example:
role:push button and name:Clear
To address element within parent element > can be used, for example:
name:Find Purchase Orders > name:NumberField
Some keywords accept element as an parameter in place of locator.
New locator type strict has been added in rpaframework==12.5.0. Currently property values of string type have been evaluated with startsWith which can match several property values. With strict set in the locator string, all locator on the right side of this definition will be matched using strict (equal matching), example:
# without strict, name can be 'Type', 'Type1', 'Type of'... Get Elements role:push button and name:Type # name must be equal to 'Type' Get Elements role:push button and strict:True and name:Type
Keyword Get Elements has extra parameter strict, which when set to True forces all locator value matches to be strict, example:
# without strict, name can be 'Type', 'Type1', 'Type of'... Get Elements role:push button and name:Type # name must be equal to 'Type' and role must be equal to 'text' Get Elements role:text and name:Type strict=True
About JavaElement object
The JavaElement was added in rpaframework==12.3.0 for easy access into ContextNode objects which have been returned by Get Elements keyword.
Keyword Get Elements still returns ContextNode objects, but with parameter java_elements=True the keyword returns JavaElement objects instead (they still contain reference to ContextNode object via node property, e.g. JavaObject.node).
Properties and methods included in the JavaElement:
- name: str
- role: str
- states: list # list presentation of states (string)
- checked: bool
- selected: bool
- visible: bool
- enabled: bool
- states_string: str
- x: int # left coordinate of the element
- y: int # top coordinate of the element
- width: int
- height: int
- node: ContextNode # original ContextNode
- row: int # table row, -1 if element is not member of table
- col: int # table column, -1 if element is not member of table
- text: str # text content of the element
- column_count: int # table column count
- visible_children: list #visible_children elements of this element
- click() # method for clicking element center
- type_text() # method for typing text into element (if possible)
Interacting with elements
By default application elements are interacted with Actions supported by the element. Most common example is click action supported by an button element.
But because application and technology support for the actions might be limited, it is also possible to opt for interaction elements by their coordinates by giving keyword parameter action=False if parameter is available.
Examples
robotframework
*** Settings *** Library RPA.JavaAccessBridge Library Process *** Tasks *** Write text into Swing application Start Process java -jar BasicSwing.jar ... shell=${TRUE} ... cwd=${CURDIR} Select Window Chat Frame Type Text role:text ... text for the textarea Type Text role:text ... text for the input field ... index=1 ... clear=${TRUE} Click Element role:push button and name:Send
Python
from RPA.JavaAccessBridge import JavaAccessBridge import subprocess jab = JavaAccessBridge() subprocess.Popen( ["java", "-jar", "BasicSwing.jar"], shell=True, cwd=".", close_fds=True ) jab.select_window("Chat Frame") jab.type_text( "role:text", "text for the textarea", enter=True ) jab.type_text( "role:text", "text for the input field", index=1, clear=True ) jab.click_element("role:push button and name:Send")