Cloud machine learning (ML) APIs example robot

Supporting advanced data analytics or decision logic beyond simple deterministic rules often requires machine learning (ML) solutions. One way to provide the needed intelligence is to use commercial cloud services that can be applied for a number of typical use cases. Such services include, for example, image analysis, natural language processing, document analysis, and optical character recognition (OCR).

These examples demonstrate how to use cloud-based ML APIs with RPA Framework. Currently, a selected set of services from AWS (RPA.Cloud.AWS), Microsoft Azure (RPA.Cloud.Azure) and Google Cloud (RPA.Cloud.Google) are available.

You can access cloud ML services that are not supported by RPA Framework by writing custom Robot Framework or Python libraries to connect them via REST APIs or service-specific SDKs. If your automation team has in-house machine learning expertise, the possibilities for self-developed ML services from the cloud are almost limitless. A separate article on this topic will be published later.

Prerequisites

To use cloud ML APIs, you need:

• An account for the cloud provider of your choice.
• The required ML service(s) activated and accessible from the public internet.
• Credentials to the service(s) - the format depends on the cloud provider.

Setup the access keys

The authentication method varies depending on the cloud service:

• AWS: AWS key and AWS key ID (AWS_KEY and AWS_KEY_ID)
• Azure: Azure subscription key (AZURE_SUBSCRIPTION_KEY)
• Google Cloud: Service credentials file

There are three different ways to set the access credentials - See the documentation of the RPA.Cloud libraries. Here we present an approach that works both in a local and Control Room setting.

For local robot runs, we use a vault.json file placed outside the repository, e.g., in /Users/<username>/vault.json. For AWS and Azure, we can save the key values directly in the file. For Google Cloud, the content of the service credentials file must be mapped to a string, with special characters escaped. This can be done, e.g., with the following Python script.

This will result in the following JSON file:

If the robot is run in Control Room, store the above credential information in your workspace vault with the names aws, azure and gcp.

Configure file vault support

To run in your local or development environment, edit the path to the vault file in devdata/env.json.

Sample data

In this example, we will use images of an invoice and a sphygmomanometer to showcase the image processing functionalities. For natural language processing, we will be working on a piece of text given as a string.

The sample invoice and picture are shown below.

Using ML services with RPA Framework

The robot code resides in the tasks.robot file. In the *** Settings *** section, we define which libraries are used and configure the cloud libraries.

Match the region argument of RPA.Cloud.AWS and RPA.Cloud.Azure to your region in AWS or Azure.

The basic functionality of RPA.Cloud libraries is to send a request to the API and return the response as JSON without any modification to the content. The JSON data can then be processed further based on the needs of the automated process. Such operations are not currently fully supported by RPA Framework since each service has a different response syntax.

AWS

To analyze a scanned invoice, we use the Textract service from AWS.

After configuring the client and getting the answer from AWS, we use the Get Tables keyword to collect tables from the response. In this example, table data is correctly found.

The full response from AWS is stored in a JSON file (here output/textract.json). Parsing the other data, like key-value pairs, from the JSON response needs to be done by separate functions. The syntax of the Textract response and code examples are available on the Textract developer guide.

Azure

The usage of (RPA.Cloud.Azure) is demonstrated by Text analytics service. In the following task, the robot recognizes the language, key phrases, and sentiment of a sample text.

The outcome of the analysis is stored in JSON files (here output/text_lang.json, output/text_phrases.json, and output/text_sentiment.json). As a summary, we find the following information:

Google Cloud

From RPA.Cloud.Google we try Google Vision AI which can be used, for example, to label an image or detect the text.

The full responses are stored in JSON files (here output/vision_labels.json and output/vision_text.json). The summary of the outcome is shown below. Note that in order to get "sphygmomanometer" among the labels, one needs to train a model since such a term is not included in Google Vision AI. The recognized text blocks cover almost all the text appearing in the image. The full response data also contains the bounding box information.

Other examples

• Process PDF invoices with Amazon Textract
• Process purchase invoices automatically using Aito.ai and Google Sheets
• IMDB Movie Review Sentiment Analysis With Amazon Comprehend