JP7568371B2 - System and method for activity target selection for robotic process automation - Patents.com - Google Patents

Description

The present invention relates to robotic process automation (RPA), and more particularly to a system and method for automatically identifying user interface elements that are the target of an activity such as a mouse click or text entry.

RPA is an emerging field of information technology that aims to improve productivity by automating repetitive computing tasks, thus freeing human operators to perform intellectually sophisticated and/or creative activities. Notable tasks targeted for automation include extracting structured data from documents and interacting with user interfaces, for example to fill out forms, among others.

A clear direction of RPA development is aimed at simplifying the programming and management of software robots, with the ultimate goal of expanding the reach of RPA technology to users who lack advanced programming skills or training. One way to make RPA more accessible is the development of RPA-oriented integrated development environments (IDEs) that allow programming of robots via graphical user interface (GUI) tools rather than coding itself.

However, automating interactions with a user interface poses substantial technical challenges, e.g., unambiguously identifying target elements such as buttons or form fields. Furthermore, RPA applications may fail due to changes in the appearance of the interface (e.g., positioning of various elements, color schemes, fonts, etc.) that occur between the design and run-time of the respective software robot. There is therefore a continuing interest in developing robust and scalable software robots that are immune to such changes.

According to one aspect, the method includes employing at least one hardware processor of a computer system, which performs, in response to receiving an RPA script comprising a set of target functions and a set of anchor functions, automatically identifying a runtime instance of a target element in a runtime user interface (UI) exposed by the computer system, where the target function is a property of the target element of the target UI and the anchor function is a property of the anchor element of the target UI. The method further includes automatically performing an operation that reproduces a result of an interaction between a human operator and the runtime instance of the target element, where the operation is determined according to the RPA script. The set of target functions comprises a target ID indicating a location of the target element in a tree representation of the target UI, a target image comprising an image of the target element in the target UI, and a target text comprising a sequence of characters displayed by the target element in the target UI. The set of anchor functions comprises an anchor ID indicating a location of the anchor element in a tree representation of the target UI, an anchor image comprising an image of the anchor element in the target UI, and an anchor text comprising a sequence of characters displayed by the anchor element in the target UI. The method includes identifying a runtime instance of a target element according to a target ID, a target image, a target text, an anchor ID, an anchor image, and an anchor text.

According to another aspect, a computer system comprises at least one hardware processor configured to execute an automated target application and an RPA robot. The automated target application is configured to expose a runtime UI. The RPA robot is configured to automatically identify a runtime instance of a target element in the runtime UI exposed by the computer system in response to receiving an RPA script comprising a set of target functions and a set of anchor functions, where the target function is a property of the target element of the target UI and the anchor function is a property of the anchor element of the target UI. The RPA robot is further configured to automatically execute an operation that reproduces the result of an interaction between a human operator and the runtime instance of the target element, where the operation is determined according to the RPA script. The set of target functions comprises a target ID indicating a position of the target element in a tree representation of the target UI, a target image comprising an image of the target element in the target UI, and a target text comprising a sequence of characters displayed by the target element in the target UI. The set of anchor features includes an anchor ID indicating a location of the anchor element in a tree representation of the target UI, an anchor image comprising an image of the anchor element in the target UI, and anchor text comprising a sequence of characters displayed by the anchor element in the target UI. Automatically identifying a runtime instance of the target element includes identifying a runtime instance of the target element according to the target ID, the target image, the target text, the anchor ID, the anchor image, and the anchor text.

According to another aspect, a non-transitory computer-readable medium stores instructions that, when executed by at least one hardware processor of a computer system configured to expose a runtime UI, cause the computer system to automatically identify a runtime instance of a target element in a runtime UI exposed by the computer system in response to receiving an RPA script comprising a set of target functions and a set of anchor functions, where the target function is a property of the target element of the target UI and the anchor function is a property of the anchor element of the target UI. The instructions further cause the computer system to automatically perform an operation that reproduces a result of an interaction between a human operator and the runtime instance of the target element, where the operation is determined according to the RPA script. The set of target functions comprises a target ID indicating a location of the target element in a tree representation of the target UI, a target image comprising an image of the target element in the target UI, and a target text comprising a sequence of characters displayed by the target element in the target UI. The set of anchor features includes an anchor ID indicating a location of the anchor element in a tree representation of the target UI, an anchor image comprising an image of the anchor element in the target UI, and anchor text comprising a sequence of characters displayed by the anchor element in the target UI. Automatically identifying a runtime instance of the target element includes identifying a runtime instance of the target element according to the target ID, the target image, the target text, the anchor ID, the anchor image, and the anchor text.

The foregoing aspects and advantages of the present invention will be better understood upon reading the following detailed description and upon reference to the drawings.

FIG. 1 illustrates an exemplary robotic process automation (RPA) system, according to some embodiments of the present invention.

FIG. 1 illustrates exemplary software performed on an RPA client in accordance with some embodiments of the present invention.

FIG. 2 illustrates an exemplary user interface (UI) comprising multiple UI elements in accordance with some embodiments of the present invention.

FIG. 2 illustrates an exemplary series of steps performed by a scripting application according to some embodiments of the present invention.

1 illustrates an exemplary user interface, a target element, and multiple candidate anchor elements according to some embodiments of the present invention.

FIG. 2 illustrates an exemplary series of steps performed to automatically determine an anchor element associated with a target element, according to some embodiments of the present invention.

1A-1C illustrate an exemplary user interface, an element of interest, and multiple candidate anchor placements according to some embodiments of the present invention.

FIG. 10 illustrates an alternative series of steps performed by a scripting application to automatically identify anchor elements related to a target element, according to some embodiments of the present invention.

2A-2C illustrate an example UI tree and example element IDs characterizing nodes of the UI tree, according to some embodiments of the present invention.

2A-2C are diagrams illustrating various types of data that characterize UI elements according to some embodiments of the present invention.

FIG. 1 illustrates an exemplary series of steps performed by an RPA robot, according to some embodiments of the present invention.

FIG. 1 illustrates an exemplary sequence of steps performed by an RPA robot to identify target UI elements at runtime, according to some embodiments of the present invention.

FIG. 2 illustrates a set of exemplary inter-element distances according to some embodiments of the present invention.

FIG. 13 illustrates another set of exemplary inter-element distances, according to some embodiments of the present invention.

1A-1C are diagrams illustrating exemplary inter-element angles, according to some embodiments of the present invention.

4A-4C illustrate exemplary overlap between two UI elements according to some embodiments of the present invention.

FIG. 1 illustrates an exemplary embodiment of a computing device configured to perform the methods described herein.

In the following description, it is understood that all enumerated connections between structures may be direct operational connections or indirect operational connections through intermediate structures. A set of elements includes one or more elements. An enumeration of elements is understood to refer to at least one element. A plurality of elements includes at least two elements. Any use of "or" means a negative exclusive or. Unless otherwise required, the described method steps need not necessarily be performed in the particular illustrated order. A first element (e.g., data) derived from a second element includes a first element equal to the second element, as well as a first element generated by processing the second element and optionally other data. Making a decision or judgment according to a parameter includes making a decision or judgment according to the parameter and optionally other data. Unless otherwise specified, an indicator of some quantity/data may be the quantity/data itself or an indicator different from the quantity/data itself. A computer program is a sequence of processor instructions that perform a task. Computer programs described in some embodiments of the invention may be standalone software entities or subentities (e.g., subroutines, libraries) of other computer programs. The term "database" is used herein to denote an organized, searchable collection of data. Computer-readable media encompass non-transitory media such as magnetic, optical, and semiconductor storage media (e.g., hard drives, optical disks, flash memory, DRAM), as well as communication links such as conductive cables and fiber optic links. According to some embodiments, the invention provides a computer system comprising, among other things, hardware (e.g., one or more processors) programmed to perform the methods described herein, and a computer-readable medium encoding instructions for performing the methods described herein.

The following description illustrates exemplary embodiments of the present invention and is not intended to be limiting.

1 illustrates an exemplary robotic process automation system according to some embodiments of the present invention. Each of a plurality of RPA clients 10a-e represents a computing device having at least a hardware processor, a memory unit, and a network adapter that enables the respective RPA client to connect to a computer network and/or other computing devices. The exemplary RPA clients 10a-e include, among others, personal computers, laptop and tablet computers, as well as mobile communication devices (e.g., smartphones). In an exemplary use case scenario, the RPA clients 10a-10d represent desktop computers belonging to the accounting or human resources departments of a company. The illustrated RPA clients 10a-d are interconnected by a local communication network 12, which may comprise a local area network (LAN). The clients 10a-d may further access an extended network 14, which may comprise a wide area network (WAN) and/or the Internet. In the exemplary configuration of FIG. 1, the RPA client 10e is directly connected to the extended network 14. Such clients may represent mobile computers, such as laptops, tablet computers, or mobile phones, that connect to the network 14 at various access points.

In a typical RPA scenario, company employees perform repetitive tasks using business applications (e.g., word processors, spreadsheet editors, browsers, email applications), for example to issue invoices to various business clients. To actually perform each task, the employees perform a series of operations/actions, which are considered herein as business processes. Exemplary operations forming part of an invoice issuing business process may include opening a Microsoft Excel spreadsheet, searching for the client's company details, copying the respective details into an invoice template, filling in invoice fields indicating the items purchased, switching to an email application, composing an email message to the respective client, attaching the newly created invoice to the respective email message, and clicking a "send" button. The RPA software running on the employee's computer may automate the respective business process by mimicking the set of operations performed by the respective human operators in the course of performing the respective tasks. Exemplary processes typically subject to such automation include processing payments, invoicing, communications with business clients (e.g., distribution of company newsletters and/or product offerings), internal communications (e.g., scheduling of memos, meetings and/or tasks), payroll, etc.

Mimicking a human operation/action is understood herein to encompass reproducing the sequence of computing events that occur when a human operator performs the respective operation/action on a computer, and reproducing the results of the respective operation performed by the human operator on a computer. For example, mimicking the action of clicking a button on a graphical user interface may include the operating system moving a mouse pointer to the respective button and generating a mouse click event, or may include toggling the respective GUI button itself to a clicked state.

FIG. 2 illustrates exemplary software executed on an RPA client 10 according to some embodiments of the present invention. The RPA client 10 represents any of the RPA clients 10a-e of FIG. 1. The RPA client 10 executes an operating system (OS) 40 and a set of business applications 42. The OS 40 may comprise any widely available operating system, such as Microsoft Windows, MacOS, Linux, iOS, or Android, among others, that comprises a software layer that interfaces between the applications 42 and the hardware of the RPA client 10. The business applications 42 generally represent any computer program used by a human operator of the RPA client 10 to perform tasks. Exemplary business applications 42 include, among others, word processors, spreadsheet applications, graphics applications, browsers, social media applications, and electronic communication applications. At least one business application 42 is configured to expose a user interface (UI) that is subject to automation, as described in more detail below.

In some embodiments, the RPA client 10 further executes an RPA robot 44 that comprises a set of interconnected computer programs that collectively implement business process automation. An exemplary RPA robot is built using the Windows Workflow Foundation application programming interface from Microsoft Corporation. In some embodiments, the RPA robot 44 executes within a separate dedicated virtual machine instantiated on the RPA client 10.

The components of the RPA robot 44 include an RPA agent 43 and a set of robot executors 45. The robot executors 45 are configured to receive an RPA script 50, which indicates a series of operations (also known in the art as activities) that mimic the actions of a human operator executing a business process, and to actually perform each series of operations on a respective client machine. The RPA script 50 is typically process-specific, i.e., each separate business process is described by a separate set of RPA scripts. The RPA script 50 may be formulated according to any data specification known in the art. In a preferred embodiment, the RPA script 50 is coded in a version of the Extensible Markup Language (XML), although the script 50 may also be formulated in a programming language such as C#, Visual Basic, Java, etc. Alternatively, the RPA script 50 may be specified in an RPA-specific version of bytecode, or even as a series of instructions formulated in a natural language such as English, Spanish, Japanese, etc. In some embodiments, the script 50 is pre-compiled into a set of native processor instructions (e.g., machine code).

In some embodiments, the robot executor 45 comprises an interpreter (e.g., a just-in-time interpreter or compiler) configured to convert the RPA scripts 50 into runtime packages comprising processor instructions for performing operations described in the respective scripts. Thus, executing the scripts 50 may include the executor 45 converting the RPA scripts 50, loading the resulting runtime package into memory, and instructing the processor of the RPA client 10 to launch and execute the runtime package.

The RPA agent 43 may manage the operation of the robot executor 45. For example, the RPA agent 43 may select tasks/scripts for execution by the robot executor 45 according to input from a human operator and/or according to a schedule. The agent 43 may further configure various operating parameters of the executor 45. If the robot 44 includes multiple executors 45, the agent 43 may coordinate their activities and/or inter-process communication. The RPA agent 43 may further manage communication between the RPA robot 44 and other components of the RPA system shown in FIG. 1. Such components may run on other RPA clients and/or on a set of robot management servers 11a-b. In one such example, the servers 11a-b may run a robot orchestrator service that coordinates RPA activities across multiple client machines and enables more complex scheduling and/or license management. The servers 11a-b may further receive data from the individual RPA robots indicative of various intermediate values and/or results of execution of the RPA scripts. Such data may be used to generate activity reports, enforce license agreements, and/or mitigate malfunctions.

In some embodiments, the RPA client 10 further executes a scripting application 46 configured to enable a human operator of the RPA client 10 to create the RPA script 50 and thus effectively design a robot to perform a set of activities. The creation application 46 may function like an integrated development environment (IDE) with a code editor and/or user interface that allows the operator to interact with a set of tools to model a business process. An exemplary creation application may allow a user to select a business application 42 and indicate a desired way to interact with each application, e.g., a sequence of operations to be performed by the robot 44. Exemplary operations include, for example, opening a specific Excel spreadsheet, reading data from a specific row/column of a data table, processing the respective data in a specific way, clicking a specific button, creating and sending an email message, navigating to a specific uniform repository (URL), etc. In some embodiments, the creation application 46 outputs the RPA script 50 in a format (e.g., XML) that is readable by the RPA robot 44. The RPA scripts 50 may be stored in a script repository 15 that is communicatively coupled to and accessible to the RPA clients 10a-e via networks 12 and/or 14 (see FIG. 1). In a preferred embodiment, the script repository 15 is directly linked to the robot management servers 11a-b. The script repository 15 may be organized as a database, e.g., any structured collection of data that allows selective retrieval of the scripts 50 according to a set of criteria.

Those skilled in the art will appreciate that not all of the components shown in FIG. 2 need to execute on the same physical processor or machine. In a typical RPA configuration, script development/robot design is performed on one machine (commonly known in the art as the "design side"). The resulting RPA script 50 is then distributed to multiple other users and machines for execution (commonly known as the "runtime side" or simply "runtime").

3 illustrates an exemplary user interface (UI) 58 according to some embodiments of the present invention. The UI 58 may be exposed by any of the business applications 42. A user interface is a computer interface that enables human-machine interaction, e.g., an interface configured to receive user input and respond to the respective input. A common example of a user interface is known as a graphical user interface (GUI), which enables human-machine interaction through a set of visual elements that are displayed to a user. The exemplary UI 58 includes a set of exemplary windows 60a-b and a set of exemplary UI elements including a menu indicator 62a, an icon 62b, a button 62c, and a text box 62d. Other exemplary UI elements include windows, labels, forms, individual form fields, toggles, links (e.g., hyperlinks, hypertext, or uniform resource identifiers), among others. The UI elements may display information, receive input (text, mouse events), and/or control functions of the software and/or the respective computer system.

Some UI elements are interactive in the sense that acting on them (e.g., click button 62c) triggers a behavior/response. Such behavior/response is usually specific to each element or group of elements. For example, clicking a save button and clicking a print button produce different effects. The same keyboard shortcut (e.g., Ctrl-G) may have one effect when performed in one window/application and a completely different effect when performed in another window/application. Thus, although the operation/action (performing a click, pressing a keyboard key combination, writing a series of characters, etc.) is the same, the outcome of each action may depend substantially on the operand of each operation. An operand is defined herein as a UI element that is acted upon by a current operation/action, such as a click or keyboard event, or that is otherwise selected to receive a respective user input. The terms "target" and "operand" are used interchangeably herein. Because UI element behavior is element specific, successful RPA may require unambiguously and precisely identifying the operand of each scripted RPA activity.

FIG. 4 illustrates an exemplary sequence of steps performed by the scripting application 46 according to some embodiments of the present invention. Step 101 exposes a design-side instance of a target UI, i.e., a user interface of a business application 42, that is the subject of the current automation. Step 101 may include, for example, invoking an instance of the business application 42. In step 102, the application 46 may expose a robot design interface (e.g., a GUI) that allows a user to indicate desired activities to be performed by the robot 44 on the exposed target UI. In some embodiments, the activities may be accomplished through a hierarchy of activity menus. Activities may be grouped according to various criteria, for example, according to type of business application (e.g., MS Excel activity, web activity, email activity) and/or according to type of interaction (e.g., mouse activity, hotkey activity, data grab activity, form fill activity, etc.). Step 104 receives user input indicating the respective activity. For example, step 104 may include intercepting a mouse click event and determining which menu item the user clicked to select the activity. In a further step 106, the application 46 may expose an activity configuration interface that allows the user to configure various options and/or parameters of the respective activity. One exemplary activity parameter is the operand/target UI element of the respective activity. In one example where the activity involves a mouse click, the target UI element may be a button, a menu item, a hyperlink, etc. In another example where the activity involves filling out a form, the target UI element may be a particular form field that is to accept the respective text input. The application 46 may allow the user to indicate the target UI element in various ways. For example, it may prompt the user to select the target element from a menu/list of candidate UI elements. In a preferred embodiment, the application 46 may expose an instance of the target UI (i.e., the UI of the business application with which the robot 44 is to interact, e.g., MS Excel, a browser, an email program, etc.), highlight a subset of UI elements within each UI, and prompt the user to click on one to indicate a selection. In step 108, application 46 may receive and process user input indicating the selected target element, for example, by invoking specific OS functionality to detect the mouse click and further by identifying the clicked UI element.

Next, in step 110, some embodiments may automatically determine an anchor UI element associated with the selected target element. An anchor element (or simply "anchor") is defined herein as a UI element that is displayed simultaneously with an associated target UI element, in the sense that the target and anchor are visible simultaneously in their respective user interfaces. Furthermore, anchor elements and target elements typically have a semantic connection, e.g., they both belong to the same group/container of UI elements and/or they both perform a function. Exemplary anchor elements associated with input fields include, among others, a text label displayed near the respective input field and a title of a form that includes the respective input field. Exemplary anchor elements associated with buttons include the respective buttons and text displayed on another button of the same UI. FIG. 5 illustrates an exemplary UI 58 having a target element 64 and multiple potential anchor elements 66a-e in accordance with some embodiments of the present invention.

Determining the anchor of the target element may include selecting an anchor from a set of candidate UI elements, for example as shown in FIG. 6. In step 202, application 46 may generate a set of candidate anchor elements (see, for example, items 66a-e in FIG. 5) selected from the set of UI elements displayed by UI 58. The candidate anchor elements may be selected according to the element type (e.g., button, text, input field, etc.) of the respective target element. In some embodiments, the candidate anchors may be selected according to whether they belong to the same group of elements/UI containers as the respective target. For example, if the target element is a form field, some embodiments will select anchor candidates only from among UI elements that belong to the same form field. In the case of an HTML document, some embodiments may select label candidates from the same <div> or <span> container as the target element.

Next, in step 204, application 46 may evaluate each candidate anchor element according to a set of criteria. In some embodiments, step 204 may include determining an anchor fitness score that may combine multiple sub-scores evaluated according to separate criteria. An exemplary criterion is the relative position of the candidate anchor with respect to the target element. The relative position may be determined according to a set of distances, angles, and/or overlaps between the respective target element and the candidate anchor element. Examples of such determinations are described in detail below in connection with FIGS. 13-16. Some embodiments consider UI elements that are located near and/or substantially aligned with the target element to be relatively reliable anchors. In such embodiments, such UI elements may receive a higher fitness score than other UI elements that are farther away from and/or not aligned with the selected target element.

Other exemplary anchor fitness criteria may include the image and/or text content of the respective UI element. Some embodiments prioritize text labels as anchor elements, so that UI elements that do not contain text may receive a relatively lower fitness score than other UI elements that display snippets of text. Another exemplary criterion may be the length of the text displayed by the UI element, and some embodiments may prioritize small text elements because they are more likely to be labels. In such an embodiment, relatively small text elements may receive a relatively higher fitness score compared to text elements that have a significant amount of text.

Yet another exemplary criterion may include the number of anchor candidates with similar appearance, e.g., the number of UI elements that display the same text. In one exemplary scenario, the target UI 58 includes a form designed to collect data about multiple people and that has multiple fields labeled "Last Name." In such a situation, the "Last Name" label may be highly unreliable in identifying a particular form field. Thus, some embodiments may determine whether each anchor candidate is unique (in the sense that there are no other UI elements that have a similar image or display similar text) and, if not, assign a relatively low anchor fitness score to each anchor candidate. Alternative embodiments may assign multiple anchors to the same target element, e.g., labels located near each form field and the title of each input form or input block.

Step 206 may then compare the assessed scores for the candidate anchors. If there is a clear winning candidate, then in step 210, the scripting application 46 may select the candidate element with the highest fitness score as the anchor element associated with the target element determined in step 108 (FIG. 4). In the event of a tie, i.e., multiple candidates have the same fitness score, some embodiments may prompt the user to explicitly indicate the UI element to be used as the anchor (step 208).

7-8 show an alternative method of automatically selecting an anchor UI element. In contrast to the above-mentioned method in which application 46 generates a set of candidate elements and then evaluates their suitability as an anchor according to their position relative to the target element, step 222 may generate candidate placements within UI 58, e.g., as pairs of screen coordinates {X,Y}. Such an embodiment relies on the observation that reliable anchors, such as text labels, are typically found next to their associated targets, e.g., to the left of them, or directly above or below them, depending on the default reading direction of the respective natural language of target UI 58. Some embodiments may therefore explicitly look for potential anchor elements in such placements. FIG. 7 shows multiple candidate placements 65a-d. Such candidate placements may be determined according to the screen position of the target UI element (shown as item 64 in FIG. 7) and/or according to the size of the target element. In some embodiments, the candidate placements are generated randomly, e.g., as a sum of a deterministic component and a random component.

Step 224 may then identify all UI elements that are approximately located in the candidate arrangement. In some embodiments, an element may be considered to be located in a particular arrangement if the respective arrangement is within the screen bounds of the respective element. Another embodiment may consider an element to be located in a particular arrangement if the distance between the center/centre of gravity of the respective element and the respective arrangement is less than a predefined threshold. In the example of FIG. 7, UI element 66 may be considered to be located in candidate arrangement 65a. In some embodiments, step 224 includes issuing a call to a native function of OS 40, the respective function configured to return a list of UI elements that occupy a particular region of the screen. Other methods of determining which UI elements are located in the candidate arrangements include parsing the underlying source code (e.g., HTML scripts, style sheets) of the respective UI.

If the UI elements are not placed in the respective candidate placements, some embodiments return to step 222 to generate another candidate placement. Otherwise, in step 226, the scripting application 46 may filter the identified set of UI elements according to a set of anchor suitability criteria. Such criteria may include, among others, visibility (e.g., only visible UI elements may be selected as anchors), and element type (e.g., text elements may be preferred over other types of UI elements). Other suitability criteria may be similar to those described above in connection with FIGS. 5-6. For example, the application 46 may evaluate the position score according to whether the respective UI elements are aligned with the target element, whether the respective UI elements substantially overlap with the target element, etc.

If none of the UI elements placed in the candidate arrangements are deemed suitable for the anchor (e.g., if they do not receive a suitability score above a predefined threshold), some embodiments may return to step 222 to generate another candidate arrangement. Otherwise, step 232 may select a UI element that qualifies as an anchor associated with the respective target element.

In response to identifying the target element and/or anchor UI element, in a series of steps 112-114 (FIG. 4), the scripting application 46 may determine a set of element property features for each target element and anchor element. Such element property features, according to some embodiments of the invention, are illustrated in FIG. 9 and include, among other things, a set of element IDs 80a-b, a set of element texts 82a-b, and a set of element images 84a-b that characterize the target element 64 and the anchor element 66, respectively.

The element IDs 80a-b identify each UI element to the operating system and/or respective business application 42 as a particular object within a hierarchy of objects used by the RPA client 10 to represent and/or render the respective user interface. In some embodiments, the element IDs 80a-b are included in the source code of the interface 58, e.g., as a set of attribute-value pairs. The term source code of a user interface is understood herein to indicate a programmatic representation of the content displayed by the respective user interface. Source code may encompass programs/scripts written in a programming language as well as data structures residing in the memory of the RPA client 10. Exemplary source code comprises an HTML document that is rendered as a web page by a web browser application.

In modern computing platforms, operating systems typically represent each user interface as a hierarchical data structure commonly known as a UI tree. An exemplary UI tree comprises the document object model (DOM) underlying a web page rendered by a browser application. FIG. 10 illustrates an exemplary UI tree 70 having multiple nodes 72a-e. In some embodiments, each node 72a-e comprises an object that represents a portion of the UI 58. In the exemplary UI as shown in FIG. 5, the root node 72a may represent the entire UI window. Its child nodes may represent individual UI elements (e.g., text boxes, labels, form fields, buttons, etc.), groups of elements, separate areas or blocks of the respective UI, etc. An intermediate node, such as node 72b in FIG. 10, may represent an entire form with all its input fields, labels, and buttons. For example, node 72c may represent the contents of a <form> or <fieldset> container in an HTML document. Another example of an intermediate node may represent the contents of a <div> or <span> HTML container. Yet another example of an intermediate node comprises the contents of a document header or footer. End nodes (also known in the art as leaf nodes), such as 72b, 72d, and 72e, are nodes that have no further child nodes and may represent individual UI elements (e.g., buttons, individual labels, individual input fields). In one example of a web browser UI:

In some embodiments, each node 72a-e is specified using a set of attribute-value pairs that may indicate, for example, among other things, the identity of the respective node's parent node, the identity of the respective node's child node, a name, and the type of UI element represented by the respective node.

In some embodiments, the element IDs characterizing the UI elements comprise a set of node identifiers that collectively indicate the placement of the nodes in the UI tree 70, with each node representing a respective UI element. In one such example, the element ID 80c indicates a subset of the nodes of the UI tree 70, referred to herein as a subtree (see example subtrees 74a-d in FIG. 10). Thus, the element ID 80c identifies the node/UI element as belonging to the respective subtree. For example, node 72d belongs to subtree 74c. The example element ID 80c includes a set of attribute-value pairs that identify the respective UI element as a "push button" called "Accept" that is visible in a window of an application called "uidouble.exe". The illustrated format of the element ID 80c is provided by way of example only, and one of ordinary skill in the art would understand that there may be multiple other ways of representing the placement of a particular node in a UI tree besides a list of attribute-value pairs.

In some embodiments, determining the element IDs 80a-b characterizing the target element and anchor element, respectively, includes parsing the source code (e.g., an HTML document) of the target user interface 58 and extracting the respective element IDs, e.g., as a set of attribute-value pairs associated with each UI element.

In some embodiments, each element text 82a-b (FIG. 9) comprises a computer encoding of text (a series of alphanumeric characters) to be displayed within the screen bounds of the respective UI element. In the illustrated example, element text 82a has the value NULL because target element 64 does not display any text. Element text 82b, on the other hand, consists of the text "Cash Deposit." The computer encoding of the text may comprise, for example, a series of numeric codes (e.g., Unicode), each code corresponding to a distinct character of element text 82a-b.

Embodiments of the scripting application 46 may determine the element text 82a-b using various methods. If the application 46 has access to the source code of the UI 58, the application 46 may attempt to extract the element text 82a-b from the respective source code. For example, labels displayed on buttons on a web page may be found by parsing the HTML document associated with the respective web page. For other business applications 42, the scripting application 46 may analyze data structures of the OS 40 and/or the business application 42 to determine whether the element text 82a-b is included in the source code of the UI 58.

In an alternative embodiment, application 46 may employ an image analysis tool, such as an optical character recognition (OCR) computer program, to determine element text 82a-b. In one such example, the OCR tool may input an image of a screen region including each target element and/or anchor UI element and return a set of text tokens (e.g., words) and a bounding box determined for each text token. Exemplary bounding boxes include, among other things, a polygon circumscribing each text token and a convex hull of each token. In FIG. 9, the bounding box is illustrated by a dashed rectangle that encloses the text "Cash Deposit." In response to receiving the text tokens and bounding boxes, application 46 may determine whether any bounding boxes substantially overlap the respective UI elements and, if so, select the text tokens located within the respective bounding boxes as text elements 82 that characterize the respective target element or anchor UI element. If a sufficient percentage of each bounding box (e.g., more than 50%, typically 80-100%) is located within the screen boundaries of the respective UI element, a substantial overlap can be established.

In some embodiments, each element image 84a-b (FIG. 9) characterizing a UI element includes a computer encoding of an image displayed on the screen within the boundaries of the respective UI element. The computer encoding of the image may include an array of pixel values, possibly across multiple channels (e.g., RGB), corresponding to the respective screen region, and/or a set of values calculated according to the respective array of pixel values (e.g., a JPEG or wavelet representation of the respective array of pixel values). Determining each element image 84a-b may include clipping the UI 58, i.e., grabbing the content of a limited region of the UI 58 that represents the respective UI element.

In a further step 116 (FIG. 4), the scripting application 46 may formulate an RPA script corresponding to the selected RPA activity. In other words, in step 116, the application 46 outputs the code of the robot to be used at run time, for example in a script file. The RPA script 50 may be formulated in any computer-readable encoding known in the art, for example in a version of XML, or may be compiled into a sequence of native processor instructions (for example machine code).

For each activity/automation step, the authoring application 46 outputs to the RPA script 50 an indicator of the respective activity (e.g., clicking, typing, etc.) and may further output encodings of element IDs 80a-b, element text 82a-b, and element images 84a-b that characterize the target element and anchor UI element determined in steps 108-110. The encodings of the property functions may include the property data itself and/or other representations of such data, e.g., an indicator of a network location (e.g., URL, network address) where the element property data may be accessed remotely.

In some embodiments, the application 46 may further output a set of parameter values to the RPA script 50 for configuring each activity, for example using a set of attribute-value pairs. One exemplary parameter is a match accuracy that indicates a threshold for comparing the design-time element image 84 stored in the RPA script 50 with the runtime images of the candidate UI elements (see details below in connection with FIGS. 11-12). Another exemplary parameter is a timeout threshold that indicates a maximum amount of time the robot 44 may spend at runtime and/or attempting to identify a target UI element.

Once the automation design phase is complete, the RPA script 50 may be sent to the script repository 15 and/or distributed to other RPA clients for execution (see, e.g., FIG. 1). FIG. 11 shows an exemplary sequence of steps performed by the RPA robot 44 at runtime. In response to receiving the RPA script 50, step 304 determines the type of activity to be performed according to the content of the RPA script 50. Step 304 may further determine the target UI and/or the runtime business application (e.g., MS Excel, Google Chrome, etc.) with which the respective robot is configured to interact according to the RPA script 50. In step 306, the RPA robot 44 may expose the respective target UI, for example, by invoking an instance of the respective business application on the local client machine. A further step 308 may automatically identify the runtime target UI elements of the respective activity according to the information stored in the RPA script 50. The run-time target UI elements comprise the operands of the respective activity, i.e., the UI elements of the run-time target UI that the robot 44 is configured to act on (e.g., click, enter some text, grab content, etc.). Performance of step 308 is described in more detail below. In response to successfully identifying the run-time target UI elements, step 310 may automatically execute the scripted activity, i.e., interact with the respective UI elements as shown in the RPA script 50.

12 illustrates an exemplary sequence of steps performed by the robot 44 to automatically identify run-time target UI elements according to some embodiments of the present invention. In step 312, the robot 44 may detect UI elements that match the target type of the current activity. For example, if the respective activity involves typing into form fields, step 312 may include identifying a set of form fields in the run-time UI. Step 312 may include analyzing source code underlying the run-time target UI, for example using computer vision (e.g., a neural network trained to automatically recognize various UI elements such as buttons, text boxes, input fields, etc.), and/or identifying UI elements according to an on-screen image of the run-time UI. If the intended target element and/or anchor element comprises text, some embodiments may further employ OCR technology to automatically detect text elements and further construct bounding boxes for each text element.

Next, looking within the set of UI elements returned by step 312, step 314 may attempt to identify run-time target UI elements according to element IDs (see above discussion regarding FIGS. 9-10). In some embodiments, step 314 includes determining the element ID of each UI element in the set returned by step 312 and comparing the respective element IDs with element IDs of design-side target elements (e.g., element ID 80a in FIG. 10), i.e., with element IDs specified by the RPA script 50 as characterizing the target. Step 316 may determine whether any element IDs match element IDs of intended targets of the current activity, and if so, step 318 may select the matching UI element as the run-time target. In some embodiments, step 316 determines whether there is an exact match between two element IDs. An exact match may occur when element IDs are specified using a set of attribute-value pairs, where all values of the corresponding attributes are identical.

However, due to occasional changes in the target user interface that occur between design time and run time, it may happen that a UI element in the run-time target UI does not match the design-time element ID of the intended target. For example, a form field may have been renamed. If a UI element does not match the element ID indicated in the RPA script 50, the robot 44 may automatically infer the target/operand of the current activity from available information. Some embodiments of the present invention use the element text 82 and element image 84 as alternative fallback data for identifying the run-time target when the element ID does not match.

In one such example, a series of steps 322-324 may assemble a set of candidate runtime target elements and a set of candidate runtime anchor elements according to element IDs specified in the RPA script 50 for the design-side target elements and anchor elements, respectively. The term "candidate" is used herein to denote a UI element whose element ID is the same as that of the intended target element or anchor element, respectively. Similarity may be determined in various ways. In one exemplary embodiment, the robot 44 may use regular expressions to determine whether two element IDs partially match. In an exemplary regular expression approach, two element IDs are considered similar if a certain subset of features is identical in both element IDs (e.g., the element type is the same but the element names are different). In one embodiment where the element ID indicates the location of the element in the UI tree, a partial matching strategy using regular expressions allows the robot 44 to search for candidates in a particular subtree, e.g., select only candidates that have the same root node specified in their element IDs (e.g., see the discussion above in connection with FIG. 10). For example, this situation may arise when the RPA client 10 has multiple instances of a business application running simultaneously, only one of which has the intended target element. By searching for candidate target elements using fixed nodes, the robot 44 can search all of the respective UI windows for candidates.

Another exemplary candidate selection strategy may determine whether two element IDs are similar according to a calculation of features that differ between the two element IDs. Such an approach may, for example, determine the Levenshtein distance between the two element IDs and compare the respective distances to a predefined threshold. Element IDs that are separated by a distance less than the threshold may be considered similar. In some embodiments, the threshold may be specified at design time and included in the RPA script 50. In contrast to partial matching methods using regular expressions, methods using the Levenshtein distance may not be sensitive to which features differ between the two element IDs being compared.

In response to selecting a set of candidate runtime targets and a set of candidate runtime anchor elements, some embodiments of the robot 44 may evaluate the candidates in pairs (e.g., all combinations of candidate targets and candidate anchors) to determine the most likely runtime target. In some embodiments, a series of steps 330-332 may evaluate each pair according to the relative screen positions of the respective elements and according to the content (element text and/or element image) of each member of each pair.

For each pair of candidates, some embodiments may evaluate a location score for each target and anchor candidate pair that indicates the likelihood that the candidate target is the intended run-time target element (step 330). In other words, in step 330, some embodiments determine the likelihood that the target candidate is the true intended run-time target and that the anchor candidate is the anchor element specified in the RPA script according to the relative locations of the target candidate element and the anchor candidate element.

Exemplary position scores may be determined according to various criteria, for example, according to the distance between the candidate anchor and the candidate target. FIG. 13 shows a set of exemplary distances separating a candidate target element 68 (in this example, an input field) and a candidate anchor element 69 (a label) according to some embodiments of the present invention. The distances d1 and d2 between the centers/centroids of the respective elements can be measured along the screen's primary coordinates (e.g., horizontal and vertical). For text elements detected using OCR, the distance can be measured to the center or centroid of a bounding box circumscribing the respective text element. Other exemplary inter-element distances, such as Manhattan distance, Euclidean distance, etc., can be evaluated according to d1 and d2. Some embodiments rely on the observation that an anchor element is usually placed in the vicinity of its target element, so the larger the distance between the candidate anchor and the candidate target, the less likely the respective pair represents the design-time target element and anchor element. In such an embodiment, an exemplary position score may be determined according to 1/D or (1-D/Dmax), where D represents the inter-element distance determined according to d1 and/or d2, and Dmax represents a predefined threshold above which the two UI elements are considered unlikely to be a target-anchor pair.

Another exemplary position score may be determined according to the degree of alignment between the candidate anchor element and the candidate target element. The alignment may be determined according to another set of distances, for example as shown in FIG. 14. An exemplary distance d3 separates the left edge of the anchor candidate 69 from the left edge of the target candidate 68. Meanwhile, a distance d4 separates the top edge of the anchor candidate 69 from the top edge of the target candidate 68. Since some embodiments rely on the observation that anchors are usually aligned with their target elements, a relatively small d3 or d4 distance may be associated with a relatively high likelihood that the respective anchor candidate element and target candidate element are in fact a target and anchor pair. FIG. 14 only shows distances that may be used to test for left and/or top alignment, and one skilled in the art will understand that the distance measures shown may be refined to test for right and/or bottom alignment. An exemplary fitness score may be calculated as follows:
where δ is the alignment distance determined according to d3 and/or d4, and δmax is a predefined threshold beyond which two UI elements are considered to be misaligned.

Another exemplary location score may be determined according to the angle between the candidate anchor and the candidate target. FIG. 15 shows an exemplary angle A between the anchor candidate 69 and the target candidate 68, determined as the angle of a line connecting the centers/centroids of the two respective elements. In some embodiments, the angle A is determined according to a distance measure, e.g., A=d2/d1, using the notation of FIG. 13. In some embodiments, the angle serves as a means for determining the degree of alignment of the target candidate and the anchor candidate. Some embodiments may further calculate the location score by comparing the run-time calculated angle between the target candidate and the anchor candidate with the design-time determined angle between the actual anchor element and the target element. The design-time angle may be included in the RPA script 50. A relatively small difference between the design-time angle and the run-time angle may indicate that the current target and anchor candidate pair is in approximately the same relative position as the design-time target element and anchor element, and thus may indicate a relatively high likelihood that the candidate is the truly desired run-time target element and anchor element. An exemplary position score determined according to angle may be determined according to 1/|Ad-Ar|, where Ad represents the design-time determined angle between the true anchor element (e.g., specified in the RPA script 50) and the target element, and Ar represents the run-time determined angle between the candidate target and the candidate anchor.

Yet another exemplary location score may be determined according to the overlap between the anchor candidate element and the target candidate element. FIG. 16 illustrates an exemplary overlap 67, according to some embodiments of the present invention, determined as the percentage of one element that intersects with the other element, or in other words, how much one element overlaps with the other element. In such an embodiment, two elements that do not intersect have zero overlap, while two elements where one element completely contains the other have 100% overlap. Some embodiments use the location score determined according to the overlap to identify a particular anchor, such as a button label. In one such example where the robot 44 is looking for a button-type target element, the robot may eliminate all target-anchor candidate pairs that do not have a substantial overlap (e.g., more than 90%).

In a further step 332 (FIG. 12), some embodiments of the robot 44 determine a content score for the target and anchor candidate pair. The content score may be determined according to a result of comparing the on-screen content (images and/or text) of the target and anchor candidates with the respective content of the design side targets and anchors. In some embodiments, comparing the on-screen content includes evaluating a numerical measure of similarity between the text displayed by the target candidate and the text displayed by the design side target element, and another measure of similarity between the text displayed by the anchor candidate and the text displayed by the design side anchor element. The element text of the design side target element and the anchor element are specified in the RPA 50 (see, e.g., items 82a-b in FIG. 9 and related discussion). The similarity between the two text fragments may be evaluated, for example, using the Levenshtein distance, where a relatively small distance may indicate a relatively high similarity between the compared fragments.

Step 332 may further include determining a numerical measure of similarity between the image of the target candidate and the image of the design side target element, and another measure of similarity between the image of the anchor candidate and the image of the design side anchor element. The element images of the design side target elements and anchor elements are specified in the RPA script 50 (see, e.g., items 84a-b in FIG. 9 and associated discussion). Several measures of similarity between two images are known in the art.

Text similarity may be used independently of image similarity, or the two may be combined in an aggregate content score. In situations where either the text or the image of the target or anchor element has changed between design and run time, aggregating image and text aspects may provide a robust method of identifying the run time target element. In such situations, the robot 44 may determine that two UI elements are similar according to the text content even if the image content does not match, or vice versa. Also, in situations where only the target element has changed between design and run time, while the anchor remains nearly identical, combining the content score determined for the anchor with the content score determined for the target may result in a robust method. In such situations, the robot 44 may be able to identify the run time target according to the content of the candidate anchor.

In an alternative embodiment, the robot 44 uses step 330 as a filter of object-candidate anchor pairs. In one such example, for each candidate pair, the robot 44 may evaluate a set of indicators of the relative position of the object candidate with respect to the anchor candidate, such as a set of distances as described above in connection with FIGS. 13-14. If the evaluated distances indicate that the object and anchor candidates are unlikely to be a truly desired run-time object-anchor pair, e.g., because they are too far apart and/or misaligned, the respective object-anchor candidate pair is no longer considered for content score evaluation (step 332). Such optimization may significantly reduce the computational cost of identifying run-time objects, since image analysis is typically resource-intensive.

In step 336, the robot 44 may select a runtime object from the set of object candidates identified in step 322 according to the location score and/or content score determined for each of the set of object-candidate pairs. In some embodiments, step 336 may calculate an aggregate score for each pair, the aggregate score being a combination of the location score and content score determined for each pair. The scores may be combined using various methods known in the art, for example, as a weighted average where each score is multiplied by a predetermined numerical weight. The weight value may indicate the confidence associated with each score (e.g., a relatively high weight may be given to a score that is more likely to correctly identify the runtime object).

In some embodiments, the candidate target element of the pair whose aggregate score indicates the highest similarity to the design side target and anchor pair specified in the RPA 50 is selected as the run-time target element. The robot 44 can then proceed to the scripted activity (step 320), i.e., apply the current activity to the selected run-time target.

Figure 17 illustrates an exemplary hardware configuration of a computing device programmed to perform some of the methods described herein. Each computing device may represent any of the RPA clients 10a-e of Figure 1, such as the personal computer shown in Figure 18. Other computing devices, such as mobile phones, tablet computers, and wearables, may have slightly different configurations. The processor 22 comprises a physical device (e.g., a microprocessor, a multi-core integrated circuit formed on a semiconductor substrate) configured to perform computational and/or logical operations using a set of signals and/or data. Such signals or data may be encoded and delivered to the processor 22 in the form of processor instructions, e.g., machine code. The processor 22 may include an array of central processing units (CPUs) and/or graphics processing units (GPUs).

The memory unit 24 may comprise a volatile computer-readable medium (e.g., Dynamic Random Access Memory - DRAM) that stores data/signal/instruction encodings accessed or generated by the processor 22 in the course of performing operations. The input devices 26 may include a computer keyboard, a mouse, and a microphone, among other hardware interfaces and/or adapters that allow a user to introduce data and/or instructions to the RPA client 10. The output devices 28 may include display devices such as a monitor and speakers, among other hardware interfaces/adapters such as a graphics card that allow the respective computing devices to communicate data to the user. In some embodiments, the input devices 26 and the output devices 28 share common hardware (e.g., a touch screen). The storage devices 32 include computer-readable media that allow non-volatile storage, reading, and writing of software instructions and/or data. Exemplary storage devices include magnetic disk devices, optical disk devices, and flash memory devices, as well as removable media such as CD and/or DVD disks and their drives. A network adapter 34 enables each computing device to connect to electronic communications networks (e.g., networks 12 and 14 in FIG. 1) and/or other devices/computer systems.

The controller hub 30 generally represents multiple system buses, peripheral buses, and/or chipset buses, and/or any other circuitry that enables communication between the processor 22 and the remaining hardware components of the RPA client 10. For example, the controller hub 30 may include a memory controller, an input/output (I/O) controller, and an interrupt controller. Depending on the hardware manufacturer, some such controllers may be incorporated into a single integrated circuit and/or may be integrated with the processor 22. In another example, the controller hub 30 may include a northbridge that connects the processor 22 to the memory 24 and/or a southbridge that connects the processor 22 to the devices 26, 28, 32, and 34.

The exemplary systems and methods described above facilitate RPA operations by improving the automatic identification of activity objects, i.e., user interface elements that are acted upon by the robot software. In typical RPA applications, object identification poses substantial technical challenges because the target user interfaces (e.g., e-commerce web pages, accounting interfaces, etc.) are developed and maintained independently of the robot design to interact with the respective interfaces. Thus, the functionality and/or appearance of the target UI may change without the knowledge of the RPA developer. Successful RPA may therefore depend on a robust method of identifying activity objects, i.e., a method that is relatively insensitive to variations in the design of the target user interfaces.

When designing the robot software (a phase of automation commonly known as design time), the RPA developer invokes an instance of a target UI and indicates target elements and the activity to be performed on each target element. For example, the developer may configure the robot to indicate buttons in the target UI and click on each button. In another example, the developer may configure the robot to indicate input fields and type some text into each input field. In yet another example, the developer may configure the robot to indicate text boxes in a user interface and grab the content of each text box. The resulting robot code may include an indicator of the target elements and an indicator of the respective activities. The robot code may then be delivered to the RPA client.

At another stage of automation, commonly known as run time, the client machine may execute the respective robot, which may attempt to interact with another client-side instance of the target UI. However, the client-side UI may not be identical to the design-side UI. If the target UI comprises a web interface, the respective user interface may change multiple times throughout the day, especially if the respective robots are designed to interact with complex websites. The web developers of the respective websites may tweak the appearance, for example, changing the position of buttons, changing the configuration of menus, and/or changing the color scheme, fonts, and size of various elements. Thus, the robot software may need to successfully identify the target element even if the appearance of the interface has changed.

Some conventional RPA systems identify target elements according to their names or IDs, which are specified in the underlying source code or data structures of the respective user interface (e.g., HTML code that specifies the appearance and content of a web page). However, such systems and methods can fail when the names of the respective elements change unexpectedly. Such changes can occur very frequently, especially since a significant proportion of web documents are now dynamically generated and various aspects of web documents are algorithmically controlled.

In contrast to such conventional approaches, some embodiments of the present invention further identify target elements according to their images and text displayed at design time. The design-time images and text are stored in the robot's code and sent to the RPA client. At run-time, the robot may identify multiple candidate target elements and evaluate each of the multiple candidate target elements according to the element ID and further according to the image and text displayed by each candidate element. Candidates that at least partially match the ID, image, and text of the design-time target element may be selected as run-time targets. The robot may then apply scripted activities to the selected run-time target elements.

Some embodiments may use optimization strategies to save computational resources and thus improve runtime RPA efficiency and user experience. In the first phase, the robot may attempt to identify the runtime target according to the element ID, and if such identification fails (e.g., because the name of the element has changed in the source code of the UI), it may use text and/or image matches as fallback locations. Candidate UI elements may be selected such that they partially match the element ID of the design-time target element. If there is a partial match with the element ID, the robot may search for target elements in a related subgroup of candidates (e.g., candidates that belong to the same region of the UI as the design-side target element).

To further improve the robustness of the method, some embodiments employ characteristic data (e.g., element ID, image, and text data) of another UI element of the target interface, which is co-displayed with the target element and considered as an anchor of the target element. At run-time, some embodiments may identify multiple candidate anchor elements and attempt to match each candidate with a design-time anchor according to element ID, image, and/or data. Using anchor element data in combination with target element data relies on the assumption that both the target and the anchor are unlikely to have changed between design time and run time, such that a target may be successfully identified based on data characterizing its anchor.

It will be apparent to those skilled in the art that the above embodiments can be modified in many ways without departing from the scope of the present invention. Therefore, the scope of the present invention should be determined by the following claims and their legal equivalents.

Claims (17)

1. A method comprising employing at least one hardware processor of a computer system, the at least one hardware processor of the computer system comprising:
In response to receiving a Robotic Process Automation (RPA) script having a set of target functions that are characteristic of a target element of a target UI and a set of anchor functions that are characteristic of an anchor element of the target UI, automatically identifying a runtime instance of the target element within a runtime user interface (UI) exposed by the computer system;
2. A method for automatically performing operations determined according to the RPA script that replicate results of an interaction between a human operator and the runtime instance of the target element, comprising:
The set of target features is
A target ID indicating a position of the target element within a tree representation of the target UI;
a target image comprising an image of the target element within the target UI;
a target text comprising a sequence of characters to be displayed by the target element in the target UI;
The set of anchor functions
an anchor ID indicating a location of the anchor element within the tree representation of the target UI;
an anchor image comprising an image of the anchor element within the target UI;
and anchor text comprising a series of characters displayed by the anchor element in the target UI;
The method includes identifying the runtime instance of the target element according to the target ID, target image, target text, anchor ID, anchor image, and anchor text. automatically identifying the runtime instance of the target element,
For each candidate of a plurality of candidate UI elements of the runtime UI, determining whether an element ID of the each candidate, indicating a position of the each candidate within a tree representation of the runtime UI, closely matches the target ID;
in response, designating each of the candidates as the runtime instance of the target element if the element ID of the candidate exactly matches the target ID;
2. The method of claim 1, further comprising: if none of the plurality of candidate UI elements has an element ID that closely matches the target ID, identifying the runtime instance of the target element further according to the target image and target text. selecting a candidate object from the plurality of candidate UI elements according to whether an element ID of the candidate object partially matches the object ID;
selecting a candidate anchor from the plurality of candidate UI elements according to whether an element ID of the candidate anchor partially matches the anchor ID;
3. The method of claim 2, further comprising: in response to selection of the candidate object and candidate anchor, determining whether to designate the candidate object as the runtime instance of the target element according to a result of comparing the object text with text displayed by the candidate object and further according to a result of comparing the anchor text with text displayed by the candidate anchor. The method of claim 3, further comprising: in response to the selection of the candidate object and the candidate anchor, determining whether to designate the candidate object as the runtime instance of the target element further according to a result of comparing the object image with an on-screen image of the candidate object and further according to a result of comparing the anchor image with an on-screen image of the candidate anchor. The method of claim 3, further comprising, in response to the selection of the candidate object and the candidate anchor, determining whether to designate the candidate object as the runtime instance of the target element further according to a relative on-screen position of the candidate object with respect to the candidate anchor. The method of claim 5, wherein determining the relative on-screen position includes determining the angle of a line connecting the center of the candidate object with the center of the candidate anchor. The method of claim 5, wherein determining the relative on-screen position includes determining an overlap between the candidate object and the candidate anchor. The method of claim 1, wherein the interaction comprises an item selected from the group consisting of: performing a mouse click on the runtime instance of the target element; pressing a particular combination of keyboard keys; writing a sequence of characters into the runtime instance of the target element; grabbing an on-screen image of the runtime instance of the target element; and grabbing text displayed by the runtime instance of the target element. 1. A computer system comprising at least one hardware processor configured to execute an application to be automated and a robotic process automation (RPA) robot, comprising:
the automation target application is configured to expose a run-time user interface (UI);
The RPA robot,
In response to receiving an RPA script comprising a set of target functions that are characteristic of a target element of a target UI and a set of anchor functions that are characteristic of an anchor element of the target UI, automatically identifying a runtime instance of the target element within the runtime UI;
1. A computer system configured to automatically perform operations determined according to the RPA script, the operations replicating results of an interaction between a human operator and the runtime instance of the target element, comprising:
The set of target features is
A target ID indicating a position of the target element within a tree representation of the target UI;
a target image comprising an image of the target element within the target UI;
a target text comprising a sequence of characters to be displayed by the target element in the target UI;
The set of anchor functions
an anchor ID indicating a location of the anchor element within the tree representation of the target UI;
an anchor image comprising an image of the anchor element within the target UI;
and anchor text comprising a series of characters displayed by the anchor element in the target UI;
The computer system, wherein automatically identifying the runtime instance of the target element includes identifying the runtime instance of the target element according to the target ID, target image, target text, anchor ID, anchor image, and anchor text. automatically identifying the runtime instance of the target element,
For each candidate of a plurality of candidate UI elements of the runtime UI, determining whether an element ID of the each candidate closely matches the target ID indicating a position of the each candidate within a tree representation of the runtime UI;
in response, designating each of the candidates as the runtime instance of the target element if the element ID of the candidate exactly matches the target ID;
10. The computer system of claim 9, further comprising: if none of the plurality of candidate UI elements has an element ID that closely matches the target ID, identifying the runtime instance of the target element further according to the target image and target text. The RPA robot,
selecting a candidate object from the plurality of candidate UI elements according to whether an element ID of the candidate object partially matches the object ID;
selecting a candidate anchor from the plurality of candidate UI elements according to whether an element ID of the candidate anchor partially matches the anchor ID;
11. The computer system of claim 10, further configured to: in response to selection of the candidate object and candidate anchor, determine whether to designate the candidate object as the runtime instance of the target element according to a result of comparing the object text with text displayed by the candidate object and further according to a result of comparing the anchor text with text displayed by the candidate anchor. The computer system of claim 11, further configured to perform the following in response to the selection of the candidate target and the candidate anchor: determining whether to designate the candidate target as the runtime instance of the target element further according to a result of comparing the target image with an on-screen image of the candidate target and further according to a result of comparing the anchor image with an on-screen image of the candidate anchor. The computer system of claim 11, wherein the RPA robot is further configured to determine whether to designate the candidate object as the runtime instance of the target element in response to the selection of the candidate object and the candidate anchor, further depending on a relative on-screen position of the candidate object with respect to the candidate anchor. The computer system of claim 13, wherein determining the relative on-screen position includes determining an angle of a line connecting a center of the candidate object with a center of the candidate anchor. The computer system of claim 13, wherein determining the relative on-screen position includes determining an overlap between the candidate object and the candidate anchor. The computer system of claim 9, wherein the interaction comprises an item selected from the group consisting of: performing a mouse click on the runtime instance of the target element; pressing a particular combination of keyboard keys; writing a sequence of characters into the runtime instance of the target element; grabbing an on-screen image of the runtime instance of the target element; and grabbing text displayed by the runtime instance of the target element. A non-transitory computer-readable medium storing instructions that, when executed by at least one hardware processor of a computer system configured to expose a runtime user interface (UI), provide the computer system with:
in response to receiving a robotic process automation (RPA) script comprising a set of target functions that are characteristic of a target element of a target UI and a set of anchor functions that are characteristic of an anchor element of the target UI, automatically identifying a runtime instance of the target element within the runtime UI;
1. A computer system for automatically executing operations determined according to the RPA script, the operations replicating a result of an interaction between a human operator and the runtime instance of the target element, the computer system comprising:
The set of target features is
A target ID indicating a position of the target element within a tree representation of the target UI;
a target image comprising an image of the target element within the target UI;
a target text comprising a sequence of characters to be displayed by the target element in the target UI;
The set of anchor functions
an anchor ID indicating a location of the anchor element within the tree representation of the target UI;
an anchor image comprising an image of the anchor element within the target UI;
and anchor text comprising a series of characters displayed by the anchor element in the target UI;
A non-transitory computer-readable medium, wherein automatically identifying the runtime instance of the target element includes identifying the runtime instance of the target element according to the target ID, target image, target text, anchor ID, anchor image, and anchor text.

Images