Robotic Process Automation¶
Johann Mitloehner, 2022-2023
Definition¶
Robotic Process Automation (RPA) allows businesses to automate tasks that are typically carried out by employees.
- Traditional process automation uses an application's API to perform tasks.
- In contrast, RPA emulates human interaction with software systems, e.g. by using the web interface of an application instead of its API.
RPA is usually aimed primarily at repetitive and tedious tasks in order to
- free people for more intellectually satisfying work, or
- simply cut personel cost.
In order to mimick human interaction with applications RPA needs to
- work with the user interface provided by the system, such as
- login to a system providing a web interface and follow links
- enter data in web forms
- 'understand' the system responses, such as
- extract specific parts from the content of a web page
- assign meaning to those parts, such as success or failure of login
- perform actions corresponding to specific situations
- usually pre-defined in a programmatic way, but increasingly
- involving machine learning
- maybe interact with more than one system
- extract and process data from one system
- transfer data via a different interface to another system
Related Concepts¶
Workflow Automation¶
This type of automation is usually understood to involve software that accesses the back-end of a system through its API (application programming interface). In contrast, RPA accesses the system via the front end, usually a GUI (graphical user interface), closely mimicking the human/computer interaction.
Since not all systems provide an API but need to provide a frontend for human users, there are situations where RPA is the only feasible approach to automation.
GUI Testing Tools¶
These are aimed at performing system testing with pre-defined test cases and expected results. The focus is usually not on interacting with multiple applications; however, sophisticated testing tools like the robot framework which will be used here allow for extracting data and processing it in more than one application.
Robot Software¶
This type of software is usually understood to control physical robots. RPA involves software robots i.e. programs that are not controlling a physical robot but perform actions by interacting with other software systems, not the physical world.
Artificial Intelligence¶
A confusing term that is hard to define, since even natural intelligence remains an ellusive concept. We will use the less misleading term machine learning instead to refer to a type of decision making that sometimes seems like AI but is really just another type of software.
Machine Learning¶
This usually refers to software that 'learns' from observation i.e. from data providing instances of situations and actions, e.g.
- credit card application data and corresponding decisions such as grant/reject
- measurements of petals and the corresponding species of Iris flowers (a classic dataset from 1936)
The system is usually understood to work on data from a large number of cases in an adaptive manner -- often, but not necessarily, working iteratively through the cases. It will (hopefully) adjust its behaviour towards optimal decisions, usually defined by minimising an error function.
Various approaches are used; connectionist models, especially deep learning neural nets, are currently very much in favour because of spectacular successes, particularly in image processing. However, there are a number of much simpler yet still useful approaches. Some of them will be discussed here, as it is feasible to apply them in RPA with limited resources in terms of time and coding skill.
Chatbots¶
RPA is usually aimed at back-office tasks while the automated servicing of customer request is increasingly offered using chatbot technology. Since chatbots automate front-office tasks they are often not seen as RPA; however, using the above definition chatbots qualify as RPA, since they are software robots that automate tasks otherwise carried out by employees.
Benefits of RPA¶
There is always a lot of hype surrounding new concepts in management and technology. The following list of RPA benefits is somewhat conservative and also a little critical; just some food for thought.
Cost savings. When robots do the work of people we can cut personel cost. Euphemistically this is described as freeing time for more creative work.
Resilience. While the human workforce is limited there can always be more software robots, the only limit being the performance of the computing hardware. Therefore, if demand suddenly increases, the robot army is instantly ready.
Accuracy. Once a process is defined any errors left are due to design, not the software robots. Humans make mistakes, particularly in tedious tasks; (software) robots do not. Physical robots are a different matter and can make disastrous mistakes.
Compliance. An automated process can be defined so as to be fully compliant to some regulations at all times. Humans might make exceptions that can lead to trouble; robots make no exceptions, and no trouble.
Productivity. When measured in terms of input/output relation robots are hard to beat, especially software robots who work 24/7 without any wear and tear. If needed they can be replicated at practically zero additional cost.
Employee happiness. When freed from tedious tasks people can (in theory) do more creative and fulfilling things, and that may well make them happier (at least those who still have a job).
RPA as Enterprise Software¶
This list is similar to the previous one but focuses on the development and deployment of software, which is often a huge and risky project. Fortunately, RPA is somewhat different from typical enterprise software projects:
No Disruption. The RPA only uses the front-end of the system, so any problems caused must have been present already in day-to-day operations by human users (and have hopefully been wiped out). Deployment of RPA is less likely to cause disruption of service, whereas API-based process automation can access functions not available to users, or in a manner not possible when using the front-end GUI, and thereby causing unforeseen problems.
Scalable. RPA software robots not only work faster than humans, they also run continously 24/7 all year long, thereby easily meeting with increased demand; and since they are just software programs we can have more than one instance running on one or more computers at the same time. The only limit is the performance of the computing hardware.
Small Investment. Obviously this depends on the project. However, as we will see, at least some simple RPA projects can be cheap yet useful.
Quick ROI. A simple RPA project can start generating return on investment relatively quickly since development and deployment tend to be less problematic compared to a similar process automation project based on API programming.
Downsides of RPA¶
These are compared to traditional automation based on API programming i.e. using the back-end of the system. Most of these problems do not seem overwhelming or unsolvable, at least in most situations.
Performance. Compared to API-based automation the RPA approach will tend to be slower, maybe even so much slower that it is not feasible for a particular project.
Front-End Limitations. The front-end was not designed to be used by robots, but by humans. Problems can arise that were never faced before, because human users would know how to handle exceptional situations while robots just shamble on -- you may have seen the video of the paint robots in car factories that turn on each other.
Citizen Developers. Business units can now develop bots using simple end-user tools and without the need for support by an IT team, or any involvement (or even knowledge) of the IT department. This can be seen as a benefit or a nightmare, depending on where you stand.
Examples¶
In the following we will look at examples of RPA in the following areas:
Testing. While test suites can of course be run via the API (if one is available) there may be subtle differences to actual user interaction that are not easy to cover reliably. Using RPA and the GUI closely mimicks human interaction and (hopefully) bypasses those problems.
Web Scraping. Since the Robot Framework uses XPath to access elements in HTML documents, and also allows for very simple integration of custom Python code in Robot test case files, it is easy to automatically extract content from web pages for further processing, also known as web scraping.
Customer Service. Many customer requests fall into one of very few categories and are therefore prime candidates for automation. This is an area that applies concepts from chatbots and machine learning. We will look at a simple case study using robot testing for automation and open datasets for machine learning.
Getting Started with the Robot Framework¶
The Robot Framework is available at robotframework.org. Its main purpose is automated testing, but it can be used for general process automation in the interaction with web servers. Here are some other (free and open source) options with similar applications as the robot framework:
- selenium: the robot framework is based on this library which can be used directly in Python code
- requests_html: emphasis on ease of use for downloading and parsing HTML
- beautiful soup: many functions for extracting data from HTML source
There are various types of testing frameworks; the Robot Framework uses keyword-driven testing: the idea is that the keywords
- describe the actions that need to be performed without too much detail
- are independent of the test framework being used
The approach somewhat resembles pseudo-code in algorithm design; it can be used for both manual and automated testing.
The following examples provide an introduction to the approach. The Robot Framework is written in Python, and we need to install some packages, and maybe Python itself as well.
Installation¶
Everything below was tested in Python 3.10 on Linux (Mint), and Python 3.11 on Windows 10.
Windows 10¶
The installation is somewhat more elaborate in Windows, since Python is typically not installed; however, once we have a reasonably current Python3 installation the remaining steps should be identical to Linux and MacOS. Fortunately, the web drivers for the browser no longer need separate installation.
If you already have an Anaconda Python installation you can install the robotframework as described at https://anaconda.org/conda-forge/robotframework (not tested by this author).
Installation is probably very similar or identical in Windows 11; not tested.
Python¶
- go to https://www.python.org/downloads/windows/
- download the "Windows installer (64-bit)" for the latest stable release (3.11.4 as of June 2023)
- start the installer
- click "Add python.exe to PATH" --- very important!
- "use admin rights" should already be ckecked
- click "Custom installation"
- in the next step, leave everything checked and click Next
- change directory to a shorter path, e.g. C:\Python311
- click "Install", then in "Allow this app" click Yes
- click "Disable path length limit" (maybe not necessary)
- click Close
Check Python installation¶
open a command prompt (type cmd in the the search box, or Start/Windows System/Command Prompt), and enter
py --version
you should see the version of Python that was installed, e.g. 3.11.4
then, still in the command prompt, enter
pip --version
you should see the version of pip, e.g. 23.1.2
Browser¶
If necessary, install the browser of your choice, e.g. Firefox, English Windows 64-bit MSI, download and run the installer
https://download.mozilla.org/?product=firefox-msi-latest-ssl&os=win64&lang=en-GB
Some Mac users run into trouble with the Safari browser; if so, just install Firefox or Chrome.
Robotframework¶
In the command prompt, enter
pip install robotframework
pip install robotframework-seleniumlibrary
This should download and install a number of packages.
In the command prompt, enter
robot --version
This should show the version of the robotframework, e.g. 6.1.1
In the command prompt, enter
notepad test01.txt
In the notepad window, enter the following minimal robot test script; when using copy/paste, make sure *** and below is flush left, but Open Browser and below is indented 4 spaces.
*** Settings ***
Library SeleniumLibrary
*** Test Cases ***
Test01
Open Browser http://www.w3.org Firefox
Page Should Contain W3C
[Teardown] Close Browser
In the command prompt, enter
robot test01.txt
You should see the browser window being openend and closed, and in the command prompt window the PASS after the test has finished.
File Names in Windows¶
In Windows it can be tricky to save files with extensions like .rob or .robot, so it may be easier to simply use .txt, particularly when using notepad as your editor. However, even notepad can still save files with .py extension (when Python is installed; this extension is then associated with this application).
Linux, MacOS, ChromeOS¶
This should be much easier, since a recent version of Python3 is (probably) already installed. Just
- open a terminal
- do the pip statements
- using the text editor of your choice, create the robot test script
- run robot
However, if your Python is much older than 3.10 then you may need to upgrade, as the package versions will also be older and possibly not suitable for what we are doing here.
In Case of Problems¶
The robotframework github site has detailled installation instructions for various operating systems.
Github Page webdrivermanager & Command Line Options:
https://github.com/MarketSquare/webdrivermanager/#command-line-optionsPATH variable setting:
https://github.com/robotframework/robotframework/blob/master/INSTALL.rst#configuring-path
Optional: Install Jupyter¶
If you want to use JupyterLab for your coding then you can install it on your computer by entering the following on the command line:
pip install jupyterlab
On Windows you should get a menu entry, and on Linux you can run the Jupyter Lab by entering on the command line:
jupyter-lab
A Simple Flask Web Site for Testing¶
In order to provide well-defined test cases we will build our own little web site using the Flask framework. This will allow us to iteratively refine the web application in small steps and simultaneously keep testing as we add new features.
During this process we will be in complete control of the web server, meaning that we will never be locked out, which may well happen if we continue to send robot requests to some external server.
Flask is a microframework that is contained in many Python distributions. It includes its own development web server.
We can install the package easily (if we need to):
pip install flask
Optional: Production-Level Web Server¶
For more serious projects it is preferrable to use a production-level web server; gunicorn will do the job nicely, but only on Linux. For Windows, you could use e.g. waitress.
If you are using gunicorn or waitress then do not include the line
app.run(...)
at the end of the Python file with the Flask code.
Very Basic Flask App¶
We use our favorite text editor to create a file credit.py and copy-paste the following code into the file:
from flask import Flask
app = Flask(__name__)
@app.route("/")
def credit():
return "<p>Credit App"
app.run(host='localhost', port=8080, debug=True, use_reloader=True)
Things to note about the code above:
- Flask is object-oriented and starts by creating an object for the app
- the @ decorator tells route() to call the decorated function when "/" is requested
- return value is just a bit of HTML code in this minimal example
Now we can start the web server.
Open a new terminal window and change to your project directory.
We can run our app like any other Python code (on Windows, probably py instead of python3):
python3 credit.py
Our web server is now running on the localhost (the computer we are working on) and accepting requests at the URL
If you run the web server and this notebook (the one that you are reading right now) on your own computer you can just click on the link above. It should open a new tab in your browser and show the response of your web server.
In the terminal window where you started the Flask web server you should see a line of log output like this one:
127.0.0.1 - - [23/Aug/2023 11:11:58] "GET / HTTP/1.1" 200 -
- This was a GET request (in contrast to a POST)
- the requested URL was "/", the root document
- the protocol was HTTP version 1.1
- the response code was 200 -- meaning success
With gunicorn or waitress you should get additional data:
- response length in number of bytes
- referrer (the web page we came from, if available)
- user agent (our browser, as seen by the web server)
It is always a good idea to look at the log, especially when things do not work the way we expect, e.g. try requesting the following URL:
and then look at the log output:
- the response code is now 404 -- file not found
- this code should also be displayed in the title bar of the browser
- the page displayed by the browser should be something like "Not Found -- The requested URL was not found on the server"
We are now ready to begin automated testing.
Minimal Example: Checking Web Page Contents¶
The robot framework relies on test files containing the instructions for tests. Here is a minimal example:
*** Settings *** Library SeleniumLibrary *** Test Cases *** Home Page Open Browser http://localhost:8080 Firefox Page Should Contain Credit App [Teardown] Close Browser
The test file contains test cases and is written in a somewhat unusual notation that resembles natural language.
- a test case consists of a name and one or more steps
- note the indentation
- the name of the test case is flush left
- the steps of the test case are indented
- each step contains
- commands (such as Open Browser)
- conditions (such as Page Should Contain)
- most commands and conditions expect parameters
- two or more spaces are used as delimiters to separate
- the command from the parameters
- several parameters
- nice alignment is not necessary but helpful
This test will try to open the root document of the web server running on this computer, using the Firefox browser.
Use your text editor to put this code into a file, such as t1.rob, and then run the robot on the command line (you may need to open a new terminal to get a command line that is not already running a job):
robot t1.rob
The robot should read the test file and execute the test steps
- open a new browser window (Open Browser)
- execute the test (Page Should Contain)
- close the browser window (Teardown)
- print text output showing the passed and failed tests
- exit gracefully
Sometimes the robot seems to get stuck after a few lines of output:
press ctrl-c
Starting with option --quiet or --console none also seems to help:
robot --quiet t1.rob
Other output files such as output.xml will still be written and can be analysed later.
There will be several output files in the current directory:
- a nicely formatted report.html that you can view in your browser
- an XML file for automated processing of test results
- .log files
Opening the browser window takes a few seconds, but once it is open any further actions will happen fast, so we will only see the page contents briefly.
Check the Log¶
In the terminal where you started the web server you should see the robot requests in the log outout. You can check the file using a GUI app, or on the command line enter
ls -lrt
This displays the files sorted by modification time.
After some testing we get a lot of robot log files and PNG screenshots. You can delete them using the following command:
rm *.log *.png
☆ This will permanently delete all files in the current directory with the extension .log and .png -- make sure
- which directory you are actually working in, e.g. by
- looking at the prompt
- using the pwd command
- there are no typos...
Elaborate¶
The minimal example works, now we can turn to some more useful elements of the syntax.
Use your favorite text editor to create another file, such as t2.rob, with the following content:
*** Settings *** Documentation Check for text in page Library SeleniumLibrary *** Variables *** ${URL} http://localhost:8080 ${BROWSER} Firefox *** Test Cases *** Keywords Open Browser To Home Page Page Should Contain Credit App [Teardown] Close Browser *** Keywords *** Open Browser To Home Page Open Browser ${URL} ${BROWSER}
Things to note about the code above:
a keywords/argument format is used
- default is two or more blanks for indent and separators
- if blanks are inconvenient we can also use the pipe character |
- start the line with | and then
- use | with one space left and right for indent and separators
- a trailing pipe at the end of the line is optional
in the Settings section we provide short documentation and request Selenium for web page processing
in the Variables section we define the URL and the browser
- not strictly necessary, but good practise to do this here at the top, especially in longer files
the Test Cases section contains one or more cases
- the first statement in that test case is defined in the Keywords section
- the second statement contains the pre-defined keywords "Page Should Contain"
- make sure to leave two or more blanks before the text to check for, otherwise
- the Robot Framework will not recognise the text as a parameter and
- try to treat it as another user-defined keyword phrase
- The Teardown statement makes sure the browser windows is closed, instead of leaving it open
in the Keywords section we define what we mean by saying "Open Browser To Home Page"
- we use the pre-defined phrase "Open Browser"
- and supply it with the variables from above
Expect to Fail¶
Let's try a test that we expect to fail.
Use your text editor and put the following in a file t3.rob:
*** Settings *** Documentation Check for text in page, expect fail Library SeleniumLibrary *** Variables *** ${URL} http://localhost:8080 ${BROWSER} Firefox *** Test Cases *** Fail Open Browser To Home Page Page Should Contain The Credit App [Teardown] Close Browser *** Keywords *** Open Browser To Home Page Open Browser ${URL} ${BROWSER}
Note how we introduced a small change in the text:
Our website says "Credit App", not "The Credit App"
This test should fail.
Extend the Test App with Section Headers and Links¶
Our minimal app is performing its Hello function, but now we add some more features.
We are slowly approaching a small application with database connection.
Incremental development and testing will go side by side.
Let's add a link to a list of clients.
Note that at this point we do not yet need an actual procedure for listing anything, just the link to it.
Use your text editor to change the content of the file credit.py to the following:
from flask import Flask
app = Flask(__name__)
@app.route("/")
def credit():
return """<h1>Credit App</h1>
<p><a href=clients>Clients</a>"""
app.run(host='localhost', port=8080, debug=True, use_reloader=True)
Take a look the terminal window that runs the server: you should see a line that looks like this:
* Detected change in ..., reloading
Flask has detected the change in the source file and restarted the app.
Check for headers and links¶
Our web app starts to get a little more elaborate.
We now have a few distinct elements that we can check in our robot tests.
We can use XPath expressions to find elements in the HTML and check their contents.
Use your text editor to create a file t4.rob and put the following code into that file:
*** Settings *** Documentation Check page for header and link Library SeleniumLibrary *** Variables *** ${LOGIN URL} http://localhost:8080 ${BROWSER} Firefox *** Test Cases *** Headers and Links Check Home Page [Teardown] Close Browser *** Keywords *** Check Home Page Open Browser ${LOGIN URL} ${BROWSER} Element Should Contain //h1 Credit App Element Should Contain //a Clients
We introduced the user-defined phrase "Check Home Page" in the test case and defined it in the keywords section.
The pre-defined phrase "Element Should Contain" has two parameters:
- the HTML element, specified e.g. by an XPath expression, such as //h1
- the contents of the element, such as the text of the section header
Note that only the first section header h1 in the page is checked. To check other elements which are not the first of their type we will need to specify their XPath more elaborately.
In a similar fashion we check for the link with a given text. Note that for this test to succeed we do not need a procedure actually listing anything, only a link with the specified text.
There are various types of element locators in the Selenium library which is used in the Robot Framework; XPath can get somewhat tedious but it offers sufficient power to locate elements in more complex web content.
Let's run the robot again: on the command line enter
robot t4.rob
The text output should PASS.
XPath¶
To test for the presence of a link with the text "Clients" we need to use an XPath expression that points to this link. We could simply use indexing with square brackets such as [2] but that could lead to trouble lateron when we add more links and change their order.
Instead, we delve a little deeper into XPath and use the functions contains() and text().
Use your text editor to put the following robot code into a file t5.rob:
*** Settings *** Documentation Check page for header and links Library SeleniumLibrary *** Variables *** ${LOGIN URL} http://localhost:8080/ ${BROWSER} Firefox *** Test Cases *** XPath Check Home Page [Teardown] Close Browser *** Keywords *** Check Home Page Open Browser ${LOGIN URL} ${BROWSER} Element Should Contain //h1 Credit App Element Should Contain //a Clients Element Should Contain //a[contains(text(), "Clients")] Clients
Note that in the last line we need to supply the text twice although it is already obvious from the XPath expression.
Run the robot again to see that the XPath expressions actually target the link as intended.
robot t5.rob
The text output should show PASS.
The test works as intended. Now we can move on the actually providing procedures for the functions, such as the list of clients. For that purpose, we chose a somewhat more elaborate route by introducing a database connection to our little sample app.
SQLite Database Connection¶
To make our approach extensible and reasonably realistic we add more functions to the sample app instead of just providing toy examples. Playing in the sandbox can only get us so far.
Fortunately there is a free open-source database management system that we can easily use in our sample app: SQLite. This DBMS is widely used since it is so easy to install and apply. It should be noted that it is also lacking in several respects, such as in terms of implementing standard SQL numeric data types. We happily accept these deficiencies since they do not bother us here (much).
See the documentation for more details about SQLite2 and its use in Python3:
The new version of our shop app contains new routes:
- add new clients using HTML form input
- list the existing clients
- initialise the DB table for the clients and insert some data (just to save work during testing)
Our application is now a handsome size, make sure you do not miss anything when copy and paste into the file credit.py:
from flask import Flask, request
import sqlite3
app = Flask(__name__)
def getconn():
return sqlite3.connect("credit.db")
@app.route("/")
def credit():
return """<h1>Credit App</h1>
<ul>
<li><a href=clients>Clients</a></li>
<li><a href=newclient>New Client</a></li>
<li><a href=initdb>Init DB</a></li>
</ul>"""
@app.route('/initdb')
def initdb():
conn = getconn()
cur = conn.cursor()
cur.execute("drop table if exists client")
cur.execute("create table client "
+ "(id int primary key, lim int, sex int, edu int, mar int, age int)")
conn.commit()
conn.close()
return "DB initialized."
@app.route('/clients')
def clients():
conn = getconn()
cur = conn.cursor()
rows = cur.execute("select id, lim from client limit 500")
html = "<h3>Clients</h3><table>\n"
for row in rows:
html += "<tr><td align=right> %d <td align=right> %.2f\n" % row
return html + "</table>\n"
conn.close()
@app.route('/newclient')
def newclient():
return """<h3>New Client</h3>
<form action=insertclient method=POST>
<table>
<tr><td>Client ID:<td><input type=text name=id>
<tr><td>Credit Limit:<td><input type=text name=lim>
<tr><td>Sex:<td><input type=text name=sex value=1>
<tr><td>Education:<td><input type=text name=edu value=1>
<tr><td>Marriage:<td><input type=text name=mar value=1>
<tr><td>Age:<td><input type=text name=age value=30>
</table><input type=submit value=OK></form>"""
@app.route('/insertclient', methods=['POST'])
def insertclient():
id = request.form['id']
lim = request.form['lim']
sex = request.form['sex']
edu = request.form['edu']
mar = request.form['mar']
age = request.form['age']
conn = getconn()
cur = conn.cursor()
cur.execute('insert into client (id, lim, sex, edu, mar, age) '
+ ' values (?, ?, ?, ?, ?, ?)', (id, lim, sex, edu, mar, age))
conn.commit()
conn.close()
return "Client inserted."
app.run(host='localhost', port=8080, debug=True, use_reloader=True)
The SQLite3 connection module should be part of the Python3 distribution, we just need to import it.
we define a funtion getconn() to give a DB connection whenever we need it. SQlite stores the DB in a file in the current directory, as named in the connect() function. So, we should expect a file ending in ".db" after the initdb route is execute for the first time.
in the initdb route we use that getconn() function and get a cursor from the connection. With this cursor we can execute SQL statements.
we drop the table if it exists so we can run this code multiple times. Note that in this interface we do not end SQL statements with ";"
we create a simple table
we insert a few rows into our table. For the sake of simplicity we use integers for everything; SQLite does not worry about that at all, as we will see.
the commit() is necessary here since auto-commit is only default in interactive use. Without it all changes would be lost when the connection is closed: SQLite supports basic transaction logic.
With the table initialised we can list its content:
- again we get a cursor from the connection
- for Select statements the execute() function returns the resulting rows
- we use the printf() function to force two decimals in the currency values
- without it we would see the actual float values as stored in the DB
- we make a sporting attempt at presenting the content in a nice tabular layout
- in the loop we go through the results and access the fields by index
At this point we can interact with our credit app via the browser; note that we first have to init the DB, then we can enter new clients and list them. However, we want to automate all of that.
Now we can do quite a bit of testing!
Including Python Code in Test Files¶
The following test will
- insert a new client
- list all clients and check for the newly inserted one
A new client should have an ID and a credit limit; we can easily generate those in Python.
We put our Python code into a file with the extension .py in the current directory.
Let's call it mytools.py; create it with your text editor and put the following code into the file:
import sqlite3
import random
def get_new_client_id():
cur = sqlite3.connect("credit.db").cursor()
row = cur.execute("select max(id)+1 from client").fetchone()
return "%d" % row
def get_new_client_limit():
return "%d" % (10000 * random.randint(1,5))
Do not try to run this file directly; it will not produce any useful results. It will work with the robot after the Library definition in the robot code file.
The code above defines two functions for creating new values which are then available in our robot test files as user-defined phrases.
The underscore character translates as blank in the robot code:
get_new_client_id() becomes Get New Client Id
get_new_client_limit() becomes Get New Client Limit
Both functions return values which we can capture in the robot test file.
☆ Note that our simple solution is not thread-safe. SQLite DB files can be accessed be multiple processes; there is no locking for read access, but it employs database locking for write access. Two processes running at the same time will get the same value for max(id)+1 and therefore identical client records.
The option AUTOINCREMENT set in the SQL table create statements should guarantee unique primary keys.
Another clean solution here would be sequences. Sadly, SQLite does not support them. However, we could create a table in the DB init part:
create table mycount(n int)
insert into mycount values(0)
And then use the following code via the Python API to get a new number:
update mycount set n = n + 1
select n from mycount
In theory, this solution should be thread-safe:
- some process A starts the update
- this should lock the DB
- another process B wants to do the same update
- since the DB is locked it has to wait
- until the second statement in A has finished and the lock is released
- new B can do its update and select
A and B should always see different values in their select results.
However, with respect to later development of the sample application we do not worry about this issue here, since we will bulk import external data.
In the Settings section of the robot test file we use the keyword Library to include the code from our tools module.
Now we can do a lot of testing!
We need to call initdb first, otherwise everything else will fail, since the DB table would not yet exists.
Let's put this into a file t6.rob:
*** Settings *** Documentation Check new client insert Library SeleniumLibrary Library mytools.py *** Variables *** ${LOGIN URL} http://localhost:8080 ${BROWSER} Firefox ${ID} 1 *** Test Cases *** Valid App Start Page Open Browser ${LOGIN URL} ${BROWSER} Page Should Contain Link //a[@href="clients"] Page Should Contain Link //a[@href="newclient"] Page Should Contain Link //a[@href="initdb"] Valid Init DB Go To ${LOGIN URL} Click Link //a[@href="initdb"] Page Should Contain DB initialized. Valid Insert ${LIM}= Get New Client Limit Go To ${LOGIN URL} Click Link //a[@href="newclient"] Page Should Contain New Client Input Text //input[@name="id"] ${ID} Input Text //input[@name="lim"] ${LIM} Click Element //input[@type="submit"] Page Should Contain Client inserted Set Global Variable ${LIM} Valid Listing Go To ${LOGIN URL} Click Link //a[@href="clients"] Page Should Contain ${ID} Page Should Contain ${LIM} [Teardown] Close Browser
In this test file we have introduced several test cases; divide and conquer.
To access the generated client ID in more than one test we make it global. Sadly this does not work in the Variables section as one would expect. Instead, we do this in the first test case.
The code above contains some other new features:
the user-defined phrase Get New Client Limit returns a value which we put into a variabel LIM
- note the dollar sign and curly braces with variable names
We do not (yet) use our Get New Client Id the table is empty, and the max(id) expression would result in 'None'
- we just initialized the DB;
- instead we define the client ID in the Variables section
the XPath expression //a[@href="newclient"] finds the first a element with an attribute href equal to "newclient"
we follow this link by using the pre-defined phrase Click Link
the pre-defined phrase Input Text finds the form elements and enters the values
click on the submit button and check the response
now we could go straight to the client listing, but instead
- we go back to the start page and then
- follow the link to the client listing, much like a human user would
Now we check for the name of the new client in the listing
This test will take a little longer; we will probably be able to see the new entry briefly showing in the form fields and the listing.
Performance is not the strong point of this type of automated testing. However, it is still much faster than human testers.
Run the robot:
robot t6.rob
and observe the results; you should see PASS for all tests.
Tasks vs Tests¶
The Robot Framework can easily be used for robotic process automation. This can be made explicit by using the section header Tasks instead of Test Cases; everything else works in the same way as in tests.
In order to facilitate our report processing later we will just continue to use "Test".
We cannot use both tasks and tests in the same robot file.
Arguments to User-Defined Keywords¶
When we create a task to initialise the DB and insert a few clients we do not want to go through all steps for inserting a new client again and again!
DRY: Don't Repeat Yourself.
☠
Code duplication makes it harder to maintain code. It may well be the root of all evil (in software)
We want to define a new user keywords with the required steps in one place and then use that code for repeated application, only supplying the necessary data in each call as arguments (parameters).
Put the following into a file t7.rob:
*** Settings *** Documentation Init DB and insert some clients Library SeleniumLibrary Library mytools.py *** Variables *** ${LOGIN URL} http://localhost:8080 ${BROWSER} Firefox *** Test Cases *** Init DB Open Browser ${LOGIN URL} ${BROWSER} Click Link //a[@href="initdb"] Page Should Contain DB initialized Insert Several Clients Insert Client id=1001 lim=4000 Insert Client id=1002 lim=8000 Insert Client id=2001 Insert Client id=2002 lim=6000 [Teardown] Close Browser *** Keywords *** Insert Client [Arguments] ${id} ${lim}=5000 Go To ${LOGIN URL} Click Link //a[@href="newclient"] Page Should Contain New Client Input Text //input[@name="id"] ${id} Input Text //input[@name="lim"] ${lim} Click Element //input[@type="submit"] Page Should Contain Client inserted
In the Keywords section we use named arguments with default values. This allows us to call the user-defined phrase Insert Client in the Test section. We can supply all arguments, or omit the ones with default values.
Run the robot:
robot t7.rob
and observe the results; again, all tests should PASS.
XML Reports¶
The Robot Framework provides tools for generating summaries from the XML reports for each test; however, they are somewhat cumbersome, and it is easier and more flexible to go through the reports using the XML package of plain Python.
After some study of the structure of the XML files we find that the last status element in each test contains the overall status of the test; we can get that element with the expression findall("status")[-1]
Put the following code into a file xmlrep.py (identical to optional section on Automating the RPA above):
import xml.etree.ElementTree as ET
import sys
def report(fn):
for t in ET.parse(fn).getroot().iter('test'):
s = t.findall("status")[-1]
res = s.get("status") + ' ' + t.get("name")
if s.get("status") == "FAIL": res = res + ' -- ' + s.text
print(res)
if __name__ == '__main__': report(sys.argv[1])
Here the last line is executed when the complete file is run as a script (instead of just importing the report function):
Run it on the command line to check the file output.xml in the current directory:
python3 xmlrep.py output.xml
The output should look like this:
PASS Init DB PASS Insert Several Clients
Running Several Test Case Files¶
The Robot Framework features the concept of a Test Suite for this purpose; however, with our XML reporting tools already available it is more convenient to run individual test case files and then process the reports.
To run several robot tests and direct the output to different XML files for later analysis we can add the XML output file to the robot call.
Linux: Use your text editor to create a script that runs several tests at once and then summarizes all the XML reports.
Put the following code into a file mytests.sh:
robot -o o4.xml t4.rob robot -o o5.xml t5.rob robot -o o6.xml t6.rob for file in o4.xml o5.xml o6.xml do echo $file python3 xmlrep.py $file done
Run this script by entering the following on the command line:
bash mytests.sh
Windows: .bat file, or Python script
Optional: Extracting Robot Code from Notebook cells¶
This notebook is by default saved as a JSON file, which means that it is relatively easy to process the contents of Jupyter notebook cells with our own Python code. Download the .ipynb file, not the HTML version.
Here is a little script to
- extract the cells containing robot test code from this notebook
- execute the tests with the --quiet option
- summarize the results from the XML files
import json
import sys
import os
f = open(sys.argv[1], 'r')
data = json.load(f)
t = 1
cells = data['cells']
for x in cells:
src = x['source']
typ = x['cell_type']
if typ == 'markdown' and len(src) > 2:
if src[1].startswith('*** Settings ***'):
print('Cell', t)
fout = open('tmp.rob', 'w')
fout.write( ''.join(src[1:-1]))
fout.close()
os.system('robot --quiet -o tmp' + str(t) + '.xml tmp.rob')
os.system('python3 xmlrep.py tmp' + str(t) + '.xml')
t += 1
Extracting Data from Web Pages -- Web Scraping¶
The Robot Framework can also be used for web scraping i.e. extracting data from web pages using
- XPath locators
- test case syntax, and
- user-defined Python functions.
The Robot Framework provides functions for file access. However, it is much more convenient and versatile to define our own in plain Python.
Add the following code to the mytools.py library file:
def create_my_file(fn):
fp = open(fn, 'w')
fp.close()
def append_my_file(fn, txt):
fp = open(fn, 'a')
fp.write(txt + '\n')
fp.close()
Now we can use these two new keywords in the following robot test file.
As a real-world practical example let's assume that we are interested in the titles of all Bond movies ever made (although the exact definition of that list is somewhat fuzzy), and after a bit of search we find a web site that has a relatively simple structure:
After exploring the page source (right mouse button, View Page Source in Firefox) we can create our robot file:
*** Settings ***
Library SeleniumLibrary
Library mytools.py
*** Variables ***
${url} https://www.pocket-lint.com/tv/news/148096-james-bond-007-best-movie-viewing-order-chronological-release
*** Test Cases ***
Start Movie List
Open Browser ${url} Firefox
Page Should Contain James Bond movies
Create My File list.txt
Iterate Through Movies
${elements}= Get WebElements //h3
FOR ${element} IN @{elements}
Append My File list.txt ${element.text}
END
[Teardown] Close Browser
The robot code above
- uses the library mytools.py
- defines an URL with a conveniently simple HTML structure for our movie list: simply all level 3 headers
- in the first test case we create our output file
- in the second test case we
- find all h3 elements
- loop through the results and write each item to the file
The result is not perfect but serviceable, at least at the moment. Note that the format of the web page being scraped may change at any time, and our processing will fail.
Misc: Python built-in web server¶
If for some reason you do not want to use Flask and run a very simple local web server that serves just HTML pages then you do not need to write any Python code: just enter the following on the command line:
python3 -m http.server 8080 --bind 127.0.0.1
This will start a very basic web server with the current directory as root for all HTML files. Note that the port must be free at this time; if your flask server still runs on 8080 then the above command will not work.
Exercises¶
Further study the XPath syntax and examples, from sources such as
Find external web pages with moderately complex structure
Write more robot test files to check for elements and text content
Specific challenges:
- relative paths
- position of elements
- wildcards
Make sure not to call the robot too often; leave intervals of a minute or more between calls -- some websites apply automatic exclusion procedures when hit by too many requests from the same source.