The Keras toolkit provides a number of commonly used neural net modules for use with Python. The use of a framework like Keras facilitates development; in conjunction with the default Tensorflow backend it is also easy to switch from CPU to GPU computing without any changes in the Python code.
The following example illustrates the use of Keras with the Iris dataset. The Iris dataset contains 150 observations on 5 variables. This dataset is small and well suited for demonstrating the main concepts of neural net machine learning. However, it should be noted that neural nets are typically trained on much larger datasets.
To perform your own computations you can copy the code into a script file such as ml.py and run it on the the command line with
python3 ml.py
You may need to install the packages keras and tensorflow:
pip3 install keras --user
pip3 install tensorflow --user
The example assumes that the file iris.data is present in the current directory. This file is available from the UCI Machine Learning Repository at the URL https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data
The numpy utility genfromtxt() is used to read the file.
Column 4 (with index origin 0) contains the species name which is converted with the defined function conv. Unfortunately, in Python 3 the Numpy utility genfromtxt() does not return strings for text fields, but byte sequences; therefore, the comparison needs to specify byte sequences, indicated by the b in front of the string.
The print statements check whether the import was successful; the number of rows and columns should be 150 rows, 5 columns. The last column should only contain the numbers 1.0, 2.0, and 3.0. The first and last few rows are also printed.
import numpy as np
from keras.models import Sequential
from keras.layers import Dense
def conv(s):
if s == b'Iris-setosa': return 0.0
if s == b'Iris-versicolor': return 1.0
if s == b'Iris-virginica': return 2.0
return -1
data = np.genfromtxt('iris.data', delimiter=',', converters={4:conv})
print(data.shape)
print(set(data[:,4]))
print(data[:3,:])
print(data[-3:,:])
To simplify the situation we take only the first 100 rows from the Iris data which contain the species setosa and versicolor. These two are linearly separable.
The input matrix X to the neural net consists of numeric values for sepal and petal length and width. The output vector y is the corresponding species coded as 0 and 1.
d = np.random.permutation(data[:100])
X = d[:,:4]
y = d[:,4]
print(X[:3,:])
print(y)
The neural net model consists of a densely connected hidden layer with a small number of units using the activation function tanh, followed by the densely connected output layer using a function with output between 0 and 1, such as the sigmoid function.
Keras provides several optimizers; Adam is used here, as it is generally a good choice. It adjust the learning rate during training. The loss function is mean squared error, and the metric is accuracy i.e. how often the correct class is predicted.
A small batch of observations is used in every of training step; after each step the weights are updated. When all data has been processed the next epoch begins.
The weights are initialised randomly, and the data is shuffled in each step. In most cases convergence is quick; however, in some cases the algorithm gets stuck in a local minimum of the cost function with the accuracy remaining at 0.5.
model = Sequential()
model.add(Dense(8, input_dim=4, activation='tanh'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])
model.fit(X, y, epochs=10, batch_size=5)
Since the full Iris data set contains three species the neural net needs to be modified to deal with all classes.
The output of the net will use the softmax function for vectors summing to one, in this case the prediction for the class such as 1 0 0 for setosa, 0 1 0 for versicolor, and 0 0 1 for virginica.
The training vector y need to be cast into this format which is done by the LabelEncoder in the sklearn package.
from keras.utils import np_utils
d = np.random.permutation(data)
X = d[:,:4]
y = np_utils.to_categorical(d[:,4])
print(X[:3,:])
print(y[:5])
The net needs small changes in the number of outputs and type of activation function.
The problem is now a little more complex; an accuracy of about 0.95 is feasible.
model = Sequential()
model.add(Dense(12, input_dim=4, activation='tanh'))
model.add(Dense(3, activation='softmax'))
model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])
model.fit(X, y, epochs=10, batch_size=5)
Overfitting is a problem for this type of machine learning. With a sufficient number of hidden units the net can approximate any function to any given degree. Therefore, the performance on the training data is not the most meaningful information.
For a proper evaluation we need to separate into training and validation data since we are interested in the performance on unseen data.
We use an 80/20 split: we remove the last 30 observations from the training set and use them for evaluation.
d = np.random.permutation(data)
X = d[:,:4]
y = np_utils.to_categorical(d[:,4])
Xt = X[:-30,:]
yt = y[:-30,:]
Xv = X[-30:,:]
yv = y[-30:,:]
model.fit(Xt, yt, epochs=10, batch_size=5)
model.evaluate(Xv, yv)
The result is typical for this type of machine learning: the performance on the training set is very good, but significantly worse on the test set.
Careful tuning of the parameters can achieve a good balance; however, it introduces another problem: when we are tuning in order to achieve better performance on the test data we are again performing an optimization on that data. It is therefore common practise to separate the data into three sets:
This approach only works if the evaluation on the test data is in fact final. There is a certain temptation to disregard procedure; competitions are one way to avoid this problem: participants submit the output of their approach based on the test data input; the correct output for the test data is only published after the competition is finished.