Machine learning methods tend to rely on numerical input. Many types of data such as user reviews are in textual form; some conversion is needed to go from e.g. a review text so some sort of numerical format that can be used as input for a neural net.
One of the more common approaches is discussed here.
This dataset of 2000 movie reviews categorized in positive and negative sentiment classes has been used in a number of studies, and it still provides a good base for comparison of machine learning methods.
http://www.cs.cornell.edu/people/pabo/movie-review-data/
Download and unpack the .tar file in the current directory of your notebook or Python script.
Files are organized in directories txt_sentoken/pos/ and /neg/ with reviews in individual files, such as
txt_sentoken/pos/cv754_7216.txt
kolya is one of the richest films i've seen in some time . zdenek sverak plays a confirmed old bachelor ( who's likely to remain so ) , who finds his life as a czech cellist increasingly impacted by the five-year old boy that he's taking care of . though it ends rather abruptly-- and i'm whining , 'cause i wanted to spend more time with these characters-- the acting , writing , and production values are as high as , if not higher than , comparable american dramas . this father-and-son delight-- sverak also wrote the script , while his son , jan , directed-- won a golden globe for best foreign language film and , a couple days after i saw it , walked away an oscar . in czech and russian , with english subtitles .
We observe
We also see typical problems in data from real sources such as movie reviews: The double dash '--' was not split properly. Expect things like that to happen. Never rely on anything to work perfectly.
Let's print the content of positive and negative reviews and check whether the splitting works as expected:
filelst = ['txt_sentoken/pos/cv754_7216.txt','txt_sentoken/neg/cv435_24355.txt']
for fn in filelst:
print(fn)
print(open(fn).read().split())
There is the occasional weird element, such as '[r]' but generally the data looks good.
We can now encode our data into a numerical format.
Strictly speaking we already have that, by just taking the orders ord(c) in the character set, or even the bit vector encoding each character:
[ ord(c) for c in 'films' ]
Our neural net would have a hard time learning. Nevertheless, similar methods have been applied successfully in machine learning with other network architectures.
With our simple one-layer feed-forward neural network encoding the individual words is a more common approach.
This method takes a list of words w and transforms each text t into an encoding vector e where
e[i] = 1 if t contains w[i], and 0 else
This results in a numeric vector of 0 and 1 with length equal to the number of words in w.
For a list of H hot words and N observations we get an N x H matrix.
To get some idea of what to use for our hot words we turn the token lists into sets and use intersection():
s1 = set(open(filelst[0]).read().split())
s2 = set(open(filelst[1]).read().split())
print('tokens in both:')
print(s1.intersection(s2))
Let's illustrate the idea on our samples and a very small list of hot words.
hotwords = ['golden','high','mostly','script','haphazard']
enc = []
for fn in filelst:
sents = open(fn).read().split()
enc.append([ int(hotwords[i] in sents) for i in range(len(hotwords)) ])
enc = np.asarray(enc)
print(enc)
Some words, like golden and haphazard, carry obvious meaning for sentiment. A particular challenge in this approach is to select a suitable set of hot words for encoding. We take the easy way and simply
We read the corpus by reading each file in the directory pos/ and assign a y-value of 1, and the same for /neg and 0.
import os
from collections import defaultdict
import random
data = []
y = []
d = 'txt_sentoken/'
for sd in ['pos/','neg/']:
for fn in os.listdir(d + sd):
data.append( set(open(d+sd+fn).read().split()) )
y.append( int(sd=='pos/') )
print(len(data), len(y))
To get an idea how that worked we print some data.
Since the reviews are now sets of words there is no order, and we pick random words.
random.seed(17)
print(random.sample(data[0], 10), y[0])
print(random.sample(data[-1], 10), y[-1])
For the hot words we have to come up with some selection criteria.
Let's count the number of occurrence for each word:
wcnt = defaultdict(int)
for wset in data:
for w in wset:
wcnt[w] += 1
To get an idea of the result we print the most frequent words:
print(sorted(wcnt, key=wcnt.get, reverse=True)[:100])
Now we emply a very simple strategy for hot word selection, using the number of occurrence as limits:
hotw = [ k for k in wcnt if wcnt[k] > 100 and wcnt[k] < 500 ]
print(hotw)
print(len(hotw))
Our two-layer feed-forward net is quite efficiently implemented, and an encoding vector of several hundred columns is computationally feasible for a few thousand observations, even for low-end computers.
X = np.asarray([ [ int(hotw[i] in obs) for i in range(len(hotw)) ] for obs in data ])
y = np.asarray(y)
print(X.shape, y.shape)
Our data is now converted into numerical input. The array consists of 0s and 1s only.
print(X)
How many bits are hot for the average observation:
np.mean(np.sum(X, axis=1))
About 10%, this seems reasonable.
Recall our single-layer neural net with sigmoid activation function:
f = lambda x: 1 / (1 + np.exp(-x))
f_ = lambda x: f(x) * (1 - f(x))
def nn1(X, y, l=0.01, epochs=200):
w = np.random.rand(X.shape[1]) - 0.5
for ep in range(epochs):
h = np.dot(X, w)
o = f(h)
w += np.dot(-l * (o - y) * f_(h), X)
if ep % (epochs/5) == 0: print(sum(abs(o-y))/len(X))
return w, sum(abs(o-y)/len(X))
w, e = nn1(X, y)
Even with this very simple approach both for the neural net and the encoding the error drops dramatically from the initial 50% for random guessing.
Of course this performance on the training data is not very relevant for practical problems where we want to detect the sentiment of new observations.
A more useful measure is the performance on a separate part of the dataset that serves as test data.
For this purpose we perform a split into training and testing data. Obviously we have to split X and y in the same fashion, with observations and classes still corresponding.
Here is just one way of doing this.
ix = random.sample(range(len(X)), len(X))
print(ix[:10])
X = X[ix,:]
y = y[ix]
print(y[:10])
The data are now in random order. We use a 80:20 split for training and testing, a common choice.
lim = int(0.8 * len(X))
X_train, y_train = X[:lim,], y[:lim]
X_test, y_test = X[lim:,], y[lim:]
w, e = nn1(X_train, y_train)
Now the net is trained, and we apply the weight vector to the test data i.e. we only apply the feed forward part of the network code:
def fwd(X, y, w):
h = np.dot(X, w)
o = f(h)
return sum(abs(o-y)/len(X))
fwd(X_test, y_test, w)
As expected the performance on the test data is much worse than on the training data.
EXERCISES:
The last step will really bring your programming skills to the next level.