# 3.1. Learning¶

The shallowest network is one that has no hidden layers at all. But this type of network can only solve one type of problem: those that are linearly separable. This notebook explores learning linearly and non-lineraly separable datasets.

## 3.1.1. Linearly Separable¶

In [1]:
from conx import *
import random
Using Theano backend.
conx, version 3.5.10

First, let’s construct a fake linearly-separable dataset.

In [2]:
count = 500

positives = [(i/count, i/(count * 2) + random.random()/6) for i in range(count)]
negatives = [(i/count, 0.3 + i/(count * 2) + random.random()/6) for i in range(count)]
In [3]:
scatter([
["Positive", positives],
["Negative", negatives],
],
height=8.0,
width=8.0,
symbols={"Positive": "bo", "Negative": "ro"})
In [4]:
ds = Dataset()
In [5]:
ds.load([(p, [ 1.0], "Positive") for p in positives] +
[(n, [ 0.0], "Negative") for n in negatives])
In [6]:
ds.shuffle()
In [7]:
ds.split(.1)
In [8]:
ds.summary()

Dataset Split: * training : 900 * testing : 100 * total : 1000

Input Summary: * shape : [(2,)] * range : [(0.0, 0.998)]

Target Summary: * shape : [(1,)] * range : [(0.0, 1.0)]

In [9]:
net = Network("Linearly Separable", 2, 1, activation="sigmoid")
net.compile(error="mae", optimizer="adam")
In [10]:
net.set_dataset(ds)
In [11]:
net.dashboard()
In [12]:
net.test(tolerance=0.4)
========================================================
Testing validation dataset with tolerance 0.4...
Total count: 900
correct: 326
incorrect: 574
Total percentage correct: 0.3622222222222222
In [13]:
symbols = {
"Positive (correct)": "w+",
"Positive (wrong)": "k+",
"Negative (correct)": "w_",
"Negative (wrong)": "k_",
}

net.plot_activation_map(scatter=net.test(tolerance=0.4, interactive=False),
symbols=symbols, title="Before Training")
In [14]:
if net.saved():
net.load()
net.plot_results()
else:
net.train(epochs=10000, accuracy=1.0, report_rate=50,
tolerance=0.4, batch_size=len(net.dataset.train_inputs),
plot=True, record=100, save=True)
========================================================================
|  Training |  Training |  Validate |  Validate
Epochs |     Error |  Accuracy |     Error |  Accuracy
------ | --------- | --------- | --------- | ---------
# 6713 |   0.27460 |   1.00000 |   0.26868 |   1.00000
Saving network... Saved!
In [15]:
net.plot_activation_map(scatter=net.test(tolerance=0.4, interactive=False),
symbols=symbols, title="After Training")
In [16]:
net.get_weights("output")
Out[16]:
[[[3.3102009296417236], [-6.698812961578369]], [1.610288143157959]]
In [27]:
from conx.activations import sigmoid

def output(x, y):
wts = net.get_weights("output")
return sigmoid(x * wts[0][1][0] + y * wts[0][0][0] + wts[1][0])

def ascii(f):
return "%4.1f" % f
In [28]:
for y in frange(0, 1.1, .1):
for x in frange(1.0, 0.1, -0.1):
print(ascii(output(x, y)), end=" ")
print()
0.0  0.0  0.0  0.0  0.1  0.1  0.3  0.4  0.6
0.0  0.0  0.0  0.1  0.1  0.2  0.3  0.5  0.6
0.0  0.0  0.0  0.1  0.1  0.3  0.4  0.6  0.7
0.0  0.0  0.1  0.1  0.2  0.3  0.5  0.6  0.8
0.0  0.0  0.1  0.1  0.3  0.4  0.6  0.7  0.8
0.0  0.1  0.1  0.2  0.3  0.5  0.6  0.8  0.9
0.0  0.1  0.1  0.3  0.4  0.6  0.7  0.8  0.9
0.1  0.1  0.2  0.3  0.5  0.6  0.8  0.9  0.9
0.1  0.1  0.2  0.4  0.6  0.7  0.8  0.9  0.9
0.1  0.2  0.3  0.5  0.6  0.8  0.9  0.9  1.0
0.1  0.2  0.4  0.6  0.7  0.8  0.9  0.9  1.0
In [29]:
net.playback(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
scatter=net.test(tolerance=0.4, interactive=False),
symbols=symbols,
interactive=False))
In [30]:
net.movie(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
scatter=net.test(tolerance=0.4, interactive=False),
symbols=symbols,
format="pil",
interactive=False))
Out[30]:

## 3.1.2. Non-Linearly Separable¶

In [31]:
import math
In [32]:
def distance(x1, y1, x2, y2):
return math.sqrt((x1 - x2) ** 2 + (y1 - y2) ** 2)
In [33]:
negatives = []
while len(negatives) < 500:
x = random.random()
y = random.random()
d = distance(x, y, 0.5, 0.5)
if d > 0.375 and d < 0.5:
negatives.append([x, y])
positives = []
while len(positives) < 500:
x = random.random()
y = random.random()
d = distance(x, y, 0.5, 0.5)
if d < 0.25:
positives.append([x, y])
In [34]:
scatter([
["Positive", positives],
["Negative", negatives],
],
height=8.0,
width=8.0,
symbols={"Positive": "bo", "Negative": "ro"})
In [35]:
net = Network("Non-Linearly Separable", 2, 5, 1, activation="sigmoid")
net.compile(error="mae", optimizer="adam")
In [36]:
net
Out[36]:
In [37]:
ds = Dataset()
In [38]:
ds.load([(p, [ 1.0], "Positive") for p in positives] +
[(n, [ 0.0], "Negative") for n in negatives])
In [39]:
ds.shuffle()
In [40]:
ds.split(.1)
In [41]:
net.set_dataset(ds)
In [42]:
net.test(tolerance=0.4)
========================================================
Testing validation dataset with tolerance 0.4...
Total count: 900
correct: 0
incorrect: 900
Total percentage correct: 0.0
In [43]:
net.dashboard()
In [44]:
net.plot_activation_map(scatter=net.test(interactive=False), symbols=symbols, title="Before Training")

You may want to either net.reset() or net.retrain() if the following cell doesn’t complete with 100% accuracy. Calling net.reset() may be needed if the network has landed in a local maxima; net.retrain() may be necessary if the network just needs additional training.

In [45]:
if net.saved():
net.load()
net.plot_results()
else:
net.train(epochs=10000, accuracy=1.0, report_rate=50,
tolerance=0.4, batch_size=len(net.dataset.train_inputs),
plot=True, record=100, save=True)
========================================================================
|  Training |  Training |  Validate |  Validate
Epochs |     Error |  Accuracy |     Error |  Accuracy
------ | --------- | --------- | --------- | ---------
#10000 |   0.29694 |   0.71889 |   0.22183 |   0.80000
Saving network... Saved!
In [46]:
net.plot_activation_map(scatter=net.test(interactive=False), symbols=symbols, title="After Training")
In [47]:
net.get_weights("hidden")
Out[47]:
[[[-4.994647026062012,
1.4458032846450806,
-10.851877212524414,
-7.857443809509277,
0.9621898531913757],
[-7.210960865020752,
10.02563762664795,
0.3498399555683136,
-4.398433685302734,
10.174257278442383]],
[8.520471572875977,
-2.3241660594940186,
2.1214261054992676,
2.188873529434204,
-2.1985650062561035]]
In [48]:
net.get_weights_as_image("hidden").resize((400, 200))
Out[48]:
In [49]:
net.get_weights("output")
Out[49]:
[[[-6.4147844314575195],
[6.576595783233643],
[-8.222257614135742],
[-6.249880313873291],
[6.961214542388916]],
[-1.430251121520996]]
In [50]:
net.get_weights_as_image("output").resize((500, 100))
Out[50]:
In [51]:
for y in frange(0, 1.1, .1):
for x in frange(1.0, 0.1, -0.1):
print(ascii(net.propagate([x, y])[0]), end=" ")
print()
0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
0.3  0.2  0.1  0.1  0.0  0.0  0.0  0.0  0.0
0.9  0.9  0.8  0.7  0.6  0.4  0.1  0.0  0.0
1.0  1.0  1.0  1.0  1.0  0.9  0.8  0.3  0.0
1.0  1.0  1.0  1.0  1.0  1.0  1.0  0.8  0.2
1.0  1.0  1.0  1.0  1.0  1.0  1.0  0.9  0.4
1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  0.6
1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  0.8
1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  0.8
1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  0.9
1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0
In [52]:
net.playback(lambda net, epoch: net.plot_activation_map(title="Epoch: %s" % epoch,
scatter=net.test(interactive=False),
symbols=symbols,
interactive=False))
In [44]:
net.movie(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
scatter=net.test(tolerance=0.4, interactive=False),
symbols=symbols,
format="pil",
interactive=False))
Out[44]:

## 3.1.3. Non-Linearly Separable - Deeper¶

In [53]:
net = Network("Non-Linearly Separable - Deep", 2, 5, 2, 1, activation="sigmoid")
net.compile(error="mae", optimizer="adam")
In [54]:
net.set_dataset(ds)
In [55]:
net.dashboard()
In [56]:
if net.saved():
net.load()
net.plot_results()
else:
net.train(epochs=25000, accuracy=1.0, report_rate=50,
tolerance=0.4, batch_size=len(net.dataset.train_inputs),
plot=True, record=100, save=True)
========================================================================
|  Training |  Training |  Validate |  Validate
Epochs |     Error |  Accuracy |     Error |  Accuracy
------ | --------- | --------- | --------- | ---------
# 4579 |   0.06875 |   1.00000 |   0.07678 |   1.00000
Saving network... Saved!
In [57]:
net.plot_activation_map()
net.plot_activation_map("hidden2")
In [58]:
net.playback(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
scatter=net.test(interactive=False),
symbols=symbols,
interactive=False))
In [48]:
%%time
net.movie(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
scatter=net.test(interactive=False),
symbols=symbols,
interactive=False, format="pil"),
step=1, duration=200)
CPU times: user 47.8 s, sys: 19.9 s, total: 1min 7s
Wall time: 49.9 s
Out[48]: