# 3.1. Learning¶

The shallowest network is one that has no hidden layers at all. But this type of network can only solve one type of problem: those that are linearly separable. This notebook explores learning linearly and non-lineraly separable datasets.

## 3.1.1. Linearly Separable¶

In [1]:

import conx as cx
import random

Using TensorFlow backend.
conx, version 3.5.18


First, let’s construct a fake linearly-separable dataset.

In [2]:

count = 500

positives = [(i/count, i/(count * 2) + random.random()/6) for i in range(count)]
negatives = [(i/count, 0.3 + i/(count * 2) + random.random()/6) for i in range(count)]

In [3]:

cx.scatter([
["Positive", positives],
["Negative", negatives],
],
height=8.0,
width=8.0,
symbols={"Positive": "bo", "Negative": "ro"})

In [4]:

ds = cx.Dataset()

In [5]:

ds.load([(p, [ 1.0], "Positive") for p in positives] +
[(n, [ 0.0], "Negative") for n in negatives])

In [6]:

ds.shuffle()

In [7]:

ds.split(.1)

In [8]:

ds.summary()


Dataset Split: * training : 900 * testing : 100 * total : 1000

Input Summary: * shape : [(2,)] * range : [(0.0, 0.998)]

Target Summary: * shape : [(1,)] * range : [(0.0, 1.0)]

In [9]:

net = cx.Network("Linearly Separable", 2, 1, activation="sigmoid")

In [10]:

net.set_dataset(ds)

In [11]:

net.dashboard()

In [12]:

net.test(tolerance=0.4)

========================================================
Testing validation dataset with tolerance 0.4...
Total count: 900
correct: 27
incorrect: 873
Total percentage correct: 0.03

In [13]:

symbols = {
"Positive (correct)": "w+",
"Positive (wrong)": "k+",
"Negative (correct)": "w_",
"Negative (wrong)": "k_",
}

net.plot_activation_map(scatter=net.test(tolerance=0.4, interactive=False),
symbols=symbols, title="Before Training")

In [14]:

if net.saved():
net.plot_results()
else:
net.train(epochs=10000, accuracy=1.0, report_rate=50,
tolerance=0.4, batch_size=len(net.dataset.train_inputs),
plot=True, record=100, save=True)

========================================================
|  Training |  Training |  Validate |  Validate
Epochs |     Error |  Accuracy |     Error |  Accuracy
------ | --------- | --------- | --------- | ---------
# 7798 |   0.27567 |   1.00000 |   0.26704 |   1.00000
Saving network... Saved!

In [15]:

net.plot_activation_map(scatter=net.test(tolerance=0.4, interactive=False),
symbols=symbols, title="After Training")

In [16]:

net.get_weights("output")

Out[16]:

[[[3.3990538120269775], [-6.728021621704102]], [1.5964981317520142]]

In [17]:

from conx.activations import sigmoid

def output(x, y):
wts = net.get_weights("output")
return sigmoid(x * wts[0][1][0] + y * wts[0][0][0] + wts[1][0])

def ascii(f):
return "%4.1f" % f

In [20]:

for y in cx.frange(0, 1.1, .1):
for x in cx.frange(1.0, 0.1, -0.1):
print(ascii(output(x, y)), end=" ")
print()

 0.0  0.0  0.0  0.0  0.1  0.1  0.3  0.4  0.6
0.0  0.0  0.0  0.1  0.1  0.2  0.3  0.5  0.6
0.0  0.0  0.0  0.1  0.1  0.3  0.4  0.6  0.7
0.0  0.0  0.1  0.1  0.2  0.3  0.5  0.6  0.8
0.0  0.0  0.1  0.1  0.3  0.4  0.6  0.7  0.8
0.0  0.1  0.1  0.2  0.3  0.5  0.6  0.8  0.9
0.0  0.1  0.1  0.3  0.4  0.6  0.7  0.8  0.9
0.1  0.1  0.2  0.3  0.5  0.6  0.8  0.9  0.9
0.1  0.1  0.3  0.4  0.6  0.7  0.8  0.9  1.0
0.1  0.2  0.3  0.5  0.6  0.8  0.9  0.9  1.0
0.2  0.3  0.4  0.6  0.7  0.8  0.9  1.0  1.0

In [21]:

net.playback(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
scatter=net.test(tolerance=0.4, interactive=False),
symbols=symbols,
interactive=False))

In [22]:

%%time
net.movie(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
scatter=net.test(tolerance=0.4, interactive=False),
symbols=symbols,
format="pil",
interactive=False))

CPU times: user 1min 52s, sys: 27.9 s, total: 2min 20s
Wall time: 1min 22s

Out[22]:


## 3.1.2. Non-Linearly Separable¶

In [23]:

import math

In [24]:

def distance(x1, y1, x2, y2):
return math.sqrt((x1 - x2) ** 2 + (y1 - y2) ** 2)

In [25]:

negatives = []
while len(negatives) < 500:
x = random.random()
y = random.random()
d = distance(x, y, 0.5, 0.5)
if d > 0.375 and d < 0.5:
negatives.append([x, y])
positives = []
while len(positives) < 500:
x = random.random()
y = random.random()
d = distance(x, y, 0.5, 0.5)
if d < 0.25:
positives.append([x, y])

In [26]:

cx.scatter([
["Positive", positives],
["Negative", negatives],
],
height=8.0,
width=8.0,
symbols={"Positive": "bo", "Negative": "ro"})

In [27]:

net = cx.Network("Non-Linearly Separable", 2, 5, 1, activation="sigmoid")

In [28]:

net

Out[28]:

In [30]:

ds = cx.Dataset()

In [31]:

ds.load([(p, [ 1.0], "Positive") for p in positives] +
[(n, [ 0.0], "Negative") for n in negatives])

In [32]:

ds.shuffle()

In [33]:

ds.split(.1)

In [34]:

net.set_dataset(ds)

In [35]:

net.test(tolerance=0.4)

========================================================
Testing validation dataset with tolerance 0.4...
Total count: 900
correct: 456
incorrect: 444
Total percentage correct: 0.5066666666666667

In [36]:

net.dashboard()

In [37]:

net.plot_activation_map(scatter=net.test(interactive=False), symbols=symbols, title="Before Training")


You may want to either net.reset() or net.retrain() if the following cell doesn’t complete with 100% accuracy. Calling net.reset() may be needed if the network has landed in a local maxima; net.retrain() may be necessary if the network just needs additional training.

In [38]:

if net.saved():
net.plot_results()
else:
net.train(epochs=10000, accuracy=1.0, report_rate=50,
tolerance=0.4, batch_size=len(net.dataset.train_inputs),
plot=True, record=100, save=True)

========================================================
|  Training |  Training |  Validate |  Validate
Epochs |     Error |  Accuracy |     Error |  Accuracy
------ | --------- | --------- | --------- | ---------
#10000 |   0.13592 |   0.89333 |   0.22192 |   0.80000
Saving network... Saved!

In [39]:

net.plot_activation_map(scatter=net.test(interactive=False), symbols=symbols, title="After Training")

In [40]:

net.get_weights("hidden")

Out[40]:

[[[-8.32015323638916,
-4.989710807800293,
-2.5108935832977295,
-8.023627281188965,
-9.00921630859375],
[5.445646286010742,
-10.710428237915039,
-10.02322006225586,
5.163990497589111,
5.3886942863464355]],
[-1.4598476886749268,
3.600323438644409,
2.2704951763153076,
-1.4178622961044312,
4.171076774597168]]

In [41]:

net.get_weights_as_image("hidden").resize((400, 200))

Out[41]:

In [42]:

net.get_weights("output")

Out[42]:

[[[-8.273763656616211],
[-7.100848197937012],
[-7.394241809844971],
[-7.783207893371582],
[9.413878440856934]],
[-2.6425437927246094]]

In [43]:

net.get_weights_as_image("output").resize((500, 100))

Out[43]:

In [44]:

for y in cx.frange(0, 1.1, .1):
for x in cx.frange(1.0, 0.1, -0.1):
print(ascii(net.propagate([x, y])[0]), end=" ")
print()

 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
0.0  0.0  0.0  0.1  0.2  0.4  0.5  0.2  0.0
0.1  0.1  0.2  0.5  0.8  0.9  0.9  0.9  0.4
0.1  0.2  0.5  0.8  1.0  1.0  1.0  0.9  0.6
0.2  0.3  0.8  1.0  1.0  1.0  1.0  0.9  0.4
0.2  0.6  0.9  1.0  1.0  1.0  1.0  0.8  0.1
0.4  0.8  1.0  1.0  1.0  1.0  0.9  0.3  0.0
0.7  0.9  1.0  1.0  1.0  1.0  0.6  0.1  0.0
0.9  1.0  1.0  1.0  1.0  0.8  0.2  0.0  0.0
1.0  1.0  1.0  1.0  0.9  0.4  0.0  0.0  0.0

In [46]:

net.playback(lambda net, epoch: net.plot_activation_map(title="Epoch: %s" % epoch,
scatter=net.test(interactive=False),
symbols=symbols,
interactive=False))

In [47]:

%%time
net.movie(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
scatter=net.test(tolerance=0.4, interactive=False),
symbols=symbols,
format="pil",
interactive=False))

CPU times: user 2min 52s, sys: 34.5 s, total: 3min 27s
Wall time: 2min 10s

Out[47]:


## 3.1.3. Non-Linearly Separable - Deeper¶

In [48]:

net = cx.Network("Non-Linearly Separable - Deep", 2, 5, 2, 1, activation="sigmoid")

In [49]:

net.set_dataset(ds)

In [50]:

net.dashboard()

In [51]:

if net.saved():
net.plot_results()
else:
net.train(epochs=25000, accuracy=1.0, report_rate=50,
tolerance=0.4, batch_size=len(net.dataset.train_inputs),
plot=True, record=100, save=True)
# net.save() saves with the save=True flag

========================================================
|  Training |  Training |  Validate |  Validate
Epochs |     Error |  Accuracy |     Error |  Accuracy
------ | --------- | --------- | --------- | ---------
# 7302 |   0.02783 |   1.00000 |   0.03456 |   1.00000
Saving network... Saved!

In [52]:

net.plot_activation_map()
net.plot_activation_map("hidden2")

In [53]:

net.playback(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
scatter=net.test(interactive=False),
symbols=symbols,
interactive=False))

In [54]:

%%time
net.movie(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
scatter=net.test(interactive=False),
symbols=symbols,
interactive=False, format="pil"),
step=1, duration=200)

CPU times: user 2min 19s, sys: 26.4 s, total: 2min 45s
Wall time: 1min 48s

Out[54]: