3.1. Learning

The shallowest network is one that has no hidden layers at all. But this type of network can only solve one type of problem: those that are linearly separable. This notebook explores learning linearly and non-lineraly separable datasets.

3.1.1. Linearly Separable

In [1]:
import conx as cx
import random
Using TensorFlow backend.
conx, version 3.5.18

First, let’s construct a fake linearly-separable dataset.

In [2]:
count = 500

positives = [(i/count, i/(count * 2) + random.random()/6) for i in range(count)]
negatives = [(i/count, 0.3 + i/(count * 2) + random.random()/6) for i in range(count)]
In [3]:
cx.scatter([
         ["Positive", positives],
         ["Negative", negatives],
        ],
    height=8.0,
    width=8.0,
    symbols={"Positive": "bo", "Negative": "ro"})
_images/Learning_4_0.png
In [4]:
ds = cx.Dataset()
In [5]:
ds.load([(p, [ 1.0], "Positive") for p in positives] +
        [(n, [ 0.0], "Negative") for n in negatives])
In [6]:
ds.shuffle()
In [7]:
ds.split(.1)
In [8]:
ds.summary()

Dataset Split: * training : 900 * testing : 100 * total : 1000

Input Summary: * shape : [(2,)] * range : [(0.0, 0.998)]

Target Summary: * shape : [(1,)] * range : [(0.0, 1.0)]

In [9]:
net = cx.Network("Linearly Separable", 2, 1, activation="sigmoid")
net.compile(error="mae", optimizer="adam")
In [10]:
net.set_dataset(ds)
In [11]:
net.dashboard()
In [12]:
net.test(tolerance=0.4)
========================================================
Testing validation dataset with tolerance 0.4...
Total count: 900
      correct: 27
      incorrect: 873
Total percentage correct: 0.03
In [13]:
symbols = {
    "Positive (correct)": "w+",
    "Positive (wrong)": "k+",
    "Negative (correct)": "w_",
    "Negative (wrong)": "k_",
}

net.plot_activation_map(scatter=net.test(tolerance=0.4, interactive=False),
                        symbols=symbols, title="Before Training")
_images/Learning_14_0.png
In [14]:
if net.saved():
    net.load()
    net.plot_results()
else:
    net.train(epochs=10000, accuracy=1.0, report_rate=50,
             tolerance=0.4, batch_size=len(net.dataset.train_inputs),
             plot=True, record=100, save=True)
_images/Learning_15_0.svg
========================================================
       |  Training |  Training |  Validate |  Validate
Epochs |     Error |  Accuracy |     Error |  Accuracy
------ | --------- | --------- | --------- | ---------
# 7798 |   0.27567 |   1.00000 |   0.26704 |   1.00000
Saving network... Saved!
In [15]:
net.plot_activation_map(scatter=net.test(tolerance=0.4, interactive=False),
                        symbols=symbols, title="After Training")
_images/Learning_16_0.png
In [16]:
net.get_weights("output")
Out[16]:
[[[3.3990538120269775], [-6.728021621704102]], [1.5964981317520142]]
In [17]:
from conx.activations import sigmoid

def output(x, y):
    wts = net.get_weights("output")
    return sigmoid(x * wts[0][1][0] + y * wts[0][0][0] + wts[1][0])

def ascii(f):
    return "%4.1f" % f
In [20]:
for y in cx.frange(0, 1.1, .1):
    for x in cx.frange(1.0, 0.1, -0.1):
        print(ascii(output(x, y)), end=" ")
    print()
 0.0  0.0  0.0  0.0  0.1  0.1  0.3  0.4  0.6
 0.0  0.0  0.0  0.1  0.1  0.2  0.3  0.5  0.6
 0.0  0.0  0.0  0.1  0.1  0.3  0.4  0.6  0.7
 0.0  0.0  0.1  0.1  0.2  0.3  0.5  0.6  0.8
 0.0  0.0  0.1  0.1  0.3  0.4  0.6  0.7  0.8
 0.0  0.1  0.1  0.2  0.3  0.5  0.6  0.8  0.9
 0.0  0.1  0.1  0.3  0.4  0.6  0.7  0.8  0.9
 0.1  0.1  0.2  0.3  0.5  0.6  0.8  0.9  0.9
 0.1  0.1  0.3  0.4  0.6  0.7  0.8  0.9  1.0
 0.1  0.2  0.3  0.5  0.6  0.8  0.9  0.9  1.0
 0.2  0.3  0.4  0.6  0.7  0.8  0.9  1.0  1.0
In [21]:
net.playback(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
                                                        scatter=net.test(tolerance=0.4, interactive=False),
                                                        symbols=symbols,
                                                        interactive=False))
_images/Learning_20_1.svg
In [22]:
%%time
net.movie(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
                                                     scatter=net.test(tolerance=0.4, interactive=False),
                                                     symbols=symbols,
                                                     format="pil",
                                                     interactive=False))
CPU times: user 1min 52s, sys: 27.9 s, total: 2min 20s
Wall time: 1min 22s
Out[22]:

3.1.2. Non-Linearly Separable

In [23]:
import math
In [24]:
def distance(x1, y1, x2, y2):
    return math.sqrt((x1 - x2) ** 2 + (y1 - y2) ** 2)
In [25]:
negatives = []
while len(negatives) < 500:
    x = random.random()
    y = random.random()
    d = distance(x, y, 0.5, 0.5)
    if d > 0.375 and d < 0.5:
        negatives.append([x, y])
positives = []
while len(positives) < 500:
    x = random.random()
    y = random.random()
    d = distance(x, y, 0.5, 0.5)
    if d < 0.25:
        positives.append([x, y])
In [26]:
cx.scatter([
         ["Positive", positives],
         ["Negative", negatives],
        ],
    height=8.0,
    width=8.0,
    symbols={"Positive": "bo", "Negative": "ro"})
_images/Learning_26_0.png
In [27]:
net = cx.Network("Non-Linearly Separable", 2, 5, 1, activation="sigmoid")
net.compile(error="mae", optimizer="adam")
In [28]:
net
Out[28]:
Layer: output (output) output range: (0, 1) shape = (1,) Keras class = Dense activation = sigmoidoutputWeights from hidden to output output_2/kernel:0 has shape (5, 1) output_2/bias:0 has shape (1,)Layer: hidden (hidden) output range: (0, 1) shape = (5,) Keras class = Dense activation = sigmoidhiddenWeights from input to hidden hidden/kernel:0 has shape (2, 5) hidden/bias:0 has shape (5,)Layer: input (input) output range: (-Infinity, +Infinity) shape = (2,) Keras class = InputinputNon-Linearly Separable
In [30]:
ds = cx.Dataset()
In [31]:
ds.load([(p, [ 1.0], "Positive") for p in positives] +
        [(n, [ 0.0], "Negative") for n in negatives])
In [32]:
ds.shuffle()
In [33]:
ds.split(.1)
In [34]:
net.set_dataset(ds)
In [35]:
net.test(tolerance=0.4)
========================================================
Testing validation dataset with tolerance 0.4...
Total count: 900
      correct: 456
      incorrect: 444
Total percentage correct: 0.5066666666666667
In [36]:
net.dashboard()
In [37]:
net.plot_activation_map(scatter=net.test(interactive=False), symbols=symbols, title="Before Training")
_images/Learning_36_0.png

You may want to either net.reset() or net.retrain() if the following cell doesn’t complete with 100% accuracy. Calling net.reset() may be needed if the network has landed in a local maxima; net.retrain() may be necessary if the network just needs additional training.

In [38]:
if net.saved():
    net.load()
    net.plot_results()
else:
    net.train(epochs=10000, accuracy=1.0, report_rate=50,
              tolerance=0.4, batch_size=len(net.dataset.train_inputs),
              plot=True, record=100, save=True)
_images/Learning_38_0.svg
========================================================
       |  Training |  Training |  Validate |  Validate
Epochs |     Error |  Accuracy |     Error |  Accuracy
------ | --------- | --------- | --------- | ---------
#10000 |   0.13592 |   0.89333 |   0.22192 |   0.80000
Saving network... Saved!
In [39]:
net.plot_activation_map(scatter=net.test(interactive=False), symbols=symbols, title="After Training")
_images/Learning_39_0.png
In [40]:
net.get_weights("hidden")
Out[40]:
[[[-8.32015323638916,
   -4.989710807800293,
   -2.5108935832977295,
   -8.023627281188965,
   -9.00921630859375],
  [5.445646286010742,
   -10.710428237915039,
   -10.02322006225586,
   5.163990497589111,
   5.3886942863464355]],
 [-1.4598476886749268,
  3.600323438644409,
  2.2704951763153076,
  -1.4178622961044312,
  4.171076774597168]]
In [41]:
net.get_weights_as_image("hidden").resize((400, 200))
Out[41]:
_images/Learning_41_0.png
In [42]:
net.get_weights("output")
Out[42]:
[[[-8.273763656616211],
  [-7.100848197937012],
  [-7.394241809844971],
  [-7.783207893371582],
  [9.413878440856934]],
 [-2.6425437927246094]]
In [43]:
net.get_weights_as_image("output").resize((500, 100))
Out[43]:
_images/Learning_43_0.png
In [44]:
for y in cx.frange(0, 1.1, .1):
    for x in cx.frange(1.0, 0.1, -0.1):
        print(ascii(net.propagate([x, y])[0]), end=" ")
    print()
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.1  0.2  0.4  0.5  0.2  0.0
 0.1  0.1  0.2  0.5  0.8  0.9  0.9  0.9  0.4
 0.1  0.2  0.5  0.8  1.0  1.0  1.0  0.9  0.6
 0.2  0.3  0.8  1.0  1.0  1.0  1.0  0.9  0.4
 0.2  0.6  0.9  1.0  1.0  1.0  1.0  0.8  0.1
 0.4  0.8  1.0  1.0  1.0  1.0  0.9  0.3  0.0
 0.7  0.9  1.0  1.0  1.0  1.0  0.6  0.1  0.0
 0.9  1.0  1.0  1.0  1.0  0.8  0.2  0.0  0.0
 1.0  1.0  1.0  1.0  0.9  0.4  0.0  0.0  0.0
In [46]:
net.playback(lambda net, epoch: net.plot_activation_map(title="Epoch: %s" % epoch,
                                                        scatter=net.test(interactive=False),
                                                        symbols=symbols,
                                                        interactive=False))
_images/Learning_45_1.svg
In [47]:
%%time
net.movie(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
                                                     scatter=net.test(tolerance=0.4, interactive=False),
                                                     symbols=symbols,
                                                     format="pil",
                                                     interactive=False))
CPU times: user 2min 52s, sys: 34.5 s, total: 3min 27s
Wall time: 2min 10s
Out[47]:

3.1.3. Non-Linearly Separable - Deeper

In [48]:
net = cx.Network("Non-Linearly Separable - Deep", 2, 5, 2, 1, activation="sigmoid")
net.compile(error="mae", optimizer="adam")
In [49]:
net.set_dataset(ds)
In [50]:
net.dashboard()
In [51]:
if net.saved():
    net.load()
    net.plot_results()
else:
    net.train(epochs=25000, accuracy=1.0, report_rate=50,
              tolerance=0.4, batch_size=len(net.dataset.train_inputs),
              plot=True, record=100, save=True)
    # net.save() saves with the save=True flag
_images/Learning_51_0.svg
========================================================
       |  Training |  Training |  Validate |  Validate
Epochs |     Error |  Accuracy |     Error |  Accuracy
------ | --------- | --------- | --------- | ---------
# 7302 |   0.02783 |   1.00000 |   0.03456 |   1.00000
Saving network... Saved!
In [52]:
net.plot_activation_map()
net.plot_activation_map("hidden2")
_images/Learning_52_0.png
_images/Learning_52_1.png
In [53]:
net.playback(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
                                                        scatter=net.test(interactive=False),
                                                        symbols=symbols,
                                                        interactive=False))
_images/Learning_53_1.svg
In [54]:
%%time
net.movie(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
                                                        scatter=net.test(interactive=False),
                                                        symbols=symbols,
                                                        interactive=False, format="pil"),
         step=1, duration=200)
CPU times: user 2min 19s, sys: 26.4 s, total: 2min 45s
Wall time: 1min 48s
Out[54]: