3.1. Learning

The shallowest network is one that has no hidden layers at all. But this type of network can only solve one type of problem: those that are linearly separable. This notebook explores learning linearly and non-linearly separable datasets.

3.1.1. Linearly Separable

import conx as cx
import random

Using TensorFlow backend.
Conx, version 3.6.1

First, let’s construct a fake linearly-separable dataset.

count = 500

positives = [(i/count, i/(count * 2) + random.random()/6) for i in range(count)]
negatives = [(i/count, 0.3 + i/(count * 2) + random.random()/6) for i in range(count)]

cx.scatter([
         ["Positive", positives],
         ["Negative", negatives],
        ],
    symbols={"Positive": "bo", "Negative": "ro"})

Let’s make a dataset out of these two types:

ds = cx.Dataset()

A dataset is composed of:

• inputs - the (x,y)
• targets - [0] or [1]
• labels - a string representing the category

ds.load([(p, [ 1.0], "Positive") for p in positives] +
        [(n, [ 0.0], "Negative") for n in negatives])

Although the patterns will be shuffled during training, we will also shuffle them here just to make it clear that as we view them that the target should be determined from the (x,y) values.

ds.shuffle()

We will save 10% of the patterns for testing/validation:

ds.split(.1)

And finally, we look at the dataset summary to make sure it looks right:

ds.summary()

_________________________________________________________________
Unnamed Dataset:
Patterns    Shape                 Range
=================================================================
inputs      (2,)                  (0.0, 0.998)
targets     (1,)                  (0.0, 1.0)
=================================================================
Total patterns: 1000
   Training patterns: 900
   Testing patterns: 100
_________________________________________________________________

Now we are ready to design the network. This one is fairly easy: the inputs and outputs are determined by the dataset. We need only determine the error function, the optimizer, and the activation function on the output layer:

net = cx.Network("Linearly Separable", 2, 1, activation="sigmoid")
net.compile(error="mae", optimizer="adam")

We associate the dataset with the network:

net.set_dataset(ds)

And let’s use the dashboard to see the dataset and network:

net.dashboard()

How complicated is this network?

net.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input (InputLayer)           (None, 2)                 0
_________________________________________________________________
output (Dense)               (None, 1)                 3
=================================================================
Total params: 3
Trainable params: 3
Non-trainable params: 0
_________________________________________________________________

We can get a flat array of these weights with net.to_array():

net.to_array()

[0.09843516, 1.1521822, 0.0]

Before training, let’s see how bad the network does with these random weights:

net.test(tolerance=0.4)

========================================================
Testing validation dataset with tolerance 0.4...
Total count: 900
      correct: 230
      incorrect: 670
Total percentage correct: 0.25555555555555554

Also, we can use the net.plot_activation_map() method that Lisa introduced in the “Getting Started with Conx” notebook.

Recall that the method will sample from the range of possible inputs of two units of an input layer, and will show you a “heatmap” of the resulting activation of one unit in any hidden or output layer.

By default, it will use the first two units of a layer named “input” and will propagate to the first unit of a layer called “output”. This matches our network exactly, so we can just use the defaults.

net.plot_activation_map(title="Before Training")

In addition, we can also put scatter plot on top of the activations, and give those points a legend by using the net.test() method again, but this time with interactive=False:

# net.test(tolerance=0.4, interactive=False)

This will return a list of tuples in the form:

(“LABEL (correct | wrong)”, INPUTS)

like so:

[('Negative (correct)',
  [[0.5239999890327454, 0.7271904349327087],
  ...
   [0.33000001311302185, 0.5292619466781616]]),
 ('Positive (wrong)',
  [[0.08399999886751175, 0.3982737064361572],
  ...
   [0.33000001311302185, 0.5292619466781616]]),
 ('Positive (correct)',
  [[0.08399999886751175, 0.3982737064361572],
  ...
   [0.33000001311302185, 0.5292619466781616]]),
 ('Negative (wrong)',
  [[0.08399999886751175, 0.3982737064361572],
   ...
   [0.014000000432133675, 0.05173352733254433]])]

Putting that all together, with a dictionary of symbols, gives:

symbols = {
    "Positive (correct)": "w+",
    "Positive (wrong)": "k+",
    "Negative (correct)": "w_",
    "Negative (wrong)": "k_",
}

net.plot_activation_map(scatter=net.test(tolerance=0.4, interactive=False),
                        symbols=symbols, title="Before Training")

Ok, now we are ready to train the network:

#net.delete()
#net.reset()

if net.saved():
    net.load()
    net.plot_results()
else:
    net.train(epochs=10000, accuracy=1.0, report_rate=50,
             tolerance=0.4, batch_size=len(net.dataset.train_inputs),
             plot=True, record=100, save=True)

Now we can see the trained network on the training set:

net.plot_activation_map(scatter=net.test(tolerance=0.4, interactive=False),
                        symbols=symbols, title="After Training")

Now, we can see the weights after training:

net.to_array()

[3.086318, -6.2050633, 1.4618336]

To verify the network so see that it is computing the outputs as expected, we can manually do the math.

That is, the sigmoid(sum of the inputs * weights + bias) should gives us the same output as net.propagate().

from conx.activations import sigmoid

def output(x, y):
    wts = net.get_weights("output")
    return sigmoid(x * wts[0][1][0] + y * wts[0][0][0] + wts[1][0])

def ascii(f):
    return "%4.1f" % f

We can use that to make our own ASCII heatmap:

for y in cx.frange(0, 1.1, .1):
    for x in cx.frange(1.0, 0.1, -0.1):
        print(ascii(output(x, y)), end=" ")
    print()

 0.0  0.0  0.0  0.1  0.1  0.2  0.3  0.4  0.6
 0.0  0.0  0.0  0.1  0.1  0.2  0.3  0.5  0.6
 0.0  0.0  0.1  0.1  0.2  0.3  0.4  0.6  0.7
 0.0  0.0  0.1  0.1  0.2  0.3  0.5  0.6  0.8
 0.0  0.1  0.1  0.2  0.3  0.4  0.6  0.7  0.8
 0.0  0.1  0.1  0.2  0.3  0.5  0.6  0.8  0.9
 0.1  0.1  0.2  0.3  0.4  0.6  0.7  0.8  0.9
 0.1  0.1  0.2  0.3  0.5  0.6  0.8  0.9  0.9
 0.1  0.2  0.3  0.4  0.6  0.7  0.8  0.9  0.9
 0.1  0.2  0.3  0.5  0.6  0.8  0.9  0.9  1.0
 0.2  0.3  0.4  0.6  0.7  0.8  0.9  0.9  1.0

Finally, it would be very useful to see how learning works over time. For this, we can use the net.playback() method. It takes a function of the form:

def function(network, epoch):
    return IMAGE

The function is given the network as it was at the time of EPOCH. That is, the network is transported back in time.

We use a variation of the plot_activation_map:

net.playback(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
                                                        scatter=net.test(tolerance=0.4, interactive=False),
                                                        symbols=symbols,
                                                        format="svg"))

net.set_weights_from_history(-1)

# net.movie(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
#                                                      scatter=net.test(tolerance=0.4, interactive=False),
#                                                      symbols=symbols,
#                                                      format="image"))

3.1.2. Non-Linearly Separable

import math

def distance(x1, y1, x2, y2):
    return math.sqrt((x1 - x2) ** 2 + (y1 - y2) ** 2)

negatives = []
while len(negatives) < 500:
    x = random.random()
    y = random.random()
    d = distance(x, y, 0.5, 0.5)
    if d > 0.375 and d < 0.5:
        negatives.append([x, y])
positives = []
while len(positives) < 500:
    x = random.random()
    y = random.random()
    d = distance(x, y, 0.5, 0.5)
    if d < 0.25:
        positives.append([x, y])

cx.scatter([
         ["Positive", positives],
         ["Negative", negatives],
        ],
    symbols={"Positive": "bo", "Negative": "ro"})

net = cx.Network("Non-Linearly Separable", 2, 5, 1, activation="sigmoid")
net.compile(error="mae", optimizer="adam")

net.picture()

ds = cx.Dataset()

ds.load([(p, [ 1.0], "Positive") for p in positives] +
        [(n, [ 0.0], "Negative") for n in negatives])

ds.shuffle()

ds.split(.1)

net.set_dataset(ds)

net.test(tolerance=0.4)

========================================================
Testing validation dataset with tolerance 0.4...
Total count: 900
      correct: 446
      incorrect: 454
Total percentage correct: 0.4955555555555556

net.dashboard()

net.plot_activation_map(scatter=net.test(interactive=False), symbols=symbols, title="Before Training")

You may want to either net.reset() or net.retrain() if the following cell doesn’t complete with 100% accuracy. Calling net.reset() may be needed if the network has landed in a local maxima; net.retrain() may be necessary if the network just needs additional training.

# net.delete()
# net.reset()

if net.saved():
    net.load()
    net.plot_results()
else:
    net.train(epochs=10000, accuracy=1.0, report_rate=50,
              tolerance=0.4, batch_size=256,
              plot=True, record=100, save=True)

net.plot_activation_map(scatter=net.test(interactive=False), symbols=symbols, title="After Training")

for y in cx.frange(0, 1.1, .1):
    for x in cx.frange(1.0, 0.1, -0.1):
        print(ascii(net.propagate([x, y])[0]), end=" ")
    print()

 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.2  0.3  0.4  0.3  0.1  0.1
 0.0  0.1  0.2  0.7  0.9  1.0  0.9  0.7  0.2
 0.0  0.1  0.4  0.9  1.0  1.0  1.0  1.0  0.8
 0.0  0.1  0.4  0.9  1.0  1.0  1.0  1.0  0.8
 0.0  0.1  0.4  0.9  1.0  1.0  1.0  0.9  0.4
 0.0  0.1  0.3  0.8  0.9  0.9  0.8  0.3  0.0
 0.0  0.0  0.2  0.5  0.6  0.5  0.1  0.0  0.0
 0.0  0.0  0.0  0.1  0.1  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0

net.playback(lambda net, epoch: net.plot_activation_map(title="Epoch: %s" % epoch,
                                                        scatter=net.test(interactive=False),
                                                        symbols=symbols,
                                                        format="svg"))

# net.movie(lambda net, epoch: net.plot_activation_map(title="Epoch %s" % epoch,
#                                                      scatter=net.test(tolerance=0.4, interactive=False),
#                                                      symbols=symbols,
#                                                      format="image"))

The above cell, when run, will create and display the movie. It may take a few minutes to create. If you don’t see it above, you can also click on the link Non-Lineraly_Separable-Movie.mp4 to see it.