3.6. Autoencoding¶

3.6.1. One-hot Encoder¶

In [1]:

from conx import *

Using TensorFlow backend.
/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
  return f(*args, **kwds)
conx, version 3.5.15

In [3]:

size = 5

In [4]:

net = Network("Autoencoder")
net.add(Layer("input", size)) #, minmax=(0,1)))
net.add(Layer("hidden", 5, activation="relu", visible=True))
net.add(Layer("output", size, activation="sigmoid"))
net.config["font_family"] = "monospace"

In [5]:

net.connect()

In [6]:

net.compile(error='binary_crossentropy', optimizer="adam")

In [7]:

net

Out[7]:

In [8]:

patterns = [onehot(i, size) for i in range(size)]

In [9]:

patterns[0]

Out[9]:

[1, 0, 0, 0, 0]

In [10]:

net.dataset.load([(p,p) for p in patterns])

In [11]:

net

Out[11]:

In [12]:

import time
for i in range(size):
    net.propagate(net.dataset.inputs[i], visualize=True)
    time.sleep(1)

In [13]:

net.dataset.summary()

Dataset Split: * training : 5 * testing : 0 * total : 5

Input Summary: * shape : [(5,)] * range : [(0.0, 1.0)]

Target Summary: * shape : [(5,)] * range : [(0.0, 1.0)]

In [14]:

net.reset()
net.train(accuracy=1, epochs=10000, report_rate=200, tolerance=0.4, plot=True)

========================================================================
       |  Training |  Training
Epochs |     Error |  Accuracy
------ | --------- | ---------
# 1789 |   0.09234 |   1.00000

In [15]:

net.propagate(net.dataset.inputs[0])

Out[15]:

[0.8396602272987366,
05913377180695534,
001064726267941296,
015739908441901207,
01280010212212801]

In [16]:

net.test(tolerance=0.4, show=True)

========================================================
Testing validation dataset with tolerance 0.4...
# | inputs | targets | outputs | result
---------------------------------------
0 | [[1.00,0.00,0.00,0.00,0.00]] | [[1.00,0.00,0.00,0.00,0.00]] | [0.84,0.06,0.00,0.02,0.01] | correct
1 | [[0.00,1.00,0.00,0.00,0.00]] | [[0.00,1.00,0.00,0.00,0.00]] | [0.05,0.94,0.02,0.01,0.00] | correct
2 | [[0.00,0.00,1.00,0.00,0.00]] | [[0.00,0.00,1.00,0.00,0.00]] | [0.03,0.09,0.76,0.40,0.01] | correct
3 | [[0.00,0.00,0.00,1.00,0.00]] | [[0.00,0.00,0.00,1.00,0.00]] | [0.00,0.00,0.11,0.70,0.04] | correct
4 | [[0.00,0.00,0.00,0.00,1.00]] | [[0.00,0.00,0.00,0.00,1.00]] | [0.12,0.00,0.00,0.22,0.93] | correct
Total count: 5
      correct: 5
      incorrect: 0
Total percentage correct: 1.0

In [17]:

for i in range(size):
    net.propagate(net.dataset.inputs[i], visualize=True)
    time.sleep(1)

In [18]:

net.dashboard()

3.6.2. MNIST Autoencoding¶

In [19]:

from conx import *

In [20]:

net = Network("MNIST-Autoencoder")

In [21]:

net.add(ImageLayer("input", (28,28), 1))
net.add(Conv2DLayer("conv", 3, (5,5), activation="relu"))
net.add(MaxPool2DLayer("pool", pool_size=(2,2)))
net.add(FlattenLayer("flatten"))
net.add(Layer("hidden3", 25, activation="relu"))
net.add(Layer("output", (28,28,1), activation="sigmoid"))

Out[21]:

'output'

In [22]:

net.connect()
net.compile(error="mse", optimizer="adam")
net

Out[22]:

In [23]:

net.dataset.get('mnist')
net.dataset.set_targets_from_inputs()
net.dataset.targets.reshape(0, (28 * 28))
net.dataset.summary()

WARNING: network 'MNIST-Autoencoder' target bank #0 has a multi-dimensional shape, which is not allowed

Dataset name: MNIST

Original source: http://yann.lecun.com/exdb/mnist/

The MNIST database of handwritten digits, available from this page, has 70,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image. It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting.

Dataset Split: * training : 70000 * testing : 0 * total : 70000

Input Summary: * shape : [(28, 28, 1)] * range : [(0.0, 1.0)]

Target Summary: * shape : [(784,)] * range : [(0.0, 1.0)]

In [24]:

net.dashboard()

In [25]:

net.propagate_to_features("pool", net.dataset.inputs[0], cols=1)

Out[25]:

Feature 0

Feature 1

Feature 2

In [26]:

image = net.dataset.inputs[0]
output = net.propagate_to_image("output", image)
output.size

Out[26]:

(28, 28)

In [27]:

net.propagate_to("hidden3", image)

Out[27]:

[0.0,
24985632300376892,
0,
30162084102630615,
0,
0,
0,
0,
0,
1399173140525818,
0,
0,
0,
0,
0,
0,
0,
1823977679014206,
12886810302734375,
21955928206443787,
0,
0,
15523290634155273,
0,
3935135304927826]

In [28]:

net.dataset.slice(10)

In [29]:

net.train(accuracy=0.5, epochs=1000, report_rate=100, tolerance=.4, plot=True)

========================================================================
       |  Training |  Training
Epochs |     Error |  Accuracy
------ | --------- | ---------
#  819 |   0.00063 |   0.50000

In [30]:

for i in range(10):
    net.propagate(net.dataset.inputs[i], visualize=True)

In [31]:

net.test(show_inputs=False, show_outputs=False, show=True)

========================================================
Testing validation dataset with tolerance 0.4...
# | result
---------------------------------------
0 | X
1 | X
2 | correct
3 | correct
4 | X
5 | X
6 | correct
7 | X
8 | correct
9 | correct
Total count: 10
      correct: 5
      incorrect: 5
Total percentage correct: 0.5