3.6. Autoencoding

3.6.1. One-hot Encoder

In [1]:
from conx import *
Using Theano backend.
conx, version 3.5.9
In [2]:
size = 5
In [3]:
net = Network("Autoencoder")
net.add(Layer("input", size, minmax=(0,1)))
net.add(Layer("hidden", 5, activation="relu", visible=True))
net.add(Layer("output", size, activation="sigmoid"))
net.config["font_family"] = "monospace"
In [4]:
net.connect()
In [5]:
net.compile(error='binary_crossentropy', optimizer="adam")
In [6]:
net
Out[6]:
AutoencoderLayer: output (output) shape = (5,) Keras class = Dense activation = sigmoidoutputWeights from hidden to output output/kernel has shape (5, 5) output/bias has shape (5,)Layer: hidden (hidden) shape = (5,) Keras class = Dense activation = reluhiddenWeights from input to hidden hidden/kernel has shape (5, 5) hidden/bias has shape (5,)Layer: input (input) shape = (5,) Keras class = Inputinput
In [7]:
patterns = [onehot(i, size) for i in range(size)]
In [8]:
patterns[0]
Out[8]:
[1, 0, 0, 0, 0]
In [9]:
net.dataset.load([(p,p) for p in patterns])
In [10]:
net
Out[10]:
AutoencoderLayer: output (output) shape = (5,) Keras class = Dense activation = sigmoidoutputWeights from hidden to output output/kernel has shape (5, 5) output/bias has shape (5,)Layer: hidden (hidden) shape = (5,) Keras class = Dense activation = reluhiddenWeights from input to hidden hidden/kernel has shape (5, 5) hidden/bias has shape (5,)Layer: input (input) shape = (5,) Keras class = Inputinput
In [11]:
import time
for i in range(size):
    net.propagate(net.dataset.inputs[i], visualize=True)
    time.sleep(1)
In [12]:
net.dataset.summary()

Dataset Split: * training : 5 * testing : 0 * total : 5

Input Summary: * shape : [(5,)] * range : [(0.0, 1.0)]

Target Summary: * shape : [(5,)] * range : [(0.0, 1.0)]

In [13]:
net.reset()
net.train(accuracy=1, epochs=10000, report_rate=200, tolerance=0.4, plot=True)
_images/Autoencoder_14_0.svg
========================================================================
       |  Training |  Training
Epochs |     Error |  Accuracy
------ | --------- | ---------
# 1001 |   0.10735 |   1.00000
In [14]:
net.propagate(net.dataset.inputs[0])
Out[14]:
[0.8117181062698364,
 0.013156135566532612,
 0.028108935803174973,
 0.06900336593389511,
 0.05056529864668846]
In [16]:
net.test(tolerance=0.4, show=True)
========================================================
Testing validation dataset with tolerance 0.4...
# | inputs | targets | outputs | result
---------------------------------------
0 | [[1.00,0.00,0.00,0.00,0.00]] | [[1.00,0.00,0.00,0.00,0.00]] | [0.81,0.01,0.03,0.07,0.05] | correct
1 | [[0.00,1.00,0.00,0.00,0.00]] | [[0.00,1.00,0.00,0.00,0.00]] | [0.01,0.60,0.10,0.13,0.05] | correct
2 | [[0.00,0.00,1.00,0.00,0.00]] | [[0.00,0.00,1.00,0.00,0.00]] | [0.12,0.27,0.73,0.03,0.05] | correct
3 | [[0.00,0.00,0.00,1.00,0.00]] | [[0.00,0.00,0.00,1.00,0.00]] | [0.07,0.06,0.00,0.88,0.06] | correct
4 | [[0.00,0.00,0.00,0.00,1.00]] | [[0.00,0.00,0.00,0.00,1.00]] | [0.05,0.05,0.01,0.07,0.85] | correct
Total count: 5
      correct: 5
      incorrect: 0
Total percentage correct: 1.0
In [17]:
for i in range(size):
    net.propagate(net.dataset.inputs[i], visualize=True)
    time.sleep(1)
In [18]:
net.dashboard()

3.6.2. MNIST Autoencoding

In [19]:
from conx import *
In [20]:
net = Network("MNIST-Autoencoder")
In [21]:
net.add(ImageLayer("input", (28,28), 1))
net.add(Conv2DLayer("conv", 3, (5,5), activation="relu"))
net.add(MaxPool2DLayer("pool", pool_size=(2,2)))
net.add(FlattenLayer("flatten"))
net.add(Layer("hidden3", 25, activation="relu"))
net.add(Layer("output", (28,28,1), activation="sigmoid"))
In [22]:
net.connect()
net.compile(error="mse", optimizer="adam")
net
Out[22]:
MNIST-AutoencoderLayer: output (output) shape = (28, 28, 1) Keras class = Dense activation = sigmoidoutputWeights from hidden3 to output output/kernel has shape (25, 784) output/bias has shape (784,)Layer: hidden3 (hidden) shape = (25,) Keras class = Dense activation = reluhidden3Weights from flatten to hidden3 hidden3/kernel has shape (432, 25) hidden3/bias has shape (25,)Layer: flatten (hidden) Keras class = FlattenflattenWeights from pool to flattenLayer: pool (hidden) Keras class = MaxPooling2D pool_size = (2, 2)pool30Weights from conv to poolLayer: conv (hidden) Keras class = Conv2D activation = reluconv30Weights from input to conv conv/kernel has shape (5, 5, 1, 3) conv/bias has shape (3,)Layer: input (input) shape = (28, 28, 1) Keras class = Inputinput
In [23]:
net.dataset.get('mnist')
net.dataset.set_targets_from_inputs()
net.dataset.targets.reshape(0, (28 * 28))
net.dataset.summary()
WARNING: network 'MNIST-Autoencoder' target bank #0 has a multi-dimensional shape, which is not allowed

Dataset name: MNIST

Original source: http://yann.lecun.com/exdb/mnist/

The MNIST database of handwritten digits, available from this page, has 70,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image. It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting.

Dataset Split: * training : 70000 * testing : 0 * total : 70000

Input Summary: * shape : [(28, 28, 1)] * range : [(0.0, 1.0)]

Target Summary: * shape : [(784,)] * range : [(0.0, 1.0)]

In [24]:
net.dashboard()
In [26]:
net.propagate_to_features("pool", net.dataset.inputs[0], cols=1)
Out[26]:

Feature 0

Feature 1

Feature 2
In [27]:
image = net.dataset.inputs[0]
output = net.propagate_to_image("output", image)
output.size
Out[27]:
(28, 28)
In [28]:
net.propagate_to("hidden3", image)
Out[28]:
[0.5323348045349121,
 0.0,
 0.26464763283729553,
 0.5374033451080322,
 0.0,
 0.3257647454738617,
 0.0,
 0.09645804762840271,
 0.0,
 0.0,
 0.0,
 0.0,
 0.0,
 0.19905291497707367,
 0.0,
 0.0,
 0.8087217211723328,
 0.0,
 0.0,
 0.44031110405921936,
 0.8584030270576477,
 0.0,
 0.2285786122083664,
 0.506879448890686,
 1.0062655210494995]
In [29]:
net.dataset.slice(10)
In [30]:
net.train(accuracy=0.5, epochs=1000, report_rate=100, tolerance=.4, plot=True)
_images/Autoencoder_30_0.svg
========================================================================
       |  Training |  Training
Epochs |     Error |  Accuracy
------ | --------- | ---------
#  903 |   0.00110 |   0.50000
In [31]:
for i in range(10):
    net.propagate(net.dataset.inputs[i], visualize=True)
In [32]:
net.test(show_inputs=False, show_outputs=False, show=True)
========================================================
Testing validation dataset with tolerance 0.4...
# | result
---------------------------------------
0 | X
1 | X
2 | correct
3 | correct
4 | X
5 | X
6 | correct
7 | X
8 | correct
9 | correct
Total count: 10
      correct: 5
      incorrect: 5
Total percentage correct: 0.5