3.26. VGG16 and ImageNet

ImageNet is an image classification and localization competition. VGG16 is a 16-layer network architecture and weights trained on the competition dataset by the Visual Geometry Group (VGG).

In this notebook we explore testing the network on samples images.

3.26.1. Constructing the Network

First, we import the ConX library:

[1]:
import conx as cx
Using TensorFlow backend.
ConX, version 3.6.8

We can load in a number of predefined networks, including “VGG16” and “VGG19”:

[2]:
net = cx.Network.get("vgg16")
WARNING: no such history file '/home/dblank/.keras/models/history.pickle'

We get a warning letting us know that this is a trained model, but no training history exists. That is because this model was trained outside of ConX.

Let’s see what this network looks like:

[3]:
net.picture(rotate=True)
[3]:
Layer: predictions (output) output range: (0, 1) shape = (1000,) Keras class = Dense trainable = True activation = softmax use_bias = TruepredictionsWeights from fc2 to predictions predictions/kernel:0 has shape (4096, 1000) predictions/bias:0 has shape (1000,)Layer: fc2 (hidden) output range: (0, +Infinity) shape = (4096,) Keras class = Dense trainable = True activation = relu use_bias = Truefc2Weights from fc1 to fc2 fc2/kernel:0 has shape (4096, 4096) fc2/bias:0 has shape (4096,)Layer: fc1 (hidden) output range: (0, +Infinity) shape = (4096,) Keras class = Dense trainable = True activation = relu use_bias = Truefc1Weights from flatten to fc1 fc1/kernel:0 has shape (25088, 4096) fc1/bias:0 has shape (4096,)Layer: flatten (hidden) output range: (-Infinity, +Infinity) Keras class = Flatten trainable = TrueflattenWeights from block5_pool to flattenLayer: block5_pool (hidden) output range: (-Infinity, +Infinity) Keras class = MaxPooling2D trainable = True pool_size = (2, 2) padding = valid strides = (2, 2) data_format = channels_lastblock5_pool5120Weights from block5_conv3 to block5_poolLayer: block5_conv3 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 512 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock5_conv35120Weights from block5_conv2 to block5_conv3 block5_conv3/kernel:0 has shape (3, 3, 512, 512) block5_conv3/bias:0 has shape (512,)Layer: block5_conv2 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 512 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock5_conv25120Weights from block5_conv1 to block5_conv2 block5_conv2/kernel:0 has shape (3, 3, 512, 512) block5_conv2/bias:0 has shape (512,)Layer: block5_conv1 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 512 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock5_conv15120Weights from block4_pool to block5_conv1 block5_conv1/kernel:0 has shape (3, 3, 512, 512) block5_conv1/bias:0 has shape (512,)Layer: block4_pool (hidden) output range: (-Infinity, +Infinity) Keras class = MaxPooling2D trainable = True pool_size = (2, 2) padding = valid strides = (2, 2) data_format = channels_lastblock4_pool5120Weights from block4_conv3 to block4_poolLayer: block4_conv3 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 512 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock4_conv35120Weights from block4_conv2 to block4_conv3 block4_conv3/kernel:0 has shape (3, 3, 512, 512) block4_conv3/bias:0 has shape (512,)Layer: block4_conv2 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 512 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock4_conv25120Weights from block4_conv1 to block4_conv2 block4_conv2/kernel:0 has shape (3, 3, 512, 512) block4_conv2/bias:0 has shape (512,)Layer: block4_conv1 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 512 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock4_conv15120Weights from block3_pool to block4_conv1 block4_conv1/kernel:0 has shape (3, 3, 256, 512) block4_conv1/bias:0 has shape (512,)Layer: block3_pool (hidden) output range: (-Infinity, +Infinity) Keras class = MaxPooling2D trainable = True pool_size = (2, 2) padding = valid strides = (2, 2) data_format = channels_lastblock3_pool2560Weights from block3_conv3 to block3_poolLayer: block3_conv3 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 256 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock3_conv32560Weights from block3_conv2 to block3_conv3 block3_conv3/kernel:0 has shape (3, 3, 256, 256) block3_conv3/bias:0 has shape (256,)Layer: block3_conv2 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 256 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock3_conv22560Weights from block3_conv1 to block3_conv2 block3_conv2/kernel:0 has shape (3, 3, 256, 256) block3_conv2/bias:0 has shape (256,)Layer: block3_conv1 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 256 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock3_conv12560Weights from block2_pool to block3_conv1 block3_conv1/kernel:0 has shape (3, 3, 128, 256) block3_conv1/bias:0 has shape (256,)Layer: block2_pool (hidden) output range: (-Infinity, +Infinity) Keras class = MaxPooling2D trainable = True pool_size = (2, 2) padding = valid strides = (2, 2) data_format = channels_lastblock2_pool1280Weights from block2_conv2 to block2_poolLayer: block2_conv2 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 128 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock2_conv21280Weights from block2_conv1 to block2_conv2 block2_conv2/kernel:0 has shape (3, 3, 128, 128) block2_conv2/bias:0 has shape (128,)Layer: block2_conv1 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 128 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock2_conv11280Weights from block1_pool to block2_conv1 block2_conv1/kernel:0 has shape (3, 3, 64, 128) block2_conv1/bias:0 has shape (128,)Layer: block1_pool (hidden) output range: (-Infinity, +Infinity) Keras class = MaxPooling2D trainable = True pool_size = (2, 2) padding = valid strides = (2, 2) data_format = channels_lastblock1_pool640Weights from block1_conv2 to block1_poolLayer: block1_conv2 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 64 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock1_conv2640Weights from block1_conv1 to block1_conv2 block1_conv2/kernel:0 has shape (3, 3, 64, 64) block1_conv2/bias:0 has shape (64,)Layer: block1_conv1 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 64 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock1_conv1640Weights from input_1 to block1_conv1 block1_conv1/kernel:0 has shape (3, 3, 3, 64) block1_conv1/bias:0 has shape (64,)Layer: input_1 (input) output range: (-Infinity, +Infinity) Keras class = Input batch_shape = (None, 224, 224, 3)input_130VGG16

Predefined networks have an info method describing the network:

[4]:
net.info()

Network: VGG16

  • Status: compiled
  • Layers: 23

This network architecture comes from the paper:

Very Deep Convolutional Networks for Large-Scale Image Recognition by Karen Simonyan and Andrew Zisserman.

Their network was trained on the ImageNet challenge dataset. The dataset contains 32,326 images broken down into 1,000 categories.

The network was trained for 74 epochs on the training data. This typically took 3 to 4 weeks time on a computer with 4 GPUs. This network’s weights were converted from the original Caffe model into Keras.

Sources: * https://arxiv.org/pdf/1409.1556.pdf * http://www.robots.ox.ac.uk/~vgg/research/very_deep/ * http://www.image-net.org/challenges/LSVRC/ * http://image-net.org/challenges/LSVRC/2014/ * http://image-net.org/challenges/LSVRC/2014/browse-synsets

We can see a complete summary of each layer:

[5]:
net.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         (None, 224, 224, 3)       0
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 224, 224, 64)      1792
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 224, 224, 64)      36928
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 112, 112, 64)      0
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 112, 112, 128)     73856
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 112, 112, 128)     147584
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 56, 56, 128)       0
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 56, 56, 256)       295168
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 56, 56, 256)       590080
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 56, 56, 256)       590080
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 28, 28, 256)       0
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 28, 28, 512)       1180160
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 28, 28, 512)       2359808
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 28, 28, 512)       2359808
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 14, 14, 512)       0
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 14, 14, 512)       2359808
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 14, 14, 512)       2359808
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 14, 14, 512)       2359808
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 7, 7, 512)         0
_________________________________________________________________
flatten (Flatten)            (None, 25088)             0
_________________________________________________________________
fc1 (Dense)                  (None, 4096)              102764544
_________________________________________________________________
fc2 (Dense)                  (None, 4096)              16781312
_________________________________________________________________
predictions (Dense)          (None, 1000)              4097000
=================================================================
Total params: 138,357,544
Trainable params: 138,357,544
Non-trainable params: 0
_________________________________________________________________

This network is huge! 138 million weights!

But why does this have 23 layers when it is described as having 16? The original description only counted layers that have trainable weights. So, that doesn’t include the input layer (1) nor the pooling layers (5) or the flatten layer (1). So:

[6]:
23 - 1 - 5 - 1
[6]:
16

3.26.2. Testing

Let’s see what the network can do. Let’s grab an image and propagate it through the network (I’ll show you where this image came from in just a moment).

We download the picture:

[7]:
cx.download("http://farm4.static.flickr.com/3426/3817878004_7dec9cdfbd.jpg", filename="geyser-1.jpg")
Using cached http://farm4.static.flickr.com/3426/3817878004_7dec9cdfbd.jpg as './geyser-1.jpg'.

In order to propagate it through the network, it needs to be resized to 224 x 224, so we can do that as we open it:

[8]:
img = cx.image("geyser-1.jpg", resize=(224, 224))
img
[8]:
_images/VGG16_and_ImageNet_18_0.png

To propagate it through the network it should be a 3D matrix:

[29]:
array = cx.image_to_array(img)

This network comes with a special preprocess() method for getting the image values into the proper range:

[30]:
array2 = net.preprocess(array)

And now, we are ready to propagate the array through the network:

[31]:
output = net.propagate(array2)

What is this output?

[32]:
len(output)
[32]:
1000

We can pretty-format the output to see condensed view of the values:

[33]:
print(net.pf(output))
[0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.01,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.01,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.01,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.03,0.00,0.09,0.00,0.76, 0.00,0.00,0.00,0.00,0.03,0.01,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00, 0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00]

This network comes with a special method postprocess() that will take this values and look them up, giving us the category ID, and a nice human-readable label:

[34]:
net.postprocess(output)
[34]:
[('n09288635', 'geyser', 0.7626754641532898),
 ('n09246464', 'cliff', 0.08688867092132568),
 ('n09193705', 'alp', 0.03189372643828392),
 ('n09468604', 'valley', 0.027246791869401932),
 ('n03160309', 'dam', 0.01493859849870205)]

Indeed, this is a picture of a geyser!

We can see a picture of the entire network, with activations:

[35]:
net.picture(array2, scale=0.25)
[35]:
Layer: predictions (output) output range: (0, 1) shape = (1000,) Keras class = Dense trainable = True activation = softmax use_bias = TruepredictionsWeights from fc2 to predictions predictions/kernel:0 has shape (4096, 1000) predictions/bias:0 has shape (1000,)Layer: fc2 (hidden) output range: (0, +Infinity) shape = (4096,) Keras class = Dense trainable = True activation = relu use_bias = Truefc2Weights from fc1 to fc2 fc2/kernel:0 has shape (4096, 4096) fc2/bias:0 has shape (4096,)Layer: fc1 (hidden) output range: (0, +Infinity) shape = (4096,) Keras class = Dense trainable = True activation = relu use_bias = Truefc1Weights from flatten to fc1 fc1/kernel:0 has shape (25088, 4096) fc1/bias:0 has shape (4096,)Layer: flatten (hidden) output range: (-Infinity, +Infinity) Keras class = Flatten trainable = TrueflattenWeights from block5_pool to flattenLayer: block5_pool (hidden) output range: (-Infinity, +Infinity) Keras class = MaxPooling2D trainable = True pool_size = (2, 2) padding = valid strides = (2, 2) data_format = channels_lastblock5_pool5120Weights from block5_conv3 to block5_poolLayer: block5_conv3 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 512 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock5_conv35120Weights from block5_conv2 to block5_conv3 block5_conv3/kernel:0 has shape (3, 3, 512, 512) block5_conv3/bias:0 has shape (512,)Layer: block5_conv2 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 512 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock5_conv25120Weights from block5_conv1 to block5_conv2 block5_conv2/kernel:0 has shape (3, 3, 512, 512) block5_conv2/bias:0 has shape (512,)Layer: block5_conv1 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 512 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock5_conv15120Weights from block4_pool to block5_conv1 block5_conv1/kernel:0 has shape (3, 3, 512, 512) block5_conv1/bias:0 has shape (512,)Layer: block4_pool (hidden) output range: (-Infinity, +Infinity) Keras class = MaxPooling2D trainable = True pool_size = (2, 2) padding = valid strides = (2, 2) data_format = channels_lastblock4_pool5120Weights from block4_conv3 to block4_poolLayer: block4_conv3 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 512 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock4_conv35120Weights from block4_conv2 to block4_conv3 block4_conv3/kernel:0 has shape (3, 3, 512, 512) block4_conv3/bias:0 has shape (512,)Layer: block4_conv2 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 512 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock4_conv25120Weights from block4_conv1 to block4_conv2 block4_conv2/kernel:0 has shape (3, 3, 512, 512) block4_conv2/bias:0 has shape (512,)Layer: block4_conv1 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 512 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock4_conv15120Weights from block3_pool to block4_conv1 block4_conv1/kernel:0 has shape (3, 3, 256, 512) block4_conv1/bias:0 has shape (512,)Layer: block3_pool (hidden) output range: (-Infinity, +Infinity) Keras class = MaxPooling2D trainable = True pool_size = (2, 2) padding = valid strides = (2, 2) data_format = channels_lastblock3_pool2560Weights from block3_conv3 to block3_poolLayer: block3_conv3 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 256 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock3_conv32560Weights from block3_conv2 to block3_conv3 block3_conv3/kernel:0 has shape (3, 3, 256, 256) block3_conv3/bias:0 has shape (256,)Layer: block3_conv2 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 256 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock3_conv22560Weights from block3_conv1 to block3_conv2 block3_conv2/kernel:0 has shape (3, 3, 256, 256) block3_conv2/bias:0 has shape (256,)Layer: block3_conv1 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 256 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock3_conv12560Weights from block2_pool to block3_conv1 block3_conv1/kernel:0 has shape (3, 3, 128, 256) block3_conv1/bias:0 has shape (256,)Layer: block2_pool (hidden) output range: (-Infinity, +Infinity) Keras class = MaxPooling2D trainable = True pool_size = (2, 2) padding = valid strides = (2, 2) data_format = channels_lastblock2_pool1280Weights from block2_conv2 to block2_poolLayer: block2_conv2 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 128 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock2_conv21280Weights from block2_conv1 to block2_conv2 block2_conv2/kernel:0 has shape (3, 3, 128, 128) block2_conv2/bias:0 has shape (128,)Layer: block2_conv1 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 128 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock2_conv11280Weights from block1_pool to block2_conv1 block2_conv1/kernel:0 has shape (3, 3, 64, 128) block2_conv1/bias:0 has shape (128,)Layer: block1_pool (hidden) output range: (-Infinity, +Infinity) Keras class = MaxPooling2D trainable = True pool_size = (2, 2) padding = valid strides = (2, 2) data_format = channels_lastblock1_pool640Weights from block1_conv2 to block1_poolLayer: block1_conv2 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 64 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock1_conv2640Weights from block1_conv1 to block1_conv2 block1_conv2/kernel:0 has shape (3, 3, 64, 64) block1_conv2/bias:0 has shape (64,)Layer: block1_conv1 (hidden) output range: (0, +Infinity) Keras class = Conv2D trainable = True filters = 64 kernel_size = (3, 3) strides = (1, 1) padding = same data_format = channels_last dilation_rate = (1, 1) activation = relu use_bias = Trueblock1_conv1640Weights from input_1 to block1_conv1 block1_conv1/kernel:0 has shape (3, 3, 3, 64) block1_conv1/bias:0 has shape (64,)Layer: input_1 (input) output range: (-Infinity, +Infinity) Keras class = Input batch_shape = (None, 224, 224, 3)input_130VGG16

Why is the first layer gray? Because it is a regular input layer, with 3 channels, and it is showing the first channel.

How does the network represent a picture? It re-represents the picture at every layer. Many layers are composed of features. For example, we can look at the first layer, input_1:

[36]:
net.propagate_to_features("input_1", array2)
[36]:

Feature 0

Feature 1

Feature 2

In this example, a feature is a color channel. So feature 0 is Red, feature 1 is Green, and feature 2 is Blue.

We can view all channels together by propagating to the input layer, and then viewing the output as an image:

[37]:
features = net.propagate_to("input_1", array2) ## this should be identical to array2
cx.array_to_image(features)
[37]:
_images/VGG16_and_ImageNet_38_0.png

Notice that it looks different from the original:

[38]:
img
[38]:
_images/VGG16_and_ImageNet_40_0.png

Why the difference? Remember that the image was preprocessed.

We can also look at the first Convolutional layer, block1_conv1:

[58]:
net.propagate_to_features("block1_conv1", array2, scale=0.5, cols=10)
[58]:

Feature 0

Feature 1

Feature 2

Feature 3

Feature 4

Feature 5

Feature 6

Feature 7

Feature 8

Feature 9

Feature 10

Feature 11

Feature 12

Feature 13

Feature 14

Feature 15

Feature 16

Feature 17

Feature 18

Feature 19

Feature 20

Feature 21

Feature 22

Feature 23

Feature 24

Feature 25

Feature 26

Feature 27

Feature 28

Feature 29

Feature 30

Feature 31

Feature 32

Feature 33

Feature 34

Feature 35

Feature 36

Feature 37

Feature 38

Feature 39

Feature 40

Feature 41

Feature 42