3.3. Activation Functions

This notebook explores the activation functions that are available in conx.

First, we import a special function plot_f that is designed to plot functions. We also import all of the activation functions:

  • softmax
  • elu
  • selu
  • softplus
  • softsign
  • relu
  • tanh
  • sigmoid
  • hard_sigmoid
  • linear
In [1]:
from conx.activations import *
from conx import choice, plot_f
/usr/local/lib/python3.6/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using Theano backend.
conx, version 3.5.0

3.3.1. softmax

The softmax activation function is unique in that the value of the activation function depends on the other output activations. So, this function takes an entire tensor as input.

The output of the softmax are said to sum to zero, but often they aren’t exactly zero. These can be used as propabilities, especially using the conx choice function.

In [2]:
softmax([0.1, 0.1, 0.7, 0.0])
Out[2]:
[0.21155263483524323,
 0.21155263483524323,
 0.38547399640083313,
 0.19142073392868042]
In [3]:
sum(softmax([0.1, 0.1, 0.7, 0.0]))
Out[3]:
1.0
In [4]:
choice(p=softmax([0.1, 0.1, 0.7, 0.0]))
Out[4]:
1

Let’s see how softmax can be used to make probabilistic choices, and see if they match our expectations:

In [5]:
bin = [0] * 4
for x in range(100):
    pick = choice([0, 1, 2, 3], p=softmax([0.1, 0.1, 0.7, 0.0]))
    bin[pick] += 1
print("softmax:", softmax([0.1, 0.1, 0.7, 0.0]))
print("picks  :", [b/100 for b in bin])
softmax: [0.21155263483524323, 0.21155263483524323, 0.38547399640083313, 0.19142073392868042]
picks  : [0.21, 0.24, 0.36, 0.19]
In [6]:
f = lambda x: softmax([x, .5, .1, 1])
In [7]:
plot_f(f, frange=(-5, 5, .1), xlabel="input", ylabel="output", title="softmax()")
_images/ActivationFunctions_10_0.png

3.3.2. elu

In [10]:
elu(0.5)
Out[10]:
0.5
In [8]:
plot_f(elu, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="elu()")
_images/ActivationFunctions_13_0.png
In [12]:
elu(0.5, .3)
Out[12]:
0.5
In [9]:
plot_f(lambda x: elu(x, .3), frange=(-15, 15, 1), xlabel="input", ylabel="output", title="elu(alpha=.3)")
_images/ActivationFunctions_15_0.png

3.3.3. selu

In [14]:
selu(0.5)
Out[14]:
0.5253505110740662
In [16]:
plot_f(selu, frange=(-5, 5, 0.5), xlabel="input", ylabel="output", title="selu()")
_images/ActivationFunctions_18_0.png

3.3.4. softplus

In [14]:
softplus(0.5)
Out[14]:
0.9740769863128662
In [17]:
plot_f(softplus, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="softplus()")
_images/ActivationFunctions_21_0.png

3.3.5. softsign

In [16]:
softsign(0.5)
Out[16]:
0.3333333432674408
In [18]:
plot_f(softsign, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="softsign()")
_images/ActivationFunctions_24_0.png

3.3.6. relu

In [18]:
relu(0.5)
Out[18]:
0.5
In [19]:
plot_f(relu, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="relu()")
_images/ActivationFunctions_27_0.png

3.3.7. tanh

In [20]:
tanh(0.5)
Out[20]:
0.46211716532707214
In [20]:
plot_f(tanh, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="tanh()")
_images/ActivationFunctions_30_0.png

3.3.8. sigmoid

In [22]:
sigmoid(0.5)
Out[22]:
0.622459352016449
In [21]:
plot_f(sigmoid, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="sigmoid()")
_images/ActivationFunctions_33_0.png

3.3.9. hard_sigmoid

In [24]:
hard_sigmoid(0.5)
Out[24]:
0.6000000238418579
In [22]:
plot_f(hard_sigmoid, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="hard_sigmoid()")
_images/ActivationFunctions_36_0.png

3.3.10. linear

In [26]:
linear(0.5)
Out[26]:
0.5
In [23]:
plot_f(linear, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="linear()")
_images/ActivationFunctions_39_0.png

3.3.11. Comparison

In [2]:
from conx import frange, plot
In [7]:
functions = [hard_sigmoid, sigmoid, tanh, relu,
             softsign, softplus, elu, selu, linear]
float_range = frange(-2, 2, .1)
lines = {}
symbols = {}
for v in float_range:
    for f in functions:
        if f.__name__ not in lines:
            lines[f.__name__] = []
            symbols[f.__name__] = "-"
        lines[f.__name__].append(f(v))
In [12]:
plot(list(lines.items()), xs=float_range, symbols=symbols, height=8.0, width=12.0,
     title="Activation Functions", xlabel="input", ylabel="output")
_images/ActivationFunctions_43_0.png