3.4. Activation Functions¶
This notebook explores the activation functions that are available in conx.
First, we import a special function plot_f that is designed to plot
functions. We also import all of the activation functions:
- softmax
- elu
- selu
- softplus
- softsign
- relu
- tanh
- sigmoid
- hard_sigmoid
- linear
In [1]:
from conx.activations import *
from conx import choice, plot_f
Using Theano backend.
conx, version 3.4.3
3.4.1. softmax¶
The softmax activation function is unique in that the value of the
activation function depends on the other output activations. So, this
function takes an entire tensor as input.
The output of the softmax are said to sum to zero, but often they aren’t
exactly zero. These can be used as propabilities, especially using the
conx choice function.
In [2]:
softmax([0.1, 0.1, 0.7, 0.0])
Out[2]:
[0.21155263483524323,
0.21155263483524323,
0.38547399640083313,
0.19142073392868042]
In [3]:
sum(softmax([0.1, 0.1, 0.7, 0.0]))
Out[3]:
1.0
In [4]:
choice(p=softmax([0.1, 0.1, 0.7, 0.0]))
Out[4]:
1
Let’s see how softmax can be used to make probabilistic choices, and see if they match our expectations:
In [5]:
bin = [0] * 4
for x in range(100):
pick = choice([0, 1, 2, 3], p=softmax([0.1, 0.1, 0.7, 0.0]))
bin[pick] += 1
print("softmax:", softmax([0.1, 0.1, 0.7, 0.0]))
print("picks :", [b/100 for b in bin])
softmax: [0.21155263483524323, 0.21155263483524323, 0.38547399640083313, 0.19142073392868042]
picks : [0.21, 0.24, 0.36, 0.19]
In [6]:
f = lambda x: softmax([x, .5, .1, 1])
In [7]:
plot_f(f, frange=(-5, 5, .1), xlabel="input", ylabel="output", title="softmax()")
3.4.2. elu¶
In [10]:
elu(0.5)
Out[10]:
0.5
In [8]:
plot_f(elu, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="elu()")
In [12]:
elu(0.5, .3)
Out[12]:
0.5
In [9]:
plot_f(lambda x: elu(x, .3), frange=(-15, 15, 1), xlabel="input", ylabel="output", title="elu(alpha=.3)")
3.4.3. selu¶
In [14]:
selu(0.5)
Out[14]:
0.5253505110740662
In [16]:
plot_f(selu, frange=(-5, 5, 0.5), xlabel="input", ylabel="output", title="selu()")
3.4.4. softplus¶
In [14]:
softplus(0.5)
Out[14]:
0.9740769863128662
In [17]:
plot_f(softplus, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="softplus()")
3.4.5. softsign¶
In [16]:
softsign(0.5)
Out[16]:
0.3333333432674408
In [18]:
plot_f(softsign, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="softsign()")
3.4.6. relu¶
In [18]:
relu(0.5)
Out[18]:
0.5
In [19]:
plot_f(relu, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="relu()")
3.4.7. tanh¶
In [20]:
tanh(0.5)
Out[20]:
0.46211716532707214
In [20]:
plot_f(tanh, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="tanh()")
3.4.8. sigmoid¶
In [22]:
sigmoid(0.5)
Out[22]:
0.622459352016449
In [21]:
plot_f(sigmoid, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="sigmoid()")
3.4.9. hard_sigmoid¶
In [24]:
hard_sigmoid(0.5)
Out[24]:
0.6000000238418579
In [22]:
plot_f(hard_sigmoid, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="hard_sigmoid()")
3.4.10. linear¶
In [26]:
linear(0.5)
Out[26]:
0.5
In [23]:
plot_f(linear, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="linear()")
