Activation Functions

This notebook explores the activation functions that are available in conx.

First, we import a special function plot_f that is designed to plot functions. We also import all of the activation functions:

  • softmax
  • elu
  • selu
  • softplus
  • softsign
  • relu
  • tanh
  • sigmoid
  • hard_sigmoid
  • linear
In [26]:
from conx.graphs import plot_f
from conx.activations import *
from conx import choice

softmax

The softmax activation function is unique in that the value of the activation function depends on the other output activations. So, this function takes an entire tensor as input.

The output of the softmax are said to sum to zero, but often they aren’t exactly zero. These can be used as propabilities, especially using the conx choice function.

In [34]:
softmax([0.1, 0.1, 0.7, 0.0])
Out[34]:
[0.21155263483524323,
 0.21155263483524323,
 0.38547399640083313,
 0.19142073392868042]
In [35]:
sum(softmax([0.1, 0.1, 0.7, 0.0]))
Out[35]:
1.0
In [36]:
choice([0, 1, 2, 3], p=softmax([0.1, 0.1, 0.7, 0.0]))
Out[36]:
2

Let’s see how softmax can be used to make probabilistic choices, and see if they match our expectations:

In [37]:
bin = [0] * 4
for x in range(100):
    pick = choice([0, 1, 2, 3], p=softmax([0.1, 0.1, 0.7, 0.0]))
    bin[pick] += 1
print("softmax:", softmax([0.1, 0.1, 0.7, 0.0]))
print("picks  :", [b/100 for b in bin])
softmax: [0.21155263483524323, 0.21155263483524323, 0.38547399640083313, 0.19142073392868042]
picks  : [0.21, 0.27, 0.36, 0.16]
In [38]:
f = lambda x: softmax([x, .5, .1, 1])
In [39]:
plot_f(f, frange=(-5, 5, .1))
_images/ActivationFunctions_10_0.png

elu

In [40]:
elu(0.5)
Out[40]:
0.5
In [41]:
plot_f(elu, frange=(-15, 15, 1))
_images/ActivationFunctions_13_0.png
In [42]:
elu(0.5, .3)
Out[42]:
0.5
In [43]:
plot_f(lambda x: elu(x, .3), frange=(-15, 15, 1))
_images/ActivationFunctions_15_0.png

selu

In [44]:
selu(0.5)
Out[44]:
0.5253505110740662
In [45]:
plot_f(selu, frange=(-1000, 1000, 100))
_images/ActivationFunctions_18_0.png

softplus

In [46]:
softplus(0.5)
Out[46]:
0.9740769863128662
In [47]:
plot_f(softplus, frange=(-15, 15, 1))
_images/ActivationFunctions_21_0.png

softsign

In [48]:
softsign(0.5)
Out[48]:
0.3333333432674408
In [49]:
plot_f(softsign, frange=(-15, 15, 1))
_images/ActivationFunctions_24_0.png

relu

In [50]:
relu(0.5)
Out[50]:
0.5
In [51]:
plot_f(relu, frange=(-15, 15, 1))
_images/ActivationFunctions_27_0.png

tanh

In [52]:
tanh(0.5)
Out[52]:
0.46211716532707214
In [53]:
plot_f(tanh, frange=(-15, 15, 1))
_images/ActivationFunctions_30_0.png

sigmoid

In [54]:
sigmoid(0.5)
Out[54]:
0.622459352016449
In [55]:
plot_f(sigmoid, frange=(-15, 15, 1))
_images/ActivationFunctions_33_0.png

hard_sigmoid

In [56]:
hard_sigmoid(0.5)
Out[56]:
0.6000000238418579
In [57]:
plot_f(hard_sigmoid, frange=(-15, 15, 1))
_images/ActivationFunctions_36_0.png

linear

In [58]:
linear(0.5)
Out[58]:
0.5
In [59]:
plot_f(linear, frange=(-15, 15, 1))
_images/ActivationFunctions_39_0.png