# 3.4. Activation Functions¶

This notebook explores the activation functions that are available in conx.

First, we import a special function plot_f that is designed to plot functions. We also import all of the activation functions:

• softmax
• elu
• selu
• softplus
• softsign
• relu
• tanh
• sigmoid
• hard_sigmoid
• linear
In [1]:

from conx.activations import *
from conx import choice, plot_f

/usr/local/lib/python3.6/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Using Theano backend.
conx, version 3.5.0


## 3.4.1. softmax¶

The softmax activation function is unique in that the value of the activation function depends on the other output activations. So, this function takes an entire tensor as input.

The output of the softmax are said to sum to zero, but often they aren’t exactly zero. These can be used as propabilities, especially using the conx choice function.

In [2]:

softmax([0.1, 0.1, 0.7, 0.0])

Out[2]:

[0.21155263483524323,
0.21155263483524323,
0.38547399640083313,
0.19142073392868042]

In [3]:

sum(softmax([0.1, 0.1, 0.7, 0.0]))

Out[3]:

1.0

In [4]:

choice(p=softmax([0.1, 0.1, 0.7, 0.0]))

Out[4]:

1


Let’s see how softmax can be used to make probabilistic choices, and see if they match our expectations:

In [5]:

bin = [0] * 4
for x in range(100):
pick = choice([0, 1, 2, 3], p=softmax([0.1, 0.1, 0.7, 0.0]))
bin[pick] += 1
print("softmax:", softmax([0.1, 0.1, 0.7, 0.0]))
print("picks  :", [b/100 for b in bin])

softmax: [0.21155263483524323, 0.21155263483524323, 0.38547399640083313, 0.19142073392868042]
picks  : [0.21, 0.24, 0.36, 0.19]

In [6]:

f = lambda x: softmax([x, .5, .1, 1])

In [7]:

plot_f(f, frange=(-5, 5, .1), xlabel="input", ylabel="output", title="softmax()")


## 3.4.2. elu¶

In [10]:

elu(0.5)

Out[10]:

0.5

In [8]:

plot_f(elu, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="elu()")

In [12]:

elu(0.5, .3)

Out[12]:

0.5

In [9]:

plot_f(lambda x: elu(x, .3), frange=(-15, 15, 1), xlabel="input", ylabel="output", title="elu(alpha=.3)")


## 3.4.3. selu¶

In [14]:

selu(0.5)

Out[14]:

0.5253505110740662

In [16]:

plot_f(selu, frange=(-5, 5, 0.5), xlabel="input", ylabel="output", title="selu()")


## 3.4.4. softplus¶

In [14]:

softplus(0.5)

Out[14]:

0.9740769863128662

In [17]:

plot_f(softplus, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="softplus()")


## 3.4.5. softsign¶

In [16]:

softsign(0.5)

Out[16]:

0.3333333432674408

In [18]:

plot_f(softsign, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="softsign()")


## 3.4.6. relu¶

In [18]:

relu(0.5)

Out[18]:

0.5

In [19]:

plot_f(relu, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="relu()")


## 3.4.7. tanh¶

In [20]:

tanh(0.5)

Out[20]:

0.46211716532707214

In [20]:

plot_f(tanh, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="tanh()")


## 3.4.8. sigmoid¶

In [22]:

sigmoid(0.5)

Out[22]:

0.622459352016449

In [21]:

plot_f(sigmoid, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="sigmoid()")


## 3.4.9. hard_sigmoid¶

In [24]:

hard_sigmoid(0.5)

Out[24]:

0.6000000238418579

In [22]:

plot_f(hard_sigmoid, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="hard_sigmoid()")


## 3.4.10. linear¶

In [26]:

linear(0.5)

Out[26]:

0.5

In [23]:

plot_f(linear, frange=(-15, 15, 1), xlabel="input", ylabel="output", title="linear()")


# 3.5. Comparison¶

In [2]:

from conx import frange, plot

In [7]:

functions = [hard_sigmoid, sigmoid, tanh, relu,
softsign, softplus, elu, selu, linear]
float_range = frange(-2, 2, .1)
lines = {}
symbols = {}
for v in float_range:
for f in functions:
if f.__name__ not in lines:
lines[f.__name__] = []
symbols[f.__name__] = "-"
lines[f.__name__].append(f(v))

In [12]:

plot(list(lines.items()), xs=float_range, symbols=symbols, height=8.0, width=12.0,
title="Activation Functions", xlabel="input", ylabel="output")