something very simple (and which is very good news):
is a linear function of the changes and in the weights
and bias. This linearity makes it easy to choose small changes
in the weights and biases to achieve any desired small change
in the output. So while sigmoid neurons have much of the same
qualitative behaviour as perceptrons, they make it much easier
to figure out how changing the weights and biases will change
the output.
If it's the shape of which really matters, and not its exact
form, then why use the particular form used for in Equation
(3)? In fact, later in the book we will occasionally consider
neurons where the output is for some other
activation function . The main thing that changes when we
use a different activation function is that the particular values
for the partial derivatives in Equation (5) change. It turns out
that when we compute those partial derivatives later, using
will simplify the algebra, simply because exponentials have
lovely properties when differentiated. In any case, is
commonly-used in work on neural nets, and is the activation
function we'll use most often in this book.
How should we interpret the output from a sigmoid neuron?
Obviously, one big difference between perceptrons and sigmoid
neurons is that sigmoid neurons don't just output or . They
can have as output any real number between and , so values
such as and are legitimate outputs. This can
be useful, for example, if we want to use the output value to
represent the average intensity of the pixels in an image input
to a neural network. But sometimes it can be a nuisance.
Suppose we want the output from the network to indicate
either "the input image is a 9" or "the input image is not a 9".
Obviously, it'd be easiest to do this if the output was a or a ,
as in a perceptron. But in practice we can set up a convention
to deal with this, for example, by deciding to interpret any
output of at least as indicating a "9", and any output less
than as indicating "not a 9". I'll always explicitly state when
we're using such a convention, so it shouldn't cause any
confusion.
Exercises
Sigmoid neurons simulating perceptrons, part I