Activation Function

Artificial Intelligence (AI)

Lesson 14 Activation Function

Some types of activation functions are proposed. Now we are going to see some of them.

Step Function outputs 1 for 0 or more as input, and 0 for less than 0. The graph is as follows.

The formula of Sigmoid Function is as follows.

f (u) = \frac{1}{1 + e^{- u}}

The graph is as follows.

The formula of Tanh Function is as follows.

f (u) = \frac{e^{u} - e^{- u}}{e^{u} + e^{- u}}

The graph is as follows.

The formula of Rectified Linear Unit is as follows.

f (u) = m a x (0, u)

The graph is as follows.

The final type is Softmax Function. When you want to detect whether dog or cat, it’s valid to get output values with the total value of 1. Therefore, we get the probability values from the following function.

f (u_{c}) = \frac{e^{u_{c}}}{\sum_{k = 1}^{n} e^{u_{k}}}

The formula itself must be difficult to be understood, and the following example should help you. Suppose there are four outputs, and the values are (0.9, 0.1, 0.2, 0.4). Where, the first output value is

u_{1} = 0.9, u_{2} = 0.1, u_{3} = 0.2, u_{4} = 0.4

f (u_{1}) = \frac{e^{u_{1}}}{e^{u_{1}} + e^{u_{2}} + e^{u_{3}} + e^{u_{4}}} \approx 0.39

The other activation functions are proposed. The total time for learning can be shorter by selecting appropriate function for each question.

To Next Article

To Contents Page

Innovation by DX

IT for Society

Lesson 14 Activation Function