Is sigmoid function only applicable after dense() layer?

0

Issue

I am making a network which is similar to SE-Net(https://github.com/titu1994/keras-squeeze-excite-network/blob/master/se.py)
using keras, but quite different with it.

Suppose that I want to make some layer sequence like :

import keras

Input = keras.model.Input((None,None,3))
x1 = keras.layers.Conv2d(filters = 32, kernel_size = (3,3))(Input)
x_gp = keras.layers.GlobalAveragePooling()(x1)
x2 = keras.layers.Conv2d(filters = 32, kernel_size = (1,1))(x_gp)
x3 = keras.layers.Conv2d(filters = 8, kernel_size = (1,1))(x2)
x2_ = keras.layers.Conv2d(filters = 32, kernel_size = (1,1))(x3)
x_se = keras.activation.sigmoid()(x2_)

I want to know that applying x_se like this is programmable. Please tell me if I am doing wrong.

Solution

you can for sure experiment sigmoid as an activation for cnn layers too but the reason why sigmoid is not used with cnn layers are:

1. Sigmoid function is monotonic but it’s derivative is not therefore there is a possibility that your training can be stuck

2. Sigmoid range:[0,1]

if you are experimenting sigmoid with cnn layers then I would suggest you to use it only for few layers.
You can give swish a try.

Answered By – keertika jain

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0

Leave A Reply

Your email address will not be published.

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More