ncdl.nn.SELU

class ncdl.nn.SELU(*args: Any, **kwargs: Any)

Applied element-wise, as:

\[\text{SELU}(x) = \text{scale} * (\max(0,x) + \min(0, \alpha * (\exp(x) - 1)))\]

with \(\alpha = 1.6732632423543772848170429916717\) and \(\text{scale} = 1.0507009873554804934193349852946\).

Warning

When using kaiming_normal or kaiming_normal_ for initialisation, nonlinearity='linear' should be used instead of nonlinearity='selu' in order to get Self-Normalizing Neural Networks. See torch.nn.init.calculate_gain() for more information.

More details can be found in the paper Self-Normalizing Neural Networks .

Parameters:

inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:
  • Input: \((*)\), where \(*\) means any number of dimensions.

  • Output: \((*)\), same shape as the input.

scripts/activation_images/SELU.png

Examples:

>>> m = nn.SELU()
>>> input = torch.randn(2)
>>> output = m(input)
__init__(inplace: bool = False) None

Methods

__init__([inplace])

extra_repr()

forward(input)

Attributes

inplace