BatchNormalization¶

BatchNormalization - 15¶

Version¶

name: BatchNormalization (GitHub)
domain: main
since_version: 15
function: False
support_level: SupportType.COMMON
shape inference: True

This version of the operator has been available since version 15.

Summary¶

Carries out batch normalization as described in the paper https://arxiv.org/abs/1502.03167. Depending on the mode it is being run, There are five required inputs ‘X’, ‘scale’, ‘B’, ‘input_mean’ and ‘input_var’. Note that ‘input_mean’ and ‘input_var’ are expected to be the estimated statistics in inference mode (training_mode=False, default), and the running statistics in training mode (training_mode=True). There are multiple cases for the number of outputs, which we list below:

Output case #1: Y, running_mean, running_var (training_mode=True)
Output case #2: Y (training_mode=False)

When training_mode=False, extra outputs are invalid. The outputs are updated as follows when training_mode=True:

running_mean = input_mean * momentum + current_mean * (1 - momentum)
running_var = input_var * momentum + current_var * (1 - momentum)

Y = (X - current_mean) / sqrt(current_var + epsilon) * scale + B

where:

current_mean = ReduceMean(X, axis=all_except_channel_index)
current_var =  ReduceVar(X, axis=all_except_channel_index)

Notice that ReduceVar refers to the population variance, and it equals to sum(sqrd(x_i - x_avg)) / N where N is the population size (this formula does not use sample size N - 1).

The computation of ReduceMean and ReduceVar uses float to avoid overflow for float16 inputs.

When training_mode=False:

Y = (X - input_mean) / sqrt(input_var + epsilon) * scale + B

For previous (depreciated) non-spatial cases, implementors are suggested to flatten the input shape to (N x C * D1 * D2 * … * Dn) before a BatchNormalization Op. This operator has optional inputs/outputs. See ONNX IR for more details about the representation of optional arguments. An empty string may be used in the place of an actual argument’s name to indicate a missing argument. Trailing optional arguments (those not followed by an argument that is present) may also be simply omitted.

Attributes¶

epsilon - FLOAT (default is '1e-05'):

The epsilon value to use to avoid division by zero.
momentum - FLOAT (default is '0.9'):

Factor used in computing the running mean and variance.e.g., running_mean = running_mean * momentum + mean * (1 - momentum).
training_mode - INT (default is '0'):

If set to true, it indicates BatchNormalization is being used for training, and outputs 1 and 2 are to be computed.

Inputs¶

X (heterogeneous) - T:

Input data tensor from the previous operator; dimensions are in the form of (N x C x D1 x D2 … Dn), where N is the batch size, C is the number of channels. Statistics are computed for every channel of C over N and D1 to Dn dimensions. For image data, input dimensions become (N x C x H x W). The op also accepts single dimension input of size N in which case C is assumed to be 1
scale (heterogeneous) - T1:

Scale tensor of shape ©.
B (heterogeneous) - T1:

Bias tensor of shape ©.
input_mean (heterogeneous) - T2:

running (training) or estimated (testing) mean tensor of shape ©.
input_var (heterogeneous) - T2:

running (training) or estimated (testing) variance tensor of shape ©.

Outputs¶

Between 1 and 3 outputs.

Y (heterogeneous) - T:

The output tensor of the same shape as X
running_mean (optional, heterogeneous) - T2:

The running mean after the BatchNormalization operator.
running_var (optional, heterogeneous) - T2:

The running variance after the BatchNormalization operator. This op uses the population size (N) for calculating variance, and not the sample size N-1.

Type Constraints¶

T in ( tensor(bfloat16), tensor(double), tensor(float), tensor(float16) ):

Constrain input and output types to float tensors.
T1 in ( tensor(bfloat16), tensor(double), tensor(float), tensor(float16) ):

Constrain scale and bias types to float tensors.
T2 in ( tensor(bfloat16), tensor(double), tensor(float), tensor(float16) ):

Constrain mean and variance types to float tensors.

BatchNormalization - 14 vs 15

BatchNormalization - 14¶

Version¶

name: BatchNormalization (GitHub)
domain: main
since_version: 14
function: False
support_level: SupportType.COMMON
shape inference: True

This version of the operator has been available since version 14.

Summary¶

Output case #1: Y, running_mean, running_var (training_mode=True) Output case #2: Y (training_mode=False)

When training_mode=False, extra outputs are invalid. The outputs are updated as follows when training_mode=True:

running_mean = input_mean * momentum + current_mean * (1 - momentum)
running_var = input_var * momentum + current_var * (1 - momentum)

Y = (X - current_mean) / sqrt(current_var + epsilon) * scale + B

where:

current_mean = ReduceMean(X, axis=all_except_channel_index)
current_var =  ReduceVar(X, axis=all_except_channel_index)

Notice that ReduceVar refers to the population variance, and it equals to
sum(sqrd(x_i - x_avg)) / N
where N is the population size (this formula does not use sample size N - 1).

When training_mode=False:

Y = (X - input_mean) / sqrt(input_var + epsilon) * scale + B

Attributes¶

epsilon - FLOAT (default is '1e-05'):

The epsilon value to use to avoid division by zero.
momentum - FLOAT (default is '0.9'):

Factor used in computing the running mean and variance.e.g., running_mean = running_mean * momentum + mean * (1 - momentum).
training_mode - INT (default is '0'):

If set to true, it indicates BatchNormalization is being used for training, and outputs 1, 2, 3, and 4 would be populated.

Inputs¶

X (heterogeneous) - T:

Input data tensor from the previous operator; dimensions are in the form of (N x C x D1 x D2 … Dn), where N is the batch size, C is the number of channels. Statistics are computed for every channel of C over N and D1 to Dn dimensions. For image data, input dimensions become (N x C x H x W). The op also accepts single dimension input of size N in which case C is assumed to be 1
scale (heterogeneous) - T:

Scale tensor of shape ©.
B (heterogeneous) - T:

Bias tensor of shape ©.
input_mean (heterogeneous) - U:

running (training) or estimated (testing) mean tensor of shape ©.
input_var (heterogeneous) - U:

running (training) or estimated (testing) variance tensor of shape ©.

Outputs¶

Between 1 and 3 outputs.

Y (heterogeneous) - T:

The output tensor of the same shape as X
running_mean (optional, heterogeneous) - U:

The running mean after the BatchNormalization operator.
running_var (optional, heterogeneous) - U:

The running variance after the BatchNormalization operator. This op uses the population size (N) for calculating variance, and not the sample size N-1.

Type Constraints¶

T in ( tensor(bfloat16), tensor(double), tensor(float), tensor(float16) ):

Constrain input and output types to float tensors.
U in ( tensor(bfloat16), tensor(double), tensor(float), tensor(float16) ):

Constrain mean and variance types to float tensors. It allows all float type for U.

BatchNormalization - 9¶

Version¶

name: BatchNormalization (GitHub)
domain: main
since_version: 9
function: False
support_level: SupportType.COMMON
shape inference: True

This version of the operator has been available since version 9.

Summary¶

Carries out batch normalization as described in the paper https://arxiv.org/abs/1502.03167. Depending on the mode it is being run, there are multiple cases for the number of outputs, which we list below:

Output case #1: Y, mean, var, saved_mean, saved_var (training mode) Output case #2: Y (test mode)

For previous (depreciated) non-spatial cases, implementors are suggested to flatten the input shape to (N x CD1D2 …*Dn) before a BatchNormalization Op. This operator has optional inputs/outputs. See ONNX IR for more details about the representation of optional arguments. An empty string may be used in the place of an actual argument’s name to indicate a missing argument. Trailing optional arguments (those not followed by an argument that is present) may also be simply omitted.

Attributes¶

epsilon - FLOAT (default is '1e-05'):

The epsilon value to use to avoid division by zero.
momentum - FLOAT (default is '0.9'):

Factor used in computing the running mean and variance.e.g., running_mean = running_mean * momentum + mean * (1 - momentum).

Inputs¶

X (heterogeneous) - T:

Input data tensor from the previous operator; dimensions are in the form of (N x C x D1 x D2 … Dn), where N is the batch size, C is the number of channels. Statistics are computed for every channel of C over N and D1 to Dn dimensions. For image data, input dimensions become (N x C x H x W). The op also accepts single dimension input of size N in which case C is assumed to be 1
scale (heterogeneous) - T:

Scale tensor of shape ©.
B (heterogeneous) - T:

Bias tensor of shape ©.
mean (heterogeneous) - T:

running (training) or estimated (testing) mean tensor of shape ©.
var (heterogeneous) - T:

running (training) or estimated (testing) variance tensor of shape ©.

Outputs¶

Between 1 and 5 outputs.

Y (heterogeneous) - T:

The output tensor of the same shape as X
mean (optional, heterogeneous) - T:

The running mean after the BatchNormalization operator.
var (optional, heterogeneous) - T:

The running variance after the BatchNormalization operator.
saved_mean (optional, heterogeneous) - T:

Saved mean used during training to speed up gradient computation.
saved_var (optional, heterogeneous) - T:

Saved variance used during training to speed up gradient computation.

Type Constraints¶

T in ( tensor(double), tensor(float), tensor(float16) ):

Constrain input and output types to float tensors.

BatchNormalization - 7¶

Version¶

name: BatchNormalization (GitHub)
domain: main
since_version: 7
function: False
support_level: SupportType.COMMON
shape inference: True

This version of the operator has been available since version 7.

Summary¶

Output case #1: Y, mean, var, saved_mean, saved_var (training mode) Output case #2: Y (test mode) This operator has optional inputs/outputs. See ONNX IR for more details about the representation of optional arguments. An empty string may be used in the place of an actual argument’s name to indicate a missing argument. Trailing optional arguments (those not followed by an argument that is present) may also be simply omitted.

Attributes¶

epsilon - FLOAT (default is '1e-05'):

The epsilon value to use to avoid division by zero.
momentum - FLOAT (default is '0.9'):

Factor used in computing the running mean and variance.e.g., running_mean = running_mean * momentum + mean * (1 - momentum).
spatial - INT (default is '1'):

If true, compute the mean and variance across per activation. If false, compute the mean and variance across per feature over each mini-batch.

Inputs¶

X (heterogeneous) - T:

Input data tensor from the previous operator; dimensions for image case are (N x C x H x W), where N is the batch size, C is the number of channels, and H and W are the height and the width of the data. For non image case, the dimensions are in the form of (N x C x D1 x D2 … Dn), where N is the batch size.
scale (heterogeneous) - T:

If spatial is true, the dimension of scale is ©. If spatial is false, the dimensions of scale are (C x D1 x … x Dn)
B (heterogeneous) - T:

If spatial is true, the dimension of bias is ©. If spatial is false, the dimensions of bias are (C x D1 x … x Dn)
mean (heterogeneous) - T:

If spatial is true, the dimension of the running mean (training) or the estimated mean (testing) is ©. If spatial is false, the dimensions of the running mean (training) or the estimated mean (testing) are (C x D1 x … x Dn).
var (heterogeneous) - T:

If spatial is true, the dimension of the running variance(training) or the estimated variance (testing) is ©. If spatial is false, the dimensions of the running variance(training) or the estimated variance (testing) are (C x D1 x … x Dn).

Outputs¶

Between 1 and 5 outputs.

Y (heterogeneous) - T:

The output tensor of the same shape as X
mean (optional, heterogeneous) - T:

The running mean after the BatchNormalization operator.
var (optional, heterogeneous) - T:

The running variance after the BatchNormalization operator.
saved_mean (optional, heterogeneous) - T:

Saved mean used during training to speed up gradient computation.
saved_var (optional, heterogeneous) - T:

Saved variance used during training to speed up gradient computation.

Type Constraints¶

T in ( tensor(double), tensor(float), tensor(float16) ):

Constrain input and output types to float tensors.

BatchNormalization - 6¶

Version¶

name: BatchNormalization (GitHub)
domain: main
since_version: 6
function: False
support_level: SupportType.COMMON
shape inference: True

This version of the operator has been available since version 6.

Summary¶

Output case #1: Y, mean, var, saved_mean, saved_var (training mode) Output case #2: Y (test mode)

Attributes¶

epsilon - FLOAT (default is '1e-05'):

The epsilon value to use to avoid division by zero, default is 1e-5f.
is_test - INT (default is '0'):

If set to nonzero, run spatial batch normalization in test mode, default is 0.
momentum - FLOAT (default is '0.9'):

Factor used in computing the running mean and variance.e.g., running_mean = running_mean * momentum + mean * (1 - momentum), default is 0.9f.
spatial - INT (default is '1'):

If true, compute the mean and variance across all spatial elements If false, compute the mean and variance across per feature.Default is 1.

Inputs¶

X (heterogeneous) - T:

Input data tensor from the previous operator; dimensions for image case are (N x C x H x W), where N is the batch size, C is the number of channels, and H and W are the height and the width of the data. For non image case, the dimensions are in the form of (N x C x D1 x D2 … Dn), where N is the batch size.
scale (heterogeneous) - T:

The scale as a 1-dimensional tensor of size C to be applied to the output.
B (heterogeneous) - T:

The bias as a 1-dimensional tensor of size C to be applied to the output.
mean (heterogeneous) - T:

The running mean (training) or the estimated mean (testing) as a 1-dimensional tensor of size C.
var (heterogeneous) - T:

The running variance (training) or the estimated variance (testing) as a 1-dimensional tensor of size C.

Outputs¶

Between 1 and 5 outputs.

Y (heterogeneous) - T:

The output tensor of the same shape as X.
mean (optional, heterogeneous) - T:

The running mean after the BatchNormalization operator. Must be in-place with the input mean. Should not be used for testing.
var (optional, heterogeneous) - T:

The running variance after the BatchNormalization operator. Must be in-place with the input var. Should not be used for testing.
saved_mean (optional, heterogeneous) - T:

Saved mean used during training to speed up gradient computation. Should not be used for testing.
saved_var (optional, heterogeneous) - T:

Saved variance used during training to speed up gradient computation. Should not be used for testing.

Type Constraints¶

T in ( tensor(double), tensor(float), tensor(float16) ):

Constrain input and output types to float tensors.

BatchNormalization - 1¶

Version¶

name: BatchNormalization (GitHub)
domain: main
since_version: 1
function: False
support_level: SupportType.COMMON
shape inference: False

This version of the operator has been available since version 1.

Summary¶

Output case #1: Y, mean, var, saved_mean, saved_var (training mode) Output case #2: Y (test mode)

Attributes¶

consumed_inputs - INTS (required) :

legacy optimization attribute.
epsilon - FLOAT (default is '1e-05'):

The epsilon value to use to avoid division by zero, default is 1e-5f.
is_test - INT (default is '0'):

If set to nonzero, run spatial batch normalization in test mode, default is 0.
momentum - FLOAT (default is '0.9'):

Factor used in computing the running mean and variance.e.g., running_mean = running_mean * momentum + mean * (1 - momentum), default is 0.9f.
spatial - INT (default is '1'):

If true, compute the mean and variance across all spatial elements If false, compute the mean and variance across per feature.Default is 1.

Inputs¶

X (heterogeneous) - T:

The input 4-dimensional tensor of shape NCHW.
scale (heterogeneous) - T:

The scale as a 1-dimensional tensor of size C to be applied to the output.
B (heterogeneous) - T:

The bias as a 1-dimensional tensor of size C to be applied to the output.
mean (heterogeneous) - T:

The running mean (training) or the estimated mean (testing) as a 1-dimensional tensor of size C.
var (heterogeneous) - T:

The running variance (training) or the estimated variance (testing) as a 1-dimensional tensor of size C.

Outputs¶

Between 1 and 5 outputs.

Y (heterogeneous) - T:

The output 4-dimensional tensor of the same shape as X.
mean (optional, heterogeneous) - T:

The running mean after the BatchNormalization operator. Must be in-place with the input mean. Should not be used for testing.
var (optional, heterogeneous) - T:

The running variance after the BatchNormalization operator. Must be in-place with the input var. Should not be used for testing.
saved_mean (optional, heterogeneous) - T:

Saved mean used during training to speed up gradient computation. Should not be used for testing.
saved_var (optional, heterogeneous) - T:

Saved variance used during training to speed up gradient computation. Should not be used for testing.

Type Constraints¶

T in ( tensor(double), tensor(float), tensor(float16) ):

Constrain input and output types to float tensors.