.. _l-onnx-op-batchnormalization-d1-14:

BatchNormalization - 1 vs 14
============================

Next section compares an older to a newer version of the same operator 
after both definition are converted into markdown text.
Green means an addition to the newer version, red means a deletion.
Anything else is unchanged.

.. raw:: html


    <div id="div_BatchNormalization_6"></div>
    <link rel="stylesheet" type="text/css" href="../_static/diff2html.min.css" />
    <script type="text/javascript" src="../_static/diff2html-ui-slim.min.js"></script>
    <script>
    const diffString = `
    --- a/BatchNormalization1
    +++ b/BatchNormalization14
    @@ -1 +1 @@
      Carries out batch normalization as described in the paper
      https://arxiv.org/abs/1502.03167. Depending on the mode it is being run,
    + There are five required inputs 'X', 'scale', 'B', 'input_mean' and
    + 'input_var'.
    + Note that 'input_mean' and 'input_var' are expected to be the estimated
    + statistics in inference mode (training_mode=False, default),
    + and the running statistics in training mode (training_mode=True).
    - there are multiple cases for the number of outputs, which we list below:
    ? ^
    + There are multiple cases for the number of outputs, which we list below:
    ? ^
  
    - Output case #1: Y, mean, var, saved_mean, saved_var (training mode)
    - Output case #2: Y (test mode)
    + Output case #1: Y, running_mean, running_var (training_mode=True)
    + Output case #2: Y (training_mode=False)
    + 
    + When training_mode=False, extra outputs are invalid.
    + The outputs are updated as follows when training_mode=True:
    + 
    + running_mean = input_mean * momentum + current_mean * (1 - momentum)
    + running_var = input_var * momentum + current_var * (1 - momentum)
    + 
    + Y = (X - current_mean) / sqrt(current_var + epsilon) * scale + B
    + 
    + where:
    + 
    + current_mean = ReduceMean(X, axis=all_except_channel_index)
    + current_var =  ReduceVar(X, axis=all_except_channel_index)
    + 
    + Notice that ReduceVar refers to the population variance, and it equals to
    + sum(sqrd(x_i - x_avg)) / N
    + where N is the population size (this formula does not use sample size N - 1).
    + 
    + 
    + 
    + When training_mode=False:
    + 
    + Y = (X - input_mean) / sqrt(input_var + epsilon) * scale + B
    + 
    + 
    + For previous (depreciated) non-spatial cases, implementors are suggested
    + to flatten the input shape to (N x C * D1 * D2 * ... * Dn) before a BatchNormalization Op.
    + This operator has **optional** inputs/outputs. See [ONNX IR](https://github.com/onnx/onnx/blob/main/docs/IR.md) for more details about the representation of optional arguments. An empty string may be used in the place of an actual argument's name to indicate a missing argument. Trailing optional arguments (those not followed by an argument that is present) may also be simply omitted.
  
      ### Attributes
  
    - * **consumed_inputs - INTS** (required) :
    - 
    -   legacy optimization attribute.
    - 
      * **epsilon - FLOAT** (default is '1e-05'):
  
    -   The epsilon value to use to avoid division by zero, default is 1e-5f.
    ?                                                     ------------------
    +   The epsilon value to use to avoid division by zero.
    - 
    - * **is_test - INT** (default is '0'):
    - 
    -   If set to nonzero, run spatial batch normalization in test mode, default is 0.
  
      * **momentum - FLOAT** (default is '0.9'):
  
    -   Factor used in computing the running mean and variance.e.g., running_mean = running_mean * momentum + mean * (1 - momentum), default is 0.9f.
    ?                                                                                                                              -----------------
    +   Factor used in computing the running mean and variance.e.g., running_mean = running_mean * momentum + mean * (1 - momentum).
  
    - * **spatial - INT** (default is '1'):
    ?     ^^ - ^^                      ^
    + * **training_mode - INT** (default is '0'):
    ?     ^^  ^^^^^^^^^                      ^
  
    -   If true, compute the mean and variance across all spatial elements If false, compute the mean and variance across per feature.Default is 1.
    +   If set to true, it indicates BatchNormalization is being used for training, and outputs 1, 2, 3, and 4 would be populated.
  
      ### Inputs
  
      - **X** (heterogeneous) - **T**:
  
    -   The input 4-dimensional tensor of shape NCHW.
    +   Input data tensor from the previous operator; dimensions are in the form of (N x C x D1 x D2 ... Dn), where N is the batch size, C is the number of channels. Statistics are computed for every channel of C over N and D1 to Dn dimensions. For image data, input dimensions become (N x C x H x W). The op also accepts single dimension input of size N in which case C is assumed to be 1
      - **scale** (heterogeneous) - **T**:
  
    -   The scale as a 1-dimensional tensor of size C to be applied to the output.
    +   Scale tensor of shape (C).
      - **B** (heterogeneous) - **T**:
  
    -   The bias as a 1-dimensional tensor of size C to be applied to the output.
    +   Bias tensor of shape (C).
    - - **mean** (heterogeneous) - **T**:
    ?                                ^
    + - **input_mean** (heterogeneous) - **U**:
    ?     ++++++                           ^
  
    -   The running mean (training) or the estimated mean (testing) as a 1-dimensional tensor of size C.
    ?  ----        -----               ----         -----           ---------   ------            ^^
    +   running (training) or estimated (testing) mean tensor of shape (C).
    ?                                               +             ^^^  + +
    - - **var** (heterogeneous) - **T**:
    ?                               ^
    + - **input_var** (heterogeneous) - **U**:
    ?     ++++++                          ^
  
    -   The running variance (training) or the estimated variance (testing) as a 1-dimensional tensor of size C.
    +   running (training) or estimated (testing) variance tensor of shape (C).
  
      ### Outputs
  
    - Between 1 and 5 outputs.
    ?               ^
    + Between 1 and 3 outputs.
    ?               ^
  
      - **Y** (heterogeneous) - **T**:
  
    -   The output 4-dimensional tensor of the same shape as X.
    ?             --------------                              -
    +   The output tensor of the same shape as X
    - - **mean** (optional, heterogeneous) - **T**:
    ?                                          ^
    + - **running_mean** (optional, heterogeneous) - **U**:
    ?     ++++++++                                     ^
  
    -   The running mean after the BatchNormalization operator. Must be in-place with the input mean. Should not be used for testing.
    +   The running mean after the BatchNormalization operator.
    - - **var** (optional, heterogeneous) - **T**:
    ?                                         ^
    + - **running_var** (optional, heterogeneous) - **U**:
    ?     ++++++++                                    ^
  
    +   The running variance after the BatchNormalization operator. This op uses the population size (N) for calculating variance, and not the sample size N-1.
    -   The running variance after the BatchNormalization operator. Must be in-place with the input var. Should not be used for testing.
    - - **saved_mean** (optional, heterogeneous) - **T**:
    - 
    -   Saved mean used during training to speed up gradient computation. Should not be used for testing.
    - - **saved_var** (optional, heterogeneous) - **T**:
    - 
    -   Saved variance used during training to speed up gradient computation. Should not be used for testing.
  
      ### Type Constraints
  
    - * **T** in ( tensor(double), tensor(float), tensor(float16) ):
    + * **T** in ( tensor(bfloat16), tensor(double), tensor(float), tensor(float16) ):
    ?             ++++++++++++++++++
  
    -   Constrain input and output types to float tensors.+   Constrain input and output types to float tensors.
    ?                                                     +
    + * **U** in ( tensor(bfloat16), tensor(double), tensor(float), tensor(float16) ):
    + 
    +   Constrain mean and variance types to float tensors. It allows all float type for U.
    `;

    document.addEventListener('DOMContentLoaded', function () {
    var targetElement = document.getElementById('div_BatchNormalization_6');
    var configuration = {
        drawFileList: true,
        fileListToggle: false,
        fileListStartVisible: false,
        fileContentToggle: false,
        matching: 'lines',
        outputFormat: 'line-by-line',
        synchronisedScroll: true,
        highlight: true,
        renderNothingWhenEmpty: false,
    };
    var diff2htmlUi = new Diff2HtmlUI(targetElement, diffString, configuration);
    diff2htmlUi.draw();
    diff2htmlUi.highlightCode();
    });
    </script>