.. _l-onnx-op-batchnormalization-d14-15:

BatchNormalization - 14 vs 15
=============================

Next section compares an older to a newer version of the same operator 
after both definition are converted into markdown text.
Green means an addition to the newer version, red means a deletion.
Anything else is unchanged.

.. raw:: html


    <div id="div_BatchNormalization_2"></div>
    <link rel="stylesheet" type="text/css" href="../_static/diff2html.min.css" />
    <script type="text/javascript" src="../_static/diff2html-ui-slim.min.js"></script>
    <script>
    const diffString = `
    --- a/BatchNormalization14
    +++ b/BatchNormalization15
    @@ -1 +1 @@
      Carries out batch normalization as described in the paper
      https://arxiv.org/abs/1502.03167. Depending on the mode it is being run,
      There are five required inputs 'X', 'scale', 'B', 'input_mean' and
      'input_var'.
      Note that 'input_mean' and 'input_var' are expected to be the estimated
      statistics in inference mode (training_mode=False, default),
      and the running statistics in training mode (training_mode=True).
      There are multiple cases for the number of outputs, which we list below:
  
    - Output case #1: Y, running_mean, running_var (training_mode=True)
    + * Output case #1: Y, running_mean, running_var (training_mode=True)
    ? ++
    - Output case #2: Y (training_mode=False)
    + * Output case #2: Y (training_mode=False)
    ? ++
  
      When training_mode=False, extra outputs are invalid.
      The outputs are updated as follows when training_mode=True:
  
      running_mean = input_mean * momentum + current_mean * (1 - momentum)
      running_var = input_var * momentum + current_var * (1 - momentum)
  
      Y = (X - current_mean) / sqrt(current_var + epsilon) * scale + B
  
      where:
  
      current_mean = ReduceMean(X, axis=all_except_channel_index)
      current_var =  ReduceVar(X, axis=all_except_channel_index)
  
      Notice that ReduceVar refers to the population variance, and it equals to
      sum(sqrd(x_i - x_avg)) / N
      where N is the population size (this formula does not use sample size N - 1).
  
    - 
    + The computation of ReduceMean and ReduceVar uses float to avoid overflow for float16 inputs.
  
      When training_mode=False:
  
      Y = (X - input_mean) / sqrt(input_var + epsilon) * scale + B
  
  
      For previous (depreciated) non-spatial cases, implementors are suggested
      to flatten the input shape to (N x C * D1 * D2 * ... * Dn) before a BatchNormalization Op.
      This operator has **optional** inputs/outputs. See [ONNX IR](https://github.com/onnx/onnx/blob/main/docs/IR.md) for more details about the representation of optional arguments. An empty string may be used in the place of an actual argument's name to indicate a missing argument. Trailing optional arguments (those not followed by an argument that is present) may also be simply omitted.
  
      ### Attributes
  
      * **epsilon - FLOAT** (default is '1e-05'):
  
        The epsilon value to use to avoid division by zero.
  
      * **momentum - FLOAT** (default is '0.9'):
  
        Factor used in computing the running mean and variance.e.g., running_mean = running_mean * momentum + mean * (1 - momentum).
  
      * **training_mode - INT** (default is '0'):
  
    -   If set to true, it indicates BatchNormalization is being used for training, and outputs 1, 2, 3, and 4 would be populated.
    ?                                                                                            -------     ^ ^ ---    ^   --
    +   If set to true, it indicates BatchNormalization is being used for training, and outputs 1 and 2 are to be computed.
    ?                                                                                                 ^ ^^^^^     ^ +
  
      ### Inputs
  
      - **X** (heterogeneous) - **T**:
  
        Input data tensor from the previous operator; dimensions are in the form of (N x C x D1 x D2 ... Dn), where N is the batch size, C is the number of channels. Statistics are computed for every channel of C over N and D1 to Dn dimensions. For image data, input dimensions become (N x C x H x W). The op also accepts single dimension input of size N in which case C is assumed to be 1
    - - **scale** (heterogeneous) - **T**:
    + - **scale** (heterogeneous) - **T1**:
    ?                                  +
  
        Scale tensor of shape (C).
    - - **B** (heterogeneous) - **T**:
    + - **B** (heterogeneous) - **T1**:
    ?                              +
  
        Bias tensor of shape (C).
    - - **input_mean** (heterogeneous) - **U**:
    ?                                      ^
    + - **input_mean** (heterogeneous) - **T2**:
    ?                                      ^^
  
        running (training) or estimated (testing) mean tensor of shape (C).
    - - **input_var** (heterogeneous) - **U**:
    ?                                     ^
    + - **input_var** (heterogeneous) - **T2**:
    ?                                     ^^
  
        running (training) or estimated (testing) variance tensor of shape (C).
  
      ### Outputs
  
      Between 1 and 3 outputs.
  
      - **Y** (heterogeneous) - **T**:
  
        The output tensor of the same shape as X
    - - **running_mean** (optional, heterogeneous) - **U**:
    ?                                                  ^
    + - **running_mean** (optional, heterogeneous) - **T2**:
    ?                                                  ^^
  
        The running mean after the BatchNormalization operator.
    - - **running_var** (optional, heterogeneous) - **U**:
    ?                                                 ^
    + - **running_var** (optional, heterogeneous) - **T2**:
    ?                                                 ^^
  
        The running variance after the BatchNormalization operator. This op uses the population size (N) for calculating variance, and not the sample size N-1.
  
      ### Type Constraints
  
      * **T** in ( tensor(bfloat16), tensor(double), tensor(float), tensor(float16) ):
  
        Constrain input and output types to float tensors.
    - * **U** in ( tensor(bfloat16), tensor(double), tensor(float), tensor(float16) ):
    ?     ^
    + * **T1** in ( tensor(bfloat16), tensor(double), tensor(float), tensor(float16) ):
    ?     ^^
  
    +   Constrain scale and bias types to float tensors.
    + * **T2** in ( tensor(bfloat16), tensor(double), tensor(float), tensor(float16) ):
    + 
    -   Constrain mean and variance types to float tensors. It allows all float type for U.?                                                      --------------------------------
    +   Constrain mean and variance types to float tensors.
    `;

    document.addEventListener('DOMContentLoaded', function () {
    var targetElement = document.getElementById('div_BatchNormalization_2');
    var configuration = {
        drawFileList: true,
        fileListToggle: false,
        fileListStartVisible: false,
        fileContentToggle: false,
        matching: 'lines',
        outputFormat: 'line-by-line',
        synchronisedScroll: true,
        highlight: true,
        renderNothingWhenEmpty: false,
    };
    var diff2htmlUi = new Diff2HtmlUI(targetElement, diffString, configuration);
    diff2htmlUi.draw();
    diff2htmlUi.highlightCode();
    });
    </script>