QuantizeLinear

QuantizeLinear - 25

Version

  • name: QuantizeLinear (GitHub)

  • domain: main

  • since_version: 25

  • function: False

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 25.

Summary

The linear quantization operator consumes a high-precision tensor, a scale, and a zero point to compute the low-precision/quantized tensor. The scale factor and zero point must have the same shape, determining the quantization granularity. The quantization formula is y = saturate((x / y_scale) + y_zero_point).

Saturation is done according to:

  • uint16: [0, 65535]

  • int16: [-32768, 32767]

  • uint8: [0, 255]

  • int8: [-128, 127]

  • uint4: [0, 15]

  • int4: [-8, 7]

  • uint2: [0, 3]

  • int2: [-2, 1]

For (x / y_scale), it rounds to the nearest even. Refer to https://en.wikipedia.org/wiki/Rounding for details.

y_zero_point and y must have the same type. y_zero_point is usually not used for quantization to float8 and 4bit types, but the quantization formula remains the same for consistency, and the type of the attribute y_zero_point still determines the quantization type. x and y_scale are allowed to have different types. The type of y_scale determines the precision of the division operation between x and y_scale, unless the precision attribute is specified.

There are three supported quantization granularities, determined by the shape of y_scale. In all cases, y_zero_point must have the same shape as y_scale.

  • Per-tensor (per-layer) quantization: y_scale is a scalar.

  • Per-axis quantization: The scale must be a 1-D tensor, with the length of the quantization axis. For an input shape (D0, ..., Di, ..., Dn) and axis=i, y_scale is a 1-D tensor of length Di.

  • Blocked quantization: The scale’s shape is identical to the input’s shape, except for one dimension, in which blocking is performed. Given x shape (D0, ..., Di, ..., Dn), axis=i, and block size B: y_scale shape is (D0, ..., ceil(Di/B), ..., Dn).

Attributes

  • axis - INT (default is 1):

    (Optional) The axis of the dequantizing dimension of the input tensor. Used only for per-axis and blocked quantization. Negative value means counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(input). When the rank of the input is 1, per-tensor quantization is applied, rendering the axis unnecessary in this scenario.

  • block_size - INT (default is 0):

    (Optional) The size of the quantization block (number of times every scale is replicated). Used only for blocked quantization. The block size is a positive integer. Given x shape (D0, ..., Di, ..., Dn), y_scale shape (S0, ... Si, ...Sn) and axis=i, the accepted range is [ceil(Di/Si), ceil(Di/(Si-1))-1]

  • output_dtype - INT (default is 0):

    (Optional) The output data type. If not supplied, the output data type is inferred from y_zero_point data type (T3). If neither output_dtype nor y_zero_point are supplied, output data type is uint8. If both output_dtype and y_zero_point are specified, output_dtype must be T3.

  • precision - INT (default is 0):

    (Optional) The precision of the division operation between x and y_scale. If not provided, it will be the same as the type of y_scale.

  • saturate - INT (default is 1):

    The parameter defines how the conversion behaves if an input value is out of range of the destination type. It only applies for float 8 quantization (float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz). It is true by default. All cases are fully described in two tables inserted in the operator description.

Inputs

Between 2 and 3 inputs.

  • x (heterogeneous) - T1:

    N-D full precision Input tensor to be quantized.

  • y_scale (heterogeneous) - T2:

    Scale for doing quantization to get y. For per-tensor/layer quantization the scale is a scalar, for per-axis quantization it is a 1-D Tensor and for blocked quantization it has the same shape as the input, except for one dimension in which blocking is performed.

  • y_zero_point (optional, heterogeneous) - T3:

    Zero point for doing quantization to get y. Shape must match y_scale. Default is uint8 with zero point of 0 if it’s not specified.

Outputs

  • y (heterogeneous) - T3:

    N-D quantized output tensor. It has same shape as input x.

Type Constraints

  • T1 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(int32) ):

    The type of the input ‘x’.

  • T2 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(float8e8m0), tensor(int32) ):

    The type of the input ‘y_scale’.

  • T3 in ( tensor(float4e2m1), tensor(float8e4m3fn), tensor(float8e4m3fnuz), tensor(float8e5m2), tensor(float8e5m2fnuz), tensor(int16), tensor(int2), tensor(int4), tensor(int8), tensor(uint16), tensor(uint2), tensor(uint4), tensor(uint8) ):

    The type of the input y_zero_point and the output y.

Examples

default

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
)

x = np.array([0, 2, 3, 1000, -254, -1000]).astype(np.float32)
y_scale = np.float32(2)
y_zero_point = np.uint8(128)
y = np.array([128, 129, 130, 255, 1, 0]).astype(np.uint8)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear",
)

_axis

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
)

x = np.array(
    [
        [
            [[-162, 10], [-100, 232], [-20, -50]],
            [[-76, 0], [0, 252], [32, -44]],
            [[245, -485], [-960, -270], [-375, -470]],
        ],
    ],
    dtype=np.float32,
)
y_scale = np.array([2, 4, 5], dtype=np.float32)
y_zero_point = np.array([84, 24, 196], dtype=np.uint8)
y = (x / y_scale.reshape(1, 3, 1, 1) + y_zero_point.reshape(1, 3, 1, 1)).astype(
    np.uint8
)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_axis",
)

_e4m3fn

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
)

x = np.array([0.0, 1.0, 2.0, 100000.0, 200.0]).astype(np.float32)
y_scale = np.float32(2)
y_zero_point = make_tensor("y_zero_point", TensorProto.FLOAT8E4M3FN, [1], [0])
y = make_tensor("y", TensorProto.FLOAT8E4M3FN, [5], [0, 0.5, 1, 448, 96])

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_e4m3fn",
)

_e5m2

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
)

x = np.array([0.0, 1.0, 2.0, 100000.0, 200.0]).astype(np.float32)
y_scale = np.float32(2)
y_zero_point = make_tensor("y_zero_point", TensorProto.FLOAT8E5M2, [1], [0.0])
y = make_tensor("y", TensorProto.FLOAT8E5M2, [5], [0, 0.5, 1, 49152, 96])

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_e5m2",
)

_uint16

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
)

x = np.array(
    [
        0.0,
        -128.0,
        3.0,
        -3.0,
        2.9,
        -2.9,
        3.1,
        -3.1,
        65536.0,
        -65534.0,
        70000.0,
        -70000.0,
    ]
).astype(np.float32)
y_scale = np.float32(2.0)
y_zero_point = np.uint16(32767)
y = np.array(
    [
        32767,
        32703,
        32769,
        32765,
        32768,
        32766,
        32769,
        32765,
        65535,
        0,
        65535,
        0,
    ]
).astype(np.uint16)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_uint16",
)

_int16

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
)

x = np.array(
    [
        0.0,
        -514.0,
        3.0,
        -3.0,
        2.9,
        -2.9,
        3.1,
        -3.1,
        65022.0,
        -66046.0,
        65023.0,
        -66047.0,
        65024.0,
        -66048.0,
        70000.0,
        -70000.0,
    ]
).astype(np.float32)
y_scale = np.float32(2.0)
y_zero_point = np.int16(256)
y = np.array(
    [
        256,
        -1,
        258,
        254,
        257,
        255,
        258,
        254,
        32767,
        -32767,
        32767,
        -32768,
        32767,
        -32768,
        32767,
        -32768,
    ]
).astype(np.int16)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_int16",
)

_uint4

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
    axis=0,
)

x = np.array(
    [
        [0.0, 2.5, 4.8, 8.6],
        [-30, -20, 6, 9],
        [12, 15, 16, 40],
    ]
).astype(np.float32)

y_scale = np.asarray([2.0, 3.0, 4.0], dtype=np.float32)
y_zero_point = make_tensor(
    "y_zero_point", TensorProto.UINT4, y_scale.shape, np.ones_like(y_scale)
)
y = make_tensor(
    "y", TensorProto.UINT4, x.shape, [1, 2, 3, 5, 0, 0, 3, 4, 4, 5, 5, 11]
)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_uint4",
)

_int4

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
    axis=0,
)

x = np.array(
    [
        [0.0, 2.5, 4.8, 8.6],
        [-30, -20, 6, 9],
        [12, 15, 16, 40],
    ]
).astype(np.float32)

y_scale = np.asarray([2.0, 3.0, 4.0], dtype=np.float32)
y_zero_point = make_tensor(
    "y_zero_point", TensorProto.INT4, y_scale.shape, np.ones_like(y_scale)
)
y = make_tensor(
    "y", TensorProto.INT4, x.shape, [1, 2, 3, 5, -8, -6, 3, 4, 4, 5, 5, 7]
)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_int4",
)

_uint2

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
    axis=0,
)

x = np.array(
    [
        [0.0, 2.5, 4.8, 8.6],
        [-2.0, -1.0, 1.0, 3.0],
        [4.0, 5.0, 6.0, 7.0],
    ],
    dtype=np.float32,
)
y_scale = np.asarray([2.0, 3.0, 4.0], dtype=np.float32)
y_zero_point = make_tensor(
    "y_zero_point", TensorProto.UINT2, y_scale.shape, np.zeros_like(y_scale)
)
y = make_tensor(
    "y", TensorProto.UINT2, x.shape, [0, 1, 2, 3, 0, 0, 0, 1, 1, 1, 2, 2]
)
expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_uint2",
)

_int2

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
    axis=0,
)
x = np.array(
    [
        [0.0, 2.5, 4.8, 8.6],
        [-4.0, -3.0, 1.0, 2.0],
        [-0.0, -2.5, -4.8, -8.6],
    ],
    dtype=np.float32,
)
y_scale = np.asarray([2.0, 3.0, 4.0], dtype=np.float32)
y_zero_point = make_tensor(
    "y_zero_point", TensorProto.INT2, y_scale.shape, np.zeros_like(y_scale)
)
y = make_tensor(
    "y", TensorProto.INT2, x.shape, [0, 1, 1, 1, -1, -1, 0, 1, 0, -1, -1, -2]
)
expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_int2",
)

_float4e2m1

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
    axis=0,
)

x = np.array(
    [
        [0.0, 2.5, 4.8, 8.6],
        [-30, -20, 6, 9],
        [-0.0, -2.5, -4.8, -8.6],
    ]
).astype(np.float32)

y_scale = np.asarray([2.0, 3.0, 4.0], dtype=np.float32)
y_zero_point = make_tensor(
    "y_zero_point",
    TensorProto.FLOAT4E2M1,
    y_scale.shape,
    np.zeros_like(y_scale),
)
y = make_tensor(
    "y",
    TensorProto.FLOAT4E2M1,
    x.shape,
    [0, 1, 2, 4, -6, -6, 2, 3, 0, -0.5, -1, -2],
)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_float4e2m1",
)

_blocked_asymmetric

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
    axis=1,
    block_size=2,
)

x = np.array(
    [
        [6.0, 12.0, 50.0, 5.0],
        [1.0, 8.0, 4.0, 5.0],
        [0.0, 20.0, 10.0, 4.0],
    ],
    dtype=np.float32,
)
y_scale = np.array(
    [
        [1.5, 2.5],
        [3.0, 4.9],
        [5.1, 6.9],
    ],
    dtype=np.float32,
)
y_zero_point = np.array(
    [
        [0, 1],
        [1, 0],
        [2, 3],
    ],
    dtype=np.uint8,
)
# x.shape = (3, 4)
# y_scale.shape = (3, 2)
assert y_scale.shape == y_zero_point.shape
block_axis = 1
# The block shape is [x.shape[i] // y_scale.shape[i] for i in range(len(x.shape))] = (1, 2)
assert all(
    x.shape[i] == y_scale.shape[i]
    for i in range(len(x.shape))
    if i != block_axis
)
assert x.shape[block_axis] % y_scale.shape[block_axis] == 0
repeats = x.shape[block_axis] // y_scale.shape[block_axis]

# Create element-wise scale and zero point
y_scale_elementwise = np.repeat(y_scale, repeats=repeats, axis=block_axis)
y_zero_point_elementwise = np.repeat(
    y_zero_point, repeats=repeats, axis=block_axis
)

y = np.rint(x / y_scale_elementwise + y_zero_point_elementwise).astype(np.uint8)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_blocked_asymmetric",
)

_blocked_symmetric

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale"],
    outputs=["y"],
    axis=1,
    block_size=2,
    output_dtype=TensorProto.INT16,
)

x = np.array(
    [
        [6.0, -8, -10, 5.0],
        [1.0, 8.0, 4.0, 5.0],
        [0.0, 20.0, 10.0, 4.0],
    ],
    dtype=np.float32,
)

y_scale = np.array(
    [
        [1.5, 2.5],
        [3.0, 4.9],
        [5.1, 6.9],
    ],
    dtype=np.float32,
)

# x.shape = (3, 4)
# y_scale.shape = (3, 2)

block_axis = 1
# The block shape is [x.shape[i] // y_scale.shape[i] for i in range(len(x.shape))] = (1, 2)
assert all(
    x.shape[i] == y_scale.shape[i]
    for i in range(len(x.shape))
    if i != block_axis
)
assert x.shape[block_axis] % y_scale.shape[block_axis] == 0
repeats = x.shape[block_axis] // y_scale.shape[block_axis]

# Create element-wise scale and zero point
y_scale_elementwise = np.repeat(y_scale, repeats=repeats, axis=block_axis)

y_val = np.clip(
    np.rint(x / y_scale_elementwise), a_min=-32768, a_max=32767
).astype(np.int16)
y = make_tensor(
    "y",
    TensorProto.INT16,
    x.shape,
    y_val,
)
expect(
    node,
    inputs=[x, y_scale],
    outputs=[y],
    name="test_quantizelinear_blocked_symmetric",
)

QuantizeLinear - 24

Version

  • name: QuantizeLinear (GitHub)

  • domain: main

  • since_version: 24

  • function: False

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 24.

Summary

The linear quantization operator consumes a high-precision tensor, a scale, and a zero point to compute the low-precision/quantized tensor. The scale factor and zero point must have the same shape, determining the quantization granularity. The quantization formula is y = saturate((x / y_scale) + y_zero_point).

Saturation is done according to:

  • uint16: [0, 65535]

  • int16: [-32768, 32767]

  • uint8: [0, 255]

  • int8: [-128, 127]

  • uint4: [0, 15]

  • int4: [-8, 7]

For (x / y_scale), it rounds to the nearest even. Refer to https://en.wikipedia.org/wiki/Rounding for details.

y_zero_point and y must have the same type. y_zero_point is usually not used for quantization to float8 and 4bit types, but the quantization formula remains the same for consistency, and the type of the attribute y_zero_point still determines the quantization type. x and y_scale are allowed to have different types. The type of y_scale determines the precision of the division operation between x and y_scale, unless the precision attribute is specified.

There are three supported quantization granularities, determined by the shape of y_scale. In all cases, y_zero_point must have the same shape as y_scale.

  • Per-tensor (per-layer) quantization: y_scale is a scalar.

  • Per-axis quantization: The scale must be a 1-D tensor, with the length of the quantization axis. For an input shape (D0, ..., Di, ..., Dn) and axis=i, y_scale is a 1-D tensor of length Di.

  • Blocked quantization: The scale’s shape is identical to the input’s shape, except for one dimension, in which blocking is performed. Given x shape (D0, ..., Di, ..., Dn), axis=i, and block size B: y_scale shape is (D0, ..., ceil(Di/B), ..., Dn).

Attributes

  • axis - INT (default is 1):

    (Optional) The axis of the dequantizing dimension of the input tensor. Used only for per-axis and blocked quantization. Negative value means counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(input). When the rank of the input is 1, per-tensor quantization is applied, rendering the axis unnecessary in this scenario.

  • block_size - INT (default is 0):

    (Optional) The size of the quantization block (number of times every scale is replicated). Used only for blocked quantization. The block size is a positive integer. Given x shape (D0, ..., Di, ..., Dn), y_scale shape (S0, ... Si, ...Sn) and axis=i, the accepted range is [ceil(Di/Si), ceil(Di/(Si-1))-1]

  • output_dtype - INT (default is 0):

    (Optional) The output data type. If not supplied, the output data type is inferred from y_zero_point data type (T3). If neither output_dtype nor y_zero_point are supplied, output data type is uint8. If both output_dtype and y_zero_point are specified, output_dtype must be T3.

  • precision - INT (default is 0):

    (Optional) The precision of the division operation between x and y_scale. If not provided, it will be the same as the type of y_scale.

  • saturate - INT (default is 1):

    The parameter defines how the conversion behaves if an input value is out of range of the destination type. It only applies for float 8 quantization (float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz). It is true by default. All cases are fully described in two tables inserted in the operator description.

Inputs

Between 2 and 3 inputs.

  • x (heterogeneous) - T1:

    N-D full precision Input tensor to be quantized.

  • y_scale (heterogeneous) - T2:

    Scale for doing quantization to get y. For per-tensor/layer quantization the scale is a scalar, for per-axis quantization it is a 1-D Tensor and for blocked quantization it has the same shape as the input, except for one dimension in which blocking is performed.

  • y_zero_point (optional, heterogeneous) - T3:

    Zero point for doing quantization to get y. Shape must match y_scale. Default is uint8 with zero point of 0 if it’s not specified.

Outputs

  • y (heterogeneous) - T3:

    N-D quantized output tensor. It has same shape as input x.

Type Constraints

  • T1 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(int32) ):

    The type of the input ‘x’.

  • T2 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(float8e8m0), tensor(int32) ):

    The type of the input ‘y_scale’.

  • T3 in ( tensor(float4e2m1), tensor(float8e4m3fn), tensor(float8e4m3fnuz), tensor(float8e5m2), tensor(float8e5m2fnuz), tensor(int16), tensor(int4), tensor(int8), tensor(uint16), tensor(uint4), tensor(uint8) ):

    The type of the input y_zero_point and the output y.

QuantizeLinear - 23

Version

  • name: QuantizeLinear (GitHub)

  • domain: main

  • since_version: 23

  • function: False

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 23.

Summary

The linear quantization operator consumes a high-precision tensor, a scale, and a zero point to compute the low-precision/quantized tensor. The scale factor and zero point must have the same shape, determining the quantization granularity. The quantization formula is y = saturate((x / y_scale) + y_zero_point).

Saturation is done according to:

  • uint16: [0, 65535]

  • int16: [-32768, 32767]

  • uint8: [0, 255]

  • int8: [-128, 127]

  • uint4: [0, 15]

  • int4: [-8, 7]

For (x / y_scale), it rounds to the nearest even. Refer to https://en.wikipedia.org/wiki/Rounding for details.

y_zero_point and y must have the same type. y_zero_point is usually not used for quantization to float8 and 4bit types, but the quantization formula remains the same for consistency, and the type of the attribute y_zero_point still determines the quantization type. x and y_scale are allowed to have different types. The type of y_scale determines the precision of the division operation between x and y_scale, unless the precision attribute is specified.

There are three supported quantization granularities, determined by the shape of y_scale. In all cases, y_zero_point must have the same shape as y_scale.

  • Per-tensor (per-layer) quantization: y_scale is a scalar.

  • Per-axis quantization: The scale must be a 1-D tensor, with the length of the quantization axis. For an input shape (D0, ..., Di, ..., Dn) and axis=i, y_scale is a 1-D tensor of length Di.

  • Blocked quantization: The scale’s shape is identical to the input’s shape, except for one dimension, in which blocking is performed. Given x shape (D0, ..., Di, ..., Dn), axis=i, and block size B: y_scale shape is (D0, ..., ceil(Di/B), ..., Dn).

Attributes

  • axis - INT (default is 1):

    (Optional) The axis of the dequantizing dimension of the input tensor. Used only for per-axis and blocked quantization. Negative value means counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(input). When the rank of the input is 1, per-tensor quantization is applied, rendering the axis unnecessary in this scenario.

  • block_size - INT (default is 0):

    (Optional) The size of the quantization block (number of times every scale is replicated). Used only for blocked quantization. The block size is a positive integer. Given x shape (D0, ..., Di, ..., Dn), y_scale shape (S0, ... Si, ...Sn) and axis=i, the accepted range is [ceil(Di/Si), ceil(Di/(Si-1))-1]

  • output_dtype - INT (default is 0):

    (Optional) The output data type. If not supplied, the output data type is inferred from y_zero_point data type (T3). If neither output_dtype nor y_zero_point are supplied, output data type is uint8. If both output_dtype and y_zero_point are specified, output_dtype must be T3.

  • precision - INT (default is 0):

    (Optional) The precision of the division operation between x and y_scale. If not provided, it will be the same as the type of y_scale.

  • saturate - INT (default is 1):

    The parameter defines how the conversion behaves if an input value is out of range of the destination type. It only applies for float 8 quantization (float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz). It is true by default. All cases are fully described in two tables inserted in the operator description.

Inputs

Between 2 and 3 inputs.

  • x (heterogeneous) - T1:

    N-D full precision Input tensor to be quantized.

  • y_scale (heterogeneous) - T2:

    Scale for doing quantization to get y. For per-tensor/layer quantization the scale is a scalar, for per-axis quantization it is a 1-D Tensor and for blocked quantization it has the same shape as the input, except for one dimension in which blocking is performed.

  • y_zero_point (optional, heterogeneous) - T3:

    Zero point for doing quantization to get y. Shape must match y_scale.Default is uint8 with zero point of 0 if it’s not specified.

Outputs

  • y (heterogeneous) - T3:

    N-D quantized output tensor. It has same shape as input x.

Type Constraints

  • T1 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(int32) ):

    The type of the input ‘x’.

  • T2 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(int32) ):

    The type of the input ‘y_scale’.

  • T3 in ( tensor(float4e2m1), tensor(float8e4m3fn), tensor(float8e4m3fnuz), tensor(float8e5m2), tensor(float8e5m2fnuz), tensor(int16), tensor(int4), tensor(int8), tensor(uint16), tensor(uint4), tensor(uint8) ):

    The type of the input y_zero_point and the output y.

QuantizeLinear - 21

Version

  • name: QuantizeLinear (GitHub)

  • domain: main

  • since_version: 21

  • function: False

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 21.

Summary

The linear quantization operator consumes a high-precision tensor, a scale, and a zero point to compute the low-precision/quantized tensor. The scale factor and zero point must have the same shape, determining the quantization granularity. The quantization formula is y = saturate((x / y_scale) + y_zero_point). Saturation is done according to:

  • uint16: [0, 65535]

  • int16: [-32768, 32767]

  • uint8: [0, 255]

  • int8: [-128, 127]

  • uint4: [0, 15]

  • int4: [-8, 7] For (x / y_scale), it rounds to the nearest even. Refer to https://en.wikipedia.org/wiki/Rounding for details. y_zero_point and y must have the same type. y_zero_point is usually not used for quantization to float8 types, but the quantization formula remains the same for consistency, and the type of the attribute y_zero_point still determines the quantization type. There are three supported quantization granularities, determined by the shape of y_scale. In all cases, y_zero_point must have the same shape as y_scale.

  • Per-tensor (per-layer) quantization: y_scale is a scalar.

  • Per-axis quantization: The scale must be a 1-D tensor, with the length of the quantization axis. For an input shape (D0, ..., Di, ..., Dn) and axis=i, y_scale is a 1-D tensor of length Di.

  • Blocked quantization: The scale’s shape is identical to the input’s shape, except for one dimension, in which blocking is performed. Given x shape (D0, ..., Di, ..., Dn), axis=i, and block size B: y_scale shape is (D0, ..., ceil(Di/B), ..., Dn).

Attributes

  • axis - INT (default is 1):

    (Optional) The axis of the dequantizing dimension of the input tensor. Used only for per-axis and blocked quantization. Negative value means counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(input). When the rank of the input is 1, per-tensor quantization is applied, rendering the axis unnecessary in this scenario.

  • block_size - INT (default is 0):

    (Optional) The size of the quantization block (number of times every scale is replicated). Used only for blocked quantization. The block size is a positive integer. Given x shape (D0, ..., Di, ..., Dn), y_scale shape (S0, ... Si, ...Sn) and axis=i, the accepted range is [ceil(Di/Si), ceil(Di/(Si-1))-1]

  • output_dtype - INT (default is 0):

    (Optional) The output data type. If not supplied, the output data type is inferred from y_zero_point data type (T2). If neither output_dtype nor y_zero_point are supplied, output data type is uint8. If both output_dtype and y_zero_point are specified, output_dtype must be T2.

  • saturate - INT (default is 1):

    The parameter defines how the conversion behaves if an input value is out of range of the destination type. It only applies for float 8 quantization (float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz). It is true by default. All cases are fully described in two tables inserted in the operator description.

Inputs

Between 2 and 3 inputs.

  • x (heterogeneous) - T1:

    N-D full precision Input tensor to be quantized.

  • y_scale (heterogeneous) - T1:

    Scale for doing quantization to get y. For per-tensor/layer quantization the scale is a scalar, for per-axis quantization it is a 1-D Tensor and for blocked quantization it has the same shape as the input, except for one dimension in which blocking is performed.

  • y_zero_point (optional, heterogeneous) - T2:

    Zero point for doing quantization to get y. Shape must match y_scale.Default is uint8 with zero point of 0 if it’s not specified.

Outputs

  • y (heterogeneous) - T2:

    N-D quantized output tensor. It has same shape as input x.

Type Constraints

  • T1 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(int32) ):

    The type of the input ‘x’.

  • T2 in ( tensor(float8e4m3fn), tensor(float8e4m3fnuz), tensor(float8e5m2), tensor(float8e5m2fnuz), tensor(int16), tensor(int4), tensor(int8), tensor(uint16), tensor(uint4), tensor(uint8) ):

    The type of the input y_zero_point and the output y.

QuantizeLinear - 19

Version

  • name: QuantizeLinear (GitHub)

  • domain: main

  • since_version: 19

  • function: False

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 19.

Summary

The linear quantization operator. It consumes a high precision tensor, a scale, and a zero point to compute the low precision / quantized tensor. The scale factor and zero point must have same shape, and can be either a scalar for per-tensor / per layer quantization, or a 1-D tensor for per-axis quantization. The quantization formula is y = saturate ((x / y_scale) + y_zero_point). For saturation, it saturates to [0, 255] if it’s uint8, or [-128, 127] if it’s int8. For (x / y_scale), it’s rounding to the nearest even. Refer to https://en.wikipedia.org/wiki/Rounding for details. ‘y_zero_point’ and ‘y’ must have same type. ‘y_zero_point’ is usually not used for quantization to float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz, but the quantization formula remains the same for consistency and the type of the attribute ‘y_zero_point’ still determines the quantization type.

Attributes

  • axis - INT (default is 1):

    (Optional) The axis of the quantization dimension of the input tensor. Ignored for per-tensor quantization. Negative value means counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(input).

  • saturate - INT (default is 1):

    The parameter defines how the conversion behaves if an input value is out of range of the destination type. It only applies for float 8 quantization (float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz). It is true by default. All cases are fully described in two tables inserted in the operator description.

Inputs

Between 2 and 3 inputs.

  • x (heterogeneous) - T1:

    N-D full precision Input tensor to be quantized.

  • y_scale (heterogeneous) - T1:

    Scale for doing quantization to get ‘y’. It can be a scalar, which means per-tensor/layer quantization, or a 1-D Tensor for per-axis quantization.

  • y_zero_point (optional, heterogeneous) - T2:

    Zero point for doing quantization to get ‘y’. Shape must match y_scale. Default is uint8 with zero point of 0 if it’s not specified.

Outputs

  • y (heterogeneous) - T2:

    N-D quantized output tensor. It has same shape as input ‘x’.

Type Constraints

  • T1 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(int32) ):

    Constrain ‘x’ to float, float16, bfloat16 or int32 tensor.

  • T2 in ( tensor(float8e4m3fn), tensor(float8e4m3fnuz), tensor(float8e5m2), tensor(float8e5m2fnuz), tensor(int8), tensor(uint8) ):

    Constrain ‘y_zero_point’ and ‘y’ to 8-bit integer/float tensor.

QuantizeLinear - 13

Version

  • name: QuantizeLinear (GitHub)

  • domain: main

  • since_version: 13

  • function: False

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 13.

Summary

The linear quantization operator. It consumes a high precision tensor, a scale, and a zero point to compute the low precision / quantized tensor. The scale factor and zero point must have same shape, and can be either a scalar for per-tensor / per layer quantization, or a 1-D tensor for per-axis quantization. The quantization formula is y = saturate ((x / y_scale) + y_zero_point). For saturation, it saturates to [0, 255] if it’s uint8, or [-128, 127] if it’s int8. For (x / y_scale), it’s rounding to the nearest even. Refer to https://en.wikipedia.org/wiki/Rounding for details. ‘y_zero_point’ and ‘y’ must have same type.

Attributes

  • axis - INT (default is 1):

    (Optional) The axis of the quantization dimension of the input tensor. Ignored for per-tensor quantization. Negative value means counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(input).

Inputs

Between 2 and 3 inputs.

  • x (heterogeneous) - T1:

    N-D full precision Input tensor to be quantized.

  • y_scale (heterogeneous) - tensor(float):

    Scale for doing quantization to get ‘y’. It can be a scalar, which means per-tensor/layer quantization, or a 1-D Tensor for per-axis quantization.

  • y_zero_point (optional, heterogeneous) - T2:

    Zero point for doing quantization to get ‘y’. Shape must match y_scale. Default is uint8 with zero point of 0 if it’s not specified.

Outputs

  • y (heterogeneous) - T2:

    N-D quantized output tensor. It has same shape as input ‘x’.

Type Constraints

  • T1 in ( tensor(float), tensor(int32) ):

    Constrain ‘x’ to float or int32 tensor.

  • T2 in ( tensor(int8), tensor(uint8) ):

    Constrain ‘y_zero_point’ and ‘y’ to 8-bit integer tensor.

QuantizeLinear - 10

Version

  • name: QuantizeLinear (GitHub)

  • domain: main

  • since_version: 10

  • function: False

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 10.

Summary

The linear per-tensor/layer quantization operator. It consumes a high precision tensor, a scale, a zero point to compute the low precision / quantized tensor. The quantization formula is y = saturate ((x / y_scale) + y_zero_point). For saturation, it saturates to [0, 255] if it’s uint8, or [-128, 127] if it’s int8. For (x / y_scale), it’s rounding to the nearest even. Refer to https://en.wikipedia.org/wiki/Rounding for details. ‘y_zero_point’ and ‘y’ must have same type.

Inputs

Between 2 and 3 inputs.

  • x (heterogeneous) - T1:

    N-D full precision Input tensor to be quantized.

  • y_scale (heterogeneous) - tensor(float):

    Scale for doing quantization to get ‘y’. It’s a scalar, which means a per-tensor/layer quantization.

  • y_zero_point (optional, heterogeneous) - T2:

    Zero point for doing quantization to get ‘y’. It’s a scalar, which means a per-tensor/layer quantization. Default value is uint8 typed 0 if it’s not specified.

Outputs

  • y (heterogeneous) - T2:

    N-D quantized output tensor. It has same shape as input ‘x’.

Type Constraints

  • T1 in ( tensor(float), tensor(int32) ):

    Constrain ‘x’ to float or int32 tensor.

  • T2 in ( tensor(int8), tensor(uint8) ):

    Constrain ‘y_zero_point’ and ‘y’ to 8-bit integer tensor.