QuantizeLinear

QuantizeLinear - 28

Version

  • name: QuantizeLinear (GitHub)

  • domain: main

  • since_version: 28

  • function: False

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 28.

Summary

The linear quantization operator consumes a high-precision tensor, a scale, and a zero point to compute the low-precision/quantized tensor. The scale factor and zero point must have the same shape, determining the quantization granularity. The quantization formula is y = saturate((x / y_scale) + y_zero_point).

Saturation is done according to:

  • uint16: [0, 65535]

  • int16: [-32768, 32767]

  • uint8: [0, 255]

  • int8: [-128, 127]

  • uint4: [0, 15]

  • int4: [-8, 7]

  • uint2: [0, 3]

  • int2: [-2, 1]

For (x / y_scale), it rounds to the nearest even. Refer to https://en.wikipedia.org/wiki/Rounding for details.

y_zero_point and y must have the same type. y_zero_point is usually not used for quantization to float8 and 4bit types, but the quantization formula remains the same for consistency, and the type of the attribute y_zero_point still determines the quantization type. x and y_scale are allowed to have different types. The type of y_scale determines the precision of the division operation between x and y_scale, unless the precision attribute is specified.

There are three supported quantization granularities, determined by the shape of y_scale. In all cases, y_zero_point must have the same shape as y_scale.

  • Per-tensor (per-layer) quantization: y_scale is a scalar.

  • Per-axis quantization: The scale must be a 1-D tensor, with the length of the quantization axis. For an input shape (D0, ..., Di, ..., Dn) and axis=i, y_scale is a 1-D tensor of length Di.

  • Blocked quantization: The scale’s shape is identical to the input’s shape, except for one dimension, in which blocking is performed. Given x shape (D0, ..., Di, ..., Dn), axis=i, and block size B: y_scale shape is (D0, ..., ceil(Di/B), ..., Dn).

Attributes

  • axis - INT (default is 1):

    (Optional) The axis of the dequantizing dimension of the input tensor. Used only for per-axis and blocked quantization. Negative value means counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(input). When the rank of the input is 1, per-tensor quantization is applied, rendering the axis unnecessary in this scenario.

  • block_size - INT (default is 0):

    (Optional) The size of the quantization block (number of times every scale is replicated). Used only for blocked quantization. The block size is a positive integer. Given x shape (D0, ..., Di, ..., Dn), y_scale shape (S0, ... Si, ...Sn) and axis=i, the accepted range is [ceil(Di/Si), ceil(Di/(Si-1))-1]

  • output_dtype - INT (default is 0):

    (Optional) The output data type. If not supplied, the output data type is inferred from y_zero_point data type (T3). If neither output_dtype nor y_zero_point are supplied, output data type is uint8. If both output_dtype and y_zero_point are specified, output_dtype must be T3.

  • precision - INT (default is 0):

    (Optional) The precision of the division operation between x and y_scale. If not provided, it will be the same as the type of y_scale.

  • saturate - INT (default is 1):

    The parameter defines how the conversion behaves if an input value is out of range of the destination type. It only applies for float 8 quantization (float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz). It is true by default. All cases are fully described in two tables inserted in the operator description. It has no effect for float4e2m1, float6e2m3, or float6e3m2, since those types have no non-saturating (infinity-representable) encoding to fall back to.

Inputs

Between 2 and 3 inputs.

  • x (heterogeneous) - T1:

    N-D full precision Input tensor to be quantized.

  • y_scale (heterogeneous) - T2:

    Scale for doing quantization to get y. For per-tensor/layer quantization the scale is a scalar, for per-axis quantization it is a 1-D Tensor and for blocked quantization it has the same shape as the input, except for one dimension in which blocking is performed.

  • y_zero_point (optional, heterogeneous) - T3:

    Zero point for doing quantization to get y. Shape must match y_scale. Default is uint8 with zero point of 0 if it’s not specified.

Outputs

  • y (heterogeneous) - T3:

    N-D quantized output tensor. It has same shape as input x.

Type Constraints

  • T1 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(int32) ):

    The type of the input ‘x’.

  • T2 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(float8e8m0), tensor(int32) ):

    The type of the input ‘y_scale’.

  • T3 in ( tensor(float4e2m1), tensor(float6e2m3), tensor(float6e3m2), tensor(float8e4m3fn), tensor(float8e4m3fnuz), tensor(float8e5m2), tensor(float8e5m2fnuz), tensor(int16), tensor(int2), tensor(int4), tensor(int8), tensor(uint16), tensor(uint2), tensor(uint4), tensor(uint8) ):

    The type of the input y_zero_point and the output y.

Examples

default

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
)

x = np.array([0, 2, 3, 1000, -254, -1000]).astype(np.float32)
y_scale = np.float32(2)
y_zero_point = np.uint8(128)
y = np.array([128, 129, 130, 255, 1, 0]).astype(np.uint8)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear",
)

_axis

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
)

x = np.array(
    [
        [
            [[-162, 10], [-100, 232], [-20, -50]],
            [[-76, 0], [0, 252], [32, -44]],
            [[245, -485], [-960, -270], [-375, -470]],
        ],
    ],
    dtype=np.float32,
)
y_scale = np.array([2, 4, 5], dtype=np.float32)
y_zero_point = np.array([84, 24, 196], dtype=np.uint8)
y = (x / y_scale.reshape(1, 3, 1, 1) + y_zero_point.reshape(1, 3, 1, 1)).astype(
    np.uint8
)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_axis",
)

_e4m3fn

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
)

x = np.array([0.0, 1.0, 2.0, 100000.0, 200.0]).astype(np.float32)
y_scale = np.float32(2)
y_zero_point = make_tensor("y_zero_point", TensorProto.FLOAT8E4M3FN, [1], [0])
y = make_tensor("y", TensorProto.FLOAT8E4M3FN, [5], [0, 0.5, 1, 448, 96])

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_e4m3fn",
)

_e5m2

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
)

x = np.array([0.0, 1.0, 2.0, 100000.0, 200.0]).astype(np.float32)
y_scale = np.float32(2)
y_zero_point = make_tensor("y_zero_point", TensorProto.FLOAT8E5M2, [1], [0.0])
y = make_tensor("y", TensorProto.FLOAT8E5M2, [5], [0, 0.5, 1, 49152, 96])

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_e5m2",
)

_uint16

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
)

x = np.array(
    [
        0.0,
        -128.0,
        3.0,
        -3.0,
        2.9,
        -2.9,
        3.1,
        -3.1,
        65536.0,
        -65534.0,
        70000.0,
        -70000.0,
    ]
).astype(np.float32)
y_scale = np.float32(2.0)
y_zero_point = np.uint16(32767)
y = np.array(
    [
        32767,
        32703,
        32769,
        32765,
        32768,
        32766,
        32769,
        32765,
        65535,
        0,
        65535,
        0,
    ]
).astype(np.uint16)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_uint16",
)

_int16

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
)

x = np.array(
    [
        0.0,
        -514.0,
        3.0,
        -3.0,
        2.9,
        -2.9,
        3.1,
        -3.1,
        65022.0,
        -66046.0,
        65023.0,
        -66047.0,
        65024.0,
        -66048.0,
        70000.0,
        -70000.0,
    ]
).astype(np.float32)
y_scale = np.float32(2.0)
y_zero_point = np.int16(256)
y = np.array(
    [
        256,
        -1,
        258,
        254,
        257,
        255,
        258,
        254,
        32767,
        -32767,
        32767,
        -32768,
        32767,
        -32768,
        32767,
        -32768,
    ]
).astype(np.int16)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_int16",
)

_uint4

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
    axis=0,
)

x = np.array(
    [
        [0.0, 2.5, 4.8, 8.6],
        [-30, -20, 6, 9],
        [12, 15, 16, 40],
    ]
).astype(np.float32)

y_scale = np.asarray([2.0, 3.0, 4.0], dtype=np.float32)
y_zero_point = make_tensor(
    "y_zero_point", TensorProto.UINT4, y_scale.shape, np.ones_like(y_scale)
)
y = make_tensor(
    "y", TensorProto.UINT4, x.shape, [1, 2, 3, 5, 0, 0, 3, 4, 4, 5, 5, 11]
)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_uint4",
)

_int4

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
    axis=0,
)

x = np.array(
    [
        [0.0, 2.5, 4.8, 8.6],
        [-30, -20, 6, 9],
        [12, 15, 16, 40],
    ]
).astype(np.float32)

y_scale = np.asarray([2.0, 3.0, 4.0], dtype=np.float32)
y_zero_point = make_tensor(
    "y_zero_point", TensorProto.INT4, y_scale.shape, np.ones_like(y_scale)
)
y = make_tensor(
    "y", TensorProto.INT4, x.shape, [1, 2, 3, 5, -8, -6, 3, 4, 4, 5, 5, 7]
)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_int4",
)

_uint2

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
    axis=0,
)

x = np.array(
    [
        [0.0, 2.5, 4.8, 8.6],
        [-2.0, -1.0, 1.0, 3.0],
        [4.0, 5.0, 6.0, 7.0],
    ],
    dtype=np.float32,
)
y_scale = np.asarray([2.0, 3.0, 4.0], dtype=np.float32)
y_zero_point = make_tensor(
    "y_zero_point", TensorProto.UINT2, y_scale.shape, np.zeros_like(y_scale)
)
y = make_tensor(
    "y", TensorProto.UINT2, x.shape, [0, 1, 2, 3, 0, 0, 0, 1, 1, 1, 2, 2]
)
expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_uint2",
)

_int2

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
    axis=0,
)
x = np.array(
    [
        [0.0, 2.5, 4.8, 8.6],
        [-4.0, -3.0, 1.0, 2.0],
        [-0.0, -2.5, -4.8, -8.6],
    ],
    dtype=np.float32,
)
y_scale = np.asarray([2.0, 3.0, 4.0], dtype=np.float32)
y_zero_point = make_tensor(
    "y_zero_point", TensorProto.INT2, y_scale.shape, np.zeros_like(y_scale)
)
y = make_tensor(
    "y", TensorProto.INT2, x.shape, [0, 1, 1, 1, -1, -1, 0, 1, 0, -1, -1, -2]
)
expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_int2",
)

_float4e2m1

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
    axis=0,
)

x = np.array(
    [
        [0.0, 2.5, 4.8, 8.6],
        [-30, -20, 6, 9],
        [-0.0, -2.5, -4.8, -8.6],
    ]
).astype(np.float32)

y_scale = np.asarray([2.0, 3.0, 4.0], dtype=np.float32)
y_zero_point = make_tensor(
    "y_zero_point",
    TensorProto.FLOAT4E2M1,
    y_scale.shape,
    np.zeros_like(y_scale),
)
y = make_tensor(
    "y",
    TensorProto.FLOAT4E2M1,
    x.shape,
    [0, 1, 2, 4, -6, -6, 2, 3, 0, -0.5, -1, -2],
)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_float4e2m1",
)

_blocked_asymmetric

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale", "y_zero_point"],
    outputs=["y"],
    axis=1,
    block_size=2,
)

x = np.array(
    [
        [6.0, 12.0, 50.0, 5.0],
        [1.0, 8.0, 4.0, 5.0],
        [0.0, 20.0, 10.0, 4.0],
    ],
    dtype=np.float32,
)
y_scale = np.array(
    [
        [1.5, 2.5],
        [3.0, 4.9],
        [5.1, 6.9],
    ],
    dtype=np.float32,
)
y_zero_point = np.array(
    [
        [0, 1],
        [1, 0],
        [2, 3],
    ],
    dtype=np.uint8,
)
# x.shape = (3, 4)
# y_scale.shape = (3, 2)
assert y_scale.shape == y_zero_point.shape
block_axis = 1
# The block shape is [x.shape[i] // y_scale.shape[i] for i in range(len(x.shape))] = (1, 2)
assert all(
    x.shape[i] == y_scale.shape[i]
    for i in range(len(x.shape))
    if i != block_axis
)
assert x.shape[block_axis] % y_scale.shape[block_axis] == 0
repeats = x.shape[block_axis] // y_scale.shape[block_axis]

# Create element-wise scale and zero point
y_scale_elementwise = np.repeat(y_scale, repeats=repeats, axis=block_axis)
y_zero_point_elementwise = np.repeat(
    y_zero_point, repeats=repeats, axis=block_axis
)

y = np.rint(x / y_scale_elementwise + y_zero_point_elementwise).astype(np.uint8)

expect(
    node,
    inputs=[x, y_scale, y_zero_point],
    outputs=[y],
    name="test_quantizelinear_blocked_asymmetric",
)

_blocked_symmetric

import numpy as np
import onnx

node = onnx.helper.make_node(
    "QuantizeLinear",
    inputs=["x", "y_scale"],
    outputs=["y"],
    axis=1,
    block_size=2,
    output_dtype=TensorProto.INT16,
)

x = np.array(
    [
        [6.0, -8, -10, 5.0],
        [1.0, 8.0, 4.0, 5.0],
        [0.0, 20.0, 10.0, 4.0],
    ],
    dtype=np.float32,
)

y_scale = np.array(
    [
        [1.5, 2.5],
        [3.0, 4.9],
        [5.1, 6.9],
    ],
    dtype=np.float32,
)

# x.shape = (3, 4)
# y_scale.shape = (3, 2)

block_axis = 1
# The block shape is [x.shape[i] // y_scale.shape[i] for i in range(len(x.shape))] = (1, 2)
assert all(
    x.shape[i] == y_scale.shape[i]
    for i in range(len(x.shape))
    if i != block_axis
)
assert x.shape[block_axis] % y_scale.shape[block_axis] == 0
repeats = x.shape[block_axis] // y_scale.shape[block_axis]

# Create element-wise scale and zero point
y_scale_elementwise = np.repeat(y_scale, repeats=repeats, axis=block_axis)

y_val = np.clip(
    np.rint(x / y_scale_elementwise), a_min=-32768, a_max=32767
).astype(np.int16)
y = make_tensor(
    "y",
    TensorProto.INT16,
    x.shape,
    y_val,
)
expect(
    node,
    inputs=[x, y_scale],
    outputs=[y],
    name="test_quantizelinear_blocked_symmetric",
)

QuantizeLinear - 25

Version

  • name: QuantizeLinear (GitHub)

  • domain: main

  • since_version: 25

  • function: False

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 25.

Summary

The linear quantization operator consumes a high-precision tensor, a scale, and a zero point to compute the low-precision/quantized tensor. The scale factor and zero point must have the same shape, determining the quantization granularity. The quantization formula is y = saturate((x / y_scale) + y_zero_point).

Saturation is done according to:

  • uint16: [0, 65535]

  • int16: [-32768, 32767]

  • uint8: [0, 255]

  • int8: [-128, 127]

  • uint4: [0, 15]

  • int4: [-8, 7]

  • uint2: [0, 3]

  • int2: [-2, 1]

For (x / y_scale), it rounds to the nearest even. Refer to https://en.wikipedia.org/wiki/Rounding for details.

y_zero_point and y must have the same type. y_zero_point is usually not used for quantization to float8 and 4bit types, but the quantization formula remains the same for consistency, and the type of the attribute y_zero_point still determines the quantization type. x and y_scale are allowed to have different types. The type of y_scale determines the precision of the division operation between x and y_scale, unless the precision attribute is specified.

There are three supported quantization granularities, determined by the shape of y_scale. In all cases, y_zero_point must have the same shape as y_scale.

  • Per-tensor (per-layer) quantization: y_scale is a scalar.

  • Per-axis quantization: The scale must be a 1-D tensor, with the length of the quantization axis. For an input shape (D0, ..., Di, ..., Dn) and axis=i, y_scale is a 1-D tensor of length Di.

  • Blocked quantization: The scale’s shape is identical to the input’s shape, except for one dimension, in which blocking is performed. Given x shape (D0, ..., Di, ..., Dn), axis=i, and block size B: y_scale shape is (D0, ..., ceil(Di/B), ..., Dn).

Attributes

  • axis - INT (default is 1):

    (Optional) The axis of the dequantizing dimension of the input tensor. Used only for per-axis and blocked quantization. Negative value means counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(input). When the rank of the input is 1, per-tensor quantization is applied, rendering the axis unnecessary in this scenario.

  • block_size - INT (default is 0):

    (Optional) The size of the quantization block (number of times every scale is replicated). Used only for blocked quantization. The block size is a positive integer. Given x shape (D0, ..., Di, ..., Dn), y_scale shape (S0, ... Si, ...Sn) and axis=i, the accepted range is [ceil(Di/Si), ceil(Di/(Si-1))-1]

  • output_dtype - INT (default is 0):

    (Optional) The output data type. If not supplied, the output data type is inferred from y_zero_point data type (T3). If neither output_dtype nor y_zero_point are supplied, output data type is uint8. If both output_dtype and y_zero_point are specified, output_dtype must be T3.

  • precision - INT (default is 0):

    (Optional) The precision of the division operation between x and y_scale. If not provided, it will be the same as the type of y_scale.

  • saturate - INT (default is 1):

    The parameter defines how the conversion behaves if an input value is out of range of the destination type. It only applies for float 8 quantization (float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz). It is true by default. All cases are fully described in two tables inserted in the operator description.

Inputs

Between 2 and 3 inputs.

  • x (heterogeneous) - T1:

    N-D full precision Input tensor to be quantized.

  • y_scale (heterogeneous) - T2:

    Scale for doing quantization to get y. For per-tensor/layer quantization the scale is a scalar, for per-axis quantization it is a 1-D Tensor and for blocked quantization it has the same shape as the input, except for one dimension in which blocking is performed.

  • y_zero_point (optional, heterogeneous) - T3:

    Zero point for doing quantization to get y. Shape must match y_scale. Default is uint8 with zero point of 0 if it’s not specified.

Outputs

  • y (heterogeneous) - T3:

    N-D quantized output tensor. It has same shape as input x.

Type Constraints

  • T1 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(int32) ):

    The type of the input ‘x’.

  • T2 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(float8e8m0), tensor(int32) ):

    The type of the input ‘y_scale’.

  • T3 in ( tensor(float4e2m1), tensor(float8e4m3fn), tensor(float8e4m3fnuz), tensor(float8e5m2), tensor(float8e5m2fnuz), tensor(int16), tensor(int2), tensor(int4), tensor(int8), tensor(uint16), tensor(uint2), tensor(uint4), tensor(uint8) ):

    The type of the input y_zero_point and the output y.

QuantizeLinear - 24

Version

  • name: QuantizeLinear (GitHub)

  • domain: main

  • since_version: 24

  • function: False

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 24.

Summary

The linear quantization operator consumes a high-precision tensor, a scale, and a zero point to compute the low-precision/quantized tensor. The scale factor and zero point must have the same shape, determining the quantization granularity. The quantization formula is y = saturate((x / y_scale) + y_zero_point).

Saturation is done according to:

  • uint16: [0, 65535]

  • int16: [-32768, 32767]

  • uint8: [0, 255]

  • int8: [-128, 127]

  • uint4: [0, 15]

  • int4: [-8, 7]

For (x / y_scale), it rounds to the nearest even. Refer to https://en.wikipedia.org/wiki/Rounding for details.

y_zero_point and y must have the same type. y_zero_point is usually not used for quantization to float8 and 4bit types, but the quantization formula remains the same for consistency, and the type of the attribute y_zero_point still determines the quantization type. x and y_scale are allowed to have different types. The type of y_scale determines the precision of the division operation between x and y_scale, unless the precision attribute is specified.

There are three supported quantization granularities, determined by the shape of y_scale. In all cases, y_zero_point must have the same shape as y_scale.

  • Per-tensor (per-layer) quantization: y_scale is a scalar.

  • Per-axis quantization: The scale must be a 1-D tensor, with the length of the quantization axis. For an input shape (D0, ..., Di, ..., Dn) and axis=i, y_scale is a 1-D tensor of length Di.

  • Blocked quantization: The scale’s shape is identical to the input’s shape, except for one dimension, in which blocking is performed. Given x shape (D0, ..., Di, ..., Dn), axis=i, and block size B: y_scale shape is (D0, ..., ceil(Di/B), ..., Dn).

Attributes

  • axis - INT (default is 1):

    (Optional) The axis of the dequantizing dimension of the input tensor. Used only for per-axis and blocked quantization. Negative value means counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(input). When the rank of the input is 1, per-tensor quantization is applied, rendering the axis unnecessary in this scenario.

  • block_size - INT (default is 0):

    (Optional) The size of the quantization block (number of times every scale is replicated). Used only for blocked quantization. The block size is a positive integer. Given x shape (D0, ..., Di, ..., Dn), y_scale shape (S0, ... Si, ...Sn) and axis=i, the accepted range is [ceil(Di/Si), ceil(Di/(Si-1))-1]

  • output_dtype - INT (default is 0):

    (Optional) The output data type. If not supplied, the output data type is inferred from y_zero_point data type (T3). If neither output_dtype nor y_zero_point are supplied, output data type is uint8. If both output_dtype and y_zero_point are specified, output_dtype must be T3.

  • precision - INT (default is 0):

    (Optional) The precision of the division operation between x and y_scale. If not provided, it will be the same as the type of y_scale.

  • saturate - INT (default is 1):

    The parameter defines how the conversion behaves if an input value is out of range of the destination type. It only applies for float 8 quantization (float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz). It is true by default. All cases are fully described in two tables inserted in the operator description.

Inputs

Between 2 and 3 inputs.

  • x (heterogeneous) - T1:

    N-D full precision Input tensor to be quantized.

  • y_scale (heterogeneous) - T2:

    Scale for doing quantization to get y. For per-tensor/layer quantization the scale is a scalar, for per-axis quantization it is a 1-D Tensor and for blocked quantization it has the same shape as the input, except for one dimension in which blocking is performed.

  • y_zero_point (optional, heterogeneous) - T3:

    Zero point for doing quantization to get y. Shape must match y_scale. Default is uint8 with zero point of 0 if it’s not specified.

Outputs

  • y (heterogeneous) - T3:

    N-D quantized output tensor. It has same shape as input x.

Type Constraints

  • T1 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(int32) ):

    The type of the input ‘x’.

  • T2 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(float8e8m0), tensor(int32) ):

    The type of the input ‘y_scale’.

  • T3 in ( tensor(float4e2m1), tensor(float8e4m3fn), tensor(float8e4m3fnuz), tensor(float8e5m2), tensor(float8e5m2fnuz), tensor(int16), tensor(int4), tensor(int8), tensor(uint16), tensor(uint4), tensor(uint8) ):

    The type of the input y_zero_point and the output y.

QuantizeLinear - 23

Version

  • name: QuantizeLinear (GitHub)

  • domain: main

  • since_version: 23

  • function: False

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 23.

Summary

The linear quantization operator consumes a high-precision tensor, a scale, and a zero point to compute the low-precision/quantized tensor. The scale factor and zero point must have the same shape, determining the quantization granularity. The quantization formula is y = saturate((x / y_scale) + y_zero_point).

Saturation is done according to:

  • uint16: [0, 65535]

  • int16: [-32768, 32767]

  • uint8: [0, 255]

  • int8: [-128, 127]

  • uint4: [0, 15]

  • int4: [-8, 7]

For (x / y_scale), it rounds to the nearest even. Refer to https://en.wikipedia.org/wiki/Rounding for details.

y_zero_point and y must have the same type. y_zero_point is usually not used for quantization to float8 and 4bit types, but the quantization formula remains the same for consistency, and the type of the attribute y_zero_point still determines the quantization type. x and y_scale are allowed to have different types. The type of y_scale determines the precision of the division operation between x and y_scale, unless the precision attribute is specified.

There are three supported quantization granularities, determined by the shape of y_scale. In all cases, y_zero_point must have the same shape as y_scale.

  • Per-tensor (per-layer) quantization: y_scale is a scalar.

  • Per-axis quantization: The scale must be a 1-D tensor, with the length of the quantization axis. For an input shape (D0, ..., Di, ..., Dn) and axis=i, y_scale is a 1-D tensor of length Di.

  • Blocked quantization: The scale’s shape is identical to the input’s shape, except for one dimension, in which blocking is performed. Given x shape (D0, ..., Di, ..., Dn), axis=i, and block size B: y_scale shape is (D0, ..., ceil(Di/B), ..., Dn).

Attributes

  • axis - INT (default is 1):

    (Optional) The axis of the dequantizing dimension of the input tensor. Used only for per-axis and blocked quantization. Negative value means counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(input). When the rank of the input is 1, per-tensor quantization is applied, rendering the axis unnecessary in this scenario.

  • block_size - INT (default is 0):

    (Optional) The size of the quantization block (number of times every scale is replicated). Used only for blocked quantization. The block size is a positive integer. Given x shape (D0, ..., Di, ..., Dn), y_scale shape (S0, ... Si, ...Sn) and axis=i, the accepted range is [ceil(Di/Si), ceil(Di/(Si-1))-1]

  • output_dtype - INT (default is 0):

    (Optional) The output data type. If not supplied, the output data type is inferred from y_zero_point data type (T3). If neither output_dtype nor y_zero_point are supplied, output data type is uint8. If both output_dtype and y_zero_point are specified, output_dtype must be T3.

  • precision - INT (default is 0):

    (Optional) The precision of the division operation between x and y_scale. If not provided, it will be the same as the type of y_scale.

  • saturate - INT (default is 1):

    The parameter defines how the conversion behaves if an input value is out of range of the destination type. It only applies for float 8 quantization (float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz). It is true by default. All cases are fully described in two tables inserted in the operator description.

Inputs

Between 2 and 3 inputs.

  • x (heterogeneous) - T1:

    N-D full precision Input tensor to be quantized.

  • y_scale (heterogeneous) - T2:

    Scale for doing quantization to get y. For per-tensor/layer quantization the scale is a scalar, for per-axis quantization it is a 1-D Tensor and for blocked quantization it has the same shape as the input, except for one dimension in which blocking is performed.

  • y_zero_point (optional, heterogeneous) - T3:

    Zero point for doing quantization to get y. Shape must match y_scale.Default is uint8 with zero point of 0 if it’s not specified.

Outputs

  • y (heterogeneous) - T3:

    N-D quantized output tensor. It has same shape as input x.

Type Constraints

  • T1 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(int32) ):

    The type of the input ‘x’.

  • T2 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(int32) ):

    The type of the input ‘y_scale’.

  • T3 in ( tensor(float4e2m1), tensor(float8e4m3fn), tensor(float8e4m3fnuz), tensor(float8e5m2), tensor(float8e5m2fnuz), tensor(int16), tensor(int4), tensor(int8), tensor(uint16), tensor(uint4), tensor(uint8) ):

    The type of the input y_zero_point and the output y.

QuantizeLinear - 21

Version

  • name: QuantizeLinear (GitHub)

  • domain: main

  • since_version: 21

  • function: False

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 21.

Summary

The linear quantization operator consumes a high-precision tensor, a scale, and a zero point to compute the low-precision/quantized tensor. The scale factor and zero point must have the same shape, determining the quantization granularity. The quantization formula is y = saturate((x / y_scale) + y_zero_point). Saturation is done according to:

  • uint16: [0, 65535]

  • int16: [-32768, 32767]

  • uint8: [0, 255]

  • int8: [-128, 127]

  • uint4: [0, 15]

  • int4: [-8, 7] For (x / y_scale), it rounds to the nearest even. Refer to https://en.wikipedia.org/wiki/Rounding for details. y_zero_point and y must have the same type. y_zero_point is usually not used for quantization to float8 types, but the quantization formula remains the same for consistency, and the type of the attribute y_zero_point still determines the quantization type. There are three supported quantization granularities, determined by the shape of y_scale. In all cases, y_zero_point must have the same shape as y_scale.

  • Per-tensor (per-layer) quantization: y_scale is a scalar.

  • Per-axis quantization: The scale must be a 1-D tensor, with the length of the quantization axis. For an input shape (D0, ..., Di, ..., Dn) and axis=i, y_scale is a 1-D tensor of length Di.

  • Blocked quantization: The scale’s shape is identical to the input’s shape, except for one dimension, in which blocking is performed. Given x shape (D0, ..., Di, ..., Dn), axis=i, and block size B: y_scale shape is (D0, ..., ceil(Di/B), ..., Dn).

Attributes

  • axis - INT (default is 1):

    (Optional) The axis of the dequantizing dimension of the input tensor. Used only for per-axis and blocked quantization. Negative value means counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(input). When the rank of the input is 1, per-tensor quantization is applied, rendering the axis unnecessary in this scenario.

  • block_size - INT (default is 0):

    (Optional) The size of the quantization block (number of times every scale is replicated). Used only for blocked quantization. The block size is a positive integer. Given x shape (D0, ..., Di, ..., Dn), y_scale shape (S0, ... Si, ...Sn) and axis=i, the accepted range is [ceil(Di/Si), ceil(Di/(Si-1))-1]

  • output_dtype - INT (default is 0):

    (Optional) The output data type. If not supplied, the output data type is inferred from y_zero_point data type (T2). If neither output_dtype nor y_zero_point are supplied, output data type is uint8. If both output_dtype and y_zero_point are specified, output_dtype must be T2.

  • saturate - INT (default is 1):

    The parameter defines how the conversion behaves if an input value is out of range of the destination type. It only applies for float 8 quantization (float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz). It is true by default. All cases are fully described in two tables inserted in the operator description.

Inputs

Between 2 and 3 inputs.

  • x (heterogeneous) - T1:

    N-D full precision Input tensor to be quantized.

  • y_scale (heterogeneous) - T1:

    Scale for doing quantization to get y. For per-tensor/layer quantization the scale is a scalar, for per-axis quantization it is a 1-D Tensor and for blocked quantization it has the same shape as the input, except for one dimension in which blocking is performed.

  • y_zero_point (optional, heterogeneous) - T2:

    Zero point for doing quantization to get y. Shape must match y_scale.Default is uint8 with zero point of 0 if it’s not specified.

Outputs

  • y (heterogeneous) - T2:

    N-D quantized output tensor. It has same shape as input x.

Type Constraints

  • T1 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(int32) ):

    The type of the input ‘x’.

  • T2 in ( tensor(float8e4m3fn), tensor(float8e4m3fnuz), tensor(float8e5m2), tensor(float8e5m2fnuz), tensor(int16), tensor(int4), tensor(int8), tensor(uint16), tensor(uint4), tensor(uint8) ):

    The type of the input y_zero_point and the output y.

QuantizeLinear - 19

Version

  • name: QuantizeLinear (GitHub)

  • domain: main

  • since_version: 19

  • function: False

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 19.

Summary

The linear quantization operator. It consumes a high precision tensor, a scale, and a zero point to compute the low precision / quantized tensor. The scale factor and zero point must have same shape, and can be either a scalar for per-tensor / per layer quantization, or a 1-D tensor for per-axis quantization. The quantization formula is y = saturate ((x / y_scale) + y_zero_point). For saturation, it saturates to [0, 255] if it’s uint8, or [-128, 127] if it’s int8. For (x / y_scale), it’s rounding to the nearest even. Refer to https://en.wikipedia.org/wiki/Rounding for details. ‘y_zero_point’ and ‘y’ must have same type. ‘y_zero_point’ is usually not used for quantization to float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz, but the quantization formula remains the same for consistency and the type of the attribute ‘y_zero_point’ still determines the quantization type.

Attributes

  • axis - INT (default is 1):

    (Optional) The axis of the quantization dimension of the input tensor. Ignored for per-tensor quantization. Negative value means counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(input).

  • saturate - INT (default is 1):

    The parameter defines how the conversion behaves if an input value is out of range of the destination type. It only applies for float 8 quantization (float8e4m3fn, float8e4m3fnuz, float8e5m2, float8e5m2fnuz). It is true by default. All cases are fully described in two tables inserted in the operator description.

Inputs

Between 2 and 3 inputs.

  • x (heterogeneous) - T1:

    N-D full precision Input tensor to be quantized.

  • y_scale (heterogeneous) - T1:

    Scale for doing quantization to get ‘y’. It can be a scalar, which means per-tensor/layer quantization, or a 1-D Tensor for per-axis quantization.

  • y_zero_point (optional, heterogeneous) - T2:

    Zero point for doing quantization to get ‘y’. Shape must match y_scale. Default is uint8 with zero point of 0 if it’s not specified.

Outputs

  • y (heterogeneous) - T2:

    N-D quantized output tensor. It has same shape as input ‘x’.

Type Constraints

  • T1 in ( tensor(bfloat16), tensor(float), tensor(float16), tensor(int32) ):

    Constrain ‘x’ to float, float16, bfloat16 or int32 tensor.

  • T2 in ( tensor(float8e4m3fn), tensor(float8e4m3fnuz), tensor(float8e5m2), tensor(float8e5m2fnuz), tensor(int8), tensor(uint8) ):

    Constrain ‘y_zero_point’ and ‘y’ to 8-bit integer/float tensor.

QuantizeLinear - 13

Version

  • name: QuantizeLinear (GitHub)

  • domain: main

  • since_version: 13

  • function: False

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 13.

Summary

The linear quantization operator. It consumes a high precision tensor, a scale, and a zero point to compute the low precision / quantized tensor. The scale factor and zero point must have same shape, and can be either a scalar for per-tensor / per layer quantization, or a 1-D tensor for per-axis quantization. The quantization formula is y = saturate ((x / y_scale) + y_zero_point). For saturation, it saturates to [0, 255] if it’s uint8, or [-128, 127] if it’s int8. For (x / y_scale), it’s rounding to the nearest even. Refer to https://en.wikipedia.org/wiki/Rounding for details. ‘y_zero_point’ and ‘y’ must have same type.

Attributes

  • axis - INT (default is 1):

    (Optional) The axis of the quantization dimension of the input tensor. Ignored for per-tensor quantization. Negative value means counting dimensions from the back. Accepted range is [-r, r-1] where r = rank(input).

Inputs

Between 2 and 3 inputs.

  • x (heterogeneous) - T1:

    N-D full precision Input tensor to be quantized.

  • y_scale (heterogeneous) - tensor(float):

    Scale for doing quantization to get ‘y’. It can be a scalar, which means per-tensor/layer quantization, or a 1-D Tensor for per-axis quantization.

  • y_zero_point (optional, heterogeneous) - T2:

    Zero point for doing quantization to get ‘y’. Shape must match y_scale. Default is uint8 with zero point of 0 if it’s not specified.

Outputs

  • y (heterogeneous) - T2:

    N-D quantized output tensor. It has same shape as input ‘x’.

Type Constraints

  • T1 in ( tensor(float), tensor(int32) ):

    Constrain ‘x’ to float or int32 tensor.

  • T2 in ( tensor(int8), tensor(uint8) ):

    Constrain ‘y_zero_point’ and ‘y’ to 8-bit integer tensor.

QuantizeLinear - 10

Version

  • name: QuantizeLinear (GitHub)

  • domain: main

  • since_version: 10

  • function: False

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 10.

Summary

The linear per-tensor/layer quantization operator. It consumes a high precision tensor, a scale, a zero point to compute the low precision / quantized tensor. The quantization formula is y = saturate ((x / y_scale) + y_zero_point). For saturation, it saturates to [0, 255] if it’s uint8, or [-128, 127] if it’s int8. For (x / y_scale), it’s rounding to the nearest even. Refer to https://en.wikipedia.org/wiki/Rounding for details. ‘y_zero_point’ and ‘y’ must have same type.

Inputs

Between 2 and 3 inputs.

  • x (heterogeneous) - T1:

    N-D full precision Input tensor to be quantized.

  • y_scale (heterogeneous) - tensor(float):

    Scale for doing quantization to get ‘y’. It’s a scalar, which means a per-tensor/layer quantization.

  • y_zero_point (optional, heterogeneous) - T2:

    Zero point for doing quantization to get ‘y’. It’s a scalar, which means a per-tensor/layer quantization. Default value is uint8 typed 0 if it’s not specified.

Outputs

  • y (heterogeneous) - T2:

    N-D quantized output tensor. It has same shape as input ‘x’.

Type Constraints

  • T1 in ( tensor(float), tensor(int32) ):

    Constrain ‘x’ to float or int32 tensor.

  • T2 in ( tensor(int8), tensor(uint8) ):

    Constrain ‘y_zero_point’ and ‘y’ to 8-bit integer tensor.