GeGLU

GeGLU - 29

Version

  • name: GeGLU (GitHub)

  • domain: main

  • since_version: 29

  • function: True

  • support_level: SupportType.COMMON

  • shape inference: True

This version of the operator has been available since version 29.

Summary

GeGLU is a gated activation that takes two inputs, a gate A and a linear (value) input B, and produces one output Y. It applies the Gelu activation to the gate and multiplies the result elementwise by the linear input:

Y = Gelu(A) * B

The gate activation is the Gelu operator with the same approximate attribute. With approximate = "none" (the default) it is the exact form

Gelu(a) = 0.5 * a * (1 + erf(a / sqrt(2)))

and with approximate = "tanh" it is the tanh approximation

Gelu(a) = 0.5 * a * (1 + tanh(sqrt(2 / pi) * (a + 0.044715 * a^3)))

Any other value of approximate is invalid. For float16 and bfloat16 inputs the function body computes Gelu(A) * B in float32 and casts the result back to the input type, because Gelu evaluated in float16 loses nearly all precision for negative gate values.

Inputs A and B must have identical shapes; broadcasting is not applied and the output Y has the same shape as the inputs. A model whose A and B shapes differ at runtime is invalid, even though the Mul in the function body would broadcast them.

Exporters typically produce A and B in one of two ways: for the common two-projection form wire the two projection outputs directly to A (gate) and B (value); for a fused/packed single projection, split it upstream into A and B with Split (contiguous layout) or Slice/Gather (interleaved layout).

GeGLU was introduced in “GLU Variants Improve Transformer” (Shazeer, 2020, https://arxiv.org/abs/2002.05202) and is used as the feed-forward gate in models such as T5 v1.1 and Gemma.

Attributes

  • approximate - STRING (default is none):

    Gelu approximation algorithm used for the gate: "none" (default) or "tanh". Any other value is invalid.

Inputs

  • A (heterogeneous) - T:

    Gate input tensor

  • B (heterogeneous) - T:

    Linear (value) input tensor

Outputs

  • Y (heterogeneous) - T:

    Output tensor

Type Constraints

  • T in ( tensor(bfloat16), tensor(double), tensor(float), tensor(float16) ):

    Constrain input and output types to float tensors.

Examples

default

import numpy as np
import onnx

node = onnx.helper.make_node(
    "GeGLU",
    inputs=["a", "b"],
    outputs=["y"],
)
a = np.array(
    [
        [[1.0, -2.0, 3.0], [4.0, -1.0, 0.5]],
        [[-3.0, 2.0, -0.5], [-4.0, 0.0, 1.5]],
    ],
    dtype=np.float32,
)
b = np.array(
    [
        [[0.5, 1.0, -1.0], [2.0, 2.0, -1.0]],
        [[0.5, 1.0, 1.5], [-2.0, 3.0, 0.25]],
    ],
    dtype=np.float32,
)
y = np.array(
    [
        [
            [0.4206724, -0.045500264, -2.9959502],
            [7.999747, -0.3173105, -0.34573123],
        ],
        [
            [-0.002024847, 1.9544997, -0.23140316],
            [0.00025336994, 0.0, 0.3499473],
        ],
    ],
    dtype=np.float32,
)
expect(
    node,
    inputs=[a, b],
    outputs=[y],
    name="test_geglu",
    opset_imports=[onnx.helper.make_opsetid("", 29)],
)

_approximate_none

import numpy as np
import onnx

node = onnx.helper.make_node(
    "GeGLU",
    inputs=["a", "b"],
    outputs=["y"],
    approximate="none",
)
a = np.array(
    [
        [[1.0, -2.0, 3.0], [4.0, -1.0, 0.5]],
        [[-3.0, 2.0, -0.5], [-4.0, 0.0, 1.5]],
    ],
    dtype=np.float32,
)
b = np.array(
    [
        [[0.5, 1.0, -1.0], [2.0, 2.0, -1.0]],
        [[0.5, 1.0, 1.5], [-2.0, 3.0, 0.25]],
    ],
    dtype=np.float32,
)
y = np.array(
    [
        [
            [0.4206724, -0.045500264, -2.9959502],
            [7.999747, -0.3173105, -0.34573123],
        ],
        [
            [-0.002024847, 1.9544997, -0.23140316],
            [0.00025336994, 0.0, 0.3499473],
        ],
    ],
    dtype=np.float32,
)
expect(
    node,
    inputs=[a, b],
    outputs=[y],
    name="test_geglu_approximate_none",
    opset_imports=[onnx.helper.make_opsetid("", 29)],
)

_tanh

import numpy as np
import onnx

node = onnx.helper.make_node(
    "GeGLU",
    inputs=["a", "b"],
    outputs=["y"],
    approximate="tanh",
)
y = np.array(
    [
        [
            [0.420596, -0.045402307, -2.9963627],
            [7.9998593, -0.31761602, -0.345714],
        ],
        [
            [-0.001818696, 1.9545977, -0.23142898],
            [0.0001404919, 0.0, 0.34989288],
        ],
    ],
    dtype=np.float32,
)
a = np.array(
    [
        [[1.0, -2.0, 3.0], [4.0, -1.0, 0.5]],
        [[-3.0, 2.0, -0.5], [-4.0, 0.0, 1.5]],
    ],
    dtype=np.float32,
)
b = np.array(
    [
        [[0.5, 1.0, -1.0], [2.0, 2.0, -1.0]],
        [[0.5, 1.0, 1.5], [-2.0, 3.0, 0.25]],
    ],
    dtype=np.float32,
)
expect(
    node,
    inputs=[a, b],
    outputs=[y],
    name="test_geglu_tanh",
    opset_imports=[onnx.helper.make_opsetid("", 29)],
)

_float16

import numpy as np
import onnx

node = onnx.helper.make_node(
    "GeGLU",
    inputs=["a", "b"],
    outputs=["y"],
)
a = np.array(
    [[-3.0, -4.0, -1.0, 2.0], [-0.5, 1.5, -2.5, 0.75]], dtype=np.float16
)
b = np.array([[0.5, 1.0, -1.0, 2.0], [2.0, -1.0, 0.5, 1.0]], dtype=np.float16)
y = np.array(
    [
        [
            -0.002025604248046875,
            -0.00012671947479248047,
            0.15869140625,
            3.908203125,
        ],
        [-0.30859375, -1.3994140625, -0.007762908935546875, 0.580078125],
    ],
    dtype=np.float16,
)
expect(
    node,
    inputs=[a, b],
    outputs=[y],
    name="test_geglu_float16",
    opset_imports=[onnx.helper.make_opsetid("", 29)],
)

_bfloat16

import numpy as np
import onnx

node = onnx.helper.make_node(
    "GeGLU",
    inputs=["a", "b"],
    outputs=["y"],
)
a = np.array(
    [[-3.0, -4.0, -1.0, 2.0], [-0.5, 1.5, -2.5, 0.75]], dtype=ml_dtypes.bfloat16
)
b = np.array(
    [[0.5, 1.0, -1.0, 2.0], [2.0, -1.0, 0.5, 1.0]], dtype=ml_dtypes.bfloat16
)
y = np.array(
    [
        [-0.0020294189453125, -0.00012683868408203125, 0.158203125, 3.90625],
        [-0.30859375, -1.3984375, -0.00775146484375, 0.578125],
    ],
    dtype=ml_dtypes.bfloat16,
)
expect(
    node,
    inputs=[a, b],
    outputs=[y],
    name="test_geglu_bfloat16",
    opset_imports=[onnx.helper.make_opsetid("", 29)],
)

_double

import numpy as np
import onnx

node = onnx.helper.make_node(
    "GeGLU",
    inputs=["a", "b"],
    outputs=["y"],
)
a = np.array(
    [[-3.0, -4.0, -1.0, 2.0], [-0.5, 1.5, -2.5, 0.75]], dtype=np.float64
)
b = np.array([[0.5, 1.0, -1.0, 2.0], [2.0, -1.0, 0.5, 1.0]], dtype=np.float64)
y = np.array(
    [
        [
            -0.002024847047445155,
            -0.00012668496733247991,
            0.15865525393145707,
            3.908999472207283,
        ],
        [
            -0.3085375387259869,
            -1.399789198096713,
            -0.007762081657220199,
            0.5800294857173488,
        ],
    ],
    dtype=np.float64,
)
expect(
    node,
    inputs=[a, b],
    outputs=[y],
    name="test_geglu_double",
    opset_imports=[onnx.helper.make_opsetid("", 29)],
)