(l-onnx-doc-GeGLU)= # GeGLU (l-onnx-op-geglu-29)= ## GeGLU - 29 ### Version - **name**: [GeGLU (GitHub)](https://github.com/onnx/onnx/blob/main/docs/Operators.md#GeGLU) - **domain**: `main` - **since_version**: `29` - **function**: `True` - **support_level**: `SupportType.COMMON` - **shape inference**: `True` This version of the operator has been available **since version 29**. ### Summary GeGLU is a gated activation that takes two inputs, a gate `A` and a linear (value) input `B`, and produces one output `Y`. It applies the Gelu activation to the gate and multiplies the result elementwise by the linear input: ``` Y = Gelu(A) * B ``` The gate activation is the `Gelu` operator with the same `approximate` attribute. With `approximate = "none"` (the default) it is the exact form ``` Gelu(a) = 0.5 * a * (1 + erf(a / sqrt(2))) ``` and with `approximate = "tanh"` it is the tanh approximation ``` Gelu(a) = 0.5 * a * (1 + tanh(sqrt(2 / pi) * (a + 0.044715 * a^3))) ``` Any other value of `approximate` is invalid. For float16 and bfloat16 inputs the function body computes `Gelu(A) * B` in float32 and casts the result back to the input type, because Gelu evaluated in float16 loses nearly all precision for negative gate values. Inputs `A` and `B` must have identical shapes; broadcasting is not applied and the output `Y` has the same shape as the inputs. A model whose `A` and `B` shapes differ at runtime is invalid, even though the `Mul` in the function body would broadcast them. Exporters typically produce `A` and `B` in one of two ways: for the common two-projection form wire the two projection outputs directly to `A` (gate) and `B` (value); for a fused/packed single projection, split it upstream into `A` and `B` with `Split` (contiguous layout) or `Slice`/`Gather` (interleaved layout). GeGLU was introduced in "GLU Variants Improve Transformer" (Shazeer, 2020, https://arxiv.org/abs/2002.05202) and is used as the feed-forward gate in models such as T5 v1.1 and Gemma. ### Attributes * **approximate - STRING** (default is `none`): Gelu approximation algorithm used for the gate: `"none"` (default) or `"tanh"`. Any other value is invalid. ### Inputs - **A** (heterogeneous) - **T**: Gate input tensor - **B** (heterogeneous) - **T**: Linear (value) input tensor ### Outputs - **Y** (heterogeneous) - **T**: Output tensor ### Type Constraints * **T** in ( `tensor(bfloat16)`, `tensor(double)`, `tensor(float)`, `tensor(float16)` ): Constrain input and output types to float tensors. ### Examples #### default ```python import numpy as np import onnx node = onnx.helper.make_node( "GeGLU", inputs=["a", "b"], outputs=["y"], ) a = np.array( [ [[1.0, -2.0, 3.0], [4.0, -1.0, 0.5]], [[-3.0, 2.0, -0.5], [-4.0, 0.0, 1.5]], ], dtype=np.float32, ) b = np.array( [ [[0.5, 1.0, -1.0], [2.0, 2.0, -1.0]], [[0.5, 1.0, 1.5], [-2.0, 3.0, 0.25]], ], dtype=np.float32, ) y = np.array( [ [ [0.4206724, -0.045500264, -2.9959502], [7.999747, -0.3173105, -0.34573123], ], [ [-0.002024847, 1.9544997, -0.23140316], [0.00025336994, 0.0, 0.3499473], ], ], dtype=np.float32, ) expect( node, inputs=[a, b], outputs=[y], name="test_geglu", opset_imports=[onnx.helper.make_opsetid("", 29)], ) ``` #### _approximate_none ```python import numpy as np import onnx node = onnx.helper.make_node( "GeGLU", inputs=["a", "b"], outputs=["y"], approximate="none", ) a = np.array( [ [[1.0, -2.0, 3.0], [4.0, -1.0, 0.5]], [[-3.0, 2.0, -0.5], [-4.0, 0.0, 1.5]], ], dtype=np.float32, ) b = np.array( [ [[0.5, 1.0, -1.0], [2.0, 2.0, -1.0]], [[0.5, 1.0, 1.5], [-2.0, 3.0, 0.25]], ], dtype=np.float32, ) y = np.array( [ [ [0.4206724, -0.045500264, -2.9959502], [7.999747, -0.3173105, -0.34573123], ], [ [-0.002024847, 1.9544997, -0.23140316], [0.00025336994, 0.0, 0.3499473], ], ], dtype=np.float32, ) expect( node, inputs=[a, b], outputs=[y], name="test_geglu_approximate_none", opset_imports=[onnx.helper.make_opsetid("", 29)], ) ``` #### _tanh ```python import numpy as np import onnx node = onnx.helper.make_node( "GeGLU", inputs=["a", "b"], outputs=["y"], approximate="tanh", ) y = np.array( [ [ [0.420596, -0.045402307, -2.9963627], [7.9998593, -0.31761602, -0.345714], ], [ [-0.001818696, 1.9545977, -0.23142898], [0.0001404919, 0.0, 0.34989288], ], ], dtype=np.float32, ) a = np.array( [ [[1.0, -2.0, 3.0], [4.0, -1.0, 0.5]], [[-3.0, 2.0, -0.5], [-4.0, 0.0, 1.5]], ], dtype=np.float32, ) b = np.array( [ [[0.5, 1.0, -1.0], [2.0, 2.0, -1.0]], [[0.5, 1.0, 1.5], [-2.0, 3.0, 0.25]], ], dtype=np.float32, ) expect( node, inputs=[a, b], outputs=[y], name="test_geglu_tanh", opset_imports=[onnx.helper.make_opsetid("", 29)], ) ``` #### _float16 ```python import numpy as np import onnx node = onnx.helper.make_node( "GeGLU", inputs=["a", "b"], outputs=["y"], ) a = np.array( [[-3.0, -4.0, -1.0, 2.0], [-0.5, 1.5, -2.5, 0.75]], dtype=np.float16 ) b = np.array([[0.5, 1.0, -1.0, 2.0], [2.0, -1.0, 0.5, 1.0]], dtype=np.float16) y = np.array( [ [ -0.002025604248046875, -0.00012671947479248047, 0.15869140625, 3.908203125, ], [-0.30859375, -1.3994140625, -0.007762908935546875, 0.580078125], ], dtype=np.float16, ) expect( node, inputs=[a, b], outputs=[y], name="test_geglu_float16", opset_imports=[onnx.helper.make_opsetid("", 29)], ) ``` #### _bfloat16 ```python import numpy as np import onnx node = onnx.helper.make_node( "GeGLU", inputs=["a", "b"], outputs=["y"], ) a = np.array( [[-3.0, -4.0, -1.0, 2.0], [-0.5, 1.5, -2.5, 0.75]], dtype=ml_dtypes.bfloat16 ) b = np.array( [[0.5, 1.0, -1.0, 2.0], [2.0, -1.0, 0.5, 1.0]], dtype=ml_dtypes.bfloat16 ) y = np.array( [ [-0.0020294189453125, -0.00012683868408203125, 0.158203125, 3.90625], [-0.30859375, -1.3984375, -0.00775146484375, 0.578125], ], dtype=ml_dtypes.bfloat16, ) expect( node, inputs=[a, b], outputs=[y], name="test_geglu_bfloat16", opset_imports=[onnx.helper.make_opsetid("", 29)], ) ``` #### _double ```python import numpy as np import onnx node = onnx.helper.make_node( "GeGLU", inputs=["a", "b"], outputs=["y"], ) a = np.array( [[-3.0, -4.0, -1.0, 2.0], [-0.5, 1.5, -2.5, 0.75]], dtype=np.float64 ) b = np.array([[0.5, 1.0, -1.0, 2.0], [2.0, -1.0, 0.5, 1.0]], dtype=np.float64) y = np.array( [ [ -0.002024847047445155, -0.00012668496733247991, 0.15865525393145707, 3.908999472207283, ], [ -0.3085375387259869, -1.399789198096713, -0.007762081657220199, 0.5800294857173488, ], ], dtype=np.float64, ) expect( node, inputs=[a, b], outputs=[y], name="test_geglu_double", opset_imports=[onnx.helper.make_opsetid("", 29)], ) ```