GeGLU¶
GeGLU - 29¶
Version¶
name: GeGLU (GitHub)
domain:
mainsince_version:
29function:
Truesupport_level:
SupportType.COMMONshape inference:
True
This version of the operator has been available since version 29.
Summary¶
GeGLU is a gated activation that takes two inputs, a gate A and a linear (value)
input B, and produces one output Y. It applies the Gelu activation to the gate
and multiplies the result elementwise by the linear input:
Y = Gelu(A) * B
The gate activation is the Gelu operator with the same approximate attribute.
With approximate = "none" (the default) it is the exact form
Gelu(a) = 0.5 * a * (1 + erf(a / sqrt(2)))
and with approximate = "tanh" it is the tanh approximation
Gelu(a) = 0.5 * a * (1 + tanh(sqrt(2 / pi) * (a + 0.044715 * a^3)))
Any other value of approximate is invalid. For float16 and bfloat16 inputs the
function body computes Gelu(A) * B in float32 and casts the result back to the
input type, because Gelu evaluated in float16 loses nearly all precision for
negative gate values.
Inputs A and B must have identical shapes; broadcasting is not applied and the
output Y has the same shape as the inputs. A model whose A and B shapes
differ at runtime is invalid, even though the Mul in the function body would
broadcast them.
Exporters typically produce A and B in one of two ways: for the common
two-projection form wire the two projection outputs directly to A (gate) and B
(value); for a fused/packed single projection, split it upstream into A and B
with Split (contiguous layout) or Slice/Gather (interleaved layout).
GeGLU was introduced in “GLU Variants Improve Transformer” (Shazeer, 2020, https://arxiv.org/abs/2002.05202) and is used as the feed-forward gate in models such as T5 v1.1 and Gemma.
Attributes¶
approximate - STRING (default is
none):Gelu approximation algorithm used for the gate:
"none"(default) or"tanh". Any other value is invalid.
Inputs¶
A (heterogeneous) - T:
Gate input tensor
B (heterogeneous) - T:
Linear (value) input tensor
Outputs¶
Y (heterogeneous) - T:
Output tensor
Type Constraints¶
T in (
tensor(bfloat16),tensor(double),tensor(float),tensor(float16)):Constrain input and output types to float tensors.
Examples¶
default¶
import numpy as np
import onnx
node = onnx.helper.make_node(
"GeGLU",
inputs=["a", "b"],
outputs=["y"],
)
a = np.array(
[
[[1.0, -2.0, 3.0], [4.0, -1.0, 0.5]],
[[-3.0, 2.0, -0.5], [-4.0, 0.0, 1.5]],
],
dtype=np.float32,
)
b = np.array(
[
[[0.5, 1.0, -1.0], [2.0, 2.0, -1.0]],
[[0.5, 1.0, 1.5], [-2.0, 3.0, 0.25]],
],
dtype=np.float32,
)
y = np.array(
[
[
[0.4206724, -0.045500264, -2.9959502],
[7.999747, -0.3173105, -0.34573123],
],
[
[-0.002024847, 1.9544997, -0.23140316],
[0.00025336994, 0.0, 0.3499473],
],
],
dtype=np.float32,
)
expect(
node,
inputs=[a, b],
outputs=[y],
name="test_geglu",
opset_imports=[onnx.helper.make_opsetid("", 29)],
)
_approximate_none¶
import numpy as np
import onnx
node = onnx.helper.make_node(
"GeGLU",
inputs=["a", "b"],
outputs=["y"],
approximate="none",
)
a = np.array(
[
[[1.0, -2.0, 3.0], [4.0, -1.0, 0.5]],
[[-3.0, 2.0, -0.5], [-4.0, 0.0, 1.5]],
],
dtype=np.float32,
)
b = np.array(
[
[[0.5, 1.0, -1.0], [2.0, 2.0, -1.0]],
[[0.5, 1.0, 1.5], [-2.0, 3.0, 0.25]],
],
dtype=np.float32,
)
y = np.array(
[
[
[0.4206724, -0.045500264, -2.9959502],
[7.999747, -0.3173105, -0.34573123],
],
[
[-0.002024847, 1.9544997, -0.23140316],
[0.00025336994, 0.0, 0.3499473],
],
],
dtype=np.float32,
)
expect(
node,
inputs=[a, b],
outputs=[y],
name="test_geglu_approximate_none",
opset_imports=[onnx.helper.make_opsetid("", 29)],
)
_tanh¶
import numpy as np
import onnx
node = onnx.helper.make_node(
"GeGLU",
inputs=["a", "b"],
outputs=["y"],
approximate="tanh",
)
y = np.array(
[
[
[0.420596, -0.045402307, -2.9963627],
[7.9998593, -0.31761602, -0.345714],
],
[
[-0.001818696, 1.9545977, -0.23142898],
[0.0001404919, 0.0, 0.34989288],
],
],
dtype=np.float32,
)
a = np.array(
[
[[1.0, -2.0, 3.0], [4.0, -1.0, 0.5]],
[[-3.0, 2.0, -0.5], [-4.0, 0.0, 1.5]],
],
dtype=np.float32,
)
b = np.array(
[
[[0.5, 1.0, -1.0], [2.0, 2.0, -1.0]],
[[0.5, 1.0, 1.5], [-2.0, 3.0, 0.25]],
],
dtype=np.float32,
)
expect(
node,
inputs=[a, b],
outputs=[y],
name="test_geglu_tanh",
opset_imports=[onnx.helper.make_opsetid("", 29)],
)
_float16¶
import numpy as np
import onnx
node = onnx.helper.make_node(
"GeGLU",
inputs=["a", "b"],
outputs=["y"],
)
a = np.array(
[[-3.0, -4.0, -1.0, 2.0], [-0.5, 1.5, -2.5, 0.75]], dtype=np.float16
)
b = np.array([[0.5, 1.0, -1.0, 2.0], [2.0, -1.0, 0.5, 1.0]], dtype=np.float16)
y = np.array(
[
[
-0.002025604248046875,
-0.00012671947479248047,
0.15869140625,
3.908203125,
],
[-0.30859375, -1.3994140625, -0.007762908935546875, 0.580078125],
],
dtype=np.float16,
)
expect(
node,
inputs=[a, b],
outputs=[y],
name="test_geglu_float16",
opset_imports=[onnx.helper.make_opsetid("", 29)],
)
_bfloat16¶
import numpy as np
import onnx
node = onnx.helper.make_node(
"GeGLU",
inputs=["a", "b"],
outputs=["y"],
)
a = np.array(
[[-3.0, -4.0, -1.0, 2.0], [-0.5, 1.5, -2.5, 0.75]], dtype=ml_dtypes.bfloat16
)
b = np.array(
[[0.5, 1.0, -1.0, 2.0], [2.0, -1.0, 0.5, 1.0]], dtype=ml_dtypes.bfloat16
)
y = np.array(
[
[-0.0020294189453125, -0.00012683868408203125, 0.158203125, 3.90625],
[-0.30859375, -1.3984375, -0.00775146484375, 0.578125],
],
dtype=ml_dtypes.bfloat16,
)
expect(
node,
inputs=[a, b],
outputs=[y],
name="test_geglu_bfloat16",
opset_imports=[onnx.helper.make_opsetid("", 29)],
)
_double¶
import numpy as np
import onnx
node = onnx.helper.make_node(
"GeGLU",
inputs=["a", "b"],
outputs=["y"],
)
a = np.array(
[[-3.0, -4.0, -1.0, 2.0], [-0.5, 1.5, -2.5, 0.75]], dtype=np.float64
)
b = np.array([[0.5, 1.0, -1.0, 2.0], [2.0, -1.0, 0.5, 1.0]], dtype=np.float64)
y = np.array(
[
[
-0.002024847047445155,
-0.00012668496733247991,
0.15865525393145707,
3.908999472207283,
],
[
-0.3085375387259869,
-1.399789198096713,
-0.007762081657220199,
0.5800294857173488,
],
],
dtype=np.float64,
)
expect(
node,
inputs=[a, b],
outputs=[y],
name="test_geglu_double",
opset_imports=[onnx.helper.make_opsetid("", 29)],
)