SwiGLU¶

SwiGLU - 28¶

Version¶

name: SwiGLU (GitHub)
domain: main
since_version: 28
function: True
support_level: SupportType.COMMON
shape inference: True

This version of the operator has been available since version 28.

Summary¶

SwiGLU is a gated activation that takes two inputs, a gate A and a linear (value) input B, and produces one output Y. It applies the Swish activation to the gate and multiplies the result elementwise by the linear input:

Y = Swish_alpha(A) * B

The gate activation Swish_alpha is exactly the Swish operator with the same alpha, i.e. Swish_alpha(a) = a * Sigmoid(alpha * a). Inputs A and B must have identical shapes; broadcasting is not applied and the output Y has the same shape as the inputs.

Exporters typically produce A and B in one of two ways: for the common two-projection form (e.g. Llama’s gate_proj/up_proj) wire the two projection outputs directly to A (gate) and B (value); for a fused/packed single projection, split it upstream into A and B with Split (contiguous layout) or Slice/Gather (interleaved layout).

Function Body¶

The function definition for this operator.

&lt;
  domain: &#34;&#34;,
  opset_import: [&#34;&#34; : 28]
&gt;
SwiGLU &lt;alpha&gt;(A, B) =&gt; (Y)
{
   SwishGate = Swish &lt;alpha: float = @alpha&gt; (A)
   Y = Mul (SwishGate, B)
}

Attributes¶

alpha - FLOAT (default is 1.0):

Coefficient that scales the gate input inside the sigmoid of the Swish activation. The default value is 1.0.

Inputs¶

A (heterogeneous) - T:

Gate input tensor
B (heterogeneous) - T:

Linear (value) input tensor

Outputs¶

Y (heterogeneous) - T:

Output tensor

Type Constraints¶

T in ( tensor(bfloat16), tensor(double), tensor(float), tensor(float16) ):

Constrain input and output types to float tensors.

Examples¶

default¶

import numpy as np
import onnx

node = onnx.helper.make_node(
    "SwiGLU",
    inputs=["a", "b"],
    outputs=["y"],
)

a = np.array([[1.0, -2.0, 3.0, 4.0], [-1.0, 2.0, -3.0, 0.5]], dtype=np.float32)
b = np.array([[0.5, 1.0, -1.0, 2.0], [2.0, -1.0, 0.5, 1.0]], dtype=np.float32)
y = swiglu(a, b, alpha=1.0)

expect(
    node,
    inputs=[a, b],
    outputs=[y],
    name="test_swiglu",
    opset_imports=[onnx.helper.make_opsetid("", 28)],
)

_alpha¶

import numpy as np
import onnx

node = onnx.helper.make_node(
    "SwiGLU",
    inputs=["a", "b"],
    outputs=["y"],
    alpha=0.5,  # pass alpha as attribute
)

a = np.array([[1.0, -2.0, 3.0, 4.0], [-1.0, 2.0, -3.0, 0.5]], dtype=np.float32)
b = np.array([[0.5, 1.0, -1.0, 2.0], [2.0, -1.0, 0.5, 1.0]], dtype=np.float32)
y = swiglu(a, b, alpha=0.5)

expect(
    node,
    inputs=[a, b],
    outputs=[y],
    name="test_swiglu_alpha",
    opset_imports=[onnx.helper.make_opsetid("", 28)],
)

_float16¶

import numpy as np
import onnx

node = onnx.helper.make_node(
    "SwiGLU",
    inputs=["a", "b"],
    outputs=["y"],
)

a = np.array([[1.0, -2.0, 3.0, 4.0], [-1.0, 2.0, -3.0, 0.5]], dtype=np.float16)
b = np.array([[0.5, 1.0, -1.0, 2.0], [2.0, -1.0, 0.5, 1.0]], dtype=np.float16)
y = swiglu(a.astype(np.float32), b.astype(np.float32), alpha=1.0).astype(
    np.float16
)

expect(
    node,
    inputs=[a, b],
    outputs=[y],
    name="test_swiglu_float16",
    opset_imports=[onnx.helper.make_opsetid("", 28)],
)