SwiGLU¶

SwiGLU - 28¶

Version¶

name: SwiGLU (GitHub)
domain: main
since_version: 28
function: True
support_level: SupportType.COMMON
shape inference: True

This version of the operator has been available since version 28.

Summary¶

SwiGLU is a gated activation that takes two inputs, a gate A and a linear (value) input B, and produces one output Y. It applies the Swish activation to the gate and multiplies the result elementwise by the linear input:

Y = Swish_alpha(A) * B

The gate activation Swish_alpha is exactly the Swish operator with the same alpha, i.e. Swish_alpha(a) = a * Sigmoid(alpha * a). Inputs A and B must have identical shapes; broadcasting is not applied and the output Y has the same shape as the inputs.

Exporters typically produce A and B in one of two ways: for the common two-projection form (e.g. Llama’s gate_proj/up_proj) wire the two projection outputs directly to A (gate) and B (value); for a fused/packed single projection, split it upstream into A and B with Split (contiguous layout) or Slice/Gather (interleaved layout).

Function Body¶

The function definition for this operator.

&lt;
  domain: &#34;&#34;,
  opset_import: [&#34;&#34; : 28]
&gt;
SwiGLU &lt;alpha&gt;(A, B) =&gt; (Y)
{
   SwishGate = Swish &lt;alpha: float = @alpha&gt; (A)
   Y = Mul (SwishGate, B)
}

Attributes¶

alpha - FLOAT (default is 1.0):

Coefficient that scales the gate input inside the sigmoid of the Swish activation. The default value is 1.0.

Inputs¶

A (heterogeneous) - T:

Gate input tensor
B (heterogeneous) - T:

Linear (value) input tensor

Outputs¶

Y (heterogeneous) - T:

Output tensor

Type Constraints¶

T in ( tensor(bfloat16), tensor(double), tensor(float), tensor(float16) ):

Constrain input and output types to float tensors.