(l-onnx-doc-SwiGLU)=

# SwiGLU


(l-onnx-op-swiglu-28)=

## SwiGLU - 28

### Version

- **name**: [SwiGLU (GitHub)](https://github.com/onnx/onnx/blob/main/docs/Operators.md#SwiGLU)
- **domain**: `main`
- **since_version**: `28`
- **function**: `True`
- **support_level**: `SupportType.COMMON`
- **shape inference**: `True`

This version of the operator has been available
**since version 28**.

### Summary

SwiGLU is a gated activation that takes two inputs, a gate `A` and a linear (value)
input `B`, and produces one output `Y`. It applies the Swish activation to the gate
and multiplies the result elementwise by the linear input:

```
Y = Swish_alpha(A) * B
```

The gate activation `Swish_alpha` is exactly the `Swish` operator with the same
`alpha`, i.e. `Swish_alpha(a) = a * Sigmoid(alpha * a)`. Inputs `A` and `B` must
have identical shapes; broadcasting is not applied and the output `Y` has the same
shape as the inputs.

Exporters typically produce `A` and `B` in one of two ways: for the common
two-projection form (e.g. Llama's `gate_proj`/`up_proj`) wire the two projection
outputs directly to `A` (gate) and `B` (value); for a fused/packed single
projection, split it upstream into `A` and `B` with `Split` (contiguous layout)
or `Slice`/`Gather` (interleaved layout).

#### Function Body

The function definition for this operator.

```
<
  domain: "",
  opset_import: ["" : 28]
>
SwiGLU <alpha>(A, B) => (Y)
{
   SwishGate = Swish <alpha: float = @alpha> (A)
   Y = Mul (SwishGate, B)
}
```

### Attributes

* **alpha - FLOAT** (default is `1.0`):

  Coefficient that scales the gate input inside the sigmoid of the Swish activation. The default value is 1.0.

### Inputs

- **A** (heterogeneous) - **T**:

  Gate input tensor
- **B** (heterogeneous) - **T**:

  Linear (value) input tensor

### Outputs

- **Y** (heterogeneous) - **T**:

  Output tensor

### Type Constraints

* **T** in ( `tensor(bfloat16)`, `tensor(double)`, `tensor(float)`, `tensor(float16)` ):

  Constrain input and output types to float tensors.

### Examples

#### default

```python
import numpy as np
import onnx

node = onnx.helper.make_node(
    "SwiGLU",
    inputs=["a", "b"],
    outputs=["y"],
)

a = np.array([[1.0, -2.0, 3.0, 4.0], [-1.0, 2.0, -3.0, 0.5]], dtype=np.float32)
b = np.array([[0.5, 1.0, -1.0, 2.0], [2.0, -1.0, 0.5, 1.0]], dtype=np.float32)
y = swiglu(a, b, alpha=1.0)

expect(
    node,
    inputs=[a, b],
    outputs=[y],
    name="test_swiglu",
    opset_imports=[onnx.helper.make_opsetid("", 28)],
)
```

#### _alpha

```python
import numpy as np
import onnx

node = onnx.helper.make_node(
    "SwiGLU",
    inputs=["a", "b"],
    outputs=["y"],
    alpha=0.5,  # pass alpha as attribute
)

a = np.array([[1.0, -2.0, 3.0, 4.0], [-1.0, 2.0, -3.0, 0.5]], dtype=np.float32)
b = np.array([[0.5, 1.0, -1.0, 2.0], [2.0, -1.0, 0.5, 1.0]], dtype=np.float32)
y = swiglu(a, b, alpha=0.5)

expect(
    node,
    inputs=[a, b],
    outputs=[y],
    name="test_swiglu_alpha",
    opset_imports=[onnx.helper.make_opsetid("", 28)],
)
```

#### _float16

```python
import numpy as np
import onnx

node = onnx.helper.make_node(
    "SwiGLU",
    inputs=["a", "b"],
    outputs=["y"],
)

a = np.array([[1.0, -2.0, 3.0, 4.0], [-1.0, 2.0, -3.0, 0.5]], dtype=np.float16)
b = np.array([[0.5, 1.0, -1.0, 2.0], [2.0, -1.0, 0.5, 1.0]], dtype=np.float16)
y = swiglu(a.astype(np.float32), b.astype(np.float32), alpha=1.0).astype(
    np.float16
)

expect(
    node,
    inputs=[a, b],
    outputs=[y],
    name="test_swiglu_float16",
    opset_imports=[onnx.helper.make_opsetid("", 28)],
)
```