(l-onnx-doc-SwiGLU)=

# SwiGLU


(l-onnx-op-swiglu-28)=

## SwiGLU - 28

### Version

- **name**: [SwiGLU (GitHub)](https://github.com/onnx/onnx/blob/main/docs/Operators.md#SwiGLU)
- **domain**: `main`
- **since_version**: `28`
- **function**: `True`
- **support_level**: `SupportType.COMMON`
- **shape inference**: `True`

This version of the operator has been available
**since version 28**.

### Summary

SwiGLU is a gated activation that takes two inputs, a gate `A` and a linear (value)
input `B`, and produces one output `Y`. It applies the Swish activation to the gate
and multiplies the result elementwise by the linear input:

```
Y = Swish_alpha(A) * B
```

The gate activation `Swish_alpha` is exactly the `Swish` operator with the same
`alpha`, i.e. `Swish_alpha(a) = a * Sigmoid(alpha * a)`. Inputs `A` and `B` must
have identical shapes; broadcasting is not applied and the output `Y` has the same
shape as the inputs.

Exporters typically produce `A` and `B` in one of two ways: for the common
two-projection form (e.g. Llama's `gate_proj`/`up_proj`) wire the two projection
outputs directly to `A` (gate) and `B` (value); for a fused/packed single
projection, split it upstream into `A` and `B` with `Split` (contiguous layout)
or `Slice`/`Gather` (interleaved layout).

#### Function Body

The function definition for this operator.

```
<
  domain: "",
  opset_import: ["" : 28]
>
SwiGLU <alpha>(A, B) => (Y)
{
   SwishGate = Swish <alpha: float = @alpha> (A)
   Y = Mul (SwishGate, B)
}
```

### Attributes

* **alpha - FLOAT** (default is `1.0`):

  Coefficient that scales the gate input inside the sigmoid of the Swish activation. The default value is 1.0.

### Inputs

- **A** (heterogeneous) - **T**:

  Gate input tensor
- **B** (heterogeneous) - **T**:

  Linear (value) input tensor

### Outputs

- **Y** (heterogeneous) - **T**:

  Output tensor

### Type Constraints

* **T** in ( `tensor(bfloat16)`, `tensor(double)`, `tensor(float)`, `tensor(float16)` ):

  Constrain input and output types to float tensors.