SoftmaxCrossEntropyLoss

SoftmaxCrossEntropyLoss - 13

Version

This version of the operator has been available since version 13.

Summary

Loss function that measures the softmax cross entropy between ‘scores’ and ‘labels’. This operator first computes a loss tensor whose shape is identical to the labels input. If the input is 2-D with shape (N, C), the loss tensor may be a N-element vector L = (l_1, l_2, …, l_N). If the input is N-D tensor with shape (N, C, D1, D2, …, Dk), the loss tensor L may have (N, D1, D2, …, Dk) as its shape and L[i,][j_1][j_2]…[j_k] denotes a scalar element in L. After L is available, this operator can optionally do a reduction operator.

  • shape(scores): (N, C) where C is the number of classes, or (N, C, D1, D2,…, Dk), with K >= 1 in case of K-dimensional loss.

  • shape(labels): (N) where each value is 0 <= labels[i] <= C-1, or (N, D1, D2,…, Dk), with K >= 1 in case of K-dimensional loss.

The loss for one sample, l_i, can calculated as follows:

l[i][d1][d2]...[dk] = -y[i][c][d1][d2]..[dk], where i is the index of classes.

or

l[i][d1][d2]...[dk] = -y[i][c][d1][d2]..[dk] * weights[c], if &#39;weights&#39; is provided.

loss is zero for the case when label-value equals ignore_index.

l[i][d1][d2]...[dk]  = 0, when labels[n][d1][d2]...[dk] = ignore_index

where:

p = Softmax(scores)
y = Log(p)
c = labels[i][d1][d2]...[dk]

Finally, L is optionally reduced:

  • If reduction = ‘none’, the output is L with shape (N, D1, D2, …, Dk).

  • If reduction = ‘sum’, the output is scalar: Sum(L).

  • If reduction = ‘mean’, the output is scalar: ReduceMean(L), or if weight is provided: ReduceSum(L) / ReduceSum(W), where tensor W is of shape (N, D1, D2, ..., Dk) and W[n][d1][d2]...[dk] = weights[labels[i][d1][d2]...[dk]].

Attributes

  • ignore_index - INT :

    Specifies a target value that is ignored and does not contribute to the input gradient. It’s an optional value.

  • reduction - STRING (default is mean):

    Type of reduction to apply to loss: none, sum, mean(default). ‘none’: no reduction will be applied, ‘sum’: the output will be summed. ‘mean’: the sum of the output will be divided by the number of elements in the output.

Inputs

Between 2 and 3 inputs.

  • scores (heterogeneous) - T:

    The predicted outputs with shape [batch_size, class_size], or [batch_size, class_size, D1, D2 , …, Dk], where K is the number of dimensions.

  • labels (heterogeneous) - Tind:

    The ground truth output tensor, with shape [batch_size], or [batch_size, D1, D2, …, Dk], where K is the number of dimensions. Labels element value shall be in range of [0, C). If ignore_index is specified, it may have a value outside [0, C) and the label values should either be in the range [0, C) or have the value ignore_index.

  • weights (optional, heterogeneous) - T:

    A manual rescaling weight given to each class. If given, it has to be a 1D Tensor assigning weight to each of the classes. Otherwise, it is treated as if having all ones.

Outputs

Between 1 and 2 outputs.

  • output (heterogeneous) - T:

    Weighted loss float Tensor. If reduction is ‘none’, this has the shape of [batch_size], or [batch_size, D1, D2, …, Dk] in case of K-dimensional loss. Otherwise, it is a scalar.

  • log_prob (optional, heterogeneous) - T:

    Log probability tensor. If the output of softmax is prob, its value is log(prob).

Type Constraints

  • T in ( tensor(bfloat16), tensor(double), tensor(float), tensor(float16) ):

    Constrain input and output types to float tensors.

  • Tind in ( tensor(int32), tensor(int64) ):

    Constrain target to integer types

Examples

_softmaxcrossentropy_none

import numpy as np
import onnx

# Define operator attributes.
reduction = "none"

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z"],
    reduction=reduction,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3,)).astype(np.int64)

# Compute SoftmaxCrossEntropyLoss
sce = softmaxcrossentropy(x, labels, reduction="none")

# Check results
expect(node, inputs=[x, labels], outputs=[sce], name="test_sce_none")

_softmaxcrossentropy_none_log_prob

import numpy as np
import onnx

# Define operator attributes.
reduction = "none"

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z", "log_prob"],
    reduction=reduction,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3,)).astype(np.int64)

# Compute SoftmaxCrossEntropyLoss
loss, log_prob = softmaxcrossentropy(
    x, labels, reduction="none", get_log_prob=True
)

# Check results
expect(
    node,
    inputs=[x, labels],
    outputs=[loss, log_prob],
    name="test_sce_none_log_prob",
)

_softmaxcrossentropy_none_weights

import numpy as np
import onnx

# Define operator attributes.
reduction = "none"

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z"],
    reduction=reduction,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3,)).astype(np.int64)
weights = np.array([0.9, 0.7, 0.8, 0.9, 0.9], dtype=np.float32)

# Compute SoftmaxCrossEntropyLoss
sce = softmaxcrossentropy(x, labels, weight=weights, reduction="none")

# Check results
expect(
    node,
    inputs=[x, labels, weights],
    outputs=[sce],
    name="test_sce_none_weights",
)

_softmaxcrossentropy_none_weights_log_prob

import numpy as np
import onnx

# Define operator attributes.
reduction = "none"

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z", "log_prob"],
    reduction=reduction,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3,)).astype(np.int64)
weights = np.array([0.9, 0.7, 0.8, 0.9, 0.9], dtype=np.float32)

# Compute SoftmaxCrossEntropyLoss
loss, log_prob = softmaxcrossentropy(
    x, labels, weight=weights, reduction="none", get_log_prob=True
)

# Check results
expect(
    node,
    inputs=[x, labels, weights],
    outputs=[loss, log_prob],
    name="test_sce_none_weights_log_prob",
)

_softmaxcrossentropy_sum

import numpy as np
import onnx

# Define operator attributes.
reduction = "sum"

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z"],
    reduction=reduction,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3,)).astype(np.int64)

# Compute SoftmaxCrossEntropyLoss
sce = softmaxcrossentropy(x, labels, reduction="sum")

# Check results
expect(node, inputs=[x, labels], outputs=[sce], name="test_sce_sum")

_softmaxcrossentropy_sum_log_prob

import numpy as np
import onnx

# Define operator attributes.
reduction = "sum"

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z", "log_prob"],
    reduction=reduction,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3,)).astype(np.int64)

# Compute SoftmaxCrossEntropyLoss
loss, log_prob = softmaxcrossentropy(
    x, labels, reduction="sum", get_log_prob=True
)

# Check results
expect(
    node,
    inputs=[x, labels],
    outputs=[loss, log_prob],
    name="test_sce_sum_log_prob",
)

_softmaxcrossentropy_mean

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z"],
    reduction=reduction,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3,)).astype(np.int64)

# Compute SoftmaxCrossEntropyLoss
sce = softmaxcrossentropy(x, labels)

# Check results
expect(node, inputs=[x, labels], outputs=[sce], name="test_sce_mean")

_softmaxcrossentropy_mean_log_prob

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z", "log_prob"],
    reduction=reduction,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3,)).astype(np.int64)

# Compute SoftmaxCrossEntropyLoss
loss, log_prob = softmaxcrossentropy(x, labels, get_log_prob=True)

# Check results
expect(
    node,
    inputs=[x, labels],
    outputs=[loss, log_prob],
    name="test_sce_mean_log_prob",
)

_softmaxcrossentropy_mean_3d

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z"],
    reduction=reduction,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5, 2).astype(np.float32)
y = np.random.randint(0, high=5, size=(3, 2)).astype(np.int64)

# Compute SoftmaxCrossEntropyLoss
sce = softmaxcrossentropy(x, y)

# Check results
expect(node, inputs=[x, y], outputs=[sce], name="test_sce_mean_3d")

_softmaxcrossentropy_mean_3d_log_prob

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z", "log_prob"],
    reduction=reduction,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5, 2).astype(np.float32)
y = np.random.randint(0, high=5, size=(3, 2)).astype(np.int64)

# Compute SoftmaxCrossEntropyLoss
loss, log_prob = softmaxcrossentropy(x, y, get_log_prob=True)

# Check results
expect(
    node,
    inputs=[x, y],
    outputs=[loss, log_prob],
    name="test_sce_mean_3d_log_prob",
)

_softmaxcrossentropy_mean_weights

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z"],
    reduction=reduction,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3,)).astype(np.int64)
weights = np.array([0.9, 0.7, 0.8, 0.9, 0.9], dtype=np.float32)

# Compute SoftmaxCrossEntropyLoss
sce = softmaxcrossentropy(x, labels, weight=weights)

# Check results
expect(
    node,
    inputs=[x, labels, weights],
    outputs=[sce],
    name="test_sce_mean_weight",
)

_softmaxcrossentropy_mean_weights_log_prob

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z", "log_prob"],
    reduction=reduction,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3,)).astype(np.int64)
weights = np.array([0.9, 0.7, 0.8, 0.9, 0.9], dtype=np.float32)

# Compute SoftmaxCrossEntropyLoss
loss, log_prob = softmaxcrossentropy(
    x, labels, weight=weights, get_log_prob=True
)

# Check results
expect(
    node,
    inputs=[x, labels, weights],
    outputs=[loss, log_prob],
    name="test_sce_mean_weight_log_prob",
)

_softmaxcrossentropy_mean_weights_ii

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"
ignore_index = np.int64(0)

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z"],
    reduction=reduction,
    ignore_index=ignore_index,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3,)).astype(np.int64)
labels[0] = np.int64(0)
weights = np.array([0.9, 0.7, 0.8, 0.9, 0.9], dtype=np.float32)

# Compute SoftmaxCrossEntropyLoss
sce = softmaxcrossentropy(x, labels, weight=weights, ignore_index=ignore_index)

# Check results
expect(
    node,
    inputs=[x, labels, weights],
    outputs=[sce],
    name="test_sce_mean_weight_ii",
)

_softmaxcrossentropy_mean_weights_ii_log_prob

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"
ignore_index = np.int64(0)

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z", "log_prob"],
    reduction=reduction,
    ignore_index=ignore_index,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3,)).astype(np.int64)
labels[0] = np.int64(0)
weights = np.array([0.9, 0.7, 0.8, 0.9, 0.9], dtype=np.float32)

# Compute SoftmaxCrossEntropyLoss
loss, log_prob = softmaxcrossentropy(
    x, labels, weight=weights, ignore_index=ignore_index, get_log_prob=True
)

# Check results
expect(
    node,
    inputs=[x, labels, weights],
    outputs=[loss, log_prob],
    name="test_sce_mean_weight_ii_log_prob",
)

_softmaxcrossentropy_mean_no_weights_ii

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"
ignore_index = np.int64(2)

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z"],
    reduction=reduction,
    ignore_index=ignore_index,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3,)).astype(np.int64)
labels[0] = np.int64(2)

# Compute SoftmaxCrossEntropyLoss
sce = softmaxcrossentropy(x, labels, ignore_index=ignore_index)

# Check results
expect(
    node, inputs=[x, labels], outputs=[sce], name="test_sce_mean_no_weight_ii"
)

_softmaxcrossentropy_mean_no_weights_ii_log_prob

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"
ignore_index = np.int64(2)

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z", "log_prob"],
    reduction=reduction,
    ignore_index=ignore_index,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3,)).astype(np.int64)
labels[0] = np.int64(2)

# Compute SoftmaxCrossEntropyLoss
loss, log_prob = softmaxcrossentropy(
    x, labels, ignore_index=ignore_index, get_log_prob=True
)

# Check results
expect(
    node,
    inputs=[x, labels],
    outputs=[loss, log_prob],
    name="test_sce_mean_no_weight_ii_log_prob",
)

_softmaxcrossentropy_mean_weights_ii_3d

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"
ignore_index = np.int64(1)

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z"],
    reduction=reduction,
    ignore_index=ignore_index,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5, 2).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3, 2)).astype(np.int64)
labels[0][0] = np.int64(1)
weights = np.array([0.2, 0.3, 0.6, 0.1, 0.5], dtype=np.float32)

# Compute SoftmaxCrossEntropyLoss
sce = softmaxcrossentropy(x, labels, weight=weights, ignore_index=ignore_index)

# Check results
expect(
    node,
    inputs=[x, labels, weights],
    outputs=[sce],
    name="test_sce_mean_weight_ii_3d",
)

_softmaxcrossentropy_mean_weights_ii_3d_log_prob

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"
ignore_index = np.int64(1)

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z", "log_prob"],
    reduction=reduction,
    ignore_index=ignore_index,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5, 2).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3, 2)).astype(np.int64)
labels[0][0] = np.int64(1)
weights = np.array([0.2, 0.3, 0.6, 0.1, 0.5], dtype=np.float32)

# Compute SoftmaxCrossEntropyLoss
loss, log_prob = softmaxcrossentropy(
    x, labels, weight=weights, ignore_index=ignore_index, get_log_prob=True
)

# Check results
expect(
    node,
    inputs=[x, labels, weights],
    outputs=[loss, log_prob],
    name="test_sce_mean_weight_ii_3d_log_prob",
)

_softmaxcrossentropy_mean_no_weights_ii_3d

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"
ignore_index = np.int64(2)

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z"],
    reduction=reduction,
    ignore_index=ignore_index,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5, 2).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3, 2)).astype(np.int64)
labels[0][0] = np.int64(2)

# Compute SoftmaxCrossEntropyLoss
sce = softmaxcrossentropy(x, labels, ignore_index=ignore_index)

# Check results
expect(
    node,
    inputs=[x, labels],
    outputs=[sce],
    name="test_sce_mean_no_weight_ii_3d",
)

_softmaxcrossentropy_mean_no_weights_ii_3d_log_prob

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"
ignore_index = np.int64(2)

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z", "log_prob"],
    reduction=reduction,
    ignore_index=ignore_index,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5, 2).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3, 2)).astype(np.int64)
labels[0][0] = np.int64(2)

# Compute SoftmaxCrossEntropyLoss
loss, log_prob = softmaxcrossentropy(
    x, labels, ignore_index=ignore_index, get_log_prob=True
)

# Check results
expect(
    node,
    inputs=[x, labels],
    outputs=[loss, log_prob],
    name="test_sce_mean_no_weight_ii_3d_log_prob",
)

_softmaxcrossentropy_mean_weights_ii_4d

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"
ignore_index = np.int64(2)

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z"],
    reduction=reduction,
    ignore_index=ignore_index,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5, 2, 7).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3, 2, 7)).astype(np.int64)
labels[0][0][0] = np.int64(2)
weights = np.array([0.2, 0.3, 0.6, 0.1, 0.5], dtype=np.float32)

# Compute SoftmaxCrossEntropyLoss
sce = softmaxcrossentropy(
    x, labels, reduction=reduction, weight=weights, ignore_index=ignore_index
)

# Check results
expect(
    node,
    inputs=[x, labels, weights],
    outputs=[sce],
    name="test_sce_mean_weight_ii_4d",
)

_softmaxcrossentropy_mean_weights_ii_4d_log_prob

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"
ignore_index = np.int64(2)

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z", "log_prob"],
    reduction=reduction,
    ignore_index=ignore_index,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5, 2, 7).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3, 2, 7)).astype(np.int64)
labels[0][0][0] = np.int64(2)
weights = np.array([0.2, 0.3, 0.6, 0.1, 0.5], dtype=np.float32)

# Compute SoftmaxCrossEntropyLoss
loss, log_prob = softmaxcrossentropy(
    x,
    labels,
    reduction=reduction,
    weight=weights,
    ignore_index=ignore_index,
    get_log_prob=True,
)

# Check results
expect(
    node,
    inputs=[x, labels, weights],
    outputs=[loss, log_prob],
    name="test_sce_mean_weight_ii_4d_log_prob",
)

_softmaxcrossentropy_mean_no_weights_ii_4d

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"
ignore_index = np.int64(2)

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z"],
    reduction=reduction,
    ignore_index=ignore_index,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5, 2, 7).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3, 2, 7)).astype(np.int64)
labels[0][0][0] = np.int64(2)

# Compute SoftmaxCrossEntropyLoss
sce = softmaxcrossentropy(
    x, labels, reduction=reduction, ignore_index=ignore_index
)

# Check results
expect(
    node,
    inputs=[x, labels],
    outputs=[sce],
    name="test_sce_mean_no_weight_ii_4d",
)

_softmaxcrossentropy_mean_no_weights_ii_4d_log_prob

import numpy as np
import onnx

# Define operator attributes.
reduction = "mean"
ignore_index = np.int64(2)

# Create operator.
node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z", "log_prob"],
    reduction=reduction,
    ignore_index=ignore_index,
)

# Define operator inputs.
np.random.seed(0)
x = np.random.rand(3, 5, 2, 7).astype(np.float32)
labels = np.random.randint(0, high=5, size=(3, 2, 7)).astype(np.int64)
labels[0][0][0] = np.int64(2)

# Compute SoftmaxCrossEntropyLoss
loss, log_prob = softmaxcrossentropy(
    x, labels, reduction=reduction, ignore_index=ignore_index, get_log_prob=True
)

# Check results
expect(
    node,
    inputs=[x, labels],
    outputs=[loss, log_prob],
    name="test_sce_mean_no_weight_ii_4d_log_prob",
)

_input_shape_is_NCd1d2d3d4d5_mean_weight

import numpy as np
import onnx

reduction = "mean"

node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z"],
    reduction=reduction,
)

N, C, dim1, dim2, dim3, dim4, dim5 = 3, 5, 6, 6, 5, 3, 4
np.random.seed(0)
x = np.random.rand(N, C, dim1, dim2, dim3, dim4, dim5).astype(np.float32)
labels = np.random.randint(
    0, high=C, size=(N, dim1, dim2, dim3, dim4, dim5)
).astype(np.int64)
weight = np.random.rand(C).astype(np.float32)

sce = softmaxcrossentropy(x, labels, weight=weight, reduction=reduction)

expect(
    node,
    inputs=[x, labels, weight],
    outputs=[sce],
    name="test_sce_NCd1d2d3d4d5_mean_weight",
)

_input_shape_is_NCd1d2d3d4d5_mean_weight_log_prob

import numpy as np
import onnx

reduction = "mean"

node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z", "log_prob"],
    reduction=reduction,
)

N, C, dim1, dim2, dim3, dim4, dim5 = 3, 5, 6, 6, 5, 3, 4
np.random.seed(0)
x = np.random.rand(N, C, dim1, dim2, dim3, dim4, dim5).astype(np.float32)
labels = np.random.randint(
    0, high=C, size=(N, dim1, dim2, dim3, dim4, dim5)
).astype(np.int64)
weight = np.random.rand(C).astype(np.float32)

loss, log_prob = softmaxcrossentropy(
    x, labels, weight=weight, reduction=reduction, get_log_prob=True
)

expect(
    node,
    inputs=[x, labels, weight],
    outputs=[loss, log_prob],
    name="test_sce_NCd1d2d3d4d5_mean_weight_log_prob",
)

_input_shape_is_NCd1d2d3d4d5_none_no_weight

import numpy as np
import onnx

reduction = "none"

node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z"],
    reduction=reduction,
)

N, C, dim1, dim2, dim3, dim4, dim5 = 3, 5, 6, 6, 5, 3, 4
np.random.seed(0)
x = np.random.rand(N, C, dim1, dim2, dim3, dim4, dim5).astype(np.float32)
labels = np.random.randint(
    0, high=C, size=(N, dim1, dim2, dim3, dim4, dim5)
).astype(np.int64)

sce = softmaxcrossentropy(x, labels, reduction=reduction)

expect(
    node,
    inputs=[x, labels],
    outputs=[sce],
    name="test_sce_NCd1d2d3d4d5_none_no_weight",
)

_input_shape_is_NCd1d2d3d4d5_none_no_weight_log_prob

import numpy as np
import onnx

reduction = "none"

node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z", "log_prob"],
    reduction=reduction,
)

N, C, dim1, dim2, dim3, dim4, dim5 = 3, 5, 6, 6, 5, 3, 4
np.random.seed(0)
x = np.random.rand(N, C, dim1, dim2, dim3, dim4, dim5).astype(np.float32)
labels = np.random.randint(
    0, high=C, size=(N, dim1, dim2, dim3, dim4, dim5)
).astype(np.int64)

loss, log_prob = softmaxcrossentropy(
    x, labels, reduction=reduction, get_log_prob=True
)

expect(
    node,
    inputs=[x, labels],
    outputs=[loss, log_prob],
    name="test_sce_NCd1d2d3d4d5_none_no_weight_log_prob",
)

_input_shape_is_NCd1_mean_weight_negative_ii

import numpy as np
import onnx

reduction = "mean"
ignore_index = np.int64(-1)

node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z"],
    reduction=reduction,
    ignore_index=ignore_index,
)

N, C, dim1 = 3, 5, 6
np.random.seed(0)
x = np.random.rand(N, C, dim1).astype(np.float32)
labels = np.random.randint(0, high=C, size=(N, dim1)).astype(np.int64)
labels[0][0] = -1
weight = np.random.rand(C).astype(np.float32)

sce = softmaxcrossentropy(
    x, labels, weight=weight, reduction=reduction, ignore_index=ignore_index
)

expect(
    node,
    inputs=[x, labels, weight],
    outputs=[sce],
    name="test_sce_NCd1_mean_weight_negative_ii",
)

_input_shape_is_NCd1_mean_weight_negative_ii_log_prob

import numpy as np
import onnx

reduction = "mean"
ignore_index = np.int64(-1)

node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z", "log_prob"],
    reduction=reduction,
    ignore_index=ignore_index,
)

N, C, dim1 = 3, 5, 6
np.random.seed(0)
x = np.random.rand(N, C, dim1).astype(np.float32)
labels = np.random.randint(0, high=C, size=(N, dim1)).astype(np.int64)
labels[0][0] = -1
weight = np.random.rand(C).astype(np.float32)

loss, log_prob = softmaxcrossentropy(
    x,
    labels,
    weight=weight,
    reduction=reduction,
    ignore_index=ignore_index,
    get_log_prob=True,
)

expect(
    node,
    inputs=[x, labels, weight],
    outputs=[loss, log_prob],
    name="test_sce_NCd1_mean_weight_negative_ii_log_prob",
)

_input_shape_is_NCd1d2d3_none_no_weight_negative_ii

import numpy as np
import onnx

reduction = "none"
ignore_index = np.int64(-5)

node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z"],
    reduction=reduction,
    ignore_index=ignore_index,
)

N, C, dim1, dim2, dim3 = 3, 5, 6, 6, 5
np.random.seed(0)
x = np.random.rand(N, C, dim1, dim2, dim3).astype(np.float32)
labels = np.random.randint(0, high=C, size=(N, dim1, dim2, dim3)).astype(
    np.int64
)
labels[0][0][0][0] = -5

sce = softmaxcrossentropy(
    x, labels, reduction=reduction, ignore_index=ignore_index
)

expect(
    node,
    inputs=[x, labels],
    outputs=[sce],
    name="test_sce_NCd1d2d3_none_no_weight_negative_ii",
)

_input_shape_is_NCd1d2d3_none_no_weight_negative_ii_log_prob

import numpy as np
import onnx

reduction = "none"
ignore_index = np.int64(-5)

node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y"],
    outputs=["z", "log_prob"],
    reduction=reduction,
    ignore_index=ignore_index,
)

N, C, dim1, dim2, dim3 = 3, 5, 6, 6, 5
np.random.seed(0)
x = np.random.rand(N, C, dim1, dim2, dim3).astype(np.float32)
labels = np.random.randint(0, high=C, size=(N, dim1, dim2, dim3)).astype(
    np.int64
)
labels[0][0][0][0] = -5

loss, log_prob = softmaxcrossentropy(
    x, labels, reduction=reduction, ignore_index=ignore_index, get_log_prob=True
)

expect(
    node,
    inputs=[x, labels],
    outputs=[loss, log_prob],
    name="test_sce_NCd1d2d3_none_no_weight_negative_ii_log_prob",
)

_input_shape_is_NCd1d2d3_sum_weight_high_ii

import numpy as np
import onnx

reduction = "sum"
ignore_index = np.int64(10)

node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z"],
    reduction=reduction,
    ignore_index=ignore_index,
)

N, C = 3, 5
np.random.seed(0)
x = np.random.rand(N, C).astype(np.float32)
labels = np.random.randint(0, high=C, size=(N)).astype(np.int64)
labels[0] = 10
weight = np.random.rand(C).astype(np.float32)

sce = softmaxcrossentropy(
    x, labels, weight=weight, reduction=reduction, ignore_index=ignore_index
)

expect(
    node,
    inputs=[x, labels, weight],
    outputs=[sce],
    name="test_sce_NCd1d2d3_sum_weight_high_ii",
)

_input_shape_is_NCd1d2d3_sum_weight_high_ii_log_prob

import numpy as np
import onnx

reduction = "sum"
ignore_index = np.int64(10)

node = onnx.helper.make_node(
    "SoftmaxCrossEntropyLoss",
    inputs=["x", "y", "w"],
    outputs=["z", "log_prob"],
    reduction=reduction,
    ignore_index=ignore_index,
)

N, C = 3, 5
np.random.seed(0)
x = np.random.rand(N, C).astype(np.float32)
labels = np.random.randint(0, high=C, size=(N)).astype(np.int64)
labels[0] = 10
weight = np.random.rand(C).astype(np.float32)

loss, log_prob = softmaxcrossentropy(
    x,
    labels,
    weight=weight,
    reduction=reduction,
    ignore_index=ignore_index,
    get_log_prob=True,
)

expect(
    node,
    inputs=[x, labels, weight],
    outputs=[loss, log_prob],
    name="test_sce_NCd1d2d3_sum_weight_high_ii_log_prob",
)

SoftmaxCrossEntropyLoss - 12

Version

This version of the operator has been available since version 12.

Summary

Loss function that measures the softmax cross entropy between ‘scores’ and ‘labels’. This operator first computes a loss tensor whose shape is identical to the labels input. If the input is 2-D with shape (N, C), the loss tensor may be a N-element vector L = (l_1, l_2, …, l_N). If the input is N-D tensor with shape (N, C, D1, D2, …, Dk), the loss tensor L may have (N, D1, D2, …, Dk) as its shape and L[i,][j_1][j_2]…[j_k] denotes a scalar element in L. After L is available, this operator can optionally do a reduction operator.

shape(scores): (N, C) where C is the number of classes, or (N, C, D1, D2,…, Dk), with K >= 1 in case of K-dimensional loss. shape(labels): (N) where each value is 0 <= labels[i] <= C-1, or (N, D1, D2,…, Dk), with K >= 1 in case of K-dimensional loss.

The loss for one sample, l_i, can calculated as follows: l[i][d1][d2]…[dk] = -y[i][c][d1][d2]…[dk], where i is the index of classes. or l[i][d1][d2]…[dk] = -y[i][c][d1][d2]…[dk] * weights[c], if ‘weights’ is provided.

loss is zero for the case when label-value equals ignore_index. l[i][d1][d2]…[dk] = 0, when labels[n][d1][d2]…[dk] = ignore_index

where: p = Softmax(scores) y = Log(p) c = labels[i][d1][d2]…[dk]

Finally, L is optionally reduced: If reduction = ‘none’, the output is L with shape (N, D1, D2, …, Dk). If reduction = ‘sum’, the output is scalar: Sum(L). If reduction = ‘mean’, the output is scalar: ReduceMean(L), or if weight is provided: ReduceSum(L) / ReduceSum(W), where tensor W is of shape (N, D1, D2, …, Dk) and W[n][d1][d2]…[dk] = weights[labels[i][d1][d2]…[dk]].

Attributes

  • ignore_index - INT :

    Specifies a target value that is ignored and does not contribute to the input gradient. It’s an optional value.

  • reduction - STRING (default is mean):

    Type of reduction to apply to loss: none, sum, mean(default). ‘none’: no reduction will be applied, ‘sum’: the output will be summed. ‘mean’: the sum of the output will be divided by the number of elements in the output.

Inputs

Between 2 and 3 inputs.

  • scores (heterogeneous) - T:

    The predicted outputs with shape [batch_size, class_size], or [batch_size, class_size, D1, D2 , …, Dk], where K is the number of dimensions.

  • labels (heterogeneous) - Tind:

    The ground truth output tensor, with shape [batch_size], or [batch_size, D1, D2, …, Dk], where K is the number of dimensions. Labels element value shall be in range of [0, C). If ignore_index is specified, it may have a value outside [0, C) and the label values should either be in the range [0, C) or have the value ignore_index.

  • weights (optional, heterogeneous) - T:

    A manual rescaling weight given to each class. If given, it has to be a 1D Tensor assigning weight to each of the classes. Otherwise, it is treated as if having all ones.

Outputs

Between 1 and 2 outputs.

  • output (heterogeneous) - T:

    Weighted loss float Tensor. If reduction is ‘none’, this has the shape of [batch_size], or [batch_size, D1, D2, …, Dk] in case of K-dimensional loss. Otherwise, it is a scalar.

  • log_prob (optional, heterogeneous) - T:

    Log probability tensor. If the output of softmax is prob, its value is log(prob).

Type Constraints

  • T in ( tensor(double), tensor(float), tensor(float16) ):

    Constrain input and output types to float tensors.

  • Tind in ( tensor(int32), tensor(int64) ):

    Constrain target to integer types