Foundations

Categorical Cross-Entropy

MediumRaw PyTorchAbout 20 min2 public examples plus private tests

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Your task

For each included target, measure how little probability its class receives: the loss is the negative natural log of that class’s softmax probability. Apply optional per-target weights, then return the unreduced losses, their sum, or the specified weighted mean.

Function signature

categorical_cross_entropy(
    logits: torch.Tensor,
    targets: torch.Tensor,
    mask: torch.Tensor | None = None,
    weights: torch.Tensor | None = None,
    reduction: str = 'mean',
    dim: int = -1,
    eps: float = 1e-08,
    ignore_index: int = -100,
) -> torch.Tensor

Worked example

From a target probability to its loss

Take logits=[[0,log(3)]], targets=[1], dim=-1 and reduction="none", with mask and weights omitted. The class exponential weights are [1,3], so class 1 has probability 3/4. Return [−log(3/4)] = [log(4/3)] ≈ [0.287682]. Choosing class 0 instead would give loss log(4) ≈ 1.386294. A prediction that assigns more probability to the correct class has lower loss.

Parameters

Parameter	Shape and meaning
`logits`torch.Tensor	Any shape with a positive class axis at dim.Unnormalized classification scores.
`targets`torch.Tensor	logits.shape with the class axis removed.Integer class ID per example; ignore_index marks excluded targets.
`mask`torch.Tensor \| NoneDefault `None`	Any shape broadcastable to targets.shape.Optional example validity mask; True includes a nonignored target.
`weights`torch.Tensor \| NoneDefault `None`	Any shape broadcastable to targets.shape.Optional nonnegative element weights; omission means one.
`reduction`strDefault `'mean'`	Choose an unreduced result, sum, or mean as allowed by this task.
`dim`intDefault `-1`	Class axis of logits; remove it to obtain targets.shape. Negative indices count from the end.
`eps`floatDefault `1e-08`	Finite nonnegative offset added once to the total included weight only for reduction="mean". It does not modify class probabilities or individual losses.
`ignore_index`intDefault `-100`	Target ID excluded from the loss.

Return

torch.Tensor · targets.shape for reduction="none"; scalar tensor, shape (), for "sum" or "mean".

Return Masked weighted loss after the requested reduction.

Required behavior

Categorical targets are integer class IDs. Remove the class axis dim (default $-1$ ) from logits to obtain the target and unreduced-loss shape. Ignore target values equal to ignore_index (default $-100$ ). Compute $\log\sum_c e^{z_c}-z_y$ with a shifted log-sum-exp.
mask is optional, boolean, and broadcastable to targets; True includes an element. Sanitize ignored logits/targets before arithmetic. weights is optional, nonnegative, and broadcastable to the unreduced loss; omission means one.
reduction="none" returns the masked weighted element losses; "sum" returns their scalar sum. "mean" (default) divides that sum by the total valid weight plus eps (default $10^{-8}$ ); return exact scalar zero when the valid weight sum is zero.
Floating logits, optional weights remain differentiable. Ignore masks and class IDs are nondifferentiable.

Why this matters

Working in logit space avoids saturated probabilities and preserves useful gradients even for confident incorrect predictions.

Starter code

You write the body of this function in the BioTorch workspace. Your code runs in an isolated PyTorch sandbox; Run checks the public examples and Submit grades the full test suite, including gradients.

import torch


def categorical_cross_entropy(
    logits: torch.Tensor,  # Any shape with a positive class axis at dim.
    targets: torch.Tensor,  # logits.shape with the class axis removed.
    mask: torch.Tensor | None = None,  # Any shape broadcastable to targets.shape.
    weights: torch.Tensor | None = None,  # Any shape broadcastable to targets.shape.
    reduction: str = 'mean',
    dim: int = -1,
    eps: float = 1e-08,
    ignore_index: int = -100,
) -> torch.Tensor:
    """Return a tensor with shape targets.shape for reduction="none"; scalar tensor, shape
        (), for "sum" or "mean"."""
    # Implement the operation. Inputs and test setup are supplied by BioTorch.
    raise NotImplementedError("Implement categorical_cross_entropy")

Ready to implement it?

3 progressive hints, a solution walkthrough, and grading that checks values, shapes and gradients are waiting in the workspace.

Write and run your solution Sign in

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Related exercises

Sources and further reading

A primitive exercise, not an end-to-end reproduction of the named models. No affiliation or endorsement is implied.