Foundations

Layer Normalization

EasyRaw PyTorchAbout 10 min2 public examples plus private tests

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Your task

For each final-axis feature vector, subtract its mean and divide every centered channel by the same stabilized standard deviation. Then multiply by the supplied channel scale and add optional bias; each output channel stays at the corresponding input position.

Function signature

layer_norm(
    x: torch.Tensor,
    scale: torch.Tensor,
    bias: torch.Tensor | None = None,
    eps: float = 1e-05,
) -> torch.Tensor

Worked example

A variable row and a constant row

For [1,3], the mean is 2 and population variance is 1. With eps=1, normalize to $[-1,1]/\sqrt2$ , then multiply by [2,-1] and add [0.5,1], giving approximately [-0.914214,0.292893]. The constant row [4,4] centers to zero and returns the bias [0.5,1].

import torch

x = torch.tensor([[1.0, 3.0], [4.0, 4.0]], dtype=torch.float64, requires_grad=True)
scale = torch.tensor([2.0, -1.0], dtype=torch.float64, requires_grad=True)
bias = torch.tensor([0.5, 1.0], dtype=torch.float64, requires_grad=True)
eps = 1.0

result = layer_norm(x=x, scale=scale, bias=bias, eps=eps)

Expected return

torch.tensor([[-0.9142135623730949, 0.29289321881345254], [0.5, 1.0]], dtype=torch.float64)

Parameters

Parameter	Shape and meaning
`x`torch.Tensor	$[\ldots, C]$ Feature vectors to normalize over their final channel axis.
`scale`torch.Tensor	$[C]$ Learned multiplier for each feature channel.
`bias`torch.Tensor \| NoneDefault `None`	$[C]$ Optional learned offset for each channel; omission means zero.
`eps`floatDefault `1e-05`	Positive number added to each row’s population variance before taking the square root.

Return

torch.Tensor · Exactly x.shape.

Return Normalized and affine-transformed feature vectors, with exactly x’s shape.

Required behavior

The final channel axis has positive width; every preceding axis indexes an independent vector. Empty leading axes are supported.
Compute $\mu=\mathrm{mean}_{-1}(x)$ , $v=\mathrm{mean}_{-1}((x-\mu)^2)$ , then $y=(x-\mu)/\sqrt{v+\varepsilon}\cdot\mathrm{scale}+\mathrm{bias}$ . Use population variance.
eps is a positive Python number, default 1e-5, added inside the square root. Scale has shape (C,); omitted bias means zero.
Constant rows return bias; the zero row returns bias.

Why this matters

LayerNorm removes a row’s mean before scaling its spread; RMSNorm scales its magnitude without removing the mean.

Starter code

You write the body of this function in the BioTorch workspace. Your code runs in an isolated PyTorch sandbox; Run checks the public examples and Submit grades the full test suite, including gradients.

import torch


def layer_norm(
    x: torch.Tensor,  # (..., C)
    scale: torch.Tensor,  # (C,)
    bias: torch.Tensor | None = None,  # (C,)
    eps: float = 1e-05,
) -> torch.Tensor:
    """Return a tensor with shape x.shape."""
    # Implement the operation. Inputs and test setup are supplied by BioTorch.
    raise NotImplementedError("Implement layer_norm")

Ready to implement it?

3 progressive hints, a solution walkthrough, and grading that checks values, shapes and gradients are waiting in the workspace.

Write and run your solution Sign in

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Related exercises

Sources and further reading

A primitive exercise, not an end-to-end reproduction of the named models. No affiliation or endorsement is implied.