Foundations

Masked Layer Normalization

EasyRaw PyTorchAbout 10 min1 public example plus private tests

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Your task

At each valid position, subtract that feature vector’s channel mean, divide by the square root of its population variance plus eps, then apply the supplied channel scale and bias. Each output vector describes the same position and has the same channels as x; invalid positions return zero.

Function signature

masked_layer_normalization(
    x: torch.Tensor,
    valid: torch.Tensor,
    scale: torch.Tensor,
    bias: torch.Tensor,
    eps: float = 1e-05,
) -> torch.Tensor

Worked example

The valid row has mean 2 and population variance 1. With illustrative eps=3, its denominator is 2 and its affine result is [-0.5, 0.5]. The masked row returns [0, 0], even though bias is nonzero.

import torch

x = torch.tensor([[[1.0, 3.0], [99.0, -99.0]]], dtype=torch.float64, requires_grad=True)
valid = torch.tensor([[True, False]], dtype=torch.bool)
scale = torch.tensor([2.0, -1.0], dtype=torch.float64, requires_grad=True)
bias = torch.tensor([0.5, 1.0], dtype=torch.float64, requires_grad=True)
eps = 3.0

result = masked_layer_normalization(x=x, valid=valid, scale=scale, bias=bias, eps=eps)

Expected return

torch.tensor([[[-0.5, 0.5], [0.0, 0.0]]], dtype=torch.float64)

Parameters

Parameter	Shape and meaning
`x`torch.Tensor	$[\ldots, C]$ Feature vectors to normalize independently over their final channel axis.
`valid`torch.Tensor	$[\ldots]$ True includes a feature vector; false vectors return zero.
`scale`torch.Tensor	$[C]$ Learned multiplier applied after channel normalization.
`bias`torch.Tensor	$[C]$ Learned offset added after scaling.
`eps`floatDefault `1e-05`	Positive stabilizer inside sqrt(population channel variance + eps).

Return

torch.Tensor · $[\ldots, C]$

Return Floating [...,C], normalized/modulated features, not a residual sum; invalid rows zero.

Required behavior

valid is boolean over all leading position axes. Sanitize invalid x before reductions, and zero every invalid output after any shift or bias.
Apply affine population LayerNorm to x using the supplied scale and bias.
Normalize over the final channel axis; eps defaults to $10^{-5}$ and is added inside the square root. Supplied weights use input-channel-first orientation.
Use CPU float32/float64 tensors with one shared floating dtype/device. Preserve strided layouts and do not mutate supplied tensors.
Use raw tensor operations, not a ready-made implementation of the requested primitive. Supplied parameters and random variates must not be reinitialized.
For each valid vector, let $\mu=\operatorname{mean}_C(x)$ and $v=\operatorname{mean}_C((x-\mu)^2)$ . Return $(x-\mu)/\sqrt{v+\varepsilon}\odot scale+bias$ .

Why this matters

Each position is normalized using only its own channel mean and population variance, then transformed by the supplied channel scale and bias.

Starter code

You write the body of this function in the BioTorch workspace. Your code runs in an isolated PyTorch sandbox; Run checks the public examples and Submit grades the full test suite, including gradients.

import torch


def masked_layer_normalization(
    x: torch.Tensor,  # (..., C)
    valid: torch.Tensor,  # (...)
    scale: torch.Tensor,  # (C,)
    bias: torch.Tensor,  # (C,)
    eps: float = 1e-05,
) -> torch.Tensor:
    """Return a tensor with shape (..., C)."""
    # Implement the operation. Inputs and test setup are supplied by BioTorch.
    raise NotImplementedError("Implement masked_layer_normalization")

Ready to implement it?

3 progressive hints, a solution walkthrough, and grading that checks values, shapes and gradients are waiting in the workspace.

Write and run your solution Sign in

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Related exercises

Sources and further reading

A primitive exercise, not an end-to-end reproduction of the named models. No affiliation or endorsement is implied.