Foundations

Gated Feedforward Block

MediumRaw PyTorchAbout 20 min2 public examples plus private tests

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Your task

At each input position, compute an up-projected feature vector and a separate gate vector. Activate the gate, multiply it channel by channel with the unactivated up vector, then project to output features. Return the update without a residual.

Function signature

gated_feedforward(
    x: torch.Tensor,
    w_up: torch.Tensor,
    w_gate: torch.Tensor,
    w_down: torch.Tensor,
    b_up: torch.Tensor | None = None,
    b_gate: torch.Tensor | None = None,
    b_down: torch.Tensor | None = None,
    activation: str = 'silu',
) -> torch.Tensor

Worked example

Trace the two branches

The up-projection is [1, 2]; the gate projection is [2, -2]. ReLU changes the gate to [2, 0]. Multiply the unactivated up branch by this activated gate to obtain [2, 0], then apply the down-projection: $[2,0]\begin{bmatrix}1&3\\4&2\end{bmatrix}=[2,6]$ .

import torch

x = torch.tensor([[1.0, 2.0]], dtype=torch.float64, requires_grad=True)
w_up = torch.tensor([[1.0, 0.0], [0.0, 1.0]], dtype=torch.float64, requires_grad=True)
w_gate = torch.tensor([[2.0, 0.0], [0.0, -1.0]], dtype=torch.float64, requires_grad=True)
w_down = torch.tensor([[1.0, 3.0], [4.0, 2.0]], dtype=torch.float64, requires_grad=True)
b_up = None
b_gate = None
b_down = None
activation = 'relu'

result = gated_feedforward(x=x, w_up=w_up, w_gate=w_gate, w_down=w_down, b_up=b_up, b_gate=b_gate, b_down=b_down, activation=activation)

Expected return

torch.tensor([[2.0, 6.0]], dtype=torch.float64)

Parameters

Parameter	Shape and meaning
`x`torch.Tensor	$[\ldots,C_{in}]$ Input feature vectors shared by the up and gate projections.
`w_up`torch.Tensor	$[\mathrm{Cin}, H]$ Input-to-hidden projection for the up branch.
`w_gate`torch.Tensor	$[\mathrm{Cin}, H]$ Input-to-hidden projection for the independent gate branch.
`w_down`torch.Tensor	$[H, \mathrm{Cout}]$ Hidden-to-output projection.
`b_up`torch.Tensor \| NoneDefault `None`	$[H]$ Optional up-branch hidden bias.
`b_gate`torch.Tensor \| NoneDefault `None`	$[H]$ Optional gate-branch hidden bias.
`b_down`torch.Tensor \| NoneDefault `None`	$[\mathrm{Cout}]$ Optional output bias.
`activation`strDefault `'silu'`	Choose "silu", exact erf-based "gelu", "gelu_tanh", "gelu_sigmoid", or "relu". The two GELU approximations are distinct named choices.

Return

torch.Tensor · $[\ldots,C_{out}]$

Return Feedforward update; no input residual is added.

Required behavior

Compute $u=xW_{up}+b_{up}$ , $g=xW_{gate}+b_{gate}$ , and $y=(u\odot f(g))W_{down}+b_{down}$ .
All weights use input-channel-first orientation. x has shape $[...,Cin]$ , up/gate weights(Cin,H), and down weight $[H,Cout]$ . Each bias is optional and broadcasts over leading axes.
activation defaults to silu; allowed alternatives are relu, exact erf gelu, gelu_tanh, and gelu_sigmoid. Their exact formulas are those of the corresponding activation exercises.
For a gated block, activate only the gate branch, not the up branch. Do not add a residual, normalize, or apply dropout.
Activation choices are $ReLU(x)=\max(x,0)$ , $SiLU(x)=x/(1+e^{-x})$ , and exact $GELU(x)=x[1+erf(x/\sqrt2)]/2$ .
The named GELU approximations are $x[1+\tanh(\sqrt{2/\pi}(x+0.044715x^3))]/2$ and $x\,sigmoid(1.702x)$ , respectively.

Why this matters

A gate controls a separate linear feature branch. Activating the wrong branch produces a different operation even when every shape matches.

Starter code

You write the body of this function in the BioTorch workspace. Your code runs in an isolated PyTorch sandbox; Run checks the public examples and Submit grades the full test suite, including gradients.

import torch


def gated_feedforward(
    x: torch.Tensor,  # (...,C_{in})
    w_up: torch.Tensor,  # (Cin, H)
    w_gate: torch.Tensor,  # (Cin, H)
    w_down: torch.Tensor,  # (H, Cout)
    b_up: torch.Tensor | None = None,  # (H,)
    b_gate: torch.Tensor | None = None,  # (H,)
    b_down: torch.Tensor | None = None,  # (Cout,)
    activation: str = 'silu',
) -> torch.Tensor:
    """Return a tensor with shape (...,C_{out})."""
    # Implement the operation. Inputs and test setup are supplied by BioTorch.
    raise NotImplementedError("Implement gated_feedforward")

Ready to implement it?

3 progressive hints, a solution walkthrough, and grading that checks values, shapes and gradients are waiting in the workspace.

Write and run your solution Sign in

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Related exercises

Sources and further reading

A primitive exercise, not an end-to-end reproduction of the named models. No affiliation or endorsement is implied.