Foundations

Feedforward Block

EasyRaw PyTorchAbout 10 min1 public example plus private tests

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Your task

Transform each input feature vector into hidden channels, apply the selected activation to those channels, then project them to output features. Every leading position is processed independently; return this update without adding a residual.

Function signature

feedforward(
    x: torch.Tensor,
    w_up: torch.Tensor,
    w_down: torch.Tensor,
    b_up: torch.Tensor | None = None,
    b_down: torch.Tensor | None = None,
    activation: str = 'silu',
) -> torch.Tensor

Worked example

The up-projection plus bias gives [1.5, -0.5, 2.5] and [0.5, 2.5, -1.5]. ReLU clips their negative entries. The final projection plus bias returns [[-0.5], [6.0]], shape (2, 1).

import torch

x = torch.tensor([[1.0, -1.0], [0.0, 2.0]], dtype=torch.float64, requires_grad=True)
w_up = torch.tensor([[1.0, 0.0, 1.0], [0.0, 1.0, -1.0]], dtype=torch.float64, requires_grad=True)
w_down = torch.tensor([[1.0], [2.0], [-1.0]], dtype=torch.float64, requires_grad=True)
b_up = torch.tensor([0.5, 0.5, 0.5], dtype=torch.float64, requires_grad=True)
b_down = torch.tensor([0.5], dtype=torch.float64, requires_grad=True)
activation = 'relu'

result = feedforward(x=x, w_up=w_up, w_down=w_down, b_up=b_up, b_down=b_down, activation=activation)

Expected return

torch.tensor([[-0.5], [6.0]], dtype=torch.float64)

Parameters

Parameter	Shape and meaning
`x`torch.Tensor	$[\ldots,C_{in}]$ Input feature vectors, one at each combination of leading indices.
`w_up`torch.Tensor	$[\mathrm{Cin}, H]$ Input-to-hidden projection for the up branch.
`w_down`torch.Tensor	$[H, \mathrm{Cout}]$ Hidden-to-output projection.
`b_up`torch.Tensor \| NoneDefault `None`	$[H]$ Optional up-branch hidden bias.
`b_down`torch.Tensor \| NoneDefault `None`	$[\mathrm{Cout}]$ Optional output bias.
`activation`strDefault `'silu'`	Choose "silu", exact erf-based "gelu", "gelu_tanh", "gelu_sigmoid", or "relu". The two GELU approximations are distinct named choices.

Return

torch.Tensor · $[\ldots,C_{out}]$

Return Feedforward update; no input residual is added.

Required behavior

Compute $u=xW_{up}+b_{up}$ , then $y=f(u)W_{down}+b_{down}$ . Omit a bias term when its argument is None.
Weights use input-channel-first orientation: $W_{up}$ has shape $(C_{in},H)$ , and $W_{down}$ has shape $(H,C_{out})$ . Preserve every leading axis of x.
activation defaults to silu. Alternatives are relu, exact erf gelu, gelu_tanh, and gelu_sigmoid, using the formulas from the corresponding activation exercises.
This block has one hidden branch. Do not add a gate, input residual, normalization, or dropout.
Use supplied CPU torch.float32 or torch.float64 tensors with shared floating dtype/device. Do not mutate inputs or initialize new parameters; preserve arbitrary valid leading dimensions and accept noncontiguous inputs.
The activation formulas are: ReLU max(x,0), SiLU x·sigmoid(x), exact GELU x·(1+erf(x/sqrt(2)))/2, gelu_tanh x·(1+tanh(sqrt(2/pi)·(x+0.044715x³)))/2, and gelu_sigmoid x·sigmoid(1.702x).

Why this matters

A feedforward block expands a feature vector, applies a nonlinearity, then projects it to the output width.

Starter code

You write the body of this function in the BioTorch workspace. Your code runs in an isolated PyTorch sandbox; Run checks the public examples and Submit grades the full test suite, including gradients.

import torch


def feedforward(
    x: torch.Tensor,  # (...,C_{in})
    w_up: torch.Tensor,  # (Cin, H)
    w_down: torch.Tensor,  # (H, Cout)
    b_up: torch.Tensor | None = None,  # (H,)
    b_down: torch.Tensor | None = None,  # (Cout,)
    activation: str = 'silu',
) -> torch.Tensor:
    """Return a tensor with shape (...,C_{out})."""
    # Implement the operation. Inputs and test setup are supplied by BioTorch.
    raise NotImplementedError("Implement feedforward")

Ready to implement it?

3 progressive hints, a solution walkthrough, and grading that checks values, shapes and gradients are waiting in the workspace.

Write and run your solution Sign in

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Related exercises

Sources and further reading

A primitive exercise, not an end-to-end reproduction of the named models. No affiliation or endorsement is implied.