Foundations

Exact GELU

EasyRaw PyTorchAbout 10 min1 public example plus private tests

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Your task

Multiply each input entry x by the standard normal cumulative probability Φ(x). This exact GELU smoothly attenuates entries rather than clipping all negatives to zero; return the input shape.

Function signature

gelu(x: torch.Tensor) -> torch.Tensor

Worked example

Exact GELU at negative, zero and positive inputs

At zero, the result is zero. At one, the normal CDF is approximately 0.84134, so GELU returns about 0.84134; at minus one it returns about -0.15866.

import torch

x = torch.tensor([-1.0, 0.0, 1.0], dtype=torch.float64, requires_grad=True)

result = gelu(x=x)

Expected return

torch.tensor([-0.15865525393145707, 0.0, 0.8413447460685429], dtype=torch.float64)

Parameters

Parameter	Shape and meaning
`x`torch.Tensor	Any shape, including empty axes.Values to activate independently with exact erf-based GELU.

Return

torch.Tensor · Exactly x.shape.

Return Transformed values.

Required behavior

$f(x)=\tfrac12x(1+\operatorname{erf}(x/\sqrt2))$ . This exercise uses exact erf GELU, not a tanh or sigmoid approximation.
All axes, including empty axes, are preserved. Do not introduce learned parameters or a reduction.
Use CPU float32/float64 tensors with one shared floating dtype/device. Accept noncontiguous input layouts and do not mutate supplied tensors.
Use raw tensor operations, not a ready-made implementation of the requested primitive. Supplied parameters and random variates must not be reinitialized.

Why this matters

GELU scales each value by the standard normal cumulative probability at that value, softly suppressing negative inputs.

Starter code

You write the body of this function in the BioTorch workspace. Your code runs in an isolated PyTorch sandbox; Run checks the public examples and Submit grades the full test suite, including gradients.

import torch


def gelu(
    x: torch.Tensor,  # Any shape, including empty axes.
) -> torch.Tensor:
    """Return a tensor with shape x.shape."""
    # Implement the operation. Inputs and test setup are supplied by BioTorch.
    raise NotImplementedError("Implement gelu")

Ready to implement it?

3 progressive hints, a solution walkthrough, and grading that checks values, shapes and gradients are waiting in the workspace.

Write and run your solution Sign in

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Related exercises

Sources and further reading

A primitive exercise, not an end-to-end reproduction of the named models. No affiliation or endorsement is implied.