Foundations

Tanh GELU Approximation

EasyRaw PyTorchAbout 10 min1 public example plus private tests

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Your task

Multiply each input entry by the smooth gain defined by the tanh GELU approximation below. Return one activated value at each input position, using the stated cubic term and constants.

Function signature

gelu_tanh_approximation(x: torch.Tensor) -> torch.Tensor

Worked example

Follow one activation

For $x=1$ , the tanh argument is $a=\sqrt{2/\pi}\,(1+0.044715)\approx0.833562$ . The output is $(1+\tanh a)/2\approx0.841192$ . At $x=-1$ the result is approximately $-0.158808$ ; at zero it is zero.

Parameters

Parameter	Shape and meaning
`x`torch.Tensor	Any shape, including empty axes.Values to activate independently at each position.

Return

torch.Tensor · Exactly x.shape.

Return Transformed values.

Required behavior

$f(x)=\tfrac12x[1+\tanh(\sqrt{2/\pi}(x+0.044715x^3))]$ . Use these constants and this approximation exactly.
All axes, including empty axes, are preserved. Do not introduce learned parameters or a reduction.

Why this matters

A gate controls a separate linear feature branch. Activating the wrong branch produces a different operation even when every shape matches.

Starter code

You write the body of this function in the BioTorch workspace. Your code runs in an isolated PyTorch sandbox; Run checks the public examples and Submit grades the full test suite, including gradients.

import torch


def gelu_tanh_approximation(
    x: torch.Tensor,  # Any shape, including empty axes.
) -> torch.Tensor:
    """Return a tensor with shape x.shape."""
    # Implement the operation. Inputs and test setup are supplied by BioTorch.
    raise NotImplementedError("Implement gelu_tanh_approximation")

Ready to implement it?

3 progressive hints, a solution walkthrough, and grading that checks values, shapes and gradients are waiting in the workspace.

Write and run your solution Sign in

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Related exercises

Sources and further reading

A primitive exercise, not an end-to-end reproduction of the named models. No affiliation or endorsement is implied.