Genomics and RNA

Complete Evo-style Hyena gated block

ExpertRaw PyTorchAbout 45 min1 public example plus private tests

Evo 1StripedHyena

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Your task

Turn each sequence position into three filtered streams: one drives a long recurrence together with a value stream, and the third gates its readout. Each gated[b,t,d] is that channel’s gated long-filter value before dense projection. Add the projected values and the specified MLP as residual updates to produce y, and return both continuation caches.

Function signature

hyena_operator(
    activations: dict,
    parameters: dict,
    masks: dict,
    column_split: bool = True,
    eps: float = 1e-06,
    heads: int = 1,
) -> dict[str, torch.Tensor]

Worked example

Follow one channel through both gates and caches

Short and long filters. Use one channel, two valid inputs [1,3], illustrative eps=1, pre-scale 2, and zero caches. RMS normalization gives [1,1.5]; projection weights [1,1,2] give (x2,x1,value) rows [1,1,2] and [1.5,1.5,3]. The chronological short filter [1,1] produces [1,1,2] and [2.5,2.5,5]. Thus the drives are [2,12.5]. With real pole 0.5, residue 1, and skip 0, states are $z_0=2$ , $z_1=0.5(2)+12.5=13.5$ ; multiplying by x2 gives gated=[2,33.75].

Both residuals. Identity output projection plus the input residual gives [3,36.75]. Post-scale 1 gives normalized values $n=[3/4,147/151]$ . All three scalar MLP weights are 1, so each final value is $y=r+\operatorname{GELU}(n)n$ : approximately [3.435022,37.541205]. All biases are zero. The FIR cache stores [1.5,1.5,3]; the IIR cache stores [13.5,0].

Parameters

Parameter	Shape and meaning
`activations`dict	Supplied intermediate tensors.
`parameters`dict	Supplied model parameters, keyed as shown below.
`masks`dict	Supplied masks. Each field below defines what True includes or permits.
`column_split`boolDefault `True`	True splits each head’s projected channels into x2, x1, and v before concatenating heads; False uses three contiguous channel groups.
`eps`floatDefault `1e-06`	Positive additive term outside the RMS square root: scale * x / (sqrt(mean(x²)) + eps), at both normalization stages.
`heads`intDefault `1`	Positive number of channel groups for column_split=True; must divide D. These groups organize the three projected streams.

Return

dict[str, torch.Tensor]

Return exactly the named fields applicable to this operation.

Required behavior

Implement one nonempty Evo-style block/chunk from $u$ [ $B,L,D$ ] and all supplied parameters/states. Use $N(x;s)=\frac{s\odot x}{\sqrt{\operatorname{mean}_D(x^2)}+\varepsilon}$ , with $\varepsilon=\mathrm{eps}>0$ (default $10^{-6}$ ) OUTSIDE the square root.
At the zero vector use the continuous zero output and finite $1/\varepsilon$ linear derivative; sanitize invalid $u$ first. Project $N(u)$ to $3 D$ channels with bias, mask, and apply a chronological depthwise short_kernel $[3D,K]$ with supplied fir_state[ $B,K -1$ , $3 D$ ], short_bias, and another output mask.
If column_split=True (default), reshape the $3 D$ channels as [heads, $3\cdot (D/\mathrm{heads})$ ], $\mathrm{heads}=\mathrm{heads}$ (default $1$ ), split each head into $[\mathrm{x2},\mathrm{x1},v]$ , and concatenate each group's head channels. Otherwise split three contiguous $D$ groups.
Drive each mode with $a_{t}=\mathrm{x1\_t}\cdot v_{t}$ : $z_{t}=p\cdot z_{t-1}+a_{t}$ ; $g_{t}=\mathrm{x2\_t}\cdot (\operatorname{Re}(\sum_{s} \mathrm{residues}\cdot z_{t})+\mathrm{skip}\cdot a_{t})$ . Use supplied real/imag iir_state and return its final state.
Form $\mathrm{residual}=u+g\mathbin{@}\mathrm{output\_weight}+\mathrm{output\_bias}$ and mask it.
Normalize this residual with post_scale, then add $\operatorname{GELU}(\mathrm{normalized}\mathbin{@}\mathrm{mlp\_gate\_weight})\cdot (\mathrm{normalized}\mathbin{@}\mathrm{mlp\_value\_weight}) \mathbin{@} \mathrm{mlp\_output\_weight}$ , with exact erf GELU and no MLP biases. Return $y$ , $g$ as gated, and both updated caches.
Invalid time steps advance FIR/IIR history with zero projected drive but have zero final outputs. All pole/residue components, projections, scales, biases and initial states are supplied; there is no pretrained network call or generated noise.
heads is a positive integer dividing D; it groups projected channels and is not an attention-head axis.

Why this matters

Short convolution mixes nearby tokens; a bank of recurrent modes carries longer history. Learned gates decide how much of that filtered signal reaches the output. Explicit caches let one chunk continue exactly where the previous chunk ended.

Starter code

You write the body of this function in the BioTorch workspace. Your code runs in an isolated PyTorch sandbox; Run checks the public examples and Submit grades the full test suite, including gradients.

import torch


def hyena_operator(
    activations: dict,
    parameters: dict,
    masks: dict,
    column_split: bool = True,
    eps: float = 1e-06,
    heads: int = 1,
) -> dict[str, torch.Tensor]:
    """Return dict[str, torch.Tensor]:
    
    y: (B, L, D).
    gated: (B, L, D).
    fir_state: (B, K-1, 3D).
    iir_state: (B, D, S, 2)."""
    # Implement the operation. Inputs and test setup are supplied by BioTorch.
    raise NotImplementedError("Implement hyena_operator")

Ready to implement it?

3 progressive hints, a solution walkthrough, and grading that checks values, shapes and gradients are waiting in the workspace.

Write and run your solution Sign in

Free account with email, Google or GitHub. Your code runs in an isolated PyTorch sandbox.

Sources and further reading

A primitive exercise, not an end-to-end reproduction of the named models. No affiliation or endorsement is implied.