linear_remainder

Bases: remainder

The linear remainder function.

It calculates the linear remainder, and returns remainder term of certain length. The identity remainder function can only be applied when the input and output dimensions are equal. For more general cases where the input and output dimensions are different, the linear remainder function can be used. This class inherits from the base remainder class (i.e., the remainder class in the module directory).

...

Notes

The linear remainder function \(\pi: {R}^m \to {R}^n\) just projects all inputs to themselves subject to linear transformations, i.e., $$ \begin{equation} \pi(\mathbf{x}) = \mathbf{x} \mathbf{W} \in {R}^n, \end{equation} $$ where the learnable parameter matrix \(\mathbf{W} \in R^{m \times n}\) is used for vector dimension adjustment.

By default, the remainder term will also be processed with the optional activation functions.

Attributes:

Name	Type	Description
name	str, default = 'linear_remainder'	Name of the remainder function.

Methods:

Name	Description
__init__	It initializes the linear remainder function.
forward	It implements the abstract forward method declared in the base remainder class.

Source code in tinybig/remainder/basic_remainder.py

class linear_remainder(remainder):
    r"""
    The linear remainder function.

    It calculates the linear remainder, and returns remainder term of certain length.
    The identity remainder function can only be applied when the input and output dimensions are equal.
    For more general cases where the input and output dimensions are different, the linear remainder function can be used.
    This class inherits from the base remainder class (i.e., the remainder class in the module directory).

    ...

    Notes
    ----------
    The linear remainder function $\pi: {R}^m \to {R}^n$ just projects all inputs to themselves subject to linear
    transformations, i.e.,
    $$
        \begin{equation}
            \pi(\mathbf{x}) = \mathbf{x} \mathbf{W} \in {R}^n,
        \end{equation}
    $$
    where the learnable parameter matrix $\mathbf{W} \in R^{m \times n}$ is used for vector dimension adjustment.

    By default, the remainder term will also be processed with the optional activation functions.

    Attributes
    ----------
    name: str, default = 'linear_remainder'
        Name of the remainder function.

    Methods
    ----------
    __init__
        It initializes the linear remainder function.

    forward
        It implements the abstract forward method declared in the base remainder class.
    """
    def __init__(self, name: str = 'linear_remainder', require_parameters: bool = True, enable_bias: bool = False, *args, **kwargs):
        """
        The initialization method of the linear remainder function.

        It initializes a linear remainder function object.
        This method will also call the initialization method of the base class as well.

        Parameters
        ----------
        name: str, default = 'linear_remainder'
            Name of the linear remainder function.
        require_parameters: bool, default = False
            Boolean tag of whether the function requires parameters.
        enable_bias: bool, default = False
            Boolean tag of whether the bias is enabled or not.

        Returns
        ----------
        object
            The linear remainder function object.
        """
        super().__init__(name=name, require_parameters=require_parameters, enable_bias=enable_bias, *args, **kwargs)

    def calculate_l(self, n: int, D: int):
        return n*D

    def forward(self, x: torch.Tensor, w: torch.nn.Parameter, b: torch.nn.Parameter = None, device='cpu', *args, **kwargs):
        r"""
        The forward method of the linear remainder function.

        The linear remainder function $\pi: {R}^m \to {R}^n$ just projects all inputs to themselves subject to linear
        transformations, i.e.,
        $$
            \begin{equation}
                \pi(\mathbf{x}) = \mathbf{x} \mathbf{W} \in {R}^n,
            \end{equation}
        $$
        where the learnable parameter matrix $\mathbf{W} \in R^{m \times n}$ is used for vector dimension adjustment.

        By default, the remainder term will also be processed with the optional activation functions.

        Parameters
        ----------
        x: torch.Tensor
            The input data vector.
        w: torch.nn.Parameter
            The linear transformation parameter.
        b: torch.nn.Parameter, default = None
            The linear transformation bias parameter. It will not be None if attribute "enable_bias" is assigned with "True" value at initialization.
        device: str, default = 'cpu'
            Device to calculate the remainder function.

        Returns
        ----------
        torch.Tensor
            The remainder term of the input.
        """

        if w is not None:
            x = F.linear(x, w, bias=b)
        return self.activation(x=x, device=device)

__init__(name='linear_remainder', require_parameters=True, enable_bias=False, *args, **kwargs)

The initialization method of the linear remainder function.

It initializes a linear remainder function object. This method will also call the initialization method of the base class as well.

Parameters:

Name	Type	Description	Default
name	str	Name of the linear remainder function.	'linear_remainder'
require_parameters	bool	Boolean tag of whether the function requires parameters.	True
enable_bias	bool	Boolean tag of whether the bias is enabled or not.	False

Returns:

Type	Description
object	The linear remainder function object.

Source code in tinybig/remainder/basic_remainder.py

def __init__(self, name: str = 'linear_remainder', require_parameters: bool = True, enable_bias: bool = False, *args, **kwargs):
    """
    The initialization method of the linear remainder function.

    It initializes a linear remainder function object.
    This method will also call the initialization method of the base class as well.

    Parameters
    ----------
    name: str, default = 'linear_remainder'
        Name of the linear remainder function.
    require_parameters: bool, default = False
        Boolean tag of whether the function requires parameters.
    enable_bias: bool, default = False
        Boolean tag of whether the bias is enabled or not.

    Returns
    ----------
    object
        The linear remainder function object.
    """
    super().__init__(name=name, require_parameters=require_parameters, enable_bias=enable_bias, *args, **kwargs)

forward(x, w, b=None, device='cpu', *args, **kwargs)

The forward method of the linear remainder function.

The linear remainder function \(\pi: {R}^m \to {R}^n\) just projects all inputs to themselves subject to linear transformations, i.e., $$ \begin{equation} \pi(\mathbf{x}) = \mathbf{x} \mathbf{W} \in {R}^n, \end{equation} $$ where the learnable parameter matrix \(\mathbf{W} \in R^{m \times n}\) is used for vector dimension adjustment.

By default, the remainder term will also be processed with the optional activation functions.

Parameters:

Name	Type	Description	Default
x	Tensor	The input data vector.	required
w	Parameter	The linear transformation parameter.	required
b	Parameter	The linear transformation bias parameter. It will not be None if attribute "enable_bias" is assigned with "True" value at initialization.	None
device		Device to calculate the remainder function.	'cpu'

Returns:

Type	Description
Tensor	The remainder term of the input.

Source code in tinybig/remainder/basic_remainder.py

def forward(self, x: torch.Tensor, w: torch.nn.Parameter, b: torch.nn.Parameter = None, device='cpu', *args, **kwargs):
    r"""
    The forward method of the linear remainder function.

    The linear remainder function $\pi: {R}^m \to {R}^n$ just projects all inputs to themselves subject to linear
    transformations, i.e.,
    $$
        \begin{equation}
            \pi(\mathbf{x}) = \mathbf{x} \mathbf{W} \in {R}^n,
        \end{equation}
    $$
    where the learnable parameter matrix $\mathbf{W} \in R^{m \times n}$ is used for vector dimension adjustment.

    By default, the remainder term will also be processed with the optional activation functions.

    Parameters
    ----------
    x: torch.Tensor
        The input data vector.
    w: torch.nn.Parameter
        The linear transformation parameter.
    b: torch.nn.Parameter, default = None
        The linear transformation bias parameter. It will not be None if attribute "enable_bias" is assigned with "True" value at initialization.
    device: str, default = 'cpu'
        Device to calculate the remainder function.

    Returns
    ----------
    torch.Tensor
        The remainder term of the input.
    """

    if w is not None:
        x = F.linear(x, w, bias=b)
    return self.activation(x=x, device=device)