constant_eye_reconciliation

Bases: fabrication

The constant eye parameter reconciliation function.

It performs the constant eye parameter reconciliation, and returns the constant eye parameter matrix of shape (n, D). It is a special case of the constant_reconciliation function defined above. This class inherits from the reconciliation class (i.e., the fabrication class in the module directory).

...

Notes

As a special case of the constant_reconciliation function defined above, the constant eye parameter reconciliation projects any input parameters to constant eye matrix as follows: $$ \begin{equation} \psi(\mathbf{w}) = \mathbf{I}^{n \times D} \in {R}^{n \times D}, \end{equation} $$ where the output matrix \(\mathbf{I}\) of size \(n \times D\) is returned as an eye matrix.

For constant eye parameter reconciliation, the input parameter \(\mathbf{w}\) is not required, which together with its dimension hyper-parameter \(l\) can both be set to \textit{none} in implementation.

Similar as the above constant parameter reconciliation function, the constant eye reconciliation contributes almost nothing to model learning since it involves no learnable parameters, but it provides RPN with substantial flexibility in representing and designing many models.

Attributes:

Name	Type	Description
name	str, default = 'constant_eye_reconciliation'	Name of the reconciliation function

Methods:

Name	Description
__init__	It initializes the parameter reconciliation function.
calculate_l	It calculates the length of required parameters.
forward	It implements the abstract forward method declared in the base reconciliation class.

Source code in tinybig/reconciliation/basic_reconciliation.py

class constant_eye_reconciliation(fabrication):
    r"""
    The constant eye parameter reconciliation function.

    It performs the constant eye parameter reconciliation, and returns the constant eye parameter matrix of shape (n, D).
    It is a special case of the constant_reconciliation function defined above.
    This class inherits from the reconciliation class (i.e., the fabrication class in the module directory).

    ...

    Notes
    ----------
    As a special case of the constant_reconciliation function defined above,
    the **constant eye parameter reconciliation** projects any input parameters to constant eye matrix as follows:
    $$
        \begin{equation}
            \psi(\mathbf{w}) = \mathbf{I}^{n \times D} \in {R}^{n \times D},
        \end{equation}
    $$
    where the output matrix $\mathbf{I}$ of size $n \times D$ is returned as an eye matrix.

    For constant eye parameter reconciliation, the input parameter $\mathbf{w}$ is not required,
    which together with its dimension hyper-parameter $l$ can both be set to \textit{none} in implementation.

    Similar as the above constant parameter reconciliation function, the constant eye reconciliation
    contributes almost nothing to model learning since it involves no learnable parameters, but it provides
    RPN with substantial flexibility in representing and designing many models.

    Attributes
    ----------
    name: str, default = 'constant_eye_reconciliation'
        Name of the reconciliation function

    Methods
    ----------
    __init__
        It initializes the parameter reconciliation function.

    calculate_l
        It calculates the length of required parameters.

    forward
        It implements the abstract forward method declared in the base reconciliation class.
    """
    def __init__(self, name='constant_eye_reconciliation', *args, **kwargs):
        """
        The initialization method of the constant eye parameter reconciliation function.

        It initializes a constant eye parameter reconciliation function object.
        This method will also call the initialization method of the base class as well.
        Since the constant eye parameter reconciliation doesn't require parameters, it will
        set the "require_parameters" as False in the initialization.

        Parameters
        ----------
        name: str, default = 'constant_eye_reconciliation'
            Name of the constant eye parameter reconciliation function.

        Returns
        ----------
        fabrication
            The constant eye parameter reconciliation function object.
        """
        super().__init__(name=name, require_parameters=False, *args, **kwargs)

    def calculate_l(self, n: int, D: int):
        """
        The required parameter number calculation method.

        It calculates the number of required learnable parameters, i.e., l, of the parameter reconciliation function
        based on the intermediate and output space dimensions, n and D.
        For constant eye parameter reconciliation, it doesn't require any learnable parameters, and this function
        will return the parameter number as 0 by default.

        Parameters
        ----------
        n: int
            The dimension of the output space.
        D: int
            The dimension of the intermediate expansion space.

        Returns
        -------
        int
            The number of required learnable parameters.
        """
        return 0

    def forward(self, n: int, D: int, w: torch.nn.Parameter = None, device='cpu', *args, **kwargs):
        r"""
        The forward method of the parameter reconciliation function.

        It applies the constant eye parameter reconciliation operation to the input parameter of length l,
        and returns the reconciled parameter matrix of shape (n, D) as follows:
        $$
            \begin{equation}
                \psi(\mathbf{w}) = \mathbf{I}^{n \times D} \in {R}^{n \times D},
            \end{equation}
        $$
        where the output matrix $\mathbf{I}$ of size $n \times D$ is returned as an eye matrix.

        Parameters
        ----------
        n: int
            The dimension of the output space.
        D: int
            The dimension of the intermediate expansion space.
        w: torch.nn.Parameter, default = None
            The learnable parameters of the model.
            For constant eye reconciliation, it is assigned with a default value None.
        device: str, default = 'cpu'
            Device to perform the parameter reconciliation.

        Returns
        ----------
        torch.Tensor
            The reconciled parameter matrix of shape (n, D).
        """
        return torch.eye(n=n, m=D, device=device)

__init__(name='constant_eye_reconciliation', *args, **kwargs)

The initialization method of the constant eye parameter reconciliation function.

It initializes a constant eye parameter reconciliation function object. This method will also call the initialization method of the base class as well. Since the constant eye parameter reconciliation doesn't require parameters, it will set the "require_parameters" as False in the initialization.

Parameters:

Name	Type	Description	Default
name		Name of the constant eye parameter reconciliation function.	'constant_eye_reconciliation'

Returns:

Type	Description
fabrication	The constant eye parameter reconciliation function object.

Source code in tinybig/reconciliation/basic_reconciliation.py

def __init__(self, name='constant_eye_reconciliation', *args, **kwargs):
    """
    The initialization method of the constant eye parameter reconciliation function.

    It initializes a constant eye parameter reconciliation function object.
    This method will also call the initialization method of the base class as well.
    Since the constant eye parameter reconciliation doesn't require parameters, it will
    set the "require_parameters" as False in the initialization.

    Parameters
    ----------
    name: str, default = 'constant_eye_reconciliation'
        Name of the constant eye parameter reconciliation function.

    Returns
    ----------
    fabrication
        The constant eye parameter reconciliation function object.
    """
    super().__init__(name=name, require_parameters=False, *args, **kwargs)

calculate_l(n, D)

The required parameter number calculation method.

It calculates the number of required learnable parameters, i.e., l, of the parameter reconciliation function based on the intermediate and output space dimensions, n and D. For constant eye parameter reconciliation, it doesn't require any learnable parameters, and this function will return the parameter number as 0 by default.

Parameters:

Name	Type	Description	Default
n	int	The dimension of the output space.	required
D	int	The dimension of the intermediate expansion space.	required

Returns:

Type	Description
int	The number of required learnable parameters.

Source code in tinybig/reconciliation/basic_reconciliation.py

def calculate_l(self, n: int, D: int):
    """
    The required parameter number calculation method.

    It calculates the number of required learnable parameters, i.e., l, of the parameter reconciliation function
    based on the intermediate and output space dimensions, n and D.
    For constant eye parameter reconciliation, it doesn't require any learnable parameters, and this function
    will return the parameter number as 0 by default.

    Parameters
    ----------
    n: int
        The dimension of the output space.
    D: int
        The dimension of the intermediate expansion space.

    Returns
    -------
    int
        The number of required learnable parameters.
    """
    return 0

forward(n, D, w=None, device='cpu', *args, **kwargs)

The forward method of the parameter reconciliation function.

It applies the constant eye parameter reconciliation operation to the input parameter of length l, and returns the reconciled parameter matrix of shape (n, D) as follows: $$ \begin{equation} \psi(\mathbf{w}) = \mathbf{I}^{n \times D} \in {R}^{n \times D}, \end{equation} $$ where the output matrix \(\mathbf{I}\) of size \(n \times D\) is returned as an eye matrix.

Parameters:

Name	Type	Description	Default
n	int	The dimension of the output space.	required
D	int	The dimension of the intermediate expansion space.	required
w	Parameter	The learnable parameters of the model. For constant eye reconciliation, it is assigned with a default value None.	None
device		Device to perform the parameter reconciliation.	'cpu'

Returns:

Type	Description
Tensor	The reconciled parameter matrix of shape (n, D).

Source code in tinybig/reconciliation/basic_reconciliation.py

def forward(self, n: int, D: int, w: torch.nn.Parameter = None, device='cpu', *args, **kwargs):
    r"""
    The forward method of the parameter reconciliation function.

    It applies the constant eye parameter reconciliation operation to the input parameter of length l,
    and returns the reconciled parameter matrix of shape (n, D) as follows:
    $$
        \begin{equation}
            \psi(\mathbf{w}) = \mathbf{I}^{n \times D} \in {R}^{n \times D},
        \end{equation}
    $$
    where the output matrix $\mathbf{I}$ of size $n \times D$ is returned as an eye matrix.

    Parameters
    ----------
    n: int
        The dimension of the output space.
    D: int
        The dimension of the intermediate expansion space.
    w: torch.nn.Parameter, default = None
        The learnable parameters of the model.
        For constant eye reconciliation, it is assigned with a default value None.
    device: str, default = 'cpu'
        Device to perform the parameter reconciliation.

    Returns
    ----------
    torch.Tensor
        The reconciled parameter matrix of shape (n, D).
    """
    return torch.eye(n=n, m=D, device=device)