identity_reconciliation

Bases: fabrication

The identity parameter reconciliation function.

It performs the identity parameter reconciliation, and returns the identity reconciled parameter matrix of shape (n, D). This class inherits from the reconciliation class (i.e., the fabrication class in the module directory).

...

Notes

The identity parameter reconciliation projects any input parameters to themselves as follows: $$ \begin{equation} \psi(\mathbf{w}) = \text{reshape}(\mathbf{w}) = \mathbf{W} \in {R}^{n \times D}, \end{equation} $$ where the function will reshape the parameters from vector \(\mathbf{w}\) of length \(l\) to the matrix \(\mathbf{W}\) of size \(n \times D\).

For the identity parameter reconciliation function, the required number of parameter length can be denoted as $$ \begin{equation} l = n \times D, \end{equation} $$ where \(n\) and \(D\) denote the output space and expansion space dimensions, respectively.

Identity parameter reconciliation is straightforward and may work well for some expansion functions whose output dimension \(D\) is not very large. However, when used with expansion functions that produce a large output dimension (such as the high-order Taylor's polynomial expansions), the identity parameter reconciliation function may fail due to the "curse of dimensionality" issues.

Attributes:

Name	Type	Description
name	str, default = 'identity_reconciliation'	Name of the parameter reconciliation function

Methods:

Name	Description
__init__	It initializes the parameter reconciliation function.
calculate_l	It calculates the length of required parameters.
forward	It implements the abstract forward method declared in the base reconciliation class.

Source code in tinybig/reconciliation/basic_reconciliation.py

class identity_reconciliation(fabrication):
    r"""
    The identity parameter reconciliation function.

    It performs the identity parameter reconciliation, and returns the identity reconciled parameter matrix of shape (n, D).
    This class inherits from the reconciliation class (i.e., the fabrication class in the module directory).

    ...

    Notes
    ----------
    The **identity parameter reconciliation** projects any input parameters to themselves as follows:
    $$
        \begin{equation}
            \psi(\mathbf{w}) = \text{reshape}(\mathbf{w}) = \mathbf{W} \in {R}^{n \times D},
        \end{equation}
    $$
    where the function will reshape the parameters from vector $\mathbf{w}$ of length $l$ to the matrix
    $\mathbf{W}$ of size $n \times D$.

    For the identity parameter reconciliation function, the required number of parameter length can be denoted as
    $$
        \begin{equation}
            l = n \times D,
        \end{equation}
    $$
    where $n$ and $D$ denote the output space and expansion space dimensions, respectively.

    Identity parameter reconciliation is straightforward and may work well for some expansion functions whose
    output dimension $D$ is not very large.
    However, when used with expansion functions that produce a large output dimension
    (such as the high-order Taylor's polynomial expansions), the identity parameter reconciliation
    function may fail due to the "curse of dimensionality" issues.

    Attributes
    ----------
    name: str, default = 'identity_reconciliation'
        Name of the parameter reconciliation function

    Methods
    ----------
    __init__
        It initializes the parameter reconciliation function.

    calculate_l
        It calculates the length of required parameters.

    forward
        It implements the abstract forward method declared in the base reconciliation class.
    """
    def __init__(self, name='identity_reconciliation', *args, **kwargs):
        """
        The initialization method of the identity parameter reconciliation function.

        It initializes an identity parameter reconciliation function object.
        This method will also call the initialization method of the base class as well.

        Parameters
        ----------
        name: str, default = 'identity_reconciliation'
            Name of the identity parameter reconciliation function.

        Returns
        ----------
        fabrication
            The identity parameter reconciliation function object.
        """
        super().__init__(name=name, *args, **kwargs)

    def calculate_l(self, n: int, D: int):
        r"""
        The required parameter number calculation method.

        It calculates the number of required learnable parameters, i.e., l, of the parameter reconciliation function
        based on the intermediate and output space dimensions, n and D, which can be represented as follows:
        $$
            \begin{equation}
                l = n \times D.
            \end{equation}
        $$

        Parameters
        ----------
        n: int
            The dimension of the output space.
        D: int
            The dimension of the intermediate expansion space.

        Returns
        -------
        int
            The number of required learnable parameters.
        """
        return n*D

    def forward(self, n: int, D: int, w: torch.nn.Parameter, device: str = 'cpu', *args, **kwargs):
        r"""
        The forward method of the parameter reconciliation function.

        It applies the identity parameter reconciliation operation to the input parameter of length l,
        and returns the reconciled parameter matrix of shape (n, D) as follows:
        $$
            \begin{equation}
                \psi(\mathbf{w}) = \text{reshape}(\mathbf{w}) = \mathbf{W} \in {R}^{n \times D},
            \end{equation}
        $$

        Parameters
        ----------
        n: int
            The dimension of the output space.
        D: int
            The dimension of the intermediate expansion space.
        w: torch.nn.Parameter, default = None
            The learnable parameters of the model.
        device: str, default = 'cpu'
            Device to perform the parameter reconciliation.

        Returns
        ----------
        torch.Tensor
            The reconciled parameter matrix of shape (n, D).
        """
        assert w.ndim == 2 and w.numel() == self.calculate_l(n=n, D=D)
        return w.view(n, D).to(device)

__init__(name='identity_reconciliation', *args, **kwargs)

The initialization method of the identity parameter reconciliation function.

It initializes an identity parameter reconciliation function object. This method will also call the initialization method of the base class as well.

Parameters:

Name	Type	Description	Default
name		Name of the identity parameter reconciliation function.	'identity_reconciliation'

Returns:

Type	Description
fabrication	The identity parameter reconciliation function object.

Source code in tinybig/reconciliation/basic_reconciliation.py

def __init__(self, name='identity_reconciliation', *args, **kwargs):
    """
    The initialization method of the identity parameter reconciliation function.

    It initializes an identity parameter reconciliation function object.
    This method will also call the initialization method of the base class as well.

    Parameters
    ----------
    name: str, default = 'identity_reconciliation'
        Name of the identity parameter reconciliation function.

    Returns
    ----------
    fabrication
        The identity parameter reconciliation function object.
    """
    super().__init__(name=name, *args, **kwargs)

calculate_l(n, D)

The required parameter number calculation method.

It calculates the number of required learnable parameters, i.e., l, of the parameter reconciliation function based on the intermediate and output space dimensions, n and D, which can be represented as follows: $$ \begin{equation} l = n \times D. \end{equation} $$

Parameters:

Name	Type	Description	Default
n	int	The dimension of the output space.	required
D	int	The dimension of the intermediate expansion space.	required

Returns:

Type	Description
int	The number of required learnable parameters.

Source code in tinybig/reconciliation/basic_reconciliation.py

def calculate_l(self, n: int, D: int):
    r"""
    The required parameter number calculation method.

    It calculates the number of required learnable parameters, i.e., l, of the parameter reconciliation function
    based on the intermediate and output space dimensions, n and D, which can be represented as follows:
    $$
        \begin{equation}
            l = n \times D.
        \end{equation}
    $$

    Parameters
    ----------
    n: int
        The dimension of the output space.
    D: int
        The dimension of the intermediate expansion space.

    Returns
    -------
    int
        The number of required learnable parameters.
    """
    return n*D

forward(n, D, w, device='cpu', *args, **kwargs)

The forward method of the parameter reconciliation function.

It applies the identity parameter reconciliation operation to the input parameter of length l, and returns the reconciled parameter matrix of shape (n, D) as follows: $$ \begin{equation} \psi(\mathbf{w}) = \text{reshape}(\mathbf{w}) = \mathbf{W} \in {R}^{n \times D}, \end{equation} $$

Parameters:

Name	Type	Description	Default
n	int	The dimension of the output space.	required
D	int	The dimension of the intermediate expansion space.	required
w	Parameter	The learnable parameters of the model.	required
device	str	Device to perform the parameter reconciliation.	'cpu'

Returns:

Type	Description
Tensor	The reconciled parameter matrix of shape (n, D).

Source code in tinybig/reconciliation/basic_reconciliation.py

def forward(self, n: int, D: int, w: torch.nn.Parameter, device: str = 'cpu', *args, **kwargs):
    r"""
    The forward method of the parameter reconciliation function.

    It applies the identity parameter reconciliation operation to the input parameter of length l,
    and returns the reconciled parameter matrix of shape (n, D) as follows:
    $$
        \begin{equation}
            \psi(\mathbf{w}) = \text{reshape}(\mathbf{w}) = \mathbf{W} \in {R}^{n \times D},
        \end{equation}
    $$

    Parameters
    ----------
    n: int
        The dimension of the output space.
    D: int
        The dimension of the intermediate expansion space.
    w: torch.nn.Parameter, default = None
        The learnable parameters of the model.
    device: str, default = 'cpu'
        Device to perform the parameter reconciliation.

    Returns
    ----------
    torch.Tensor
        The reconciled parameter matrix of shape (n, D).
    """
    assert w.ndim == 2 and w.numel() == self.calculate_l(n=n, D=D)
    return w.view(n, D).to(device)