random_matrix_adaption_parameterized_bilinear_interdependence

Bases: parameterized_bilinear_interdependence

A random matrix adaptation parameterized bilinear interdependence function.

This class uses random matrix adaptation to compute bilinear interdependence matrices, where the random matrix serves as a low-rank approximation.

Notes

Formally, given a data batch \(\mathbf{X} \in R^{b \times m}\), we can represent the parameterized bilinear form-based interdependence function as follows:

\[
    \begin{equation}\label{equ:bilinear_interdependence_function}
    \xi(\mathbf{X} | \mathbf{w}) = \mathbf{X}^\top \mathbf{W} \mathbf{X} = \mathbf{A} \in R^{m \times m}.
    \end{equation}
\]

Notation \(\mathbf{W} \in R^{b \times b}\) denotes the parameter matrix fabricated from the learnable parameter vector \(\mathbf{w} \in R^{l_{\xi}}\), which can be represented as follows:

\[
    \begin{equation}
    \psi(\mathbf{w}) = \Lambda_1 \mathbf{A} \Lambda_1 \mathbf{B}^\top \in R^{b \times b},
    \end{equation}
\]

Notations \(\Lambda_1\) and \(\Lambda_2\) denote two diagonal matrices \(\Lambda_1 = diag( \lambda_1) \in R^{m \times m}\) and \(\Lambda_2 = diag(\lambda_2) \in R^{r \times r}\) where the diagonal vectors \(\lambda_1\) and \(\lambda_2\) are partitioned from the parameter vector \(\mathbf{w}\). Matrices \(\mathbf{A} \in R^{b \times r}\) and \(\mathbf{B} \in R^{b \times r}\) are randomly sampled from the Gaussian distribution \(\mathcal{N}(\mathbf{0}, \mathbf{I})\).

The required length of parameter vector of this interdependence function is \(\mathbf{w}\) is \(l_{\xi} = b + r\).

Attributes:

Name	Type	Description
r	int	Rank of the random matrix approximation.

Methods:

Name	Description
__init__	Initializes the random matrix adaptation parameterized bilinear interdependence function.

Source code in tinybig/interdependence/parameterized_bilinear_interdependence.py

class random_matrix_adaption_parameterized_bilinear_interdependence(parameterized_bilinear_interdependence):
    r"""
        A random matrix adaptation parameterized bilinear interdependence function.

        This class uses random matrix adaptation to compute bilinear interdependence matrices,
        where the random matrix serves as a low-rank approximation.

        Notes
        ----------
        Formally, given a data batch $\mathbf{X} \in R^{b \times m}$, we can represent the parameterized bilinear form-based interdependence function as follows:

        $$
            \begin{equation}\label{equ:bilinear_interdependence_function}
            \xi(\mathbf{X} | \mathbf{w}) = \mathbf{X}^\top \mathbf{W} \mathbf{X} = \mathbf{A} \in R^{m \times m}.
            \end{equation}
        $$

        Notation $\mathbf{W} \in R^{b \times b}$ denotes the parameter matrix fabricated from the learnable parameter vector $\mathbf{w} \in R^{l_{\xi}}$,
        which can be represented as follows:

        $$
            \begin{equation}
            \psi(\mathbf{w}) = \Lambda_1 \mathbf{A} \Lambda_1 \mathbf{B}^\top \in R^{b \times b},
            \end{equation}
        $$

        Notations $\Lambda_1$ and $\Lambda_2$ denote two diagonal matrices $\Lambda_1 = diag( \lambda_1) \in R^{m \times m}$ and $\Lambda_2 = diag(\lambda_2) \in R^{r \times r}$
        where the diagonal vectors $\lambda_1$ and $\lambda_2$ are partitioned from the parameter vector $\mathbf{w}$.
        Matrices $\mathbf{A} \in R^{b \times r}$ and $\mathbf{B} \in R^{b \times r}$ are randomly sampled from the Gaussian distribution $\mathcal{N}(\mathbf{0}, \mathbf{I})$.

        The required length of parameter vector of this interdependence function is $\mathbf{w}$ is $l_{\xi} = b + r$.

        Attributes
        ----------
        r : int
            Rank of the random matrix approximation.

        Methods
        -------
        __init__(...)
            Initializes the random matrix adaptation parameterized bilinear interdependence function.
    """
    def __init__(self, r: int = 2, name: str = 'random_matrix_adaption_parameterized_bilinear_interdependence', *args, **kwargs):
        """
            Initializes the random matrix adaptation parameterized bilinear interdependence function.

            Parameters
            ----------
            r : int, optional
                Rank of the random matrix approximation. Defaults to 2.
            name : str, optional
                Name of the interdependence function. Defaults to 'random_matrix_adaption_parameterized_bilinear_interdependence'.
            *args : tuple
                Additional positional arguments for the parent class.
            **kwargs : dict
                Additional keyword arguments for the parent class.
        """
        super().__init__(name=name, *args, **kwargs)
        self.r = r
        self.parameter_fabrication = random_matrix_adaption_reconciliation(r=self.r)

__init__(r=2, name='random_matrix_adaption_parameterized_bilinear_interdependence', *args, **kwargs)

Initializes the random matrix adaptation parameterized bilinear interdependence function.

Parameters:

Name	Type	Description	Default
r	int	Rank of the random matrix approximation. Defaults to 2.	2
name	str	Name of the interdependence function. Defaults to 'random_matrix_adaption_parameterized_bilinear_interdependence'.	'random_matrix_adaption_parameterized_bilinear_interdependence'
*args	tuple	Additional positional arguments for the parent class.	()
**kwargs	dict	Additional keyword arguments for the parent class.	{}

Source code in tinybig/interdependence/parameterized_bilinear_interdependence.py

def __init__(self, r: int = 2, name: str = 'random_matrix_adaption_parameterized_bilinear_interdependence', *args, **kwargs):
    """
        Initializes the random matrix adaptation parameterized bilinear interdependence function.

        Parameters
        ----------
        r : int, optional
            Rank of the random matrix approximation. Defaults to 2.
        name : str, optional
            Name of the interdependence function. Defaults to 'random_matrix_adaption_parameterized_bilinear_interdependence'.
        *args : tuple
            Additional positional arguments for the parent class.
        **kwargs : dict
            Additional keyword arguments for the parent class.
    """
    super().__init__(name=name, *args, **kwargs)
    self.r = r
    self.parameter_fabrication = random_matrix_adaption_reconciliation(r=self.r)