lowrank_parameterized_bilinear_interdependence

Bases: parameterized_interdependence

A low-rank parameterized interdependence function.

Notes

Formally, given the parameter vector \(\mathbf{w} \in R^{l_{\xi}}\) and a rank hyper-parameter \(r\), we partition \(\mathbf{w}\) into two sub-vectors and subsequently reshape them into two matrices \(\mathbf{A} \in R^{m \times r}\) and \(\mathbf{B} \in R^{m' \times r}\), each possessing a rank of \(r\).

These two sub-matrices \(\mathbf{A}\) and \(\mathbf{B}\) help define the low-rank parameterized interdependence function as follows:

$$ \begin{equation} \xi(\mathbf{w}) = \mathbf{A} \mathbf{B}^\top \in R^{m \times m'}, \end{equation} $$ whose required length of vector \(\mathbf{w}\) is \(l_{\xi} = (m + m') \times r\).

Attributes:

Name	Type	Description
r	int	Rank of the low-rank approximation.

Methods:

Name	Description
__init__	Initializes the low-rank parameterized interdependence function.

Source code in tinybig/interdependence/parameterized_interdependence.py

class lowrank_parameterized_interdependence(parameterized_interdependence):
    r"""
        A low-rank parameterized interdependence function.

        Notes
        ----------

        Formally, given the parameter vector $\mathbf{w} \in R^{l_{\xi}}$ and a rank hyper-parameter $r$,
        we partition $\mathbf{w}$ into two sub-vectors and subsequently reshape them into two matrices
        $\mathbf{A} \in R^{m \times r}$ and $\mathbf{B} \in R^{m' \times r}$,
        each possessing a rank of $r$.

        These two sub-matrices $\mathbf{A}$ and $\mathbf{B}$ help define the low-rank parameterized interdependence function as follows:

        $$
            \begin{equation}
            \xi(\mathbf{w}) = \mathbf{A} \mathbf{B}^\top \in R^{m \times m'},
            \end{equation}
        $$
        whose required length of vector $\mathbf{w}$ is $l_{\xi} = (m + m') \times r$.

        Attributes
        ----------
        r : int
            Rank of the low-rank approximation.

        Methods
        -------
        __init__(...)
            Initializes the low-rank parameterized interdependence function.
    """
    def __init__(self, r: int = 2, name: str = 'lowrank_parameterized_interdependence', *args, **kwargs):
        """
            Initializes the low-rank parameterized interdependence function.

            Parameters
            ----------
            r : int, optional
                Rank of the low-rank approximation. Defaults to 2.
            name : str, optional
                Name of the interdependence function. Defaults to 'lowrank_parameterized_interdependence'.
            *args : tuple
                Additional positional arguments.
            **kwargs : dict
                Additional keyword arguments.
        """
        super().__init__(name=name, *args, **kwargs)
        self.r = r
        self.parameter_fabrication = lorr_reconciliation(r=self.r)

__init__(r=2, name='lowrank_parameterized_interdependence', *args, **kwargs)

Initializes the low-rank parameterized interdependence function.

Parameters:

Name	Type	Description	Default
r	int	Rank of the low-rank approximation. Defaults to 2.	2
name	str	Name of the interdependence function. Defaults to 'lowrank_parameterized_interdependence'.	'lowrank_parameterized_interdependence'
*args	tuple	Additional positional arguments.	()
**kwargs	dict	Additional keyword arguments.	{}

Source code in tinybig/interdependence/parameterized_interdependence.py

def __init__(self, r: int = 2, name: str = 'lowrank_parameterized_interdependence', *args, **kwargs):
    """
        Initializes the low-rank parameterized interdependence function.

        Parameters
        ----------
        r : int, optional
            Rank of the low-rank approximation. Defaults to 2.
        name : str, optional
            Name of the interdependence function. Defaults to 'lowrank_parameterized_interdependence'.
        *args : tuple
            Additional positional arguments.
        **kwargs : dict
            Additional keyword arguments.
    """
    super().__init__(name=name, *args, **kwargs)
    self.r = r
    self.parameter_fabrication = lorr_reconciliation(r=self.r)