dual_lphm_parameterized_interdependence

Bases: parameterized_interdependence

A parameterized interdependence function using dual low-rank hierarchical mapping.

Notes

Formally, given the parameter vector \(\mathbf{w} \in R^{l_{\xi}}\) and rank hyper-parameter \(r\), we partition \(\mathbf{w}\) into three sub-vectors and subsequently reshape them into four matrices \(\mathbf{P} \in R^{p \times r}\), \(\mathbf{Q} \in R^{q \times r}\), \(\mathbf{S} \in R^{\frac{m}{p} \times r}\) and \(\mathbf{T} \in R^{\frac{m'}{q} \times r}\).

These four sub-matrices \(\mathbf{A}\), \(\mathbf{S}\) and \(\mathbf{T}\) help define the dual low-rank hypercomplex parameterized interdependence function as follows:

$$ \begin{equation} \xi(\mathbf{w}) = \mathbf{A} \otimes \mathbf{B} = ( \mathbf{P} \mathbf{Q}^\top) \otimes ( \mathbf{S} \mathbf{T}^\top) \in R^{m \times m'}, \end{equation} $$ whose required length of vector \(\mathbf{w}\) is \(l_{\xi} = r \times (p + q + \frac{m}{p} + \frac{m'}{q})\).

Attributes:

Name	Type	Description
r	int	Rank of the low-rank approximation.
p	int	Number of partitions in the input dimension.
q	int	Number of partitions in the output dimension.

Methods:

Name	Description
__init__	Initializes the dual low-rank hierarchical mapping parameterized interdependence function.

Source code in tinybig/interdependence/parameterized_interdependence.py

class dual_lphm_parameterized_interdependence(parameterized_interdependence):
    r"""
        A parameterized interdependence function using dual low-rank hierarchical mapping.

        Notes
        ----------

        Formally, given the parameter vector $\mathbf{w} \in R^{l_{\xi}}$ and rank hyper-parameter $r$, we partition $\mathbf{w}$ into three sub-vectors and subsequently reshape them into four matrices
        $\mathbf{P} \in R^{p \times r}$, $\mathbf{Q} \in R^{q \times r}$,  $\mathbf{S} \in R^{\frac{m}{p} \times r}$ and $\mathbf{T} \in R^{\frac{m'}{q} \times r}$.

        These four sub-matrices $\mathbf{A}$, $\mathbf{S}$ and $\mathbf{T}$ help define the dual low-rank hypercomplex parameterized interdependence function as follows:

        $$
            \begin{equation}
            \xi(\mathbf{w}) = \mathbf{A} \otimes \mathbf{B} = ( \mathbf{P} \mathbf{Q}^\top) \otimes ( \mathbf{S} \mathbf{T}^\top) \in R^{m \times m'},
            \end{equation}
        $$
        whose required length of vector $\mathbf{w}$ is $l_{\xi} = r \times (p + q + \frac{m}{p} + \frac{m'}{q})$.

        Attributes
        ----------
        r : int
            Rank of the low-rank approximation.
        p : int
            Number of partitions in the input dimension.
        q : int
            Number of partitions in the output dimension.

        Methods
        -------
        __init__(...)
            Initializes the dual low-rank hierarchical mapping parameterized interdependence function.
    """
    def __init__(self, r: int, p: int, q: int = None, name: str = 'dual_lphm_parameterized_interdependence', *args, **kwargs):
        """
            Initializes the dual low-rank hierarchical mapping (Dual-LPHM) parameterized interdependence function.

            Parameters
            ----------
            r : int
                Rank of the low-rank approximation.
            p : int
                Number of partitions in the input dimension.
            q : int, optional
                Number of partitions in the output dimension. Defaults to `p`.
            name : str, optional
                Name of the interdependence function. Defaults to 'dual_lphm_parameterized_interdependence'.
            *args : tuple
                Additional positional arguments for the parent class.
            **kwargs : dict
                Additional keyword arguments for the parent class.

            Raises
            ------
            ValueError
                If the interdependence type is not supported.
            AssertionError
                If the input and output dimensions are not divisible by their respective partitions.
        """
        super().__init__(name=name, *args, **kwargs)

        self.r = r
        self.p = p
        self.q = q if q is not None else p

        if self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence']:
            d, d_prime = self.b, self.calculate_b_prime()
        elif self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence']:
            d, d_prime = self.m, self.calculate_m_prime()
        else:
            raise ValueError(f'Interdependence type {self.interdependence_type} not supported')
        assert d % self.p == 0 and d_prime % self.q == 0

        self.parameter_fabrication = dual_lphm_reconciliation(p=self.p, q=self.q, r=self.r)

__init__(r, p, q=None, name='dual_lphm_parameterized_interdependence', *args, **kwargs)

Initializes the dual low-rank hierarchical mapping (Dual-LPHM) parameterized interdependence function.

Parameters:

Name	Type	Description	Default
r	int	Rank of the low-rank approximation.	required
p	int	Number of partitions in the input dimension.	required
q	int	Number of partitions in the output dimension. Defaults to p.	None
name	str	Name of the interdependence function. Defaults to 'dual_lphm_parameterized_interdependence'.	'dual_lphm_parameterized_interdependence'
*args	tuple	Additional positional arguments for the parent class.	()
**kwargs	dict	Additional keyword arguments for the parent class.	{}

Raises:

Type	Description
ValueError	If the interdependence type is not supported.
AssertionError	If the input and output dimensions are not divisible by their respective partitions.

Source code in tinybig/interdependence/parameterized_interdependence.py

def __init__(self, r: int, p: int, q: int = None, name: str = 'dual_lphm_parameterized_interdependence', *args, **kwargs):
    """
        Initializes the dual low-rank hierarchical mapping (Dual-LPHM) parameterized interdependence function.

        Parameters
        ----------
        r : int
            Rank of the low-rank approximation.
        p : int
            Number of partitions in the input dimension.
        q : int, optional
            Number of partitions in the output dimension. Defaults to `p`.
        name : str, optional
            Name of the interdependence function. Defaults to 'dual_lphm_parameterized_interdependence'.
        *args : tuple
            Additional positional arguments for the parent class.
        **kwargs : dict
            Additional keyword arguments for the parent class.

        Raises
        ------
        ValueError
            If the interdependence type is not supported.
        AssertionError
            If the input and output dimensions are not divisible by their respective partitions.
    """
    super().__init__(name=name, *args, **kwargs)

    self.r = r
    self.p = p
    self.q = q if q is not None else p

    if self.interdependence_type in ['row', 'left', 'instance', 'instance_interdependence']:
        d, d_prime = self.b, self.calculate_b_prime()
    elif self.interdependence_type in ['column', 'right', 'attribute', 'attribute_interdependence']:
        d, d_prime = self.m, self.calculate_m_prime()
    else:
        raise ValueError(f'Interdependence type {self.interdependence_type} not supported')
    assert d % self.p == 0 and d_prime % self.q == 0

    self.parameter_fabrication = dual_lphm_reconciliation(p=self.p, q=self.q, r=self.r)